ML Research Wiki / Benchmarks / Language Modelling / LAMBADA

LAMBADA

Language Modelling Benchmark

Performance Over Time

📊 Showing 34 results | 📏 Metric: Accuracy

Top Performing Models

Rank	Model	Paper	Accuracy	Date	Code
1	PaLM-540B (Few-Shot)	PaLM: Scaling Language Modeling with Pathways	89.70	2022-04-05	📦 lucidrains/CoCa-pytorch 📦 lucidrains/PaLM-pytorch 📦 google/paxml
2	PaLM 2-L (one-shot)	PaLM 2 Technical Report	86.90	2023-05-17	📦 eternityyw/tram-benchmark
3	GPT-3 175B (Few-Shot)	Language Models are Few-Shot Learners	86.40	2020-05-28	📦 ggml-org/llama.cpp 📦 ggerganov/llama.cpp 📦 karpathy/llm.c
4	LLaMA-65B+CFG (Zero-Shot)	Stay on topic with Classifier-Free Guidance	84.00	2023-06-30	-
5	LLaMA-30B+CFG (zero-shot)	Stay on topic with Classifier-Free Guidance	83.90	2023-06-30	-
6	PaLM 2-M (one-shot)	PaLM 2 Technical Report	83.70	2023-05-17	📦 eternityyw/tram-benchmark
7	LLaMA-13B+CFG (zero-shot)	Stay on topic with Classifier-Free Guidance	82.20	2023-06-30	-
8	PaLM-540B (One-Shot)	PaLM: Scaling Language Modeling with Pathways	81.80	2022-04-05	📦 lucidrains/CoCa-pytorch 📦 lucidrains/PaLM-pytorch 📦 google/paxml
9	GLaM 62B/64E (One-Shot)	GLaM: Efficient Scaling of Language Models with Mixture-of-Experts	80.90	2021-12-13	-
10	PaLM 2-S (one-shot)	PaLM 2 Technical Report	80.70	2023-05-17	📦 eternityyw/tram-benchmark

All Papers (34)

PaLM: Scaling Language Modeling with Pathways

2022

PaLM-540B (Few-Shot)

lucidrains/CoCa-pytorch lucidrains/PaLM-pytorch

PaLM 2 Technical Report

2023

PaLM 2-L (one-shot)

eternityyw/tram-benchmark

Language Models are Few-Shot Learners

2020

GPT-3 175B (Few-Shot)

ggml-org/llama.cpp ggerganov/llama.cpp

Stay on topic with Classifier-Free Guidance

2023

LLaMA-65B+CFG (Zero-Shot)

Stay on topic with Classifier-Free Guidance

2023

LLaMA-30B+CFG (zero-shot)

PaLM 2 Technical Report

2023

PaLM 2-M (one-shot)

eternityyw/tram-benchmark

Stay on topic with Classifier-Free Guidance

2023

LLaMA-13B+CFG (zero-shot)

PaLM: Scaling Language Modeling with Pathways

2022

PaLM-540B (One-Shot)

lucidrains/CoCa-pytorch lucidrains/PaLM-pytorch

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

2021

GLaM 62B/64E (One-Shot)

PaLM 2 Technical Report

2023

PaLM 2-S (one-shot)

eternityyw/tram-benchmark

GLM-130B: An Open Bilingual Pre-trained Model

2022

GLM-130B (bidirectional attention)

thudm/chatglm2-6b thudm/chatglm3

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2023

SparseGPT (175B, 2:4 Sparsity)

nvidia/tensorrt-model-optimizer ist-daslab/sparsegpt

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2023

SparseGPT (175B, 4:8 Sparsity)

nvidia/tensorrt-model-optimizer ist-daslab/sparsegpt

PaLM: Scaling Language Modeling with Pathways

2022

PaLM-540B (Zero-Shot)

lucidrains/CoCa-pytorch lucidrains/PaLM-pytorch

Training Compute-Optimal Large Language Models

2022

Chinchilla (Zero-Shot)

karpathy/llama2.c nkluge-correa/teenytinyllama

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2023

SparseGPT (175B, 50% Sparsity)

nvidia/tensorrt-model-optimizer ist-daslab/sparsegpt

Language Models are Few-Shot Learners

2020

GPT-3 175B (Zero-Shot)

ggml-org/llama.cpp ggerganov/llama.cpp

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2023

OPT-175B

nvidia/tensorrt-model-optimizer ist-daslab/sparsegpt

Language Models are Few-Shot Learners

2020

GPT-3 13B (Zero-Shot)

ggml-org/llama.cpp ggerganov/llama.cpp

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

2021

GLM-XXLarge (bidirectional)

thudm/chatglm2-6b thudm/chatglm

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

2023

Pythia 12B (0-shot)

Lightning-AI/lit-gpt jzhang38/tinyllama

Language Models are Few-Shot Learners

2020

GPT-3 6.7B (Zero-Shot)

ggml-org/llama.cpp ggerganov/llama.cpp

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

2023

Mamba-2.8B

state-spaces/mamba hustvl/vim

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

2023

Pythia 6.9B (0-shot)

Lightning-AI/lit-gpt jzhang38/tinyllama

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

2021

GLM-XXLarge (unidirectional)

thudm/chatglm2-6b thudm/chatglm

Language Models are Few-Shot Learners

2020

GPT-3 2.7B (Zero-Shot)

ggml-org/llama.cpp ggerganov/llama.cpp

Universal Transformers

2018

Universal Transformer (w/ dynamic halting)

tensorflow/tensor2tensor kpot/keras-transformer

Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences

2020

Residual Shuffle-Exchange network

LUMII-Syslab/RSE Aroksak/RSE

Broad Context Language Modeling as Reading Comprehension

2016

Gated-Attention Reader (+ features)

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

2023

Pythia 6.9B(Zero-Shot)

Lightning-AI/lit-gpt jzhang38/tinyllama

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

2023

Pythia 12B(Zero-Shot)

Lightning-AI/lit-gpt jzhang38/tinyllama

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

2023

OPT-175B (50% Sparsity)

nvidia/tensorrt-model-optimizer ist-daslab/sparsegpt

Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

2019

test

yueatsprograms/ttt_cifar_release yueatsprograms/ttt_imagenet_release tejas-gokhale/AGAT

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

2022

Megatron-Turing NLG 530B (Few-Shot)

microsoft/DeepSpeed NVIDIA/NeMo-Curator

LAMBADA

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (34)

PaLM: Scaling Language Modeling with Pathways

PaLM 2 Technical Report

Language Models are Few-Shot Learners

Stay on topic with Classifier-Free Guidance

Stay on topic with Classifier-Free Guidance

PaLM 2 Technical Report

Stay on topic with Classifier-Free Guidance

PaLM: Scaling Language Modeling with Pathways

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

PaLM 2 Technical Report

GLM-130B: An Open Bilingual Pre-trained Model

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

PaLM: Scaling Language Modeling with Pathways

Training Compute-Optimal Large Language Models

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Language Models are Few-Shot Learners

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Language Models are Few-Shot Learners

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Language Models are Few-Shot Learners

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Language Models are Few-Shot Learners

Universal Transformers

Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences

Broad Context Language Modeling as Reading Comprehension

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model

Model	Paper	Accuracy	Date
PaLM-540B (Few-Shot)	PaLM: Scaling Language Modeling with Pathways	89.70	2022-04-05
PaLM 2-L (one-shot)	PaLM 2 Technical Report	86.90	2023-05-17
GPT-3 175B (Few-Shot)	Language Models are Few-Shot Learners	86.40	2020-05-28
LLaMA-65B+CFG (Zero-Shot)	Stay on topic with Classifier-Free Guidance	84.00	2023-06-30
LLaMA-30B+CFG (zero-shot)	Stay on topic with Classifier-Free Guidance	83.90	2023-06-30
PaLM 2-M (one-shot)	PaLM 2 Technical Report	83.70	2023-05-17
LLaMA-13B+CFG (zero-shot)	Stay on topic with Classifier-Free Guidance	82.20	2023-06-30
PaLM-540B (One-Shot)	PaLM: Scaling Language Modeling with Pathways	81.80	2022-04-05
GLaM 62B/64E (One-Shot)	GLaM: Efficient Scaling of Language Models with M…	80.90	2021-12-13
PaLM 2-S (one-shot)	PaLM 2 Technical Report	80.70	2023-05-17
GLM-130B (bidirectional attention)	GLM-130B: An Open Bilingual Pre-trained Model	80.20	2022-10-05
SparseGPT (175B, 2:4 Sparsity)	SparseGPT: Massive Language Models Can Be Accurat…	79.47	2023-01-02
SparseGPT (175B, 4:8 Sparsity)	SparseGPT: Massive Language Models Can Be Accurat…	78.77	2023-01-02
PaLM-540B (Zero-Shot)	PaLM: Scaling Language Modeling with Pathways	77.90	2022-04-05
Chinchilla (Zero-Shot)	Training Compute-Optimal Large Language Models	77.70	2022-03-29
SparseGPT (175B, 50% Sparsity)	SparseGPT: Massive Language Models Can Be Accurat…	76.51	2023-01-02
GPT-3 175B (Zero-Shot)	Language Models are Few-Shot Learners	76.20	2020-05-28
OPT-175B	SparseGPT: Massive Language Models Can Be Accurat…	75.59	2023-01-02
GPT-3 13B (Zero-Shot)	Language Models are Few-Shot Learners	72.50	2020-05-28
GLM-XXLarge (bidirectional)	GLM: General Language Model Pretraining with Auto…	72.35	2021-03-18
Pythia 12B (0-shot)	Pythia: A Suite for Analyzing Large Language Mode…	70.46	2023-04-03
GPT-3 6.7B (Zero-Shot)	Language Models are Few-Shot Learners	70.30	2020-05-28
Mamba-2.8B	Mamba: Linear-Time Sequence Modeling with Selecti…	69.20	2023-12-01
Pythia 6.9B (0-shot)	Pythia: A Suite for Analyzing Large Language Mode…	67.28	2023-04-03
GLM-XXLarge (unidirectional)	GLM: General Language Model Pretraining with Auto…	67.18	2021-03-18
GPT-3 2.7B (Zero-Shot)	Language Models are Few-Shot Learners	67.10	2020-05-28
Universal Transformer (w/ dynamic halting)	Universal Transformers	56.25	2018-07-10
Residual Shuffle-Exchange network	Residual Shuffle-Exchange Networks for Fast Proce…	54.34	2020-04-06
Gated-Attention Reader (+ features)	Broad Context Language Modeling as Reading Compre…	49.00	2016-10-26
Pythia 6.9B(Zero-Shot)	Pythia: A Suite for Analyzing Large Language Mode…	4.45	2023-04-03
Pythia 12B(Zero-Shot)	Pythia: A Suite for Analyzing Large Language Mode…	3.92	2023-04-03
OPT-175B (50% Sparsity)	SparseGPT: Massive Language Models Can Be Accurat…	0.02	2023-01-02
test	Test-Time Training with Self-Supervision for Gene…	0.01	2019-09-29
Megatron-Turing NLG 530B (Few-Shot)	Using DeepSpeed and Megatron to Train Megatron-Tu…		2022-01-28