ML Research Wiki / Benchmarks / Language Modelling / Text8

Text8

Language Modelling Benchmark

Performance Over Time

📊 Showing 22 results | 📏 Metric: Bit per Character (BPC)

Top Performing Models

Rank	Model	Paper	Bit per Character (BPC)	Date	Code
1	td-LSTM (Zhang et al., 2016)	Architectural Complexity Measures of Recurrent Neural Networks	1.63	2016-02-26	-
2	td-LSTM-large	Architectural Complexity Measures of Recurrent Neural Networks	1.49	2016-02-26	-
3	BFN	Bayesian Flow Networks	1.41	2023-08-14	📦 nnaisense/bayesian-flow-networks
4	Unregularised mLSTM	Multiplicative LSTM for sequence modelling	1.40	2016-09-26	📦 astakara48/python_project
5	BN LSTM	Recurrent Batch Normalization	1.36	2016-03-30	📦 cooijmanstim/recurrent-batch-normalization 📦 codedecde/Recognizing-Textual-Entailment 📦 Tetsuya-Nishikawa/ConvLSTM_DEMO
6	LayerNorm HM-LSTM	Hierarchical Multiscale Recurrent Neural Networks	1.29	2016-09-06	📦 bolducp/hierarchical-rnn 📦 kaiu85/hm-rnn 📦 nikolasthuesen/HMLSTM
7	Large RHN	Recurrent Highway Networks	1.27	2016-07-12	📦 labmlai/annotated_deep_learning_paper_implementations 📦 julian121266/RecurrentHighwayNetworks 📦 jzilly/RecurrentHighwayNetworks
8	Large mLSTM +emb +WN +VD	Multiplicative LSTM for sequence modelling	1.27	2016-09-26	📦 astakara48/python_project
9	Bipartite flows (8 flows)	Discrete Flows: Invertible Generative Models of Discrete Data	1.23	2019-05-24	📦 google/edward2 📦 TrentBrick/PyTorchDiscreteFlows
10	mLSTM + dynamic eval	Dynamic Evaluation of Neural Sequence Models	1.19	2017-09-21	📦 benkrause/dynamic-evaluation 📦 benkrause/dynamiceval-transformer 📦 sacmehta/PRU

All Papers (22)

Architectural Complexity Measures of Recurrent Neural Networks

2016

td-LSTM (Zhang et al., 2016)

Architectural Complexity Measures of Recurrent Neural Networks

2016

td-LSTM-large

Bayesian Flow Networks

2023

BFN

nnaisense/bayesian-flow-networks

Multiplicative LSTM for sequence modelling

2016

Unregularised mLSTM

astakara48/python_project

Recurrent Batch Normalization

2016

BN LSTM

cooijmanstim/recurrent-batch-normalization codedecde/Recognizing-Textual-Entailment Tetsuya-Nishikawa/ConvLSTM_DEMO

Hierarchical Multiscale Recurrent Neural Networks

2016

LayerNorm HM-LSTM

bolducp/hierarchical-rnn kaiu85/hm-rnn nikolasthuesen/HMLSTM

Recurrent Highway Networks

2016

Large RHN

labmlai/annotated_deep_learning_paper_implementations julian121266/RecurrentHighwayNetworks

Multiplicative LSTM for sequence modelling

2016

Large mLSTM +emb +WN +VD

astakara48/python_project

Discrete Flows: Invertible Generative Models of Discrete Data

2019

Bipartite flows (8 flows)

google/edward2 TrentBrick/PyTorchDiscreteFlows

Dynamic Evaluation of Neural Sequence Models

2017

mLSTM + dynamic eval

benkrause/dynamic-evaluation benkrause/dynamiceval-transformer sacmehta/PRU

Character-Level Language Modeling with Deeper Self-Attention

2018

12-layer Character Transformer Model

facebookresearch/code-prediction-transformer

Pay Attention when Required

2020

PAR Transformer 24B

NVIDIA/DeepLearningExamples Jmkernes/PAR-Transformer-XL

Character-Level Language Modeling with Deeper Self-Attention

2018

64-layer Character Transformer Model

facebookresearch/code-prediction-transformer

Adaptive Attention Span in Transformers

2019

12L Transformer + 8K adaptive span

facebookresearch/adaptive-span jerrodparker20/adaptive-transformers-in-rl

Augmenting Self-attention with Persistent Memory

2019

All-attention network - 18 layers

lucidrains/x-transformers facebookresearch/adaptive-span

BP-Transformer: Modelling Long-Range Context via Binary Partitioning

2019

BP-Transformer - 12 Layers

dmlc/dgl yzh119/BPT

Long-Short Transformer: Efficient Transformers for Language and Vision

2021

Transformer-LS (small)

keonlee9420/Comprehensive-Transformer-TTS NVIDIA/transformer-ls lucidrains/long-short-transformer

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

2019

Transformer-XL - 24 layers

huggingface/transformers labmlai/annotated_deep_learning_paper_implementations

Augmenting Self-attention with Persistent Memory

2019

All-attention network - 36 layers

lucidrains/x-transformers facebookresearch/adaptive-span

Adaptive Attention Span in Transformers

2019

24L Transformer + 8K adaptive span

facebookresearch/adaptive-span jerrodparker20/adaptive-transformers-in-rl

Dynamic Evaluation of Transformer Language Models

2019

Transformer-XL + RMS dynamic eval + decay

benkrause/dynamiceval-transformer

Focus Your Attention (with Adaptive IIR Filters)

2023

Focus

Text8

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (22)

Architectural Complexity Measures of Recurrent Neural Networks

Architectural Complexity Measures of Recurrent Neural Networks

Bayesian Flow Networks

Multiplicative LSTM for sequence modelling

Recurrent Batch Normalization

Hierarchical Multiscale Recurrent Neural Networks

Recurrent Highway Networks

Multiplicative LSTM for sequence modelling

Discrete Flows: Invertible Generative Models of Discrete Data

Dynamic Evaluation of Neural Sequence Models

Character-Level Language Modeling with Deeper Self-Attention

Pay Attention when Required

Character-Level Language Modeling with Deeper Self-Attention

Adaptive Attention Span in Transformers

Augmenting Self-attention with Persistent Memory

BP-Transformer: Modelling Long-Range Context via Binary Partitioning

Long-Short Transformer: Efficient Transformers for Language and Vision

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Augmenting Self-attention with Persistent Memory

Adaptive Attention Span in Transformers

Dynamic Evaluation of Transformer Language Models

Focus Your Attention (with Adaptive IIR Filters)

Model	Paper	Bit per Character (BPC)	Date
td-LSTM (Zhang et al., 2016)	Architectural Complexity Measures of Recurrent Ne…	1.63	2016-02-26
td-LSTM-large	Architectural Complexity Measures of Recurrent Ne…	1.49	2016-02-26
BFN	Bayesian Flow Networks	1.41	2023-08-14
Unregularised mLSTM	Multiplicative LSTM for sequence modelling	1.40	2016-09-26
BN LSTM	Recurrent Batch Normalization	1.36	2016-03-30
LayerNorm HM-LSTM	Hierarchical Multiscale Recurrent Neural Networks	1.29	2016-09-06
Large RHN	Recurrent Highway Networks	1.27	2016-07-12
Large mLSTM +emb +WN +VD	Multiplicative LSTM for sequence modelling	1.27	2016-09-26
Bipartite flows (8 flows)	Discrete Flows: Invertible Generative Models of D…	1.23	2019-05-24
mLSTM + dynamic eval	Dynamic Evaluation of Neural Sequence Models	1.19	2017-09-21
12-layer Character Transformer Model	Character-Level Language Modeling with Deeper Sel…	1.18	2018-08-09
PAR Transformer 24B	Pay Attention when Required	1.18	2020-09-09
64-layer Character Transformer Model	Character-Level Language Modeling with Deeper Sel…	1.13	2018-08-09
12L Transformer + 8K adaptive span	Adaptive Attention Span in Transformers	1.11	2019-05-19
All-attention network - 18 layers	Augmenting Self-attention with Persistent Memory	1.11	2019-07-02
BP-Transformer - 12 Layers	BP-Transformer: Modelling Long-Range Context via …	1.11	2019-11-11
Transformer-LS (small)	Long-Short Transformer: Efficient Transformers fo…	1.09	2021-07-05
Transformer-XL - 24 layers	Transformer-XL: Attentive Language Models Beyond …	1.08	2019-01-09
All-attention network - 36 layers	Augmenting Self-attention with Persistent Memory	1.08	2019-07-02
24L Transformer + 8K adaptive span	Adaptive Attention Span in Transformers	1.07	2019-05-19
Transformer-XL + RMS dynamic eval + decay	Dynamic Evaluation of Transformer Language Models	1.04	2019-04-17
Focus	Focus Your Attention (with Adaptive IIR Filters)	0.98	2023-05-24