ML Research Wiki / Benchmarks / Language Modelling / Hutter Prize

Hutter Prize

Language Modelling Benchmark

Performance Over Time

📊 Showing 18 results | 📏 Metric: Bit per Character (BPC)

Top Performing Models

Rank	Model	Paper	Bit per Character (BPC)	Date	Code
1	RHN - depth 5 [zilly2016recurrent]	Recurrent Highway Networks	1.31	2016-07-12	📦 labmlai/annotated_deep_learning_paper_implementations 📦 julian121266/RecurrentHighwayNetworks 📦 jzilly/RecurrentHighwayNetworks
2	FS-LSTM-4	Fast-Slow Recurrent Neural Networks	1.28	2017-05-24	📦 amujika/Fast-Slow-LSTM
3	Large RHN	Recurrent Highway Networks	1.27	2016-07-12	📦 labmlai/annotated_deep_learning_paper_implementations 📦 julian121266/RecurrentHighwayNetworks 📦 jzilly/RecurrentHighwayNetworks
4	Large FS-LSTM-4	Fast-Slow Recurrent Neural Networks	1.25	2017-05-24	📦 amujika/Fast-Slow-LSTM
5	Large mLSTM +emb +WN +VD	Multiplicative LSTM for sequence modelling	1.24	2016-09-26	📦 astakara48/python_project
6	3-layer AWD-LSTM	An Analysis of Neural Language Modeling at Multiple Scales	1.23	2018-03-22	📦 salesforce/awd-lstm-lm 📦 Han-JD/GRU-D 📦 jb33k/awd-lstm-lm-ThinkNet
7	Mogrifier LSTM	Mogrifier LSTM	1.12	2019-09-04	📦 deepmind/lamb 📦 RMichaelSwan/MogrifierLSTM 📦 microcoder-py/mogrifier-lstm
8	12-layer Character Transformer Model	Character-Level Language Modeling with Deeper Self-Attention	1.11	2018-08-09	📦 facebookresearch/code-prediction-transformer
9	mLSTM + dynamic eval	Dynamic Evaluation of Neural Sequence Models	1.08	2017-09-21	📦 benkrause/dynamic-evaluation 📦 benkrause/dynamiceval-transformer 📦 sacmehta/PRU
10	64-layer Character Transformer Model	Character-Level Language Modeling with Deeper Self-Attention	1.06	2018-08-09	📦 facebookresearch/code-prediction-transformer

All Papers (18)

Recurrent Highway Networks

2016

RHN - depth 5 [zilly2016recurrent]

labmlai/annotated_deep_learning_paper_implementations julian121266/RecurrentHighwayNetworks

Fast-Slow Recurrent Neural Networks

2017

FS-LSTM-4

amujika/Fast-Slow-LSTM

Recurrent Highway Networks

2016

Large RHN

labmlai/annotated_deep_learning_paper_implementations julian121266/RecurrentHighwayNetworks

Fast-Slow Recurrent Neural Networks

2017

Large FS-LSTM-4

amujika/Fast-Slow-LSTM

Multiplicative LSTM for sequence modelling

2016

Large mLSTM +emb +WN +VD

astakara48/python_project

An Analysis of Neural Language Modeling at Multiple Scales

2018

3-layer AWD-LSTM

salesforce/awd-lstm-lm Han-JD/GRU-D

Mogrifier LSTM

2019

Mogrifier LSTM

deepmind/lamb RMichaelSwan/MogrifierLSTM microcoder-py/mogrifier-lstm

Character-Level Language Modeling with Deeper Self-Attention

2018

12-layer Character Transformer Model

facebookresearch/code-prediction-transformer

Dynamic Evaluation of Neural Sequence Models

2017

mLSTM + dynamic eval

benkrause/dynamic-evaluation benkrause/dynamiceval-transformer sacmehta/PRU

Character-Level Language Modeling with Deeper Self-Attention

2018

64-layer Character Transformer Model

facebookresearch/code-prediction-transformer

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

2019

12-layer Transformer-XL

huggingface/transformers labmlai/annotated_deep_learning_paper_implementations

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

2019

18-layer Transformer-XL

huggingface/transformers labmlai/annotated_deep_learning_paper_implementations

Longformer: The Long-Document Transformer

2020

Longformer Small

huggingface/transformers mistralai/mistral-src

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

2019

24-layer Transformer-XL

huggingface/transformers labmlai/annotated_deep_learning_paper_implementations

Longformer: The Long-Document Transformer

2020

Longformer Large

huggingface/transformers mistralai/mistral-src

Mogrifier LSTM

2019

Mogrifier LSTM + dynamic eval

deepmind/lamb RMichaelSwan/MogrifierLSTM microcoder-py/mogrifier-lstm

Compressive Transformers for Long-Range Sequence Modelling

2019

Compressive Transformer

labmlai/annotated_deep_learning_paper_implementations google-deepmind/pg19

Dynamic Evaluation of Transformer Language Models

2019

Transformer-XL + RMS dynamic eval

benkrause/dynamiceval-transformer

Hutter Prize

Performance Over Time

Edit Benchmark Results

Edit Result

Top Performing Models

All Papers (18)

Recurrent Highway Networks

Fast-Slow Recurrent Neural Networks

Recurrent Highway Networks

Fast-Slow Recurrent Neural Networks

Multiplicative LSTM for sequence modelling

An Analysis of Neural Language Modeling at Multiple Scales

Mogrifier LSTM

Character-Level Language Modeling with Deeper Self-Attention

Dynamic Evaluation of Neural Sequence Models

Character-Level Language Modeling with Deeper Self-Attention

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Longformer: The Long-Document Transformer

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Longformer: The Long-Document Transformer

Mogrifier LSTM

Compressive Transformers for Long-Range Sequence Modelling

Dynamic Evaluation of Transformer Language Models

Model	Paper	Bit per Character (BPC)	Date
RHN - depth 5 [zilly2016recurrent]	Recurrent Highway Networks	1.31	2016-07-12
FS-LSTM-4	Fast-Slow Recurrent Neural Networks	1.28	2017-05-24
Large RHN	Recurrent Highway Networks	1.27	2016-07-12
Large FS-LSTM-4	Fast-Slow Recurrent Neural Networks	1.25	2017-05-24
Large mLSTM +emb +WN +VD	Multiplicative LSTM for sequence modelling	1.24	2016-09-26
3-layer AWD-LSTM	An Analysis of Neural Language Modeling at Multip…	1.23	2018-03-22
Mogrifier LSTM	Mogrifier LSTM	1.12	2019-09-04
12-layer Character Transformer Model	Character-Level Language Modeling with Deeper Sel…	1.11	2018-08-09
mLSTM + dynamic eval	Dynamic Evaluation of Neural Sequence Models	1.08	2017-09-21
64-layer Character Transformer Model	Character-Level Language Modeling with Deeper Sel…	1.06	2018-08-09
12-layer Transformer-XL	Transformer-XL: Attentive Language Models Beyond …	1.06	2019-01-09
18-layer Transformer-XL	Transformer-XL: Attentive Language Models Beyond …	1.03	2019-01-09
Longformer Small	Longformer: The Long-Document Transformer	1.00	2020-04-10
24-layer Transformer-XL	Transformer-XL: Attentive Language Models Beyond …	0.99	2019-01-09
Longformer Large	Longformer: The Long-Document Transformer	0.99	2020-04-10
Mogrifier LSTM + dynamic eval	Mogrifier LSTM	0.99	2019-09-04
Compressive Transformer	Compressive Transformers for Long-Range Sequence …	0.97	2019-11-13
Transformer-XL + RMS dynamic eval	Dynamic Evaluation of Transformer Language Models	0.94	2019-04-17