WikiText-103

Overview edit

Variants: WikiText-103

Associated Benchmarks

This dataset is used in 2 benchmarks:

• Text Generation - Metrics: Perplexity
• Language Modelling - Metrics: Test perplexity, Validation perplexity, Number of params

Recent Benchmark Submissions

Research Papers

Recent papers with results on this dataset:

• Advancing State of the Art in Language Modeling (2023) -
• Memory-efficient Stochastic methods for Memory-based Transformers (2023) -
• GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling (2023) -
• The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles (2023) -
• Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation (2023) -
• Hyena Hierarchy: Towards Larger Convolutional Language Models (2023) -
• Hungry Hungry Hippos: Towards Language Modeling with State Space Models (2022) -
• You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM (2022) -
• Mega: Moving Average Equipped Gated Attention (2022) -
• General-purpose, long-context autoregressive modeling with Perceiver AR (2022) -
• Improving language models by retrieving from trillions of tokens (2021) -
• Efficiently Modeling Long Sequences with Structured State Spaces (2021) -
• $\infty$-former: Infinite Memory Transformer (2021) -
• FNetAR: Mixing Tokens with Autoregressive Fourier Transforms (2021) -
• Differentiable Model Compression via Pseudo Quantization Noise (2021) -
• Revisiting Simple Neural Probabilistic Language Models (2021) -
• Finetuning Pretrained Transformers into RNNs (2021) -
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021) -
• Random Feature Attention (2021) -
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute (2021) -