← ML Research Wiki / 2305.11206

LIMA: Less Is More for Alignment

Chunting Zhou, Pengfei Liu, Meta Ai, Puxin Xu Srini Iyer Jiao Sun Yuning Mao Xuezhe Ma Avia Efrat Ping Yu Lili Yu Susan Zhang Gargi Ghosh Mike Lewis Luke Zettlemoyer Omer Levy, Carnegie Mellon University University of Southern California Tel Aviv University (2023)

Paper Information
arXiv ID
2305.11206
Venue
Neural Information Processing Systems
Domain
natural language processing
Reproducibility
5/10
Contents

Abstract

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences.We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling.LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history.Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data.In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback.Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.Preprint.Under review.

Summary

The paper presents LIMA (Less Is More for Alignment), a novel approach to training large language models with an emphasis on the relative importance of unsupervised pretraining versus instruction tuning. It demonstrates that the 65B parameter LLaMa model can achieve competitive performance by being fine-tuned on only 1,000 carefully curated prompts without reinforcement learning or human preference modeling. The findings suggest that a significant amount of knowledge in large language models is retained from pretraining, and a small dataset can yield high-quality outputs across diverse tasks. LIMA's performance is compared to other state-of-the-art models, showing it can be preferred or as effective as GPT-4 and other highly-tuned products in various scenarios.

Methods

This paper employs the following methods:

  • Instruction Tuning
  • Reinforcement Learning

Models Used

  • LLaMa
  • DaVinci003
  • GPT-4
  • Claude
  • Alpaca

Datasets

The following datasets were used in this research:

  • Stack Exchange
  • wikiHow
  • Pushshift Reddit Dataset
  • Super-Natural Instructions

Evaluation Metrics

  • Accuracy
  • Preference Metric

Results

  • LIMA is preferred over GPT-4 in 43% of cases and performs comparably to Bard and DaVinci003.
  • 88% of LIMA's outputs met prompt requirements, with 50% deemed excellent.
  • The model demonstrates coherence in multi-turn dialogue after fine-tuning with additional dialogue examples.

Limitations

The authors identified the following limitations:

  • Constructing high-quality examples is labor-intensive and challenging to scale.
  • LIMA may generate weak responses under certain adversarial conditions.

Technical Requirements

  • Number of GPUs: None specified
  • GPU Type: None specified

Keywords

large language models fine-tuning alignment supervised learning human feedback

Papers Using Similar Methods

External Resources

Funding: Not specified
References: 25
Influential Citations: 77