← ML Research Wiki / 2304.07327

OpenAssistant Conversations -Democratizing Large Language Model Alignment

Andreas Köpf [email protected] Christoph Schuhmann Huu Nguyen, Dimitri Von Rütte Christoph Schuhmann Huu Nguyen, Sotiris Anagnostidis Christoph Schuhmann Huu Nguyen, Zhi-Rui Tam Christoph Schuhmann Huu Nguyen, Keith Stevens Christoph Schuhmann Huu Nguyen, Abdullah Barhoum Nguyen Christoph Schuhmann Huu Nguyen, Minh Duc Christoph Schuhmann Huu Nguyen, Oliver Stanley Christoph Schuhmann Huu Nguyen, Richárd Nagyfi Christoph Schuhmann Huu Nguyen, Shahul Es Christoph Schuhmann Huu Nguyen, Sameer Suri Christoph Schuhmann Huu Nguyen, David Glushkov Christoph Schuhmann Huu Nguyen, Arnav Dantuluri Christoph Schuhmann Huu Nguyen, Andrew Maguire Christoph Schuhmann Huu Nguyen, Alexander Mattick [email protected] Christoph Schuhmann Huu Nguyen (2023)

Paper Information
arXiv ID
2304.07327
Venue
Neural Information Processing Systems
Domain
Artificial Intelligence
SOTA Claim
Yes
Code
Available
Reproducibility
8/10
Contents

Abstract

Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT.Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains.However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary.In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees.The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers.Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models.We release our code 2 and data 3 under a fully permissive licence.A list of contributors who have chosen to be acknowledged by name can be found at https://open-assistant.io/contributors.

Summary

This paper introduces OpenAssistant Conversations, a dataset aimed at democratizing the alignment of large language models (LLMs) with human preferences. It consists of 161,443 messages across 66,497 conversation trees in 35 languages, generated through a crowd-sourcing effort involving over 13,500 volunteers. The dataset is annotated with 461,292 quality ratings, enabling improved usability and effectiveness of LLMs such as Pythia and LLaMA through techniques like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). The paper highlights the importance of high-quality human feedback data for training LLMs and discusses methodology, quality control, safety measures, and the dataset's potential impact on AI alignment research and deployment. Moreover, results show that models trained on this dataset outperform baseline models on various benchmarks, promoting accessibility and inclusiveness in AI research.

Methods

This paper employs the following methods:

  • Crowd-sourcing
  • Supervised Fine-Tuning (SFT)
  • Reinforcement Learning from Human Feedback (RLHF)

Models Used

  • Pythia
  • LLaMA
  • Falcon

Datasets

The following datasets were used in this research:

  • OpenAssistant Conversations

Evaluation Metrics

  • LMEH

Results

  • Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models.

Limitations

The authors identified the following limitations:

  • Dependence on data quality for model performance.
  • Potential biases in the dataset due to contributor demographics.
  • Possibility of residual unsafe content in the dataset.

Technical Requirements

  • Number of GPUs: None specified
  • GPU Type: None specified

Keywords

Large language models Human preferences Crowd-sourcing Ethics Bias Reinforcement learning

Papers Using Similar Methods

External Resources

Funding: Not specified
References: 53
Influential Citations: 56