← ML Research Wiki / 2312.10997

Retrieval-Augmented Generation for Large Language Models: A Survey

Yunfan Gao Shanghai Research Institute for Intelligent Autonomous Systems Tongji University, Yun Xiong School of Computer Science Shanghai Key Laboratory of Data Science Fudan University, Xinyu Gao School of Computer Science Shanghai Key Laboratory of Data Science Fudan University, Kangxiang Jia School of Computer Science Shanghai Key Laboratory of Data Science Fudan University, Jinliu Pan School of Computer Science Shanghai Key Laboratory of Data Science Fudan University, Yuxi Bi College of Design and Innovation Tongji University, Yi Dai Shanghai Research Institute for Intelligent Autonomous Systems Tongji University, Jiawei Sun Shanghai Research Institute for Intelligent Autonomous Systems Tongji University, Meng Wang College of Design and Innovation Tongji University, Haofen Wang Shanghai Research Institute for Intelligent Autonomous Systems Tongji University College of Design and Innovation Tongji University (2023)

Paper Information
arXiv ID
2312.10997
Venue
arXiv.org
Domain
computer science, artificial intelligence
Reproducibility
5/10
Contents

Abstract

Large Language Models (LLMs) showcase impressive capabilities but encounter challenges like hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes.Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases.This enhances the accuracy and credibility of the generation, particularly for knowledge-intensive tasks, and allows for continuous knowledge updates and integration of domainspecific information.RAG synergistically merges LLMs' intrinsic knowledge with the vast, dynamic repositories of external databases.This comprehensive review paper offers a detailed examination of the progression of RAG paradigms, encompassing the Naive RAG, the Advanced RAG, and the Modular RAG.It meticulously scrutinizes the tripartite foundation of RAG frameworks, which includes the retrieval, the generation and the augmentation techniques.The paper highlights the state-of-theart technologies embedded in each of these critical components, providing a profound understanding of the advancements in RAG systems.Furthermore, this paper introduces up-to-date evaluation framework and benchmark.At the end, this article delineates the challenges currently faced and points out prospective avenues for research and development 1 .

Summary

This paper provides a comprehensive survey of Retrieval-Augmented Generation (RAG) for Large Language Models (LLMs). It outlines the significant advantages of RAG in enhancing LLMs by integrating external knowledge sources, addressing challenges such as hallucination, outdated information, and opaque reasoning. The paper categorizes RAG into three paradigms: Naive RAG, Advanced RAG, and Modular RAG, detailing their structures, methodologies, and the evolution influenced by the emergence of advanced LLMs like ChatGPT. The authors explore the technical intricacies of the RAG process, covering retrieval, generation, and augmentation techniques. They compile key datasets and metrics, propose evaluation frameworks, and discuss current challenges and future research directions in RAG. The survey aims to equip readers with a thorough understanding of RAG's implementation, its foundational components, and its application across various domains. Overall, it emphasizes the need for continuous innovation in RAG technologies amidst the growth of LLM capabilities.

Methods

This paper employs the following methods:

  • RAG
  • Naive RAG
  • Advanced RAG
  • Modular RAG

Models Used

  • None specified

Datasets

The following datasets were used in this research:

  • HotpotQA
  • DPR
  • SQuAD
  • WikiMultiHopQA
  • TriviaQA
  • MS Marco
  • None specified

Evaluation Metrics

  • EM
  • Accuracy
  • F1-score
  • BLEU
  • ROUGE
  • Hit Rate
  • MRR
  • NDCG
  • None specified

Results

  • RAG enhances LLM capabilities by integrating current knowledge and reducing misinformation.
  • The evolution of RAG demonstrates increased robustness and usability in practical applications.
  • Comprehensive evaluation methodologies are needed to assess RAG performance across diverse tasks.

Limitations

The authors identified the following limitations:

  • Current research on RAG lacks systematic synthesis and clarity in its broader trajectory.
  • Limited focus on evaluation methods and metrics specific to RAG.
  • Challenges in integrating quality retrieval and generation processes effectively.

Technical Requirements

  • Number of GPUs: None specified
  • GPU Type: None specified

Keywords

retrieval-augmented generation large language models RAG natural language processing knowledge-intensive tasks

Papers Using Similar Methods

External Resources

Funding: Not specified
References: 229
Influential Citations: 70