← ML Research Wiki / 2402.07927

A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

Pranab Sahoo Department of Computer Science And Engineering Indian Institute of Technology Patna, Ayush Kumar Singh Department of Computer Science And Engineering Indian Institute of Technology Patna, Sriparna Saha [email protected] Department of Computer Science And Engineering Indian Institute of Technology Patna, Vinija Jain Stanford University 3 AmazonAI, Samrat Mondal [email protected] Department of Computer Science And Engineering Indian Institute of Technology Patna, Aman Chadha Stanford University 3 AmazonAI (2024)

Paper Information
arXiv ID
2402.07927
Venue
arXiv.org
Domain
Artificial Intelligence, Natural Language Processing
Contents

Abstract

Prompt engineering has emerged as an indispensable technique for extending the capabilities of large language models (LLMs) and vision-language models (VLMs).This approach leverages task-specific instructions, known as prompts, to enhance model efficacy without modifying the core model parameters.Rather than updating the model parameters, prompts allow seamless integration of pre-trained models into downstream tasks by eliciting desired model behaviors solely based on the given prompt.Prompts can be natural language instructions that provide context to guide the model or learned vector representations that activate relevant knowledge.This burgeoning field has enabled success across various applications, from question-answering to commonsense reasoning.However, there remains a lack of systematic organization and understanding of the diverse prompt engineering methods and techniques.This survey paper addresses the gap by providing a structured overview of recent advancements in prompt engineering, categorized by application area.For each prompting approach, we provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized.We also delve into the strengths and limitations of each approach and include a taxonomy diagram and table summarizing datasets, models, and critical points of each prompting technique.This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.

Summary

This paper presents a systematic survey of prompt engineering techniques specifically tailored for large language models (LLMs) and vision-language models (VLMs). It explores the importance of prompts—task-specific instructions that guide model behavior without altering core parameters—across a diverse set of applications such as question-answering and commonsense reasoning. The authors systematically categorize and analyze 41 prompt engineering techniques by their applications, methodologies, and the models and datasets utilized. A taxonomy diagram is included to delineate the relationships of different techniques, while the strengths and limitations of each approach are discussed. The paper aims to bridge gaps in the literature concerning a systematic understanding of prompt engineering, encouraging further research and highlighting challenges and opportunities within the field.

Methods

This paper employs the following methods:

  • Zero-Shot Prompting
  • Few-Shot Prompting
  • Chain-of-Thought (CoT) Prompting
  • Automatic Chain-of-Thought (Auto-CoT) Prompting
  • Chain-of-Symbol (CoS) Prompting
  • Tree-of-Thoughts (ToT) Prompting
  • Graph-of-Thoughts (GoT) Prompting
  • System 2 Attention (S2A) Prompting
  • Thread of Thought (ThoT) Prompting
  • Chain-of-Table Prompting
  • Self-Refine Prompting
  • Code Prompting
  • Self-Harmonized Chain-of-Thought (ECHO) Prompting
  • Logic-of-Thought Prompting
  • Instance-adaptive Prompting (IAP)
  • End-to-End DAG-Path (EEDP) Prompting
  • Layer-of-Thoughts (LoT) Prompting
  • Narrative-of-Thought (NoT) Prompting
  • Buffer of Thoughts (BoT) Prompting
  • Contrastive Denoising with Noisy Chain-of-Thought (CD-CoT) Prompting
  • Reverse Chain-of-Thought (R-CoT) Prompting
  • Chain of Draft (CoD) Prompting
  • Retrieval Augmented Generation (RAG)
  • ReAct Prompting
  • Chain-of-Verification (CoVe) Prompting
  • Chain-of-Note (CoN) Prompting
  • Chain-of-Knowledge (CoK) Prompting
  • Active Prompting
  • Automatic Prompt Engineer (APE)
  • Automatic Reasoning and Tool-use (ART)
  • Contrastive Chain-of-Thought (CCoT) Prompting
  • Scratchpad Prompting
  • Program of Thoughts (PoT) Prompting
  • Structured Chain-of-Thought (SCoT) Prompting
  • Chain of Code (CoC) Prompting
  • Optimization by Prompting (OPRO)
  • Rephrase and Respond (RaR) Prompting
  • Take a Step Back Prompting

Models Used

  • LLMs
  • VLMs

Datasets

The following datasets were used in this research:

  • None specified

Evaluation Metrics

  • Accuracy
  • F1 score
  • Exact match
  • F2 score

Results

  • Prompt engineering techniques enable LLMs to enhance task adaptability and performance across a broad spectrum of applications.
  • The paper categorizes and analyzes various prompting methods, revealing insights into their strengths and weaknesses.

Limitations

The authors identified the following limitations:

  • There is a lack of systematic organization and understanding of diverse prompt engineering techniques in the literature.
  • Existing methods may encounter challenges such as biases, factual inaccuracies, and difficulties in interpretability.

Technical Requirements

  • Number of GPUs: None specified
  • GPU Type: None specified

Keywords

prompt engineering large language models LLMs techniques applications systematic review

Papers Using Similar Methods

External Resources

Funding: Not specified
References: 46
Influential Citations: 8