Domain
Artificial Intelligence
Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks.However, their unprecedented scale comes with significant computational costs.These models, often consisting of billions of parameters, require vast amounts of computational resources for execution.Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities.Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adjusting the large models over the various downstream tasks.In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large model to adapt it to a specific task or domain while minimizing the number of additional parameters introduced or computational resources required.This approach is particularly important when dealing with large-scale language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design.In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead.Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT.In addition to providing an extensive survey from an algorithmic standpoint, we also examine various real-world system designs to investigate the implementation costs associated with different PEFT approaches.This survey serves as a valuable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.
This paper provides a comprehensive survey on Parameter-Efficient Fine-Tuning (PEFT) for large models. It discusses the advancements in fine-tuning algorithms, highlighting the challenges posed by the large size and computational demands of these models. The authors categorize PEFT methods into four main types: additive, selective, reparameterized, and hybrid fine-tuning, examining their applications across various domains, including natural language processing and computer vision. The paper also outlines the computational overhead associated with these methods, evaluates their effectiveness on specific datasets and benchmarks, and suggests future research directions aimed at improving efficiency and performance in practical applications.
This paper employs the following methods:
- PEFT
- Additive Fine-Tuning
- Selective Fine-Tuning
- Reparameterized Fine-Tuning
- Hybrid Fine-Tuning
- LLaMA
- Vision Transformers
The following datasets were used in this research:
- GLUE
- OpenBookQA
- PIQA
- Social IQA
- HellaSwag
- BoolQ
- WinoGrande
- ARC-easy
- ARC-challenges
- Kinetics-400
- SSv2
- HMDB51
- MSCOCO
- ADE20K
- PASCAL VOC
- Accuracy
- Throughput
- Memory footprint
- Overview of different PEFT algorithms
- Analysis of system challenges and solutions for PEFT
- Insights into future directions for PEFT research
- Number of GPUs: None specified
- GPU Type: None specified
PEFT
Large Language Models
Fine-Tuning
Deep Learning
Transformer