Venue
Neural Information Processing Systems
Domain
Artificial Intelligence, Natural Language Processing
Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior.Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created.We hypothesize two failure modes of safety training: competing objectives and mismatched generalization.Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist.We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks.We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models.Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks.Our analysis emphasizes the need for safety-capability parity-that safety mechanisms should be as sophisticated as the underlying model-and argues against the idea that scaling alone can resolve these safety failure modes.
This paper investigates the vulnerabilities of large language models (LLMs) like GPT-4 and Claude v1.3 to safety breaches known as jailbreak attacks. The authors propose two main failure modes of safety training: competing objectives, where a model's capabilities conflict with its safety goals, and mismatched generalization, where the model's safety training fails to cover the breadth of its pretraining data. The study empirically tests various methods against these models using datasets of harmful prompts and finds that vulnerabilities persist despite ongoing safety training. The results indicate that newly crafted jailbreaks outperform existing ones, suggesting that safety mechanisms may not keep pace with model capabilities. The authors emphasize the necessity for 'safety-capability parity' in future model designs, pointing out that simply scaling models will not inherently resolve these safety challenges. Key findings suggest that current safety training methodologies are insufficient to prevent adversarial misuse, necessitating an open dialogue about vulnerabilities in AI safety protocols.
This paper employs the following methods:
- Jailbreak attacks
- Safety training analysis
- Empirical evaluation of models
The following datasets were used in this research:
- Curated set of 32 prompts
- Larger synthetic dataset of 317 harmful prompts
- Persistent vulnerabilities in GPT-4 and Claude v1.3 have been found despite extensive safety training efforts
- Newly designed attacks succeed on 100% of curated red-teaming prompts and outperform existing jailbreaks
- Safety mechanisms need to be as sophisticated as the capabilities of the models
The authors identified the following limitations:
- Number of GPUs: None specified
- GPU Type: None specified
LLM safety
jailbreaks
adversarial attacks
prompt engineering
model robustness
AI safety