ML Research Wiki / Benchmarks / Code Generation / RES-Q

RES-Q

Code Generation Benchmark

Performance Over Time

📊 Showing 9 results | 📏 Metric: pass@1

Top Performing Models

Rank Model Paper pass@1 Date Code
1 QurrentOS-coder + Claude 3.5 Sonnet RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 58.00 2024-06-24 📦 qurrent-ai/res-q
2 QurrentOS-coder + GPT-4o RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 46.00 2024-06-24 📦 qurrent-ai/res-q
3 QurrentOS-coder + GPT-4 Turbo RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 37.00 2024-06-24 📦 qurrent-ai/res-q
4 QurrentOS-coder + Claude 3 Opus RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 36.00 2024-06-24 📦 qurrent-ai/res-q
5 QurrentOS-coder + GPT-4 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 30.00 2024-06-24 📦 qurrent-ai/res-q
6 QurrentOS-coder + Gemini 1.5 Pro RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 30.00 2024-06-24 📦 qurrent-ai/res-q
7 QurrentOS-coder + DeepSeek-Coder-V2 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 29.00 2024-06-24 📦 qurrent-ai/res-q
8 QurrentOS-coder + Llama 3 70b RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 20.00 2024-06-24 📦 qurrent-ai/res-q
9 QurrentOS-coder + Qwen-72B-Instruct RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale 18.00 2024-06-24 📦 qurrent-ai/res-q

All Papers (9)

qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q
qurrent-ai/res-q