ML Research Wiki / Benchmarks / Code Generation / HumanEval

HumanEval

Code Generation Benchmark

Performance Over Time

📊 Showing 7 results | 📏 Metric: Pass@1

Top Performing Models

Rank Model Paper Pass@1 Date Code
1 DeepSeek-R1 (MGDebugger) From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging 100.00 2024-10-02 📦 YerbaPage/MGDebugger
2 LLaMA 3 Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step 99.40 2024-02-25 📦 floridsleeves/llmdebugger
3 QualityFlow (Sonnet-3.5) QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks 98.80 2025-01-20 -
4 Phi-2 Planning-Driven Programming: A Large Language Model Programming Workflow 98.20 2024-11-21 📦 you68681/lpw
5 EG-CFG (DeepSeek-V3-0324) Execution Guided Line-by-Line Code Generation 96.95 2025-06-12 📦 boazlavon/eg_cfg
6 Mistral 7B MapCoder: Multi-Agent Code Generation for Competitive Problem Solving 93.90 2024-05-18 📦 md-ashraful-pramanik/mapcoder 📦 Luoji-zju/Agents4PLC_release
7 L2MAC (GPT-4) L2MAC: Large Language Model Automatic Computer for Extensive Code Generation 90.20 2023-10-02 📦 samholt/l2mac 📦 vanderschaarlab/l2mac

All Papers (7)

YerbaPage/MGDebugger
floridsleeves/llmdebugger
you68681/lpw
boazlavon/eg_cfg
md-ashraful-pramanik/mapcoder Luoji-zju/Agents4PLC_release
samholt/l2mac vanderschaarlab/l2mac