evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksCoding

BigCodeBench

A benchmark of 1,140 (Full) / 148 (Hard) function-level Python programming tasks requiring models to compose calls across 139 diverse libraries from complex instructions, scored by calibrated Pass@1 with greedy decoding.

Codingcalibrated Pass@1Higher is better
ScoresAboutRun this benchmark

What is BigCodeBench?

A benchmark of 1,140 (Full) / 148 (Hard) function-level Python programming tasks requiring models to compose calls across 139 diverse libraries from complex instructions, scored by calibrated Pass@1 with greedy decoding. evals.report tracks reported BigCodeBench scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported BigCodeBench score: DeepSeek V3 — 40.5% (calibrated Pass@1).

ModelLabScore↓Source modelStatusDate
DeepSeek V3OpenDeepSeek40.5%—VerifiedDec 26, 2024Details
DeepSeek R1OpenDeepSeek40.5%—VerifiedJan 20, 2025Details
Gemini 2.5 ProGoogle DeepMind36.5%—VerifiedMar 25, 2025Details
DeepSeek V3 0324OpenDeepSeek35.8%—VerifiedMar 24, 2025Details
Claude 3.5 SonnetAnthropic35.1%—VerifiedJun 20, 2024Details
GPT-4oOpenAI34.5%—VerifiedMay 13, 2024Details
GPT-4.1OpenAI33.8%—VerifiedApr 14, 2025Details
Claude 3.7 SonnetAnthropic33.8%—VerifiedFeb 24, 2025Details
Gemini 2.0 FlashGoogle DeepMind33.8%—VerifiedDec 11, 2024Details
Gemini 1.5 ProGoogle DeepMind32.4%—VerifiedFeb 15, 2024Details
Llama 3.1 405BOpenMeta30.4%—VerifiedJul 23, 2024Details
Mistral LargeMistral AI29.7%—VerifiedFeb 26, 2024Details
Llama 4 MaverickOpenMeta29.1%—VerifiedApr 5, 2025Details
Llama 4 ScoutOpenMeta16.9%—VerifiedApr 5, 2025Details

Each row reports the model’s calibrated Pass@1 on BigCodeBench. Click a row for the full run context.