BigCodeBench
A benchmark of 1,140 (Full) / 148 (Hard) function-level Python programming tasks requiring models to compose calls across 139 diverse libraries from complex instructions, scored by calibrated Pass@1 with greedy decoding.
What this benchmark measures
A benchmark of 1,140 (Full) / 148 (Hard) function-level Python programming tasks requiring models to compose calls across 139 diverse libraries from complex instructions, scored by calibrated Pass@1 with greedy decoding.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is calibrated Pass@1. It should be interpreted within BigCodeBench, not compared as part of a site-wide ranking.
Frequently asked
What is BigCodeBench?
A benchmark of 1,140 (Full) / 148 (Hard) function-level Python programming tasks requiring models to compose calls across 139 diverse libraries from complex instructions, scored by calibrated Pass@1 with greedy decoding. It is a coding benchmark measured by calibrated Pass@1.
What does calibrated Pass@1 mean on BigCodeBench?
BigCodeBench reports calibrated Pass@1 (%); higher is better. Scores are shown only within BigCodeBench and are never averaged with other benchmarks.
What is the top reported BigCodeBench score?
DeepSeek V3 has the top reported score on BigCodeBench: 40.5% (calibrated Pass@1).
Why do BigCodeBench scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. BigCodeBench scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".