SWE-bench Pro
A harder public software-engineering agent benchmark built around professional repository tasks.
What this benchmark measures
A harder public software-engineering agent benchmark built around professional repository tasks.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is % resolved. It should be interpreted within SWE-bench Pro, not compared as part of a site-wide ranking.
What to be careful about
Track max turns and agent configuration because results are scaffold-dependent.
Frequently asked
What is SWE-bench Pro?
A harder public software-engineering agent benchmark built around professional repository tasks. It is a coding benchmark measured by % resolved.
What does % resolved mean on SWE-bench Pro?
SWE-bench Pro reports % resolved (%); higher is better. Scores are shown only within SWE-bench Pro and are never averaged with other benchmarks.
What is the top reported SWE-bench Pro score?
Claude Fable 5 has the top reported score on SWE-bench Pro: 80.0% (% resolved).
Why do SWE-bench Pro scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. SWE-bench Pro scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".