SWE-bench Verified
A curated SWE-bench split for evaluating systems that resolve real software engineering issues.
What this benchmark measures
SWE-bench Verified focuses on software engineering tasks derived from real repositories. A run typically asks a system to inspect an issue, modify code, and produce a patch that passes the benchmark harness.
Rows on this page are source-backed public benchmark reports. Each score keeps the source model name, benchmark version, and available run context attached to the row.
The metric shown here is benchmark-local percent resolved. It should not be averaged with reasoning, preference, or multimodal benchmark scores.
What to be careful about
Agent scaffold, tools, repository setup, and patch validation details affect comparability.
Frequently asked
What is SWE-bench Verified?
A curated SWE-bench split for evaluating systems that resolve real software engineering issues. It is a coding benchmark measured by % resolved.
What does % resolved mean on SWE-bench Verified?
SWE-bench Verified reports % resolved (%); higher is better. Scores are shown only within SWE-bench Verified and are never averaged with other benchmarks.
What is the top reported SWE-bench Verified score?
Claude Fable 5 has the top reported score on SWE-bench Verified: 95.0% (% resolved).
Why do SWE-bench Verified scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. SWE-bench Verified scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".