DeepSWE
A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers.
What this benchmark measures
A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is % resolved. It should be interpreted within DeepSWE, not compared as part of a site-wide ranking.
What to be careful about
All leaderboard scores use mini-swe-agent; store harness, reasoning effort, sample count, confidence interval, and cost metadata.
Frequently asked
What is DeepSWE?
A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers. It is a coding benchmark measured by % resolved.
What does % resolved mean on DeepSWE?
DeepSWE reports % resolved (%); higher is better. Scores are shown only within DeepSWE and are never averaged with other benchmarks.
What is the top reported DeepSWE score?
GPT-5.5 has the top reported score on DeepSWE: 70.05% (% resolved).
Are community DeepSWE runs official?
No. Community runs are independent reproductions shown separately from official scores, each labeled with its source and run caveats, and never merged with the official number.
Why do DeepSWE scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. DeepSWE scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".