SWE-Marathon
A long-horizon software-engineering benchmark of 20 realistic, multi-hour tasks (library reproductions, full-stack product clones, ML-engineering, and algorithmic optimization) that test whether frontier coding agents can autonomously complete ultra-long-horizon work; scored by binary pass@1 resolution rate with reward-hacking-resistant verifiers.
What this benchmark measures
A long-horizon software-engineering benchmark of 20 realistic, multi-hour tasks (library reproductions, full-stack product clones, ML-engineering, and algorithmic optimization) that test whether frontier coding agents can autonomously complete ultra-long-horizon work; scored by binary pass@1 resolution rate with reward-hacking-resistant verifiers.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is resolution rate (pass@1). It should be interpreted within SWE-Marathon, not compared as part of a site-wide ranking.
Frequently asked
What is SWE-Marathon?
A long-horizon software-engineering benchmark of 20 realistic, multi-hour tasks (library reproductions, full-stack product clones, ML-engineering, and algorithmic optimization) that test whether frontier coding agents can autonomously complete ultra-long-horizon work; scored by binary pass@1 resolution rate with reward-hacking-resistant verifiers. It is a agents benchmark measured by resolution rate (pass@1).
What does resolution rate (pass@1) mean on SWE-Marathon?
SWE-Marathon reports resolution rate (pass@1) (%); higher is better. Scores are shown only within SWE-Marathon and are never averaged with other benchmarks.
What is the top reported SWE-Marathon score?
Kimi K3 has the top reported score on SWE-Marathon: 42.0% (resolution rate (pass@1)).
Why do SWE-Marathon scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. SWE-Marathon scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".