SWE-rebench
A continuously updated, contamination-free agentic software-engineering benchmark from Nebius that mines fresh post-cutoff GitHub issue/PR tasks and evaluates LLM agents under a fixed ReAct scaffold, reporting the monthly decontaminated resolved rate.
What this benchmark measures
A continuously updated, contamination-free agentic software-engineering benchmark from Nebius that mines fresh post-cutoff GitHub issue/PR tasks and evaluates LLM agents under a fixed ReAct scaffold, reporting the monthly decontaminated resolved rate.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is Resolved rate (pass@1). It should be interpreted within SWE-rebench, not compared as part of a site-wide ranking.
Frequently asked
What is SWE-rebench?
A continuously updated, contamination-free agentic software-engineering benchmark from Nebius that mines fresh post-cutoff GitHub issue/PR tasks and evaluates LLM agents under a fixed ReAct scaffold, reporting the monthly decontaminated resolved rate. It is a coding benchmark measured by Resolved rate (pass@1).
What does Resolved rate (pass@1) mean on SWE-rebench?
SWE-rebench reports Resolved rate (pass@1) (%); higher is better. Scores are shown only within SWE-rebench and are never averaged with other benchmarks.
What is the top reported SWE-rebench score?
Claude Opus 4.6 has the top reported score on SWE-rebench: 65.3% (Resolved rate (pass@1)).
Why do SWE-rebench scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. SWE-rebench scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".