SWE-bench Multilingual
A software-engineering benchmark of 300 curated GitHub issue-resolution tasks spanning 42 repositories and 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust), measuring the percentage of real-world issues a model can resolve so that fail-to-pass and pass-to-pass tests succeed.
What this benchmark measures
A software-engineering benchmark of 300 curated GitHub issue-resolution tasks spanning 42 repositories and 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust), measuring the percentage of real-world issues a model can resolve so that fail-to-pass and pass-to-pass tests succeed.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is % resolved. It should be interpreted within SWE-bench Multilingual, not compared as part of a site-wide ranking.
Frequently asked
What is SWE-bench Multilingual?
A software-engineering benchmark of 300 curated GitHub issue-resolution tasks spanning 42 repositories and 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust), measuring the percentage of real-world issues a model can resolve so that fail-to-pass and pass-to-pass tests succeed. It is a coding benchmark measured by % resolved.
What does % resolved mean on SWE-bench Multilingual?
SWE-bench Multilingual reports % resolved (%); higher is better. Scores are shown only within SWE-bench Multilingual and are never averaged with other benchmarks.
What is the top reported SWE-bench Multilingual score?
Ornith-1.0-397B has the top reported score on SWE-bench Multilingual: 78.9% (% resolved).
Why do SWE-bench Multilingual scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. SWE-bench Multilingual scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".