evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksCoding

DeepSWE

A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers.

Coding% resolvedHigher is better
ScoresAboutRun this benchmark

What is DeepSWE?

A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers. evals.report tracks reported DeepSWE scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported DeepSWE score: GPT-5.5 — 70.05% (% resolved).

Community runs: 3 independent reproductions are listed separately from official scores, each shown with its own source and caveats — never merged or averaged. See how scores are labeled.

ModelLabScore↓Source modelStatusDate
GPT-5.5OpenAI70.05%gpt-5-5OfficialApr 23, 2026Details
Claude Opus 4.8Anthropic58%Claude Opus 4.8 [max]VerifiedMay 28, 2026Details
GPT-5.4OpenAI55.53%gpt-5-4OfficialMar 5, 2026Details
Claude Opus 4.7Anthropic54.20%claude-opus-4-7OfficialApr 16, 2026Details
GLM-5.2OpenZ.ai46.2%GLM-5.2VerifiedJun 16, 2026Details
Nex-N2-ProOpenNex AGI33.6%Nex-N2-ProVerifiedJun 2, 2026Details
Claude Sonnet 4.6Anthropic31.56%claude-sonnet-4-6OfficialFeb 17, 2026Details
Gemini 3.5 FlashGoogle DeepMind28.32%gemini-3-5-flashOfficialMay 19, 2026Details
Claude Opus 4.6Anthropic27.06%claude-opus-4-6OfficialFeb 5, 2026Details
Kimi K2.6OpenMoonshot AI23.89%kimi-k2-6OfficialApr 20, 2026Details
GLM-5.1OpenZ.ai17.48%glm-5-1OfficialApr 7, 2026Details
MiniMax M3OpenMiniMax13.3%MiniMax-M3 [default]CommunityJun 1, 2026Details
Gemini 3.1 Pro PreviewGoogle DeepMind9.88%gemini-3-1-pro-previewOfficialFeb 19, 2026Details
Nex-N2-miniOpenNex AGI8.0%Nex-N2-miniVerifiedJun 2, 2026Details
DeepSeek V4 ProOpenDeepSeek7.52%deepseek-v4-proOfficialApr 24, 2026Details
Community run · @ivanfioravanti (X)—5.3%▼2.2DeepSeek V4 Pro [reasoning max]CommunityApr 24, 2026Details
Gemini 3 FlashGoogle DeepMind5.16%gemini-3-flash-previewOfficialDec 17, 2025Details
Qwen 3.6 PlusAlibaba / Qwen2.65%qwen3-6-plusOfficialApr 2, 2026Details
Qwen 3.6 27BOpenAlibaba / Qwen1.79%Qwen 3.6 27B (FP8)CommunityApr 22, 2026Details
Claude Haiku 4.5Anthropic0.22%claude-haiku-4-5OfficialOct 15, 2025Details

Each row reports the model’s % resolved on DeepSWE. Click a row for the full run context.