evals.report
BenchmarksLabsCompareRun guidesIn the wild

Terminal-Bench 4.1

An agentic benchmark measuring whether an AI model can complete real command-line / terminal software tasks end-to-end inside sandboxed containers, scored by resolution rate. Version 4.1 (released 2026-11-09) keeps the 66 tasks of 4.0 and hardens their verifiers. Agent-written code now runs as an untrusted user in a separate RewardKit Worker, so it cannot read the tests or rewrite the reward file. Several overly strict verifiers were also fixed, for example music-harmony and freecad-platform-drawing. Terminal-Bench re-graded the saved 4.0 artifacts at no extra inference cost. 162 of the 11,550 trials changed score, and every agent's 4.1 score falls inside its 4.0 95% confidence interval. Trials without stored artifacts keep their 4.0 score.

Agentsresolution rateHigher is better

What is Terminal-Bench 4.1?

An agentic benchmark measuring whether an AI model can complete real command-line / terminal software tasks end-to-end inside sandboxed containers, scored by resolution rate. Version 4.1 (released 2026-11-09) keeps the 66 tasks of 4.0 and hardens their verifiers. Agent-written code now runs as an untrusted user in a separate RewardKit Worker, so it cannot read the tests or rewrite the reward file. Several overly strict verifiers were also fixed, for example music-harmony and freecad-platform-drawing. Terminal-Bench re-graded the saved 4.0 artifacts at no extra inference cost. 162 of the 11,550 trials changed score, and every agent's 4.1 score falls inside its 4.0 95% confidence interval. Trials without stored artifacts keep their 4.0 score. evals.report tracks reported Terminal-Bench 4.1 scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported Terminal-Bench 4.1 score: Claude Code + Claude Opus 5.5 — 65.8% (resolution rate).

ModelLabScore↓Source modelStatusDate
Claude Code + Claude Opus 5.5Agent systems65.8%Claude Code + Opus 5.5Official—Details
Claude Code + Claude Fable 5.2Agent systems65.8%Claude Code + Fable 5.2Official—Details
Antigravity CLI + Gemini 4 CarbonAgent systems64.5%Antigravity CLI + Gemini 4 CarbonOfficial—Details
Claude Code + Claude Sonnet 5.5Agent systems63.9%Claude Code + Sonnet 5.5Official—Details
Codex + GPT-6.1 SolAgent systems60.9%Codex + GPT-6.1 SolOfficial—Details
Codex + GPT-6 AstraAgent systems60.3%Codex + GPT-6 AstraOfficial—Details
Claude Code + Kimi K3.1 ProAgent systems59.7%Claude Code + Kimi K3.1 ProOfficial—Details
Claude Code + Claude Fable 5.1Agent systems57.6%Claude Code + Fable 5.1Official—Details
Antigravity CLI + Gemini 4 HydrogenAgent systems56.4%Antigravity CLI + Gemini 4 HydrogenOfficial—Details
Antigravity CLI + Gemini 4 ArgonAgent systems55.5%Antigravity CLI + Gemini 4 ArgonOfficial—Details
Claude Code + Claude Opus 5Agent systems53.3%Claude Code + Opus 5Official—Details
Codex + GPT-6.1 TerraAgent systems53.3%Codex + GPT-6.1 TerraOfficial—Details
Codex + GPT-6 SolAgent systems50.3%Codex + GPT-6 SolOfficial—Details
Muse Code + Muse WonderAgent systems47.0%Muse Code + Muse WonderOfficial—Details
Claude Code + Claude Fable 5Agent systems44.8%Claude Code + Fable 5Official—Details
Claude Code + GLM-5.3Agent systems41.2%Claude Code + GLM-5.3Official—Details
Claude Code + Claude Haiku 5.5Agent systems40.0%Claude Code + Haiku 5.5Official—Details
Grok Build + Grok 4.7Agent systems38.8%Grok Build + Grok 4.7Official—Details
Codex + GPT-5.6 SolAgent systems38.5%Codex + GPT-5.6 SolOfficial—Details
Claude Code + DeepSeek V4.1 Flash 1017Agent systems38.5%Claude Code + DeepSeek V4.1 Flash 1017Official—Details
Claude Code + Hy4Agent systems37.9%Claude Code + Hy4Official—Details
Claude Code + GLM-5.3-FlashAgent systems37.0%Claude Code + GLM-5.3-FlashOfficial—Details
Muse Code + Muse EmberAgent systems30.6%Muse Code + Muse EmberOfficial—Details
Claude Code + Mistral Large 4 PreviewAgent systems27.6%Claude Code + Mistral Large 4Official—Details
Claude Code + Qwen3.8 Max (0902)Agent systems27.0%Claude Code + Qwen3.8-Max-0902Official—Details
mini-SWE-agent + MiMo-V2.6-ProAgent systems27.0%mini-swe-agent + MiMo V2.6 ProOfficial—Details
Claude Code + Claude Opus 4.8Agent systems23.6%Claude Code + Opus 4.8Official—Details
Claude Code + Kimi K3Agent systems23.0%Claude Code + Kimi K3Official—Details
Codex + GPT-5.6 TerraAgent systems21.5%Codex + GPT-5.6 TerraOfficial—Details
Grok Build + Grok 4.6Agent systems19.7%Grok Build + Grok 4.6Official—Details
mini-SWE-agent + Gemini 3.8 FlashAgent systems18.2%mini-SWE-agent + Gemini 3.8 FlashOfficial—Details
Codex + GPT-5.6 LunaAgent systems17.6%Codex + GPT-5.6 LunaOfficial—Details
mini-SWE-agent + MiMo-V2.6-FlashAgent systems16.1%mini-swe-agent + MiMo V2.6 FlashOfficial—Details
Codex + GPT-6 LunaAgent systems15.2%Codex + GPT-6 LunaOfficial—Details
Muse Code + Muse Spark 1.3Agent systems14.8%Muse Code + Muse Spark 1.3Official—Details
mini-SWE-agent + Inkling 2 SmallAgent systems13.0%mini-swe-agent + Inkling Small 2Official—Details
mini-SWE-agent + Gemini 3.7 FlashAgent systems12.4%mini-SWE-agent + Gemini 3.7 FlashOfficial—Details
Grok Build + Grok 4.5Agent systems12.4%Grok Build + Grok 4.5Official—Details
Claude Code + Claude Sonnet 5Agent systems12.1%Claude Code + Sonnet 5Official—Details
mini-SWE-agent + MiniMax M3Agent systems7.6%mini-swe-agent + MiniMax M3Official—Details
Claude Code + DeepSeek V4 Pro 0813Agent systems6.7%Claude Code + DeepSeek V4 Pro 0813Official—Details
mini-SWE-agent + Beam 1.0Agent systems6.4%mini-swe-agent + Beam 1.0Official—Details
Codex + GPT-5.4 miniAgent systems1.8%Codex + GPT-5.4 MiniOfficial—Details
mini-SWE-agent + Inkling SmallAgent systems0.9%mini-swe-agent + Inkling SmallOfficial—Details
mini-SWE-agent + Nemotron 3.5 Super VLAgent systems0.3%mini-swe-agent + Nemotron 3.5 Super VLOfficial—Details
mini-SWE-agent + InklingAgent systems0.0%mini-swe-agent + InklingOfficial—Details

Each row reports the model’s resolution rate on Terminal-Bench 4.1. Click a row for the full run context.