evals.report
BenchmarksLabsCompareRun guidesIn the wild

Terminal-Bench 4.1

An agentic benchmark measuring whether an AI model can complete real command-line / terminal software tasks end-to-end inside sandboxed containers, scored by resolution rate. Version 4.1 (released 2026-11-09) keeps the 66 tasks of 4.0 and hardens their verifiers. Agent-written code now runs as an untrusted user in a separate RewardKit Worker, so it cannot read the tests or rewrite the reward file. Several overly strict verifiers were also fixed, for example music-harmony and freecad-platform-drawing. Terminal-Bench re-graded the saved 4.0 artifacts at no extra inference cost. 162 of the 11,550 trials changed score, and every agent's 4.1 score falls inside its 4.0 95% confidence interval. Trials without stored artifacts keep their 4.0 score.

Agentsresolution rateHigher is better

No run guide for this benchmark yet.