Question 1

What is Terminal-Bench 4.0?

Accepted Answer

An agentic benchmark measuring whether an AI model can complete real command-line / terminal software tasks end-to-end inside sandboxed containers, scored by resolution rate. Version 4.0 (released 2026-08-28) is a 66-task set that calibrates per-task resources (time, CPU, memory), applies fixes to 19 tasks, removes 8 tasks from 3.0 (saturation, refusals, published solutions, platform issues), and sets a flat 8-hour agent timeout. A different task set from Terminal-Bench 2.0 / 2.1, so scores are not comparable across versions. It is a agents benchmark measured by resolution rate.

Question 2

What does resolution rate mean on Terminal-Bench 4.0?

Accepted Answer

Terminal-Bench 4.0 reports resolution rate (%); higher is better. Scores are shown only within Terminal-Bench 4.0 and are never averaged with other benchmarks.

Question 3

What is the top reported Terminal-Bench 4.0 score?

Accepted Answer

Claude Code + Claude Opus 5 has the top reported score on Terminal-Bench 4.0: 51.8% (resolution rate).

Question 4

Why do Terminal-Bench 4.0 scores differ across runs?

Accepted Answer

Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.

Question 5

Does evals.report rank models across benchmarks?

Accepted Answer

No. Terminal-Bench 4.0 scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".