LLM eval scores with sources, run context, and caveats attached.
Track official benchmark scores, vendor-reported model launches, and clearly labeled community runs — each kept with its source, run context, and caveats. No composite ranking. No fake “best model.” Just the receipt.
Index 04
Benchmarks 06
See allA curated SWE-bench split for evaluating systems that resolve real software engineering issues.
A command-line agent benchmark for completing terminal tasks in reproducible task environments.
A long-horizon software-engineering benchmark with original tasks, broad repository coverage, and behavioral verifiers.
A harder public software-engineering agent benchmark built around professional repository tasks.
A broad expert-level academic question-answering benchmark for frontier reasoning systems.
GDPval evaluates AI models agentically (shell + web access via a sandbox harness) on real-world economically valuable knowledge-work deliverables — documents, spreadsheets, slides, diagrams — spanning 44 occupations across 9 major U.S. GDP industries, scored by blind pairwise quality comparison; the Artificial Analysis GDPval-AA variant reports results as an Elo rating.
Compare
Open| Benchmark | GPT-5.5OpenAI | Claude Opus 4.8Anthropic | Gemini 3.1 Pro PreviewGoogle DeepMind | DeepSeek V4 ProDeepSeek |
|---|---|---|---|---|
| SWE-bench Verified% resolved | 80.6% | 88.6% | 75.6% | 80.6% |
| DeepSWE% resolved | 70.05% | 58% | 9.88% | 5.3% |
| SWE-bench Pro% resolved | 58.6% | 69.2% | 46.10% | 55.4% |
Run guides 04
All guidesSWE-bench Verified is run locally with the official `swebench` harness (Docker-based).
Terminal-Bench evaluates AI agents on real terminal/command-line tasks inside sandboxed Docker containers.
DeepSWE is a 113-task long-horizon SWE benchmark (TypeScript, Go, Python, JavaScript, Rust) using the Harbor task format with program-based behavioral verifiers.
GPQA Diamond is a 448-question graduate-level science multiple-choice set; the score is exact-match accuracy on the A/B/C/D answer.