Arena-Hard-Auto v2.0
An automatic LLM benchmark of 500 hard real-world queries (plus 250 creative-writing prompts) sourced from Chatbot Arena, scored as a win rate against a baseline using LLM judges (GPT-4.1 and Gemini-2.5) as a cheap proxy for human preference.
What this benchmark measures
An automatic LLM benchmark of 500 hard real-world queries (plus 250 creative-writing prompts) sourced from Chatbot Arena, scored as a win rate against a baseline using LLM judges (GPT-4.1 and Gemini-2.5) as a cheap proxy for human preference.
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is % win rate. It should be interpreted within Arena-Hard-Auto v2.0, not compared as part of a site-wide ranking.
Frequently asked
What is Arena-Hard-Auto v2.0?
An automatic LLM benchmark of 500 hard real-world queries (plus 250 creative-writing prompts) sourced from Chatbot Arena, scored as a win rate against a baseline using LLM judges (GPT-4.1 and Gemini-2.5) as a cheap proxy for human preference. It is a chat preference benchmark measured by % win rate.
What does % win rate mean on Arena-Hard-Auto v2.0?
Arena-Hard-Auto v2.0 reports % win rate (%); higher is better. Scores are shown only within Arena-Hard-Auto v2.0 and are never averaged with other benchmarks.
What is the top reported Arena-Hard-Auto v2.0 score?
o3 has the top reported score on Arena-Hard-Auto v2.0: 85.9% (% win rate).
Why do Arena-Hard-Auto v2.0 scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. Arena-Hard-Auto v2.0 scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".