AnthropicClaude Sonnet
Claude 3.5 Sonnet
Anthropic · Claude Sonnet. Released Jun 20, 2024.
Claude 3.5 Sonnet is a model from Anthropic in the Claude Sonnet family, released Jun 20, 2024. evals.report tracks 33 reported Claude 3.5 Sonnet benchmark scores across GPQA Diamond, FrontierMath, AIME (OTIS Mock), MMMU-Pro, LiveCodeBench Pro, LMArena, WeirdML, Epoch Capabilities Index, and 25 more — each shown with its benchmark, metric, source status, and date, and never combined into a single ranking.
33 results
Benchmark results 33
Compare this model| Benchmark | Category | Score | Metric | Status | Date | |
|---|---|---|---|---|---|---|
| GPQA Diamond | Reasoning | 54.0% | accuracy | Official | Jun 20, 2024 | Details |
| FrontierMath | Reasoning | 1.03% | accuracy | Official | Jun 20, 2024 | Details |
| AIME (OTIS Mock) | Reasoning | 6.5% | accuracy | Official | Jun 20, 2024 | Details |
| MMMU-Pro | Multimodal | 51.5% | accuracy | Official | Jun 20, 2024 | Details |
| LiveCodeBench Pro | Coding | 572 | Codeforces Elo | Official | Jun 20, 2024 | Details |
| LMArena | Chat preference | 1297 | source-defined rating | Official | Jun 20, 2024 | Details |
| WeirdML | Coding | 40.0% | average accuracy | Official | Jun 20, 2024 | Details |
| Epoch Capabilities Index | Reasoning | 134.4 | Index | Official | Jun 20, 2024 | Details |
| Aider Polyglot | Coding | 51.6% | % correct | Official | Jun 20, 2024 | Details |
| MMLU-Pro | Reasoning | 77.2% | accuracy | Verified | Jun 20, 2024 | Details |
| MathVista | Multimodal | 67.7% | accuracy | Verified | Jun 20, 2024 | Details |
| Video-MME | Multimodal | 60.0% | accuracy | Official | Jun 20, 2024 | Details |
| LiveCodeBench | Coding | 38.1% | Pass@1 | Unverified | Jun 20, 2024 | Details |
| METR Task-Completion Time Horizons | Agents | 11.4 min | 50% time horizon | Official | Jun 20, 2024 | Details |
| SciCode | Coding | 36.6% | accuracy | Unverified | Jun 20, 2024 | Details |
| MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) | Multimodal | 68.3% | accuracy | Unverified | Jun 20, 2024 | Details |
| MultiChallenge | Reasoning | 41.4% | accuracy | Verified | Jun 20, 2024 | Details |
| LongBench v2 | Reasoning | 46.7% | accuracy | Official | Jun 20, 2024 | Details |
| Video-MMMU | Multimodal | 65.8% | accuracy | Official | Jun 20, 2024 | Details |
| Arena-Hard-Auto v2.0 | Chat preference | 33.0% | % win rate | Official | Jun 20, 2024 | Details |
| EQ-Bench Creative Writing v3 | Chat preference | 1448 | Elo | Verified | Jun 20, 2024 | Details |
| AILuminate AI Safety Benchmark | Other | Very Good | Safety grade | Verified | Jun 20, 2024 | Details |
| MASK (Model Alignment between Statements and Knowledge) | Other | 72.33 | Honesty score | Verified | Jun 20, 2024 | Details |
| OCRBench v2 | Multimodal | 47.5 | accuracy | Verified | Jun 20, 2024 | Details |
| FACTS Grounding | Reasoning | 79.4% | Grounding accuracy | Verified | Jun 20, 2024 | Details |
| BigCodeBench | Coding | 35.1% | calibrated Pass@1 | Verified | Jun 20, 2024 | Details |
| SWE-bench Multimodal | Coding | 25.34% | % resolved | Verified | Jun 20, 2024 | Details |
| SuperGPQA | Reasoning | 48.16% | accuracy | Verified | Jun 20, 2024 | Details |
| EnigmaEval | Reasoning | 0.91% | accuracy | Verified | Jun 20, 2024 | Details |
| ZeroBench | Multimodal | 0.0% (pass@1) | accuracy | Verified | Jun 20, 2024 | Details |
| Gray Swan Arena (Agent Red-Teaming / Indirect Prompt Injection) | Agents | 1.85% | Attack Success Rate (ASR) | Verified | Jun 20, 2024 | Details |
| Online-Mind2Web | Agents | 29.0% | Task success rate | Official | Jun 20, 2024 | Details |
| GSO: Software Optimization Benchmark for SWE-Agents | Coding | 4.6% | Opt@1 | Official | Jun 20, 2024 | Details |