evals.report
BenchmarksLabsCompareRun guidesIn the wild
BenchmarksCoding

WeirdML

Tests whether LLMs can do machine learning on novel, unusual datasets: each model writes and iteratively debugs PyTorch code over 5 feedback rounds in a sandboxed GPU container, scored on held-out test accuracy across 17 tasks (6 public, 13 hidden).

Codingaverage accuracyHigher is better
ScoresAboutRun this benchmark

What is WeirdML?

Tests whether LLMs can do machine learning on novel, unusual datasets: each model writes and iteratively debugs PyTorch code over 5 feedback rounds in a sandboxed GPU container, scored on held-out test accuracy across 17 tasks (6 public, 13 hidden). evals.report tracks reported WeirdML scores with the model, source, status, date, and run caveats attached — official leaderboard scores, vendor-reported launches, and clearly labeled community runs.

Top reported WeirdML score: GPT-5.5 — 84.9% (average accuracy).

ModelLabScore↓Source modelStatusDate
GPT-5.5OpenAI84.9%gpt-5.5 (xhigh)OfficialApr 23, 2026Details
Claude Opus 4.8Anthropic82.9%claude-opus-4.8 (xhigh)OfficialMay 28, 2026Details
Claude Opus 4.6Anthropic77.9%claude-opus-4.6 (high)OfficialFeb 5, 2026Details
GPT-5.3-CodexOpenAI77.9%gpt-5.3-codex (xhigh)OfficialFeb 5, 2026Details
GPT-5.4OpenAI77.7%gpt-5.4 (xhigh)OfficialMar 5, 2026Details
Claude Opus 4.7Anthropic76.4%claude-opus-4.7 (high)OfficialApr 16, 2026Details
GPT-5.2OpenAI72.2%gpt-5.2 (xhigh)OfficialDec 11, 2025Details
Gemini 3.1 Pro PreviewGoogle DeepMind72.1%gemini-3.1-pro-preview (high)OfficialFeb 19, 2026Details
Gemini 3 ProGoogle DeepMind69.9%gemini-3-pro-preview (high)OfficialNov 18, 2025Details
Claude Sonnet 4.6Anthropic66.1%claude-sonnet-4.6 (medium)OfficialFeb 17, 2026Details
Claude Opus 4.5Anthropic63.7%claude-opus-4.5 (high, 16k)OfficialNov 24, 2025Details
Gemini 3.5 FlashGoogle DeepMind62.6%gemini-3.5-flash (high)OfficialMay 19, 2026Details
Gemini 3 FlashGoogle DeepMind61.6%gemini-3-flash-preview (high)OfficialDec 17, 2025Details
GPT-5.1OpenAI60.8%gpt-5.1 (high)OfficialNov 12, 2025Details
GPT-5OpenAI60.7%gpt-5 (high)OfficialAug 7, 2025Details
GLM-5.1OpenZ.ai57.1%glm-5.1OfficialApr 7, 2026Details
Kimi K2.6OpenMoonshot AI55.9%kimi-k2.6OfficialApr 20, 2026Details
Gemini 2.5 ProGoogle DeepMind54.0%gemini-2.5-pro (thinking 16k)OfficialMar 25, 2025Details
GPT-5 miniOpenAI52.7%gpt-5-mini (high)OfficialAug 7, 2025Details
o4-miniOpenAI52.6%o4-mini (high)OfficialApr 16, 2025Details
o3OpenAI52.4%o3 (high)OfficialApr 16, 2025Details
Grok 4.20 beta reasoningxAI52.3%grok-4.20-betaOfficialMar 9, 2026Details
Grok 4.3xAI49.9%grok-4.3OfficialApr 17, 2026Details
DeepSeek V4 ProOpenDeepSeek48.9%deepseek-v4-pro (max)OfficialApr 24, 2026Details
GLM-5OpenZ.ai48.2%glm-5 (thinking)OfficialFeb 11, 2026Details
GPT-OSS-120BOpenOpenAI48.2%gpt-oss-120b (high)OfficialAug 5, 2025Details
Claude Sonnet 4.5Anthropic47.7%claude-sonnet-4.5 (thinking 16k)OfficialSep 29, 2025Details
Claude Sonnet 4Anthropic46.1%claude-4-sonnet (thinking 16k)OfficialMay 22, 2025Details
Claude Opus 4.1Anthropic45.9%claude-opus-4.1 (thinking 16k)OfficialAug 5, 2025Details
Grok 4xAI45.7%grok-4-07-09OfficialJul 9, 2025Details
Kimi K2.5OpenMoonshot AI45.6%kimi-k2.5OfficialJan 27, 2026Details
Claude Haiku 4.5Anthropic45.4%claude-haiku-4.5 (no thinking)OfficialOct 15, 2025Details
Claude Opus 4Anthropic43.7%claude-4-opus (thinking 16k)OfficialMay 22, 2025Details
NVIDIA Nemotron 3 UltraOpenNVIDIA43.5%Nemotron 3 Ultra 550B-A55BOfficialJun 4, 2026Details
Qwen 3 Coder 480BOpenAlibaba / Qwen41.2%qwen3-coderOfficialJul 22, 2025Details
Gemini 2.5 FlashGoogle DeepMind40.9%gemini-2.5-flash (thinking 16k)OfficialApr 17, 2025Details
Claude 3.5 SonnetAnthropic40.0%claude-3.6-sonnetOfficialJun 20, 2024Details
DeepSeek V3.2OpenDeepSeek39.5%deepseek-v3.2-exp (thinking)OfficialDec 1, 2025Details
Kimi K2 InstructOpenMoonshot AI39.4%kimi-k2OfficialJul 11, 2025Details
GPT-4.1OpenAI39.0%gpt-4.1OfficialApr 14, 2025Details
Qwen3 235B A22B Instruct 2507OpenAlibaba / Qwen38.7%qwen3-235b-a22b-07-25OfficialJul 21, 2025Details
DeepSeek R1OpenDeepSeek36.5%deepseek-r1OfficialJan 20, 2025Details
DeepSeek V3 0324OpenDeepSeek36.1%deepseek-v3-0324OfficialMar 24, 2025Details
Llama 4 MaverickOpenMeta24.5%llama-4-maverickOfficialApr 5, 2025Details

Each row reports the model’s average accuracy on WeirdML. Click a row for the full run context.