WeirdML
Tests whether LLMs can do machine learning on novel, unusual datasets: each model writes and iteratively debugs PyTorch code over 5 feedback rounds in a sandboxed GPU container, scored on held-out test accuracy across 17 tasks (6 public, 13 hidden).
What this benchmark measures
Tests whether LLMs can do machine learning on novel, unusual datasets: each model writes and iteratively debugs PyTorch code over 5 feedback rounds in a sandboxed GPU container, scored on held-out test accuracy across 17 tasks (6 public, 13 hidden).
Rows on this page are sourced from public benchmark artifacts, leaderboard exports, or source-linked model reports. Each row keeps benchmark version, source model name, and available run details attached to the score.
The metric shown here is average accuracy. It should be interpreted within WeirdML, not compared as part of a site-wide ranking.
What to be careful about
Scores average the per-run maximum accuracy over 5 iterations across 17 tasks; six original tasks are public and thirteen are a hidden test set. Keep the reasoning setting and bootstrap standard error attached.
Frequently asked
What is WeirdML?
Tests whether LLMs can do machine learning on novel, unusual datasets: each model writes and iteratively debugs PyTorch code over 5 feedback rounds in a sandboxed GPU container, scored on held-out test accuracy across 17 tasks (6 public, 13 hidden). It is a coding benchmark measured by average accuracy.
What does average accuracy mean on WeirdML?
WeirdML reports average accuracy (%); higher is better. Scores are shown only within WeirdML and are never averaged with other benchmarks.
What is the top reported WeirdML score?
GPT-5.5 has the top reported score on WeirdML: 84.9% (average accuracy).
Why do WeirdML scores differ across runs?
Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.
Does evals.report rank models across benchmarks?
No. WeirdML scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".