Run SimpleQA Verified
The same run guide is also available from the benchmark detail page.
SimpleQA Verified is Google DeepMind/Google Research's curated 1,000-question version of OpenAI SimpleQA, measuring short-form parametric factuality with no tools. The canonical locally-runnable harness is UK AISI's Inspect Evals, which provides an `inspect_evals/simpleqa_verified` task that runs your model on all 1k prompts and grades responses with a GPT-4.1 autorater into correct/incorrect/not-attempted. When reporting a score, state which scorer (tool-calling default vs paper-faithful `-T scorer=original`), which grader model, and which metric (Epoch uses plain proportion-correct; the paper uses an F1 harmonic mean) you used, plus the Inspect Evals task version and dataset revision.
1Install
pip install inspect-evalsexport OPENAI_API_KEY=<your-openai-key> # grader uses an OpenAI GPT-4.1 modelexport ANTHROPIC_API_KEY=<your-key> # only if evaluating an Anthropic model2Run evaluation
# Quick smoke test (default tool-calling scorer; evaluated model also grades if no grader role is bound)
inspect eval inspect_evals/simpleqa_verified --model openai/gpt-4o-mini --limit 10# Full paper-faithful reproduction run (string-matching scorer + GPT-4.1 grader)
inspect eval inspect_evals/simpleqa_verified -T scorer=original --generate-config src/inspect_evals/simpleqa/paper_config/simpleqa_verified.yaml --model-role 'grader={"model": "openai/gpt-4.1-2025-04-14", "temperature": 1.0}' --model openai/gpt-4o-mini3Score output
# Scoring is folded into the run; view results/logs with:
inspect view4Expected output
An Inspect .eval log (viewable via `inspect view`) reporting proportion correct, incorrect, not-attempted, accuracy-given-attempted, and the F-score (harmonic mean of correct and correct-given-attempted) over the 1,000 prompts. The headline 'accuracy' is the proportion correct. Do NOT compare an Inspect F-score against Epoch's leaderboard number directly: Epoch reports plain proportion-correct while the paper/Google report the F1 harmonic mean, so the two metrics differ; always state which one you used.
5Submit results
There is no open submission API for local runs. The official leaderboard is Google DeepMind's Kaggle benchmark (https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified) and Epoch AI maintains its own runs at https://epoch.ai/benchmarks/simple-qa-verified. For a self-reported score, attach: the Inspect Evals task version (e.g. changelog tag 3-B), the dataset HF revision (codelion/SimpleQA-Verified rev 5a913f57326d89935cbed0ac071494e7e624b876), the scorer used (tool-calling default vs `-T scorer=original`), the grader model (openai/gpt-4.1-2025-04-14, temp 1.0), generation settings (max-tokens), and which metric you report (proportion-correct vs F1).