Run ARC-AGI-2
The same run guide is also available from the benchmark detail page.
ARC-AGI-2 ships its public eval set (120 tasks) and training set (1000 tasks) as JSON in the arcprize/ARC-AGI-2 repo, but that repo has no harness, no scoring script, and no apps/ folder. The official arcprize/arc-agi-benchmarking harness runs your own model (configured in src/arc_agi_benchmarking/models.yml via provider adapters) over those task JSONs and scores them with exact-match correctness. You generate predictions with cli/run_all.py, then grade with scoring/scoring.py whose --task_dir points at the source taskset that holds the solutions. Keep attached to any score: which split (public_eval vs training), the model config name/version, num_attempts (ARC allows 2 trials per test input), and the arc-agi-benchmarking commit, since only the public eval set is runnable locally (semi-private/private sets are leaderboard-gated and not in the repo).
1Install
git clone https://github.com/arcprize/arc-agi-benchmarking.git# from inside the cloned arc-agi-benchmarking directory:
uv synccp .env.example .env # then fill in provider keys, e.g. OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEYgit clone https://github.com/arcprize/ARC-AGI-2.git data/arc-agi2Run evaluation
# (optional) smoke test the pipeline with the bundled sample tasks and the random baseline:
uv run cli/run_all.py --config random-baseline --data_dir data/sample/tasks --save_submission_dir submissions/random-baseline-sample --log-level INFO# Real run over the ARC-AGI-2 public evaluation set with your model config (add your model to src/arc_agi_benchmarking/models.yml, then pass its name via --config; set --num_attempts 2 to match the ARC 2-trials protocol):
uv run cli/run_all.py --config <your-model-config> --data_dir data/arc-agi/data/evaluation --save_submission_dir submissions/<your-model-config> --num_attempts 2 --log-level INFO# Single-task debug run:
uv run main.py --data_dir data/sample/tasks --config random-baseline --task_id 66e6c45b --save_submission_dir submissions/random-single --log-level INFO3Score output
uv run src/arc_agi_benchmarking/scoring/scoring.py --task_dir data/arc-agi/data/evaluation --submission_dir submissions/<your-model-config> --results_dir results/<your-model-config>4Expected output
cli/run_all.py writes per-task prediction JSONs into the --save_submission_dir (README-recommended layout <save_submission_dir>/<config>/<version>/<eval_type>/, e.g. submissions/gpt-4o-2024-11-20/v1/public_eval/). scoring.py reads those plus the source taskset (which contains the solutions) and writes aggregate results (results.json) into --results_dir, reporting exact-match accuracy: a task counts as solved only when the predicted output grid matches the validated solution exactly in shape, color, and position. This is public-eval-set accuracy; do not compare it against semi-private/private leaderboard numbers.
5Submit results
ARC-AGI-2 has no automated public submission for local runs against your own model: the official leaderboard at arcprize.org/leaderboard verifies semi-private/private sets that are not in the repo. For self-reported public-eval results you can publish your code/scores (e.g. via the ARC Prize community leaderboard, which expects a link to a public code repo and a public-set score); such scores are self-reported and unverified by ARC Prize. Always attach: split (public_eval), model config name + version, num_attempts (ARC permits 2 trials per test input), and the arc-agi-benchmarking commit.