Question 1

What is ARC-AGI-1?

Accepted Answer

The original ARC-AGI-1 abstract-reasoning puzzle benchmark (semi-private set): few-shot grid transformations that are easy for humans but resist memorization. Largely cleared by 2026 frontier reasoning models, which is what motivated the harder ARC-AGI-2. It is a reasoning benchmark measured by accuracy.

Question 2

What does accuracy mean on ARC-AGI-1?

Accepted Answer

ARC-AGI-1 reports accuracy (%); higher is better. Scores are shown only within ARC-AGI-1 and are never averaged with other benchmarks.

Question 3

What is the top reported ARC-AGI-1 score?

Accepted Answer

Gemini 3.1 Pro Preview has the top reported score on ARC-AGI-1: 98% (accuracy).

Question 4

Why do ARC-AGI-1 scores differ across runs?

Accepted Answer

Harness, scaffold, reasoning effort, and prompt setup change results, so two runs of the same model can differ. evals.report keeps each score with its run context so the differences stay visible.

Question 5

Does evals.report rank models across benchmarks?

Accepted Answer

No. ARC-AGI-1 scores are shown within their own metric; evals.report never combines benchmarks into a composite ranking or a single "best model".