Future Horizons
ARC-AGI-3 fell in five months: why saturating a benchmark still is not AGI
ARC-AGI-3 launched in March 2026 with humans at 100 per cent and frontier AI at 0.51. By 3 September GPT-6 Astra scored 99.9 through OpenAI's own harness and 62.7 through a neutral one. What that measured, and what it did not.
Gabriele Masetti ·
What ARC-AGI-3 was built to measure
On 25 March 2026 the ARC Prize Foundation released ARC-AGI-3, the first interactive benchmark in a series that had until then consisted of static grid puzzles. It ships hundreds of turn-based environments, each handcrafted by human game designers, holding thousands of levels. An agent receives no instructions and no description of the goal. It has to explore, work out what winning means, build a model of how the environment reacts, and plan a sequence of actions.
The calibration result was the whole point of the exercise.
Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%. — ARC Prize Foundation, ARC-AGI-3 technical report, March 2026
The series exists because its predecessors kept falling. The ARC Prize 2025 competition drew 1,455 teams and 15,154 entries against ARC-AGI-2, and the best score on the private evaluation set was 24 per cent — real progress from near zero, and still far below the humans the puzzles had been calibrated on. ARC-AGI-3 was built to reopen a gap that engineering kept closing.
Frontier models averaged 0.51 per cent at launch. Scoring is graded on efficiency rather than completion: a level counts in full only when an agent beats it about as economically as the human baseline did, which is why the foundation describes a perfect score as agents beating every game as efficiently as humans. The environments exclude language and outside knowledge by design, leaving what ARC calls Core Knowledge priors — objects, counting, elementary geometry.
Five months from 0.51 per cent to 99.9
On 1 May 2026 ARC Prize published a replay analysis of two frontier models on the semi-private set: GPT-5.5 scored 0.43 per cent, Claude Opus 4.7 scored 0.18 per cent. The foundation's reading of the replays was that the two failed differently — one compressed the environment wrongly, the other did not compress at all — and that the aggregate scores hid the distinction.
By midsummer the curve bent. GPT-5.6 Sol reached 7.78 per cent and Claude Opus 5 reached 30.16 per cent. On 21 August Nvidia reported that its AVO agent architecture, wrapped around an unmodified Claude Opus 5, completed all 183 levels of the 25-environment public set in 6,624 actions for a perfect efficiency score — a public-set result, as Nvidia's own post states, not a semi-private or private one.
Then, on 3 September, OpenAI released GPT-6 Astra and ARC Prize published its analysis the same day. Astra scored 62.7 per cent on the standard harness at a metered cost of $26,098, and 99.9 per cent through OpenAI's provider adapter at $18,817. On the adapter run it used fewer actions than the human baseline on 96 per cent of levels, and 51.7 per cent fewer actions per level on average. A test with no AI score above one per cent in March had a near-perfect one in September.
The harness is doing part of the work
The roughly 37-point gap between Astra's two numbers is not noise, and it is not a difference in the model's weights. The standard harness gives every model the same minimal, provider-neutral interface. The provider adapter lets a model use the context-management machinery its own lab built for it, which for Astra means carrying reasoning state across calls instead of starting cold each time.
Nvidia's result makes the same point from the other direction. Nothing inside Claude Opus 5 changed — no retraining, no fine-tuning, no adjusted weight. Persistent memory and a supervisor layer sitting outside the model took it from roughly 30 per cent to a clean sweep of the public set. Whatever is being measured, a large share of it lives in the scaffolding rather than in the network.
The distinction matters for the AGI question in a specific way. A benchmark score is a property of a system, and the system includes the harness. Reading it as a property of the model is the same error as reading an exam result as a property of a student's brain while ignoring the notes they were allowed to bring in.
One instrument is also not the field's verdict. In the same week, Artificial Analysis put Astra's Intelligence Index at 61, level with its predecessor GPT-5.6 Sol and five points behind Anthropic's Claude Fable 5.1 at 66, while Epoch AI's capabilities index ranked Astra first across more than fifty tests. A system can be the most general ever measured on one instrument and an incremental release on another, which is what happens when composite indices weight different abilities.
The people who built the test say it is not the finish line
ARC Prize's own write-up of Astra is unusually blunt about the limits of its own instrument.
While we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI. — ARC Prize Foundation, analysis of GPT-6 Astra on ARC-AGI-3, 3 September 2026
The stated reason is structural rather than diplomatic. ARC-AGI-3 has a tightly bounded scope and format, its environments have deterministic, closed-ended mechanics and goals, and it does not represent the complexity and open-endedness of the real world. Saturating it was never advertised as proof of anything, at launch or since.
François Chollet, who introduced the ARC benchmark in 2019 and expanded ARC Prize into a foundation in early 2025, did update one thing: his timing. Asked at launch how long saturation would take, he had said about a year. Astra arrived at roughly twice that pace, and when asked in September whether his standing estimate of AGI around 2030 still held, he answered sooner, given that progress was running faster than he expected.
The next instruments are already scheduled. ARC-AGI-4 is due in early 2027, on a yearly cadence Chollet says is meant to keep each new benchmark unsaturated on arrival, with ARC-AGI-5 planned after it. The foundation says it is exploring how to evaluate recursive self-improvement and open-ended innovation; reporting on Chollet's remarks places longer-horizon continual and curriculum learning in the fourth version, and open-ended invention in the fifth.
Chollet has described the endpoint in operational terms rather than philosophical ones: new versions keep coming until it is no longer possible to propose a task that ordinary people can do and machines cannot. On that reading the existence of a funded, scheduled ARC-AGI-4 is itself the current answer to whether the work is finished.
Five definitions, and the benchmark settles none of them
OpenAI's 2018 charter defines AGI as highly autonomous systems that outperform humans at most economically valuable work. A score on a closed set of games is evidence about capability, not about labour substitution, and Google DeepMind's 2023 framework explains the split: defining AGI by reaching a level of labour substitution would require real-world deployment, while defining it by the capability to substitute focuses on potential.
That DeepMind paper — Morris, Sohl-Dickstein, Fiedel, Warkentin, Dafoe, Faust, Farabet and Legg — replaces the single threshold with a grid of six performance levels crossed against narrow versus general. Its published table places frontier chat models at Level 1 general, "Emerging AGI", with every higher general tier marked as not yet achieved. No peer-reviewed re-placement of a September 2026 model on that grid has appeared.
The oldest definition still in circulation is also the loosest. Shane Legg and Marcus Hutter, writing in 2007, surveyed dozens of informal accounts of intelligence and reduced them to a single line: an agent's ability to achieve goals across a wide range of environments. Legg co-founded DeepMind and co-authored the levels paper, so the lineage is direct. The operative word in the 2007 formula is range, and a fixed set of handcrafted games bounds it by construction.
Chollet's definition is the one ARC-AGI operationalises: the intelligence of a system is a measure of its skill-acquisition efficiency over a scope of tasks, with respect to priors, experience and generalisation difficulty. On that account AGI arrives when no measurable gap in learning efficiency remains between machines and people, which is why a saturated benchmark is followed by a harder one rather than by a declaration.
A fourth definition is psychometric. "A Definition of AGI", published in October 2025 by Dan Hendrycks, Yoshua Bengio, Gary Marcus, Erik Brynjolfsson and more than two dozen co-authors, scores systems across ten cognitive domains taken from Cattell-Horn-Carroll theory. It reports 27 per cent for GPT-4 and 57 per cent for GPT-5, with the deficits concentrated in long-term memory storage. A follow-up paper argues that a flat average flatters the result and that a coherence-weighted aggregate puts GPT-5 nearer 24 per cent.
A fifth definition was a contract, and it is now dead. Microsoft's agreement with OpenAI once tied AGI to systems generating $100bn in profits; in October 2025 the declaration was handed to an independent expert panel, and the April 2026 renegotiation made revenue sharing independent of technical progress, retiring the trigger. The one definition with money attached was abandoned before anyone tried to meet it.
What the forecasters are reading
Metaculus still centres its community forecast for weakly general AI on about 2028, from roughly 1,800 forecasters, and a first fully general system on about 2033, from roughly 1,900 — figures broadly unchanged into mid-September 2026, though trackers disagree on the exact month. The crowd barely moved in the fortnight a benchmark fell.
The labs moved further. Greg Brockman, OpenAI's president, told reporters at the Astra launch that it was "not unreasonable" to feel the AGI era had begun and that calling Astra the first such model was reasonable, though OpenAI's own launch page avoided the claim. Gary Marcus called the framing marketing by people lowering the bar; Ben Goertzel, who popularised the term two decades ago, described a system that is superhuman at some things and subhuman at others.
The evidence that no benchmark can give
Three kinds of evidence are missing, and none of them is a score. The first is durable learning: a system that acquires a skill after deployment and still holds it, alongside its older ones, months later. ARC-AGI-4's stated target is roughly that, which is itself a statement that the field does not consider it demonstrated.
The second is invention rather than adaptation. Every ARC environment has a solution its designers already know, and efficiency against a human baseline is measured against that known solution. A test of open-ended innovation has to reward a system for producing a tool, rule or notation nobody specified, and grading such a thing is an unsolved research problem rather than a scheduling one.
The third is economic, and it is what the charter definition actually asks for. Outperforming humans at most economically valuable work is a claim about deployed labour, visible in firms, output and displaced tasks over years, not in a $26,098 evaluation run. The expert panel written into the OpenAI-Microsoft agreement was a partial admission of that, and the clause was retired before the panel ever had to meet.
None of which argues that 2026 was uneventful. A benchmark its authors expected to hold for a year fell in five months, and a model spent fewer moves than a person on 96 per cent of levels it had never seen.
The accurate reading is narrower than either camp wants. The field's sharpest measuring instrument has told us how fast machines now adapt inside a closed, deterministic world. Every definition of general intelligence that anyone actually argues about — the charter's, DeepMind's, Chollet's, the psychometric one — is a claim about the open one.