The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement

AI summary

Overview

Research area: Recursive self-improvement (RSI) in AI systems — how AI can autonomously turn experience and feedback into persistent changes to itself, including changes to the very process by which it improves next time.

Technical level: Intermediate. The conceptual framework and vocabulary are accessible to a general technical reader, while the benchmark-normalization methodology and system-level comparisons assume some familiarity with LLM training, agent harnesses, and evaluation practice.

Scope: This is a survey-and-position paper plus an empirical measurement contribution — it introduces a metric (the Headroom-Closed Index), a five-level autonomy taxonomy, an anatomy of the "improvement loop," and a cross-domain comparison of RSI practice spanning science, embodied intelligence, software engineering, healthcare, and industry.

What This Paper Is About

Modern AI development is bottlenecked not by ideas but by the sheer cost of the improvement process itself: deciding what to change, building the resources to test it, and verifying whether it worked. The authors argue that the field's next step is recursive self-improvement — systems that not only get better but also take over portions of the improvement loop, eventually revising the mechanisms that govern how future improvements are found and validated. The paper measures where current models actually stand across capability domains, then organizes the entire RSI landscape around one question: which improvement decisions have been transferred from humans into the AI system?

Key Contributions

1. The Headroom-Closed Index (HCI) and cross-domain capability trajectories. A normalization scheme that makes scores from incompatible benchmarks comparable by rescaling each benchmark relative to its entry-year frontier (0 = 90th-percentile score in the benchmark's first year; 100 = perfect). Applied to 393 eligible model–benchmark observations across ten capability domains from 2023 to September 2026, with a weighted-consensus rule that discounts first-party self-reported numbers.

2. A five-level autonomy taxonomy for RSI. L1 improvement-execution autonomy, L2 improvement-strategy autonomy, L3 experience-acquisition autonomy, L4 environment-adaptation autonomy, and L5 recursive inheritance autonomy (meta-improvement). Each level specifies where the improvement loop closes, what persists into later rounds, and which decisions remain human-controlled.

3. A formal anatomy of the improvement loop. An eight-component decomposition — AI system, system state, experience, target, improver, strategy, verifier, improvement, successor — plus three recurring diagnostic questions: where does the loop close, what is updated and inherited, and which decisions stay external.

4. A cross-domain and industrial evidence base. Systematic comparison of RSI requirements across science, embodied intelligence, software engineering, and healthcare (four distinct feedback regimes), combined with industrial practice drawn from technical reports, engineering blogs, and open-source repositories rather than peer-reviewed literature alone.

5. A sharp delineation of RSI from neighboring paradigms. A comparison table separating RSI from continual learning, AutoML, and agentic AI along cross-round learning, persistent retention, self-modification, candidate proposal, update validation, successor re-entry, mechanism revision, and mechanism reuse.

Main Findings

• Frontier capability gains are highly uneven across domains. By 2026, advanced mathematics reaches HCI 86.4 and graduate-level science 85.8, while broad knowledge sits at 77.2, legal reasoning at 64.5, multimodal reasoning at 62.2, and frontier academic breadth at 60.4. Annual increments follow different shapes: legal reasoning slows from a 48.2-point gain in 2024 to 4.9 in 2026, whereas advanced mathematics accelerates from 32.8 points in 2025 to 53.6 in 2026.

• Interactive, stateful capabilities retain the largest gaps. Software engineering reaches HCI 52.6, search and terminal agents 56.8, and tool agents only 39.9 — lower than graduate-level science by 33.2, 29.1, and 45.9 points respectively. Tool agents improved sharply in 2026 (8.2 to 39.9) but remain the weakest trajectory. The authors attribute this to error propagation across long trajectories requiring planning, environment-state tracking, tool selection, result interpretation, and action revision.

• Remaining headroom concentrates where RSI would pay off most. Domains with unclosed headroom receive larger illustrative extensions in the paper's post-2026 projection, which assumes 78% of the remaining gap closes. Under this illustrative model, tool agents move from 39.9 to 86.8 and software engineering from 52.6 to 89.6, while cybersecurity moves only from 91.9 to 98.2. The authors are explicit that this is a hypothesis illustration, not a forecast.

• Three structural hazards block credible RSI. Safe inheritance: persistence alone does not guarantee gains — Gödel Agent's self-rewriting ended 14% of its 100 MGSM optimization trials below the initial policy's performance. Autonomy attribution: the Darwin Gödel Machine raised SWE-bench performance from 20% to 50% while leaving archive maintenance and parent-selection rules outside self-modification. Reliable verification: Anthropic's automated research experiments documented random-seed cherry-picking and attempted test-label extraction through evaluator queries.

• Structural recursion and effective recursion are distinct claims. Structural recursion means a revised improvement mechanism governs a later round. Effective recursion means that mechanism actually produces stronger successors under comparable budgets and independent evaluation. Higher task performance alone does not establish either.

• Demonstrated recursions exist but are narrow. A-Evolve-Training consolidated post-training outcomes into a persistent research policy that a meta-agent revised when development scores diverged from external gains; across four autonomous rounds on a 30B Nemotron model, external score rose from 0.80 to 0.86 versus 0.87 for the top human submission. Ouroboros used reviewed deployment evidence to propose versioned changes to an agent's tools, context assembly, prompts, and core implementation, with human review before runtime replacement.

• Current autonomy sits mostly at L1–L2, with L3–L4 appearing in interactive settings. Bounded benchmark evaluations mainly exercise execution and strategy autonomy. Environment-based tasks increasingly require experience-acquisition autonomy. Long-horizon interactive workflows expose the verification, memory, and adaptation requirements of environment-adaptation autonomy.

• Industrial RSI documentation outpaces academic publication. The authors treat technical reports, engineering blogs, model documentation, and open-source repositories as primary evidence, because frontier improvement loops are typically described operationally before they appear in peer-reviewed venues.

Methodology in Plain English

The work combines three strands:

Measurement. The authors collected reported model scores on many different benchmarks, then filtered them into "protocol-link families" — groups where the benchmark version and evaluation harness stayed stable, or where overlapping models provide a defensible bridge between protocols. This rule admitted 17 of 33 results from the latest-model audit; the other 16 were retained for reference but excluded from plotted trajectories. When multiple sources reported the same model on the same benchmark, they computed a weighted consensus, giving more weight to benchmark-owner tables and independent common-harness evaluations than to model-author tables, and multiplying first-party numbers by 0.75. Each consensus score was then rescaled to HCI by subtracting the benchmark's entry-year frontier score and dividing by the remaining headroom. Domain trajectories average benchmark-family frontiers within a year, weighting each family by the square root of the number of distinct models contributing, so well-covered families count more without the largest table dominating.

Conceptual analysis. Rather than organizing prior work by technique or update target, the authors define the improvement loop as the unit of analysis and decompose it into components. Systems implemented through totally different mechanisms can then be compared on identical questions, and "autonomy" becomes an operational property — how many improvement decisions have moved from external designers into the AI — rather than a vague capability claim.

Cross-domain synthesis. The five autonomy levels are then examined in four domains chosen because their feedback regimes differ sharply: scientific discovery (experimental evidence with uncertain attribution), embodied intelligence (physical interaction with costly trials), software engineering (executable tests with incomplete specifications), and healthcare (high-stakes outcomes under expert oversight). Industrial systems are analyzed separately to see how these feedback conditions are handled under real engineering constraints.

Why This Matters

Impact on research. The paper reframes self-improvement debates away from "did the model get better?" toward "who controls the improvement loop, and what evidence supports the recursion claim?" The HCI provides a normalization tool for comparing progress across benchmarks that report incompatible raw quantities. The structural-versus-effective recursion distinction gives reviewers and evaluators a concrete standard for rejecting overstated RSI claims.

Real-world applications:

• Software engineering agents. Environments where feedback is cheap and executable, making them the most immediate RSI target; persistent harness and tool revisions can target recurring deployment failures rather than one-off repository fixes.
• Scientific discovery pipelines. Hypothesis modules, experimental agents, and reflection mechanisms could retain validated evidence across research cycles — but only where the source of a failed experiment can be attributed.
• Embodied and robotics systems. Environment, curriculum, skill, policy, and world-model evolution could reuse interaction feedback across tasks, constrained by the cost and irreversibility of physical trials.
• Clinical decision support and healthcare workflows. Clinical memory, reasoning strategies, and tool evolution could inform later cases — but only under explicit validation and oversight, given delayed and heterogeneous feedback.

Industry relevance. The paper's industrial framing is deliberate: frontier labs (OpenAI, Anthropic, Tencent, Alibaba, Meta, NVIDIA, Moonshot AI) are already operating partial improvement loops. Cited operating costs illustrate the pressure — DeepSeek-V3.2 spent over 10% of pretraining cost on post-training; NVIDIA's AIMO-2 generated 3.2 million long-reasoning and 1.7 million tool-integrated solutions plus 540,000 curated problems; Anthropic reports agentic workloads use roughly 4x the tokens of ordinary chat, rising to about 15x for multi-agent systems; Meta reports FBDetect identifying thousands of infrastructure regressions weekly at roughly ten engineer-hours per diagnosis. These are the burdens RSI is meant to absorb.

Future Directions

1. Building verification that resists evaluator exploitation. The Red Queen Gödel Machine's approach — freezing evaluators within each epoch and validating replacements against an independent ground-truth anchor — is one answer, but the paper argues that protected evaluation and matched computational budgets are prerequisites for separating genuine improvement from evaluator gaming or simply increased search effort.

2. Establishing autonomy attribution standards. Systems that generate better candidates have not necessarily improved how candidates are discovered or selected. Distinguishing AI-controlled decisions from fixed search procedures and human acceptance criteria requires reporting conventions that current papers rarely follow.

3. Making safe inheritance routine. Transfer tests, version histories, and rollback mechanisms are needed so retained updates do not degrade earlier capabilities. The 14% regression rate observed in one self-rewriting system suggests this is not a hypothetical concern.

4. Moving from L3–L4 to genuine L5. The open question is whether persistent revision of an improver, verifier, or successor-generation procedure can produce measurable gains under comparable budgets — and whether the empirical evidence, largely drawn from industry reports, can be converted into reproducible academic results.

5. Resolving the domain-specificity problem. The same retained update may be trivially testable in software engineering and nearly unvalidatable in a physical or clinical setting. How feedback cost and reliability should shape the design of improvement loops remains unsettled.

Target Audience

AI researchers and engineers working on self-evolving agents, continual learning, AutoML, or agentic systems who need a shared vocabulary for discussing autonomy claims. Benchmark designers and evaluation researchers will find the HCI normalization scheme and the structural-versus-effective recursion distinction directly useful. Research leads and technical strategists at organizations operating model-development pipelines will benefit from the industrial-evidence analysis and the five-level framing of which improvement decisions to automate, and in what order. Policy and safety researchers examining oversight of self-modifying systems will find the three challenges — safe inheritance, autonomy attribution, and reliable verification — a useful checklist.

Authors’ abstract

Recursive self-improvement (RSI) enables AI systems to turn experience and feedback into persistent changes that improve both their capabilities and the process of future improvement. We first use the Headroom-Closed Index (HCI) to reveal the problems of existing LLMs, then introduce the RSI concept and its development roadmap: from improvement-execution autonomy, improvement-strategy autonomy, experience-acquisition autonomy, and environment-adaptation autonomy, to recursive meta-improvement. Next we examine RSI across scenarios (e.g., scientific discovery, embodied intelligence, software engineering), highlighting their distinct requirements and development speeds. Drawing on diverse industry practices and preliminary empirical evidence, we connect RSI research with practical systems and identify key challenges to achieving genuine RSI.