Drafting Agents, Not Just Tokens: Speculative Recursive Self-Improvement for Agentic Systems

AI summary

Overview

Research area: Machine learning / automated agent self-improvement, positioned between recursive self-improvement (RSI) for agentic systems and speculative decoding for language models.

Technical level: Advanced. The paper assumes familiarity with tree search, budget-constrained optimization, and reinforcement-learning-style credit assignment.

Scope: The paper introduces Speculative Recursive Self-Improvement (SpecRSI), a two-level framework in which executable successor agents are drafted before their predecessors are fully evaluated, but adopted only after complete verification, and it reports comparative results against Janus, HGM, MetaRSI-v1, FlashEvolve, and task-specific optimizers.

What This Paper Is About

Improving an agent through successive revisions requires spending a limited budget on two competing things: proposing new edits and paying for the evaluations that prove whether those edits helped. Evaluating every intermediate version exhausts the budget before useful chains of edits pay off, while searching narrowly misses them. The paper's goal is an allocation rule that drafts successor agents ahead of their parents' full evaluation yet spends verification only where it produces verified improvement, with adoption permitted only for fully checked, admissible versions.

Key Contributions

1. A two-level framework for speculative self-improvement. The inner loop (search) extends incompletely evaluated versions but adopts only fully evaluated, admissible ones; the improver stays frozen within each search and learns from complete-search feedback between rounds.

2. Verification-Aware Tree Search (VATS). A Solver preserves affordable completion bundles (including individually low-value checks), a Scorer predicts verified gains net of subsequent costs, and a Policy selects the next draft or verification plan under the remaining budget.

3. Group Verification Optimization (GVO). Uses independent complete searches from the same start as its unit of credit; verified gains minus all search costs provide delayed credit to sampled edits and work choices, training the Drafter and Policy, while realized suffix returns and paid same-state comparisons train the Scorer. The real evaluator remains fixed.

4. A verification-grounded, cost-sensitive learning unit. The paper explicitly distinguishes its contribution from a new policy-gradient estimator, crediting independent complete successor searches rather than responses or shared-tree leaves, and charging computation, elapsed time, and failed or unfinished work.

Main Findings

• Scientific gain: In the main scientific comparison, terminal gains exceed Janus by 33.3%–40.0%, and call-axis AUBC exceeds Janus by 30.4%–33.7%, leading in all five domains (PyBaMM, AQM, FDTD, Reactor, IRP). Scientific cost indices are 158.7 for SpecRSI versus 148.1 for Janus.

• Backbone consistency: SpecRSI outperforms Janus in all 50 scientific backbone–task pairs, with terminal-gain uplifts of 17.8%–27.6% and AUBC uplifts of 15.7%–19.8%.

• System scores: Under matched D/H/M access, gains over MetaRSI-v1 are 1.6–2.9 pp. SpecRSI exceeds MetaRSI-v1 on every system backbone by 1.85–2.38 pp in four-task macro-average score across twenty backbones. Matching D+H access retains a 2.20 pp average advantage versus 2.25 pp with D/H/M. On ten backbones, SpecRSI scores higher than MetaRSI-v1 at lower target cost but 0.83–0.90% higher total-cost indices.

• Coding, prompt, and workflow profiles: SWE-Pro coding validation reaches 69.4% ± 1.2% versus HGM's 64.3% ± 1.1%, with target-cost indices of 60.8 versus 76.8 (20.8% lower). GPQA-hard100 gives 82.4% versus GEPA+Merge's 78.5%. PlanBench success is 67.8% versus EMAS's 62.7%, using 9,290 versus 10,210 task-inference tokens.

• Verification scope: At fixed full-state information, bundles raise gain from 4.8 to 6.3 pp, lower cost from 63.1 to 59.2, and reduce completion failure from 9.8% to 6.5%.

• Planning ablations: Across eight budget caps from 10% to 150%, full SpecRSI has the highest reported mean gain and lowest total-cost index of seven conditions. At 70%, full versus additive planning raises gain from 4.73 ± 0.29 to 5.99 ± 0.40 pp; planning takes 201 versus 141 ms but total cost falls from 65.1 to 58.9. Removing completion seeds reduces gain by 0.77–1.79 pp.

• Improver learning versus artifact retention: Improver updates raise reset gains from 4.1 ± 0.18 to 5.2 ± 0.21 pp and carry gains from 4.8 to 6.2 pp. Original versus updated improvers yield 5.7 versus 6.9 pp from the same released artifact.

• Component roles: Scorer-only updates yield science/code/agent gains of 5.2/5.4/5.3 pp, versus 4.7/4.9/4.8 for the drafter and 5.0/5.2/5.1 for the policy. Adding pair loss raises gain from 5.4 ± 0.23 to 6.3 ± 0.25 pp and lowers RMSE from 1.05 to 0.85, while training cost rises from 23.7 to 28.4.

• Repeated rounds: Updated SpecRSI reaches 21.7 pp by round 5, exceeding updated Meta by 3.9 pp (17.8 pp) and original-improver SpecRSI by 7.9 pp (13.8 pp). Reported incremental gains fall from 8.5 pp in round 2 to 2.1 pp in round 5.

• Resource trade-offs: Doubling offered plans from 32 to 64 raises gain from 6.3 to 6.5 pp but lowers net return from 4.7 to 4.5. Increasing workers from one to eight reduces elapsed time from 1,287 to 192 s, raising worker time from 1,287 to 1,403 s and gain from 6.3 to 6.5 pp. Batching lowers verification cost in three regimes and raises it in three. The lifecycle ledger gives SpecRSI a 5.7% lower total-cost index, 106.3 versus MetaRSI-v1's 112.7.

Methodology in Plain English

The system treats an agent as an immutable version comprising executable code plus persistent prompts, tools, and workflows. A fixed, uneditable verification contract defines a finite set of required checks and a higher-is-better score; a version counts as "admissible" only once every required check has a valid recorded outcome. Every piece of state is tracked, including which checks are outstanding, the remaining budget, attempts used, and worker count.

Within a search, three learned or fixed parts cooperate. The Solver builds bundles of checks that fit the remaining budget, honoring dependencies, shared setups, conflicts, and reserved budget, and it deliberately keeps bundles that cover individually low-value checks because those may be jointly required for adoption. The Scorer predicts the verified improvement a continuation can still deliver, net of future expense and elapsed time, without ever substituting for real evidence. The Policy then samples the next piece of work — either a draft plan (editing existing, possibly incomplete parents, at a bounded depth) or a verification plan — and ACCEPT terminates the search by returning the best fully verified version, with ties preferring the root only if it is tied for best.

Successors are drafted ahead of their parents' full evaluation: incomplete parents may expand, but each edit waits for its parent, and invalid edits are charged and end that path while preserving its valid prefix. Drafting assigns no evaluation score. Only acquired records authorize adoption.

Between search rounds, GVO collects at least two independent searches from the same root under the frozen improver. A whole-search return is computed as the verified gain minus all costs, including initial preparation and failed or unfinished work, and advantage is the difference against the mean of the other searches. Only sampled edit tokens and plan choices update the drafter and policy through a clipped objective; the scorer is trained by regression on realized, undiscounted suffix returns plus a difference-based pair loss on paid alternative continuations from the same saved state. Supervised warm-up initializes all three learned components. The verification contract and outer update rule stay fixed.

Why This Matters

Impact on research. The paper reframes the central difficulty of RSI from generating candidate revisions to spending an irreversible budget on verification. It supplies a mechanism — jointly selecting further edits and bundles of outstanding checks by predicted net verified return — that outperforms a prior co-evolution baseline (Janus) on all 50 scientific backbone–task pairs and a scheduler baseline (MetaRSI-v1) on twenty backbones, while preserving the separation between prediction and validation.

Real-world applications:

• Scientific model building and calibration, where the five tested domains (PyBaMM, AQM, FDTD, Reactor, IRP) represent costly simulation-based iteration.
• Software engineering agents, where validated repair is required and where the paper reports 69.4% coding-validation success versus HGM's 64.3%.
• Domain-expert assistants tested by harder reasoning benchmarks (GPQA-hard100 at 82.4%, AIME at 71.2%).
• Long-horizon planning agents (PlanBench at 67.8% with 9,290 task-inference tokens).

Industry relevance. The lifecycle ledger breaks total cost into acquisition (14.2 versus 14.5), training (34.8 versus 38.6), search (36.2 versus 37.2), reuse checks (4.8 versus 5.1), copies (6.1 versus 6.3), failed work (6.3 versus 6.8), and planning (3.9 versus 4.2), totaling 106.3 versus 112.7. The parallelism result — elapsed time falling from 1,287 to 192 s at the cost of worker time rising from 1,287 to 1,403 s — is directly relevant to latency-sensitive deployments. The paper reports that D+H permission (no model-weight updates) retains a 2.20 pp average advantage, which matters where weight updates are unavailable.

Future Directions

• Causal attribution and monotonicity. The paper states that GVO updates guarantee neither per-edit causal attribution nor monotonic improvement, and that intermediate edits receive only delayed credit through verified descendants. Formalizing credit assignment remains open.
• What "complete" should mean. "Complete" is defined as terminated with output and costs recorded, not every visited version fully evaluated; the consequences of this definition for search quality are left to further study.
• Where batching helps. Batching lowers verification cost in three regimes and raises it in three, so a predictive rule for when to batch is not established.
• Planning's finite useful range. The offered-plan cap improves gain up to 64 plans (6.5 pp) while net return peaks at 32 (4.7 versus 4.5), indicating an optimal planning capacity that is not characterized analytically.
• Learning the verification contract. The contract Γ and the outer update rule are held fixed by design; whether they could themselves be adapted is not addressed.

Target Audience

Researchers and engineers working on automated agent improvement, recursive self-improvement, LLM agent architecture, inference-time compute allocation, and cost-aware machine learning systems. It is most useful to readers already comfortable with tree search, budget-constrained decision making, and reinforcement-learning-style credit assignment, since the paper's distinctive claims concern where verification budget is spent rather than how models are trained from scratch. Practitioners building production agents under fixed compute or latency budgets will also find the cost and parallelism accounting directly relevant.

Authors’ abstract

Improving an agent through successive revisions requires balancing exploration and evaluation under a limited budget. A narrow search can miss useful sequences of edits, while fully evaluating every intermediate version can exhaust the budget before those sequences yield a verified improvement. We introduce Speculative Recursive Self-Improvement (SpecRSI), which drafts executable successor agents before their predecessors are fully evaluated, while requiring complete verification before adoption. Verification-Aware Tree Search (VATS) allocates the budget between further edits and outstanding checks using the predicted verified return of a continuation. Group Verification Optimization (GVO) learns the improver from completed searches while keeping the real evaluator fixed. In the main scientific comparison, terminal gains exceed Janus by 33.3%--40.0%. Additional evaluations show that SpecRSI outperforms Janus in all 50 scientific backbone--task pairs and exceeds MetaRSI-v1 by 1.85%--2.38% in macro-average system score across twenty backbones. Coding-validation success reaches 69.4% versus HGM's 64.3% at a 20.8% lower target-cost index.