SpecForge: Recursive Self-Improvement for Coding Agents via Behavioral Specifications

AI summary

Overview

Research area: Artificial intelligence — LLM-based coding agents, context-space self-improvement, and repository-level code generation.

Technical level: Advanced. The paper assumes familiarity with tree search, Pareto dominance, bandit algorithms (discounted UCB), and execution-based code evaluation.

Scope: SpecForge is a multi-frontier tree-search framework that lets a coding agent recursively improve natural-language behavioral specifications of a codebase — without touching model weights — by generating code from those specifications, testing it, and repairing the specifications from the failures.

What This Paper Is About

Coding agents built on frozen language models understand a codebase only as well as the context they are given. This paper asks whether an agent can improve that context — in the form of natural-language specifications describing what each function, class, or method does — by repeatedly generating code from the specifications, running tests, and rewriting the specifications where the code fails. The central obstacle is that program entities are interdependent: fixing one entity's specification can silently break others, producing a "whack-a-mole" effect that the authors call output coupling. The goal is a refinement loop whose gains accumulate rather than oscillate.

Key Contributions

1. Formulates specification-mediated recursive self-improvement (RSI). The paper defines behavioral specification optimization formally: quality is judged not by readability but by whether an agent can reconstruct behaviorally correct code from the specification, measured by per-entity pass rate, the set of resolved entities, and module-level pass rate. It identifies output coupling — arising from both dependency coupling (callers depend on callees) and generation coupling (the model conditions on the whole specification bundle) — as the reason naive iterative refinement fails to accumulate progress.

2. Introduces SpecForge, a multi-frontier tree search over specification states with three mechanisms: Pareto-guided state selection (an archive of non-dominated states, sampled with frequency-based weights), dependency-constrained entity selection (a callee-before-caller eligibility frontier plus a discounted-UCB bandit with gap-normalized rewards), and error-diversified expansion (multiple refinement children, each conditioned on a different subset of failing tests).

3. Demonstrates consistent gains on DevEval+ across five backbone models, with a 94.9% solve rate on Claude-4.5-Sonnet and a 5.7-percentage-point average improvement over the strongest baseline.

4. Shows the optimized specifications transfer outside the loop, improving cross-language translation (Python to Java) and new-feature implementation on SWE-Dev, where they outperform the benchmark's own Product Requirements Document.

Main Findings

• Best solve rate on every backbone. On DevEval+, SpecForge reaches 88.9% (Qwen3-235B-A22B), 89.6% (Gemini-2.5-Flash), 90.9% (GPT-4o), 91.0% (Gemini-3.1-Pro), and 94.9% (Claude-4.5-Sonnet), with corresponding module pass rates of 91.5%, 92.1%, 93.2%, 93.1%, and 96.1%.

• Margin over the strongest baseline. DocSearch is the only baseline optimizing the same objective and scores 81.5–90.4% solve rate; SpecForge exceeds it by 3.8–7.4 points in solve rate (5.7 on average) and 3.6–6.6 points in pass rate. The gap is largest on the two weakest backbones (7.4 points on both Qwen3-235B-A22B and Gemini-2.5-Flash) and narrows to 3.8–4.5 points on the two strongest — which the authors attribute to the Pareto archive enabling continuation when no single refinement passes a no-regression test.

• Static documentation is far behind. RepoAgent and DocAgent score 14.5–35.3%, exposing a documentation–specification gap: text written for humans is too imprecise to serve as a code-generation contract.

• Monolithic prompt optimizers plateau. ACE reaches 62.8% on GPT-4o; MIPROv2 and GEPA lag at 28.5–50.4%. The authors attribute this to attribution failure (the optimizer cannot localize which entity caused a failure) and coupling amplification (a single rewrite changes all entities at once).

• Commercial agents trail substantially. Cursor with both topological ordering and test feedback reaches 52.1% and Claude Code with the same augmentations reaches 61.9% on Claude-4.5-Sonnet — 33.0–42.8% below SpecForge.

• Consistent absolute gains. SpecForge delivers 23.5–49.1 percentage points of solve-rate improvement over the best iterative-refinement baseline across all models, which the authors read as evidence the method targets a structural difficulty shared across backbones.

• Convergence behavior differs sharply. Over a budget of rollouts, SpecForge rises from 41% to 93% pass rate, with the steepest gain (+29%) in the first 10 rollouts. Iterative refinement oscillates between 45–54%; Iterative+Topo reaches 57% by B=20 then plateaus; MIPROv2 plateaus at 31% by B=20; GEPA peaks at 37% at B=20 then declines to 24%; ACE reaches 64% but shows regressions such as 38% to 30% between B=10 and B=20.

• Component ablations (GPT-4o, 10-module subset, full model at 90.9% solve / 93.2% pass). Removing reverse topological ordering costs 18.6 points, the D-UCB bandit 10.4, error diversity 9.5, and the Pareto archive 5.2.

• Cross-language transfer works. On DevEval+-Java, SpecForge specifications reach 65.7% solve / 71.3% pass versus 39.7% / 50.5% for source-only translation — a 26.0-point solve-rate gain. Providing specification plus source together is best at 76.2% / 80.4%. Notably, RepoAgent documentation (33.5% / 44.6%) is worse than source-only.

• New-feature implementation beats the official PRD. On 10 SWE-Dev packages (27 instances: 15 easy, 12 hard), SpecForge specifications achieve 74.5% Pass@1 on easy and 51.7% on hard, versus 55.7% / 44.6% for the SWE-Dev PRD (+18.8 and +7.1 points) despite never accessing the PRD or target tests.

Methodology in Plain English

The agent starts with an initial specification for every entity in a module. Each search node is a complete specification bundle plus its measured per-entity pass rates. From a node, the agent generates code for the whole module using only the specifications, runs tests, and rewrites the specification of exactly one entity based on that entity's failures — keeping every other specification unchanged so feedback stays attributable.

Three rules govern the search:

• Where to expand. Instead of ranking states by a single score, the method keeps an archive of states that are not beaten on every entity simultaneously. Nodes that are best on more entity dimensions get sampled more often, so several complementary partial solutions survive rather than one trajectory being committed to early.

• What to refine. Only entities that are still unsolved and whose callees are already solved are eligible. Recursive cycles are collapsed into a single jointly refined meta-entity. Among eligible entities, a discounted UCB bandit picks the next target, scoring each by an estimated gap-closing rate multiplied by its remaining gap — so intrinsically repairable entities are preferred over ones that merely have a lot of failures left.

• How to refine. Since failing tests vary in diagnostic value and noisy tests can actively mislead, the method samples several distinct subsets of the failure set and produces one refined candidate per subset, then keeps whichever children survive dominance testing and feeds the best child's improvement back into the bandit.

The process runs for a fixed number of iterations, and the best bundle encountered is returned. Critically, final evaluation uses hidden ground-truth tests; during the loop the agent sees only LLM-generated tests that were validated by execution.

The benchmark, DevEval+, contains 20 Python modules, 247 entities, and 760 test cases, with an average dependency depth of 4.2 — drawn from DevEval and supplemented with GitHub repositories spanning finance, biological analysis, and signal processing.

Why This Matters

Impact on research. The paper reframes self-improvement as a search problem rather than a rewrite problem, and gives a concrete diagnosis (output coupling) for why language-space optimizers oscillate on structured, interdependent contexts. It extends context-space self-improvement from prompts and playbooks to entity-level artifacts with explicit dependency structure, and it proposes an evaluation criterion — reconstructability under execution — that sidesteps the weakness of documentation-similarity metrics.

Real-world applications:

• Legacy system migration. Specifications distilled from an old codebase can guide reimplementation in a new language or framework when the original source cannot be carried over.
• Cross-language porting. The DevEval+-Java results show a 26.0-point solve-rate gain over source-only translation, relevant to teams porting Python services to Java.
• Onboarding agents to unfamiliar repositories. The resulting specifications are reusable behavioral knowledge that can be shipped as context for new-feature work, where they beat hand-written PRDs on SWE-Dev.
• Documentation debt repair. The loop converts stale or human-oriented docs into machine-actionable contracts validated by execution, without human annotation.

Industry relevance. The method requires no fine-tuning, no weight access, and no human labels — only execution feedback — so it can be layered onto frozen commercial or open-weight models behind an API. The fact that gains hold across five backbones with different capability levels suggests the benefit is not tied to a particular vendor.

Future Directions

• Scaling to larger, denser repositories. DevEval+ is 20 modules and 247 entities; whether the Pareto archive and bandit remain effective when dependency depth and module count grow substantially is untested.
• Beyond Python. Only Python is used in the main loop and only Python-to-Java in transfer. Whether the specification format and refinement signals carry to statically typed or non-JVM targets, or to languages where test generation is weaker, is not reported.
• Bootstrapping without a reference implementation. The current loop derives tests from the reference implementation and keeps only execution-validated ones. A loop that must operate on code with no working reference — the realistic legacy case — would need a different source of trustworthy feedback.
• Combining context-space and weight-space improvement. SpecForge deliberately freezes the model. Whether the accumulated specifications are a useful training corpus, or whether the two substrates compose, is left open. Reducing the rollout cost of the search (the main results use 50 rollouts) is also an open engineering question.

Target Audience

Researchers working on LLM agents, self-improving systems, and language-space optimization; engineers building repository-scale coding assistants who need reliable codebase context rather than raw documentation; and practitioners in software modernization and migration who need machine-usable behavioral descriptions of existing code. Readers should be comfortable with search algorithms and bandit methods, though the motivation and experimental narrative are accessible to anyone who has run an agent against a test suite.

Authors’ abstract

Coding agents based on frozen language models can improve their performance by refining the external context they rely on, without updating model weights. We study recursive self-improvement through behavioral specifications: natural-language code documentation that describes the behavior of program entities and guides agents in generating correct code. We formulate a self-improvement loop in which an agent generates code from its current specifications, evaluates it with execution-verified tests, and refines the specifications based on observed failures. The improved specifications then guide subsequent generation, progressively improving the agent's understanding of the codebase. However, recursive refinement is challenging because program entities are interdependent. Improving the specification of one entity can inadvertently degrade the correctness of others, a phenomenon we call output coupling. This creates a whack-a-mole effect in which local improvements cause regressions elsewhere, preventing consistent progress. We introduce SpecForge, a multi-frontier tree search framework that addresses this challenge with three mechanisms: (i) Pareto-guided state selection preserves complementary specification states and prevents premature commitment; (ii) dependency-constrained entity selection prioritizes callee-before-caller refinement; and (iii) error-diversified expansion explores multiple refinements using different subsets of test failures. On DevEval+, SpecForge achieves the best performance across five backbone models, including a 94.9% solve rate with Claude-4.5-Sonnet and a 5.7-percentage-point average improvement over the strongest baseline. The improved specifications also make the agent better at tasks outside the loop: they transfer to cross-language translation and to new-feature implementation, where they outperform the benchmark-provided requirement documents.