When Requirements Interact: Scaling Long-Horizon Coding via Interdependent Specifications

AI summary

Overview

Research area: Software engineering / AI coding agents — synthetic training-data construction and reinforcement learning for long-horizon repository-level coding.

Technical level: Advanced (assumes familiarity with LLM agents, RL training such as GRPO, and SWE-bench-style benchmarks).

Scope: The paper introduces CodeSpecWeave, a pipeline that synthesizes coding tasks built around interdependent specifications (rather than independent checklists), releases a 4,670-task / 2,865-repository RL training set, and releases CodeSpecWeave-Bench, a 100-task, 1,044-specification benchmark.

What This Paper Is About

Real software changes must satisfy many requirements at once — interface compatibility, state consistency, error handling — and satisfying one can constrain how the others can be met. Existing training data for coding agents contains few specifications per task and little structural diversity, so agents learn to hit a primary goal while ignoring coupled constraints. The paper's goal is to build a scalable way to synthesize tasks where requirements genuinely interact, verify them reliably, and show that training on them improves long-horizon coding performance.

Key Contributions

1. CodeSpecWeave synthesis framework. Grounded in specification seeds mined from real developer activity, a synthesis agent iteratively co-develops task requests, fine-grained specifications, and executable tests, deliberately coupling requirements so that choices made for one constrain the others.
2. A trajectory-based validation and repair stage. A review agent inspects (query, tests, patch, trajectory) tuples produced by multiple solver models and multiple attempts, catching tests that pass incomplete implementations or reject valid alternatives; flagged candidates are repaired and revalidated.
3. A large training set and RL evidence. 4,670 training tasks across 2,865 repositories, used with GRPO to train two 310B-total / 15B-active MoE models, improving average scores across four benchmarks and raising DeepSWE-v1.1 from 12.39% to 25.30% (MiMo-V2.5) and from 54.42% to 62.98% (MiMo-V2.6-Flash-SFT).
4. CodeSpecWeave-Bench. 100 manually reviewed tasks from 100 repositories containing 1,044 specifications (mean 10.44 per task), which separates frontier agents with pass rates spanning 44.0% to 74.0%.

Main Findings

• Existing data is specification-poor. The authors' empirical analysis finds prior SWE training datasets fall short in both specification density and structural diversity, emphasizing the primary development goal while leaving compatibility, state consistency, and error handling unaddressed.
• Naive test generation is unreliable at scale. In a pilot, automated review found alignment flaws in 2,267 of 3,067 candidate tasks (73.9%): false positives only in 1,548 (50.5%), false negatives only in 426 (13.9%), both in 293 (9.6%), and none observed in 800 (26.1%).
• Filtering is aggressive. From a batch of 14,905 initial tasks, difficulty filtering retained 3,275 candidates, and review approved 766 verified tasks.
• The benchmark is hard and discriminative. Pass rates on CodeSpecWeave-Bench range from 44.0% to 74.0%; the strongest model, GPT-6 Astra, reaches 74.0% and still fails 26% of attempts.
• More tokens do not mean more success. GPT-6 Astra has the highest pass rate with the lowest mean output length (17.2k tokens); within Codex, pass rate decreases monotonically with output length across the four GPT models, from 74.0% at 17.2k tokens to 66.0% at 31.8k. Claude Opus 5 beats Qwen3.8-Max by 6.1 percentage points while producing roughly 47% fewer output tokens.
• CodeSpecWeave gives the best average across benchmarks. MiMo-V2.5 averages 50.15% versus 44.69% (vanilla), 42.05% (SWE-PI), and 49.01% (MiMo Coding); MiMo-V2.6-Flash-SFT averages 67.60% versus 60.54%, 64.66%, and 64.09%.
• The biggest gains are on long-horizon tasks. On DeepSWE-v1.1, CodeSpecWeave reaches 25.30% and 62.98%, while SWE-PI on MiMo-V2.5 actually shows negative transfer (6.78% versus 12.39% vanilla).
• Trained agents work harder and verify more. MiMo-V2.5 generates 4.8x as many thinking tokens and Flash-SFT 3.0x; modified files rise from 4.47 to 9.79 and 4.09 to 6.17, changed lines from 451 to 3,632 and 350 to 667, and edit–test loops from 3.48 to 4.34 and 2.96 to 4.90. Trajectories with at least two loops rise from 63.0% to 72.7% and 69.0% to 78.3%.
• Gains hold across specification counts and categories. Flash-SFT improves 10.2–12.5 percentage points across specification-count groups (largest in the 9–11 and 12–31 groups) and 9.5–14.9 points across the seven specification categories.
• Training is stable but the models adapt differently. Policy entropy stays in the 0.29–0.40 range while scores rise; MiMo-V2.5 more than doubles its response length (single-turn elaboration) whereas Flash-SFT nearly doubles its agent turns (multi-turn interaction).
• Two recurring failure modes. Patches target a specification but enforce it only in some boundary settings, and patches that satisfy one requirement violate another constraining the same state — e.g., a whole-state copy in Iroha shares transaction objects, so adding a signature to the copy also mutates the original.

Methodology in Plain English

The researchers start by harvesting "specification seeds" from open-source repositories — commit histories, issue discussions, and pull requests — stripping out chatter and raw code, and deduplicating to 104,182 candidate specification units across six natural languages. These seeds act as a reference guide for what realistic engineering requirements look like.

A synthesis agent (MiMo-V2.5-Pro running in a Claude Code harness) then explores a target repository and writes a task in three coupled pieces: the natural-language request, the explicit specifications, and the executable tests. Crucially, it is instructed to tie requirements to the same state transitions or core components, so that a solution satisfying one requirement in isolation still breaks the broader contract. It runs its own generated tests in the repository and refines them from runtime feedback.

Because tests written this way are frequently wrong, a second stage runs real solver models (MiMo-V2.5-Pro and MiMo-V2.5-Flash) on each candidate several times and keeps tasks with mixed success — hard but solvable. A review agent then reads the request, the tests, the produced patches, and the execution trajectories together. Runs that break several requirements but still pass reveal missing checks; correct-looking runs that fail reveal tests that over-constrain the implementation. A repair agent fixes flagged tasks, which are then re-executed, re-solved, and re-reviewed before acceptance.

For training, the accepted tasks feed GRPO with 128 batch size, 16 samples per task, temperature 1.0, top-p 0.97, and a 516,096-token response limit. Evaluation uses three samples per task with a 512K context budget, temperature 1.0, and top-p 0.95, across CodeSpecWeave-Bench, DeepSWE-v1.1 (113 tasks), SWE-bench Verified (500 tasks), and SWE-bench Pro (731 tasks). The benchmark itself was manually reviewed by the authors for realism, requirement consistency, test coverage, and absence of implementation-specific restrictions, flaky tests, and environment problems.

Why This Matters

Impact on research. The paper reframes data synthesis for coding agents: the interesting axis is not how many requirements a task has but how they interact. It provides a formal contrast between additive specifications (feasible set factorizes as a product) and interdependent ones (the set is a strict subset of the product), plus a verification protocol built on observable solver trajectories rather than static inspection — directly addressing the 73.9% flaw rate the authors measured in naive generation.

Real-world applications:

• Building coding agents for large legacy codebases where a change must preserve backward compatibility for existing callers.
• Financial or transactional systems where atomicity, rollback, and error propagation must hold simultaneously with new feature behavior.
• Protocol and API development, where interface contracts, ordering, and default arguments interact (the paper's Skycoin coin-hour allocation example is exactly this shape).
• Evaluation and procurement: teams selecting coding agents can use a benchmark that distinguishes models on coupled constraints rather than single-issue fixes.

Industry relevance. The result that better performance comes with fewer output tokens (17.2k at 74.0% versus 31.8k at 66.0%) matters for inference cost and latency. The behavior shift toward cross-file edits and repeated edit–test loops also tells engineering teams what a well-trained agent's trajectory should look like — and that token spend alone is not a proxy for capability.

Future Directions

• Reduce infrastructure burden. The authors note that each task carries a container image run in a Kubernetes pod, so training and evaluation require managing many containers concurrently; cheaper sandboxing is an open engineering question.
• Cut the filtering cost. High-quality tasks depend on strong models and multiple review rounds that discard a large fraction of candidates (14,905 → 3,275 → 766 in one batch); making synthesis yield higher without weakening verification is unresolved.
• Close the headroom on interdependent requirements. The best model still fails 26% of CodeSpecWeave-Bench attempts, and the two dominant failure modes — incomplete enforcement of a boundary and violating one requirement while satisfying another — are not yet directly targeted by the training signal.
• Extend beyond repository coding. The additive-versus-interdependent framing is stated generally over design variables and feasible spaces, but the paper only demonstrates it for Python-repository-style tasks and the four benchmarks it evaluates.

Target Audience

Researchers and engineers working on coding agents, RL post-training, and synthetic data generation for software tasks; benchmark designers who care about specification coverage and verifier reliability; and platform teams evaluating or building long-horizon coding systems. Readers without background in agent RL will still follow the problem framing and results, but the training and validation details assume intermediate-to-advanced familiarity.

Authors’ abstract

To succeed in real-world long-horizon software development, coding agents must satisfy multi-dimensional task specifications, such as interface compatibility and error handling, and do so without breaking existing dependencies. However, current works lack specification in both quantity and diversity, often emphasizing primary goals while ignoring complex constraints. Synthesizing tasks to bridge this gap presents two challenges: naively adding specifications risks reducing tasks to trivial checklists rather than compelling joint reasoning, and verifying these subtle interactions risks overlooking hidden flaws or rejecting valid alternatives. To address these challenges, we introduce CodeSpecWeave, a framework for synthesizing tasks with rich, interdependent specifications. Grounded in real developer seeds, a synthesis agent iteratively develops both coupled task requirements and executable tests, while a review agent audits solution trajectories from multiple models to guarantee test reliability. Using this pipeline, we construct 4,670 training tasks across 2,865 repositories. Applying Reinforcement learning on these tasks significantly improves performance over the base model on DeepSWE-v1.1, largely outperforming existing baselines. Furthermore, we curate CodeSpecWeave-Bench, a rigorous benchmark comprising 100 tasks and 1,044 specifications, which clearly separates frontier models and reveals significant headroom in handling interdependent engineering constraints.