FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents that resolve real software issues in code repositories (long-horizon, multi-turn tool use). Technical level: Advanced — the paper is built around GRPO-style policy optimization and advantage estimation, though its core idea is describable in plain terms. Scope: The paper proposes FC-SWE, a training framework that turns failed code patches into learning signal by training a policy to recover from its own verifier-reported failures, and evaluates it on all 500 SWE-bench Verified tasks with Qwen3.5-4B and SWE-agent.

What This Paper Is About

Repository-level software engineering agents must inspect files, edit code, run tests, and revise across many turns. Existing RL methods such as GRPO sample several independent attempts per issue and score each with a test-based verifier, but they throw away the diagnostic content of a failed attempt — the failing tests, tracebacks and error messages — instead of using it as context for a second try. FC-SWE makes failure productive: after a patch fails verification, the repository is reset and the agent gets the failed patch plus verifier feedback as context for a recovery attempt, with the training algorithm adapted so that both the failure and the recovery are learned from correctly.

Key Contributions

1. A failure-conditioned RL framework for SWE agents. FC-SWE extends policy training past failed verification: when a patch fails, the repository is restored to its original task state and a recovery trajectory is generated conditioned on the failed patch and the verifier's diagnostic output.
2. Trajectory-local reward assignment. Each attempted trajectory keeps the verifier outcome of its own patch, so a chain that fails and then succeeds keeps rewards of [0, 1]; a later recovery success cannot retroactively label an earlier rejected patch as correct.
3. Active-set advantage estimation. The comparison group for each issue pools all initial and recovery trajectories that were actually executed, while unexecuted recovery slots (chains that already succeeded, or padding rows) are excluded from both reward statistics and the policy loss.
4. A controlled empirical study on all 500 SWE-bench Verified tasks comparing Base, GRPO and FC-SWE under identical feedback-enabled controllers and attempt budgets, plus ablations on failure context, reward design, runtime penalties, and test-time scaling.

Main Findings

• Better first attempts and better recovery than GRPO. Averaged over three decoding seeds on all 500 SWE-bench Verified tasks, FC-SWE with Qwen3.5-4B and SWE-agent reaches 41.7% Resolved@1 and 52.8% Resolved@2, versus 38.9% and 48.5% for GRPO — gains of 2.9 and 4.3 percentage points. Conditional recovery among initial failures rises to 19.0%, compared with 15.7% for GRPO and 11.6% for the untrained Base policy.
• Untrained and cross-model reference points. The initial Qwen3.5-4B policy scores 27.9% / 36.3% / 11.6%; Qwen3.5-9B scores 41.2% / 50.4% / 15.6%; Nemotron-3-Nano-4B scores 28.2% / 35.8% / 10.6%.
• Failure context matters, not just extra retries. A variant that keeps failure-triggered retries and clean resets but withholds the failed patch, verifier feedback and prior history ("w/o failure context") drops to 40.4% Resolved@1, 47.9% Resolved@2 and 12.6% conditional recovery.
• Reward sharing hurts. Assigning the final chain outcome to every executed trajectory (so a [0, 1] chain gets rewards [1, 1]) causes a mid-training dip and lowers conditional recovery from 20.4% to 17.7% and Resolved@2 from 54.8% to 51.8% at the selected checkpoints (one decoding seed).
• Runtime penalties stabilize shared-reward training. Removing the −5 token-advantage penalty for invalid tool calls and malformed thinking from the shared-reward variant led to a collapse from 76.6% online success at step 28 to 2.3% at step 31; this run was not evaluated downstream.
• Feedback beats resampling only for the trained policy. At a matched two-attempt budget, feedback-conditioned retries recover less often than independent resampling for Base (11.6% vs 13.7%) and GRPO (15.7% vs 18.6%), but more often for FC-SWE (19.0% vs 17.0%).
• Verifier evidence adds value beyond the patch. For the final FC-SWE policy, second-attempt context of no failure information gives 17.0% conditional recovery, patch only gives 15.7%, and patch plus verifier evidence gives 19.0% — a 3.3-point gain over patch-only conditioning.
• Test-time scaling beyond the training horizon. Although trained with at most two attempts per chain, the same FC-SWE policy reaches 70.7% cumulative resolution with up to eleven verifier-guided attempts, versus 67.3% for GRPO and 61.1% for Base.
• Lower inference token cost at matched resolution. At the lowest-cost evaluated budgets reaching at least 50% resolution, FC-SWE, GRPO and Base use approximately 3.15M, 4.39M and 8.22M tokens per task; at 60%, 5.08M, 6.29M and 13.57M. The authors state this compares inference token cost, not training compute or measured wall-clock speed.
• Online training behaviour. Over the first 70 optimization steps, FC-SWE's first-attempt success averages 0.7 percentage points below GRPO's, but its second attempt adds 10.6 percentage points, giving a 9.9-point average advantage over GRPO's single-attempt rate.

Methodology in Plain English

The team starts from an open policy (Qwen3.5-4B) and trains it with reinforcement learning on 4,518 tasks drawn from ten R2E-Gym repositories, using OpenHands to generate training rollouts and SWE-agent for evaluation.

For each training issue, the GRPO baseline samples eight independent trajectories (G = 8), runs each one to a submitted patch, and scores it with the official SWE-bench verification procedure, which returns a binary pass/fail plus diagnostic output. FC-SWE uses the same setup but gives each of the eight initial trajectories its own chain of at most two attempts (R = 2). If the first attempt fails, the environment is reset to the original repository state — so no stray files, processes or tool state leak across attempts — and a second, recovery attempt is launched with the original issue, the previously submitted patch, and a truncated excerpt of verifier feedback (30,000-token carry-over, 6,000-character verifier excerpt, 3,000-character failure snippet). The recovery prompt explicitly allows keeping, revising or discarding the earlier solution.

Two design choices make this trainable. First, each trajectory is scored only by its own patch's verifier result; unexecuted positions are marked as such rather than treated as zero reward. Second, the "group" used to normalize rewards is the set of trajectories that actually ran for that issue, using the same leave-one-out baseline and standard-deviation normalization as the authors' GRPO implementation. Inactive batching slots never touch the statistics or the loss. Training segments were stitched across scheduler allocations by global optimization step, and the reported FC-SWE checkpoint is step 67, selected on multiple validation sets including a fixed 100-task SWE-bench Verified subset. Evaluation uses a 64K-token context, at most 100 model–tool interactions per attempt, temperature 0.6, top-p 0.95, top-k 20, and three decoding seeds (2, 3, 4) unless stated otherwise.

Ablations swap out one ingredient at a time — no failure context, no trajectory-local rewards, no runtime penalties — and separate inference-time experiments vary the second-attempt context and the attempt budget up to eleven.

Why This Matters

Impact on research. The paper shifts the unit of RL training for agents from the isolated attempt to the outcome-dependent recovery chain, and it identifies concrete pitfalls: sharing a chain's final reward corrupts credit assignment, and dropping failed trajectories removes exactly the contrast a recovery needs. It also adds a methodological caution to the literature — at fixed budget, feedback-conditioned retries did not beat plain resampling for untrained or GRPO-trained policies, so the benefit has to be trained for rather than assumed. The authors are explicit that their outcome-dependent comparison set does not inherit the unbiasedness guarantee of an action-independent baseline, and that their multi-attempt numbers measure verifier-assisted recovery rather than independent-sampling Pass@K.

Real-world applications (bullets):

• Automated issue resolution in large production codebases, where the agent can read failing CI output and retry rather than escalating immediately.
• Continuous-integration triage, turning test failures and tracebacks into a second, informed repair attempt.
• Developer-assistant tooling that explains and corrects its own rejected patches instead of repeating them.
• Any agent setting with a cheap programmatic checker (linters, type checkers, unit tests) where failure signals are abundant and informative.

Industry relevance. The work comes from NVIDIA with Tsinghua University co-authors and uses SWE-agent, OpenHands and the official SWE-bench harness, so its gains are measured on the standard benchmark and on token cost per task — the two numbers an engineering team actually budgets. Better recovery per attempt also means fewer wasted tokens: the reported token-per-task figures at fixed resolution thresholds favour FC-SWE over both GRPO and Base.

Future Directions

• Recovery with deployment-time verifiers. The conclusion names this explicitly as future work: all reported multi-attempt results use benchmark-test feedback between attempts, so recovery driven by real-world checks the agent has not seen remains open.
• Longer recovery chains. Training used R = 2 while evaluation scaled to eleven attempts; whether training on longer chains improves the returns that were observed to diminish at larger budgets is untested here.
• Correcting the advantage-estimation caveat. Because recovery trajectories exist only after failure, the active-set comparison is outcome-dependent; developing an estimator that restores the guarantees of an action-independent baseline is a natural next step.
• Verifier and feedback design. The authors deliberately kept the official verification procedure fixed to isolate the training comparison, and state this does not imply it is optimal for deployment — better verifiers and better failure-evidence formatting remain open, with additional verifier-only and shuffled-evidence controls deferred to an appendix.

Target Audience

Reinforcement learning researchers working on agentic and long-horizon training; engineers building or fine-tuning coding agents and wanting failure-aware retry policies; and practitioners who need to decide whether to spend a fixed inference budget on resampling or on feedback-conditioned retries. Readers should be comfortable with policy-gradient methods and group-relative advantage normalization to get the most from Sections 3 and 4, though the central idea and the benchmark table are accessible without that background.

Authors’ abstract

Repository-level software engineering (SWE) is a challenging long-horizon setting: agents must reason over extended interactions, use tools, and adapt to stateful environments. Recent work trains SWE agents with reinforcement learning methods such as Group Relative Policy Optimization (GRPO), which independently sample multiple trajectories per issue, test the resulting patches, and compare terminal rewards within a fixed group. However, this training setup does not reuse verifier feedback from failed patches as context for subsequent attempts, even though this feedback contains valuable diagnostic information about what went wrong. Training on recovery trajectories is challenging because the preceding outcome determines whether the next trajectory is generated, while the failed execution determines its conditioning context. We introduce FC-SWE, a failure-conditioned RL framework that incorporates recovery attempts into policy training. After a patch fails verification, FC-SWE restores the repository to its original task state and uses the failed patch and verifier feedback as context for a recovery trajectory. FC-SWE adapts GRPO to these chains of complete, multi-turn tool-use trajectories through two mechanisms. Trajectory-local rewards preserve each attempt's verifier outcome, preventing recovery success from rewarding an earlier failed patch. Active-set advantage estimation forms a comparison group from all initial and recovery trajectories actually executed for the same issue, so failed attempts remain in the group while unexecuted attempts are excluded. On all 500 SWE-bench Verified tasks under a verifier-assisted protocol, FC-SWE with Qwen3.5-4B and SWE-agent achieves 41.7% Resolved@1 and 52.8% Resolved@2, compared with 38.9% and 48.5% for GRPO. Although trained with at most two attempts per chain, FC-SWE reaches 70.7% Resolved@11 under an eleven-attempt test-time budget.