ExecCritic: Learn to Test, Test to Improve for Coding Agents

AI summary

Overview

Research area: Autonomous coding agents for repository-level bug repair, specifically the training of large language models to generate executable regression tests and to use their feedback for code revision.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO, DAPO-style dynamic sampling), reinforcement learning from verifiable rewards, agentic tool-use scaffolds, supervised fine-tuning, and the SWE-bench evaluation protocol.

Scope: The paper introduces ExecCritic, a two-agent test–verify–revise framework with role-specific reinforcement learning that separates test generation from source-code repair, and evaluates it on SWE-bench Verified and SWE-bench Pro.

What This Paper Is About

Coding agents often check their own patches by writing and running tests, but when a single trajectory produces both the fix and the test, the two can share the same blind spot: a wrong patch passes a wrong test and looks validated. ExecCritic solves this by splitting the work into two separately trained agents — a Test agent that constructs a repository-native regression test, and a Repair agent that revises only the source code against that fixed test — so the validation criteria cannot be weakened to accommodate a candidate patch. The goal is to make generated execution feedback reliably helpful rather than actively harmful.

Key Contributions

1. A test–verify–revise scaffold that decouples validation from repair. Test construction is assigned to a dedicated Test agent; a fail-closed harness validates the submitted bundle (a test patch, an exact execution command, and a structured JSON behavior contract) and requires a clean failure on the buggy repository before any repair begins. The qualified test is then frozen across all source-only Repair revisions, and the official hidden evaluator retains final authority.

2. A role-specific reinforcement learning recipe. The Test agent is trained to produce behaviorally valid, discriminative tests rewarded by Base-to-Gold passage and balanced accuracy on labeled candidate patches; the Repair agent is trained on a combined objective that rewards both direct Round-0 resolution and feedback-conditioned revision, with a larger Round-0 bonus to preserve strong initial repair behavior.

3. A controlled demonstration that execution feedback is not automatically beneficial. Holding the Repair agent fixed, test quality alone determines whether feedback raises or lowers the resolved rate, motivating the separation of roles rather than treating test writing as an off-the-shelf capability.

4. A trained two-agent system reaching 72.6% on SWE-bench Verified, an 11.4-point gain over the original no-test baseline without stronger-model or oracle feedback at evaluation time, with the code released publicly.

Main Findings

• Generated-test feedback can help or harm repair depending entirely on the test source. With the Qwen Repair agent held fixed, tests from the untrained Qwen model reduce the resolved rate from a no-test baseline of 61.2% to 57.3%, while GPT-5.6-generated tests raise it to 65.3% and Oracle fail-to-pass tests raise it to 69.4%.

• Test-agent post-training dramatically improves test reliability. Base-to-Gold success (the test must fail on the buggy repository and pass after applying the reference fix) rises from 22.2% for the untrained backbone to 39.6% after supervised fine-tuning and 62.2% after RL — comparable to Codex-5.3 at 61.0%, though still below DeepSeek-V4-Flash-0731 at 73.4% and GPT-5.6-sol at 87.8%.

• Imitation learning plateaus; on-policy RL supplies the remaining gains. Base-to-Gold success scales from roughly 22% with no teacher trajectories to 27% (1K), 35% (3K), and 39% (5K), with no further gain at 10K, while the teacher remains near 73%. RL from the 5K checkpoint reaches 62.2%.

• RL requires SFT initialization to work. Training reward stays low and oscillatory when RL starts from the base model, but rises steadily from the SFT checkpoint, because an untrained policy produces too many invalid or uniformly zero-reward rollouts to generate within-group advantage signal for GRPO.

• Repair training improves both the initial fix and the use of feedback. The no-test Round-0 resolved rate rises from 61.2% to 68.3%. With RL-35B tests, the trained Repair agent reaches 72.6%, versus 64.1% for the base Repair agent with the same tests.

• The two trained components provide complementary gains. Swapping base-generated tests for RL-generated tests adds 6.8 points for the base Repair agent and 8.0 points for the trained one; Repair training adds 7.3 points under base-test feedback and 8.5 points under RL-test feedback. The paper notes these averages do not establish a statistically significant interaction.

• A 5.0-point gap to oracle feedback remains. Holding the trained Repair agent fixed, Oracle F2P feedback reaches 77.6%, suggesting further headroom through better test quality and coverage.

• Gains come with limited revision overhead. Among the 120 trajectories that entered repair, revision required an average of only 13 additional agent turns.

• Frontier agents benefit more on Verified than on Pro. GPT-5.6 improves from 81.5% to 85.7% on Verified with generated tests (Oracle: 89.7%), but only from 61.6% to 62.3% on SWE-bench Pro even though Oracle feedback reaches 73.4%. The authors attribute this to coverage: a single bundled test targets one coherent issue behavior, while Pro instances have a median of 3 and mean of 14.43 fail-to-pass test cases, versus 1 and 3.03 for Verified.

• The direct-solve bonus matters for the final result but not the initial one. A standard Repair objective without Test-to-Improve scores 68.7 / 70.3 (Round 0 / final); Test-to-Improve without the direct bonus scores 66.4 / 71.4; the full objective scores 68.3 / 72.6.

Methodology in Plain English

The researchers start from a simple observation about the failure mode: in repository repair the official evaluator is hidden, so an agent must write its own checks, and if the same trajectory writes both the patch and the check, a wrong patch can satisfy a wrong test. They formalize this as coupling — the patch and the validation evidence are drawn from the same policy — and break it by assigning the two jobs to different agents with different write permissions.

The workflow has two stages. In Learn to Test, a Test agent explores the repository using only the issue description and the buggy checkout, then submits three artifacts: a repository-native test patch, the exact command to run it, and a structured contract describing the intended behavior. A harness validates the bundle and requires it to fail cleanly on the buggy code; the agent may retry up to five times within an episode. Critically, during training and evaluation the same bundle is also run on the reference-fixed repository and on labeled candidate patches, but those outcomes stay outside the agent's context and are used only as training signal. If no submission passes the Base gate, the instance is marked as a Base-gate failure, no repair loop is launched, and the Round-0 patch is scored anyway so the full benchmark denominator is preserved.

In Test to Improve, the qualified test is frozen. The Repair agent writes an initial patch, the harness resets an isolated workspace, applies the patch and the fixed test, and returns a PASS/FAIL verdict with bounded execution output. On FAIL the agent may revise, up to five rounds of 40 turns each; on PASS the episode ends immediately and the patch is submitted without further agent deliberation. Operational errors are reported as diagnostics, never as behavioral verdicts, and never establish local acceptance. If nothing passes, the controller force-submits the latest candidate.

Training uses Qwen-3.5-35B-A3B for both roles. The Test agent is first fine-tuned on 5,000 chain-of-thought trajectories collected from a frozen stronger teacher on SWE-ReBench — admitted for protocol validity and clean Base failure only, never filtered by gold outcomes — and then optimized with GRPO over 200 steps using DAPO-style dynamic sampling that keeps only issue groups with nonzero reward variance. Its reward is tiered: negative for failing to produce a valid submission, zero for a bundle that does not fail cleanly on Base or does not pass on the reference fix, and then positive in increasing steps as balanced accuracy on labeled candidate patches improves, with a length penalty for rollouts more than eight turns longer than the shortest. The Repair agent trains for 100 steps on issues where the base model's Round-0 accuracy is below 0.4, using a single fixed oracle fail-to-pass test case as a controlled feedback source whose source code the agent never sees. Its reward distinguishes a valid failure (0.1), a test pass without official success (0.2), a test pass with official success after revision (1.0), and a test pass with official success at Round 0 (1.5).

Evaluation uses SWE-bench Verified, with test generation run once per issue and at most one qualified bundle retained; resolved rates are averaged over three Repair runs that reuse the same bundle, and every run uses the full benchmark denominator.

Why This Matters

The paper's central claim cuts against a common assumption in agent research: adding execution feedback is not inherently good. Feedback is only as useful as the test that produces it, and an unreliable test can actively steer revision toward an incomplete reading of the issue. This reframes test generation as a trainable, specialized capability rather than a byproduct of general coding skill, and it gives a concrete mechanism — frozen, independently generated, harness-qualified tests — for preventing an agent from moving the goalposts to match a flawed patch.

Real-world applications:

• Automated maintenance of large production repositories, where issues depend on call sites, edge cases, and implementation details spread across a codebase that no single model call can inspect.
• CI/CD regression test generation, producing repository-native tests that fail on the buggy state and pass on the fixed state — a directly useful artifact independent of any patch.
• Code review and patch validation tooling, where a fixed, independently authored check prevents a contributor's own test suite from rubber-stamping a change.
• Developer assistant products and IDE integrations that need to decide when a fix is trustworthy enough to stop iterating and submit.

Industry relevance is substantial: the work comes from Microsoft Research, targets SWE-bench, the de facto industry benchmark for autonomous repository repair, and releases code publicly. The multi-agent specialization pattern — separate policies with separate write permissions for proposing a change and validating it — is directly transferable to any agentic system where an agent both acts and judges its own actions, well beyond code.

Future Directions

• Broadening behavioral coverage. The 5.0-point gap to oracle feedback and the much smaller gains on SWE-bench Pro (whose instances carry far more fail-to-pass test cases) suggest that one bundle targeting one coherent behavior per issue is a real limitation. Producing complementary bundles or multi-behavior tests is the obvious extension.
• Addressing correlated errors between roles. The paper is explicit that independence refers to generation context and write permissions, not statistical independence: the Test and Repair agents can still share the same mistaken interpretation of the issue. Whether independent sampling, adversarial test generation, or diversity-enforcing objectives reduce this residual coupling is open.
• Evolving the qualification harness itself. The authors describe an evidence-gated harness-evolution mechanism in the appendix but deliberately keep it out of the evaluated runtime; systematically relaxing or strengthening validation rules without introducing new failure modes remains unresolved.
• Compute-matched comparison and scaling. The reported 72.6% reflects additional test-generation and revision computation rather than a compute-matched improvement. Establishing the gain per unit of compute, and whether the recipe transfers to other backbones, benchmarks, and non-code domains, are natural next questions.

Target Audience

This paper is most valuable to researchers and engineers working on agentic code generation and reinforcement learning for LLM agents — particularly those building or evaluating repository-level repair systems, designing verifiable-reward training pipelines, or studying multi-agent decomposition where one component acts and another judges. Practitioners building developer tooling on top of coding agents will benefit from the concrete architectural lesson that validation criteria must be independent of the artifact being validated. Benchmark designers and anyone reasoning about why self-verification fails will find the formal treatment of trajectory coupling useful. Readers should be comfortable with RL terminology and the SWE-bench evaluation setup; those without that background will find the framework design and the result tables accessible, but the training recipe less so.

Authors’ abstract

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.