StopLoss: Teaching Agents to Predict Failure and Stop Early

AI summary

Overview

Research area: AI agent reliability and cost efficiency — specifically early stopping / failure prediction for autonomous coding agents (LLM-based software-engineering agents).

Technical level: Intermediate. The method is conceptually simple (a small fine-tuned classifier over recent tool steps), but the evaluation uses cost accounting, calibrated thresholds, and bootstrapped AUC statistics that assume some familiarity with ML benchmarking.

Scope: The paper trains small (7B–8B) external monitors on completed agent trajectories and their verifier outcomes, then uses them to abort likely-to-fail coding runs before completion, evaluating whether the saved execution cost exceeds the monitoring cost and the successes lost.

What This Paper Is About

Coding agents keep editing, running tests, and calling tools on runs that will ultimately fail verification, and the cost of those runs is only recognized after the verdict arrives. The alternative — asking a frontier model to watch the agent and judge its progress — is itself expensive, and repeated judging can cost more than it saves. StopLoss asks whether a small model trained on finished trajectories and their pass/fail labels can recognize impending failure cheaply enough that stopping early actually reduces cost per solved task.

Key Contributions

1. Outcome-supervised failure monitoring. The authors show that completed trajectories plus their verifier outcomes are sufficient to train a small external monitor that scores local execution windows and triggers early stopping, without modifying the executor or accessing its internal hidden states (unlike probe-based approaches such as Ruan et al., 2026).

2. Early recognition at low monitoring cost. Outcome supervision supplies a usable failure signal at the very first scheduled check (tool step 5), at an estimated $0.000237 per check — one to two orders of magnitude below every tested general-purpose judge.

3. A practical early-stopping evaluation protocol. The paper introduces replay-based scoring that charges each method for the checks it visits and the successes it kills, reports cost per retained solve alongside false-kill and failure-capture rates, and provides a break-even condition relating execution savings to monitoring overhead and lost successes.

4. Cross-setting evidence. Cost advantages over six judge policies are demonstrated across four executor–benchmark settings and three 7B–8B backbones, with an analysis of when learned scores and calibrated thresholds transfer across executors and benchmarks.

Main Findings

• Training turns near-chance ranking into a usable signal. For Qwen2.5-7B-Instruct, training improved mean single-model tier AUC by +0.115 on Verified/Luna and +0.148 on Pro/Luna, with ensembling adding roughly 0.014–0.016 more. The paper reports the untuned base at 0.491 and the trained ensemble at 0.606 in one framing, and 0.622 in Table 2 for the same ensemble.

• The signal exists at the first checkpoint. At tool step 5 the primary ensemble reaches ordinary AUC of 0.642 on Verified/Luna and 0.645 on Pro/Luna, versus 0.494 and 0.450 for the untuned base. On Claude Sonnet 4.6 runs the corresponding first-check AUCs are 0.604 (Verified) and 0.645 (Pro).

• Beating a step-count baseline. A baseline that scores failure purely by tool-step index without reading content gets tier AUC between 0.471 and 0.528, below every trained ensemble in the corresponding setting.

• Cheap checks. The primary ensemble costs an estimated $0.000237 per check under GPU-time accounting; GPT judges cost 9–74 times as much per check when pooled across benchmarks. On Pro/Luna, cumulative monitoring across all 213 runs was $0.15 for the Qwen2.5-Coder-7B-Instruct monitor versus $2.03–$15.36 for the judges.

• Savings on Verified/Luna, but modest. On the independent 425-run stopping cohort (306 successes, 119 failures), StopLoss retains 99.0% of successful runs and lowers cost per retained solve by 4.5% relative to full execution, with cumulative monitoring of $0.44. All three GPT-judge policies raise cost in this setting (+8.4% for GPT-5.6 Luna, +38.6% for Terra, +116.8% for Sol).

• Larger savings where execution is costlier. On Pro/Luna (213 runs, 144 successes, 69 failures) StopLoss reduces cost per retained solve by 24.2% while retaining 72.2% of successes and capturing 43.5% of failures; on Pro/Sonnet it reduces cost by 30.6% with 91.6% success retention.

• Backbone transfer. All three backbones beat every judge policy on cost per retained solve in all four settings. A Qwen3-8B ensemble on Pro/Sonnet reaches a 48.1% reduction while retaining 83.2% of successes, and Qwen2.5-Coder-7B-Instruct stops 35 of 69 failing Pro/Luna runs (50.7%) where no judge stops more than 20.

• Good ranking does not guarantee savings. On Verified/Sonnet, Claude Opus 5 has the highest AUC in the study (0.747) yet raises cost per solve by 34.7%, because its monitoring spend ($127.13) is 44.9% of the original execution budget while it avoids only 16.0% of execution cost.

• Stopping is concentrated in a few runs. In an audit of Verified/Luna, across the eight correctly stopped failures the median fraction of execution cost remaining at trigger was 64.5%, and the three largest savings accounted for 65.8% of the total execution cost avoided by correct stops.

• Thresholds transfer imperfectly. Tier AUC is similar on Verified/Luna (0.622) and Pro/Luna (0.614), but the frozen threshold yields false-kill rates of 1.0% and 27.8% respectively. Successful GPT-5.6 Luna runs have a median of 16 tool steps on Pro versus 13 on Verified, giving more chances for a threshold crossing. At tool step 8 on Verified, ordinary AUC is 0.627 on GPT-5.6 Luna runs versus 0.578 on Claude Sonnet 4.6 runs.

• Sensitivity to the cost proxy. Doubling the ensemble's monitoring charge moves the Verified/Luna estimate from −4.5% to −2.9%; at five times the charge it becomes +1.8%. Both Pro cells retain savings at ten times the proxy with unchanged stopping risk.

Methodology in Plain English

The authors start from finished agent runs where the outcome is already known. For every earlier moment in a failed run, they attach an Abort label; for the last two steps of a successful run they attach a Deliver label; for earlier moments of successful runs they attach Continue. A small pretrained language model (7B–8B parameters, adapted with LoRA) is then trained to predict these labels given a short window of recent execution: the step count plus the three most recent action–observation pairs, capped at 1,600 tokens.

At runtime the model is not asked to generate text. The failure score is simply the softmax probability assigned to the Abort answer token on a single forward pass, so a check is one pass through a small model. Checks happen at tool steps 5, 8, 11, and so on. Several trained models vote, and their scores are averaged.

To decide when to stop, the authors calibrate a threshold on successful runs only: they set it so at most a target fraction (ρ = 5%) of calibration successes would have been stopped, then freeze it. During execution, any checkpoint score at or above the threshold aborts the run; the executor itself is never modified.

Evaluation replays complete recorded trajectories on SWE-bench Verified (500 tasks) and the 266-task Python subset of SWE-bench Pro (ansible, openlibrary, qutebrowser), using GPT-5.6 Luna and Claude Sonnet 4.6 as executors. Each method is charged for execution up to the point of stopping, for every monitoring call it makes, and for any successful runs it kills. The headline metric, cost per retained solve, is total cost divided by the number of successes that survive the policy. Control conditions are six prompted judges (GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, Claude Opus 5, Claude Opus 4.6, Claude Sonnet 4.6) reading identical window text, plus untuned versions of the same small backbones and a step-count-only rule.

Why This Matters

Research impact. The paper reframes agent efficiency as a decision problem with explicit accounting: a monitor only helps if the execution it prevents exceeds its own cost plus the value of the successes it mistakenly kills (formally, CPS/CPS₀ = (1 − g + m)/(1 − r), so savings require g > m + r). That framing, and the accompanying replay methodology, gives future work in early stopping and agent monitoring a comparable yardstick — the paper shows that a monitor with the best AUC in the study can still lose money.

Real-world applications:

• Cost governance for deployed coding agents. A cheap external monitor can cap spend on doomed runs in CI pipelines or internal agent fleets without swapping the underlying model.
• Closed-model deployments. Because scoring reads only observable text (no hidden-state access), the monitor can sit in front of proprietary executors whose internals are unavailable.
• IDE and autonomous PR agents. Aborting a run early frees the user to retry with a different prompt or a stronger model rather than waiting out a hopeless trajectory.
• Budget-aware evaluation of agent products. The break-even framework lets teams decide whether adding a monitor is worth it for their specific task mix and execution costs.

Industry relevance. The result that monitoring overhead can exceed the execution it saves is directly actionable for anyone paying per-token for both agents and judges. A 4.5% cost reduction on Verified and up to ~31% on the costlier Pro subset — reaching 48.1% with a Qwen3-8B backbone on Pro/Sonnet — is meaningful at fleet scale, and the small monitor's $0.000237 per check suggests the approach is viable as a standing sidecar rather than an occasional audit.

Future Directions

• Cross-executor generalization. The monitors are trained exclusively on GPT-generated trajectories; the fixed-checkpoint gap (0.627 on GPT-5.6 Luna versus 0.578 on Claude Sonnet 4.6 at step 8 on Verified) suggests failure cues transfer only partially. Training on mixed-executor data is the obvious next step the authors name.

• Threshold transfer. Identical tier AUC produced false-kill rates of 1.0% on Verified and 27.8% on Pro, pointing to run length and score drift on successful runs as unmodeled factors. Calibration procedures that adapt to trajectory length or to per-benchmark score distributions remain open.

• Beyond Python. Training and evaluation are restricted to Python tasks, with SWE-bench Pro coverage limited to three repositories from its public Python subset. Extending to other languages and benchmarks with suitable trajectories and outcome labels is left to future work.

• Recovery rather than only stopping. The paper notes that weak outcome labels do not certify a run could not have recovered under a different continuation; combining abort decisions with restarts, redirection, or model escalation (as in SWE-Router and FailFast–RestartSmart) is a natural extension.

Target Audience

• Researchers working on LLM agent reliability, trajectory evaluation, and inference-cost reduction.
• Engineers building or operating coding-agent systems who need to control per-task spend without degrading solve rates.
• Practitioners evaluating monitors or judges who want a rigorous template for cost-aware comparison (replay, charged checks, false-kill reporting).
• Readers interested in weak supervision from verifier outcomes as an alternative to hidden-state probes or intermediate patch-test labels.

Authors’ abstract

Coding agents keep working on runs that will fail verification, and asking a frontier model to watch for this is expensive in its own right. We introduce StopLoss, a small external monitor that learns failure recognition from completed trajectories and their verifier outcomes. The name is inspired by the stop-loss order in finance: StopLoss ends a run early when the run is likely to fail, which saves resources. Final outcomes label earlier local windows, so a fine-tuned language model can score the most recent tool steps in a single forward pass, and a threshold calibrated on successful runs turns that score into an abort decision without changing the executor. Trained on GPT-family SWE-rebench trajectories and evaluated on SWE-bench Verified and a SWE-bench Pro subset with GPT and Claude executors, StopLoss is compared against six general-purpose judges reading identical windows. Outcome supervision lifts failure prediction from near chance to a useful signal at the first check, at one to two orders of magnitude lower cost per check than any tested judge. This translates into savings: relative to completing every task, StopLoss lowers cost per retained solve by 4.5\% on Verified while preserving 99\% of successful runs, and by up to 31\% on the costlier Pro subset, reaching 48\% with a Qwen3-8B backbone, and in all four settings its cost per retained solve is below that of every judge policy. Outcome-trained small monitors can make failure recognition cheap enough for early stopping to pay for itself.