Monitor Accuracy Isn't the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors

AI summary

Overview

Research area: Evaluation methodology for AI agent monitoring and failure-recovery (restart) policies, grounded in sequential decision theory and applied to software-engineering agents on SWE-bench Verified.

Technical level: Intermediate. The paper includes formal propositions with proofs and interval-based statistics, but its central argument is intuitive: classification accuracy does not tell you whether acting on an alarm helps.

Scope (one sentence): The paper argues that failure monitors for AI agents should be judged by the decision value and statistical support of the restart action they trigger, rather than by classification accuracy, true-positive rate, or false-positive rate alone.

What This Paper Is About

When an AI agent appears to be failing, a harness can abandon the trajectory and start over, but that throws away whatever chance the current attempt still had and gives the replacement only the remaining budget. The authors show that a monitor can be highly accurate at predicting failure and still make outcomes worse, because whether a restart helps depends on a separate quantity — how likely a fresh attempt is to succeed with the leftover budget — that accuracy never measures.

The paper's goal is to replace accuracy-centric monitor evaluation with a decision-value framework: an exact formula linking alarm quality to success change, a criterion for what information a progress signal must preserve, and a finite-sample audit that certifies "restart" or "continue" decisions (or explicitly abstains) using existing agent traces.

Key Contributions

1. An intervention-specific account of agent progress. The authors derive an alarm-to-action value identity relating failure true-positive rate, false-positive rate, continuation success, and residual-budget restartability, plus an information-loss criterion for progress representations and context compression.

2. A dependency-free decision audit. A finite-sample, model-free procedure estimates both action values (continue vs. one clean restart), respects live eligibility and observed horizons, uses simultaneous binomial intervals, and returns sign certificates or an explicit abstention under stated sampling assumptions — without additional model calls.

3. A reproducible retrospective analysis identifying where the observed restart effect actually comes from, together with analytical controls that separate detector accuracy, restartability, and sample resolution.

4. A sequence of externally verified agent experiments (194 model-driven executions on 19 selected SymPy issues from SWE-bench Verified) showing model-dependent restart effects and distinguishing protocol recovery from verified task completion, including retained negative results.

Main Findings

• Accuracy leaves the restart gain undetermined. With a fixed 90% accuracy monitor (continuation success q = 1/2, true-positive rate τ = 0.9, false-positive rate φ = 0.1), the success change is G = 0.5p − 0.05, where p is the restart's residual-budget success probability. That yields a five-percentage-point loss at p = 0, no change at p = 0.1, and a 35-point gain at p = 0.8 — a 40-point swing from identical alarm quality.

• A positive restart gain requires a threshold on restartability. The paper derives the condition p > qφ / ((1 − q)τ + qφ), so even a perfect failure predictor produces no additional successes when p = 0, although early termination can still save cost.

• There is an opportunity ceiling for any alarm. Under the stated assumptions, G ≤ (1 − q)p, attained by a clairvoyant alarm; the additional value over the better constant action is min{(1 − q)p, q(1 − p)}.

• No cutoff is universally dominant. For any budget B ≥ 2 and any fixed cutoff 1 ≤ c < B, there exist completion-time distributions where the uninterrupted policy strictly dominates the cutoff, and others where the cutoff strictly dominates (when ⌊B/c⌋ ≥ 2). Schedule shape alone cannot supply calibration.

• Survival alone does not identify the right action. Two latent regimes can produce the same no-success history through call c while requiring opposite decisions; for any randomized policy observing only that history, worst-case regret is at least p/(1 + p), approaching 1/2 as p approaches one.

• Compression loss has an exact form. The value lost by replacing a rich signal Z with a coarser G is ½{E|D(Z)| − E|E[D(Z)|G]|} ≥ 0, which vanishes when each coarse observation contains restart advantages of only one sign — preserving the optimal action, not the trace.

• The headline restart gain is fragile. In the retrospective audit of 130 executions with a 30-call horizon, a 2.29-point apparent restart gain traced to a single surviving failed attempt on one task.

• The effect does not replicate and is model-dependent. There was a 42.29-point loss on the second model, no positive replication of the initial gain on prospective task cohorts, and on the second model every task with a live attempt at that decision had an empirical advantage for continuation.

• Almost nothing certifies at the individual level. Across six decision times, zero of 103 live comparisons met the specified certification bar — all remained unresolved (abstain).

• Evidence is expensive at zero success. Under this procedure, bounding a zero-success rate requires n ≥ ⌈log(α/(4M)) / log(1 − ε)⌉ observations, and conditional continuation additionally requires those observations to reach the decision alive.

• Protocol recovery is not verified completion. The context-rejuvenation and semantic-resolver experiments can change agent behavior without improving the externally verified outcome; the paper retains these negative results and marks later collection stages as prospectively frozen while the decision audit itself is retrospective.

Methodology in Plain English

The authors first build a mathematical model of a single decision: an agent is at call t with some budget left, and a monitor says "this attempt is failing." Continuing keeps the current state; a clean restart throws it away and spends the same remaining budget on a fresh attempt. Writing down success probabilities for each branch gives an exact expression for the change in success rate in terms of four things: how often attempts would have succeeded anyway, how often the monitor correctly fires on failures, how often it fires on successes, and how likely a fresh attempt is to succeed with the remaining budget. Varying only that last quantity produces the accuracy-equal, outcome-different example. From there the authors prove small results about what can and cannot be inferred: no fixed cutoff wins everywhere, and two different underlying situations can look identical from the trace.

Second, they translate "does this progress signal carry the right information?" into a decision question: compress the full history into some summary, and check whether any group of histories sharing a summary contains both cases where restarting is better and cases where continuing is better. If not, the compression preserves decision value even if it reconstructs the trace poorly.

Third, they build an audit that needs only existing traces. For each task and decision time, they count how often live attempts went on to succeed, and how often fresh attempts succeeded within the remaining budget, then place conservative two-sided binomial (Clopper–Pearson) intervals on both. If the restart interval sits entirely above the continuation interval, the audit says "restart"; if entirely below, "continue"; otherwise it says "abstain." A union bound over the comparison family keeps the chance of a wrong non-abstaining call at or below a stated level.

Empirically, they run mini-swe-agent 2.4.6 on 19 selected SymPy issues from SWE-bench Verified, with tasks gated by base-fail and reference-pass checks, temperature 0.7, network-disabled Bubblewrap isolation, and an external evaluator kept outside the sandbox that checks the candidate diff after every environment action. The experimental ladder comprises restart replay (70 attempts), prospective transfer to unseen tasks (60 attempts, fixed 16 vs. fixed 30 calls), horizon extension (20 attempts, one 60 vs. repeated 30), context recovery (40 attempts, baseline vs. rejuvenation after three consecutive invalid tool responses), patch algebra (0 model calls, 19 frozen composites), and a semantic resolver (4 attempts) — 194 total model-driven executions across two models.

Why This Matters

Impact on research. Monitor and early-stopping papers commonly report accuracy, AUROC, or token savings. This paper shows those numbers are compatible with both helpful and harmful interventions, and supplies a concrete alternative: report alarm discrimination, residual-budget restartability, and realized utility jointly. It also cautions that small observed gains in agent studies can rest on a single surviving trajectory, and offers an abstention-respecting audit as a discipline for individual deployment decisions.

Real-world applications:

• Coding agents and CI-integrated assistants: deciding whether to kill a long-running repair attempt and relaunch from the original repository state, where the hidden test suite provides the external ground truth.
• Customer-support and workflow agents: deciding whether to abandon a stuck conversation or reset context, where a reset that preserves edits is a different action with a different outcome law.
• Computer-use and browser agents: loop-breaking and restart policies, where "the trace looks stuck" is not evidence about residual success.
• Agent observability and evaluation vendors: replacing accuracy dashboards with decision-value and evidence-support reporting, and surfacing abstention rather than unqualified point estimates.
• Inference cost management: terminating genuinely hopeless attempts to save calls even when restartability is zero, since early termination can save cost without adding successes.

Industry relevance. Teams deploying agents under fixed call budgets need per-decision guidance, not population-level accuracy. The audit's zero-extra-model-call design means it can run over traces a team already has, making it practical for deployment gating and for auditing claimed recovery gains before shipping a restart policy.

Future Directions

• Breaking observational aliasing with task-grounded sensors. The identification results show trace-only controllers cannot distinguish regimes that require opposite actions; the open question is which task-specific signals (test counts, code churn, verifier feedback) actually predict the residual completion law.
• Characterizing edit-preserving recovery. The independence assumption behind the identity fails when a restart retains edits, and the general form G = E[A(Y_R − Y_C)] requires outcomes for the specified action; recovery actions other than clean restart still need their own outcome laws and audits.
• Sample sizes and certification power. With zero of 103 comparisons certifying, the practical question is how much trace data (and how many live-eligible observations reaching each decision) are needed for the audit to resolve decisions, and whether tighter tests than the stated sufficient rule would change conclusions.
• Validating the framework against learned recovery systems. The authors explicitly do not train a monitor or compare against systems such as FailFast-RestartSmart or EET; pairing richer learned signals with joint evaluation of post-alarm action and remaining-budget success is the natural next step, alongside testing external validity beyond the 19 selected SymPy tasks and the fixed runtime configuration.

Target Audience

Agent-evaluation and benchmark researchers; engineers building agent harnesses, termination policies, or restart logic; ML infrastructure and observability teams deciding whether to ship a failure monitor; and applied statisticians interested in decision-theoretic evaluation. Readers need comfort with probability and basic decision theory to follow the propositions, but the core argument and the experimental results are accessible to practitioners who skip the proofs.

Authors’ abstract

Classification accuracy and true-positive and false-positive rates alone do not determine whether restarting an AI agent improves success. Using an exact alarm-to-action identity for an independent clean restart with zero monitoring overhead, we show analytically that a monitor with fixed 90% accuracy can swing from a five-percentage-point loss to a 35-point gain in success rate, depending solely on the replacement attempt's residual-budget success probability. We introduce two techniques for evaluating monitors and progress signals in decision terms. First, a decision-sufficiency criterion, a binary-action specialization of Blackwell's comparison of experiments, identifies which progress representations or context compressions preserve the information needed to choose optimally between the specified actions, independent of trace reconstruction fidelity. Second, a finite-sample, model-free decision audit uses simultaneous binomial intervals over existing agent traces, without additional model calls, to certify a preference for "restart" or "continue" under stated sampling assumptions, or return "abstain" when evidence is insufficient. In a study of 194 externally evaluated model executions on 19 selected SymPy issues from SWE-bench Verified, with a retrospective audit of 130 of these traces, we find a 2.29-point apparent restart gain traced to a single surviving failed attempt, a 42.29-point loss on a second model, no positive replication of the initial gain on prospective task cohorts, and zero of 103 live comparisons meeting the specified certification bar. These results motivate evaluating agent monitors through recovery opportunity, decision value, and statistical support, rather than classification performance alone.