How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

AI summary

Overview

Research area: Uncertainty quantification and failure prediction for LLM-based software engineering agents; inference-efficiency methods related to speculative decoding.

Technical level: Advanced. The paper assumes familiarity with speculative decoding, teacher forcing, cross-entropy/log-likelihood signals, AUROC, and calibration metrics such as ECE and Brier scores.

Scope: The paper proposes "Speculative Uncertainty" (SU), a method that recovers a failure-prediction signal for a black-box coding agent by scoring its already-generated tokens with a small draft model, and demonstrates the signal's value through a pre-execution veto gate.

What This Paper Is About

LLM coding agents frequently commit to actions that fail only after execution, forcing expensive execute-fail-retry loops — the paper calls this being "confidently wrong." Because the most capable deployed agents are closed API-served models exposing no logits, weights, or activations, and because sampling-based uncertainty estimators are too costly at agentic horizons, most existing uncertainty-quantification tools are unusable. The goal is to recover a reliable, actionable failure signal for such a black-box agent using only its output tokens, and to show that the signal is good enough to change deployment behavior.

Key Contributions

1. A black-box uncertainty estimation method. By inverting speculative decoding — a small open-weight draft model scores rather than generates the agent's committed tokens in a single forward pass — the method recovers a usable failure signal without white-box access, in exactly the regime where sampling is too costly and internals are hidden.
2. Phase-aware feature extraction. The method separates reasoning spans from action spans and computes speculative surprisal, gap, and entropy statistics separately per phase, motivated by the observation that the failure signal appears in both phases with opposite sign and cancels out when pooled.
3. Deployment guidance via a veto gate. As one downstream policy, a pre-execution veto gate cuts per-call execution error rate by 6–8 percentage points and token cost by 14–19% on SWE-Bench Verified and DA-Code, with a tunable false-veto rate and zero-shot out-of-distribution transfer.
4. Cross-agent and training-recipe findings. A distilled 4B draft is capable enough, the training objective barely matters, and a draft trained on one agent transfers to others well above the untrained baseline.

Main Findings

• Black-box signal approaches the white-box ceiling. On SWE-Bench Verified, SU reaches .77 AUROC versus .57 for verbalized confidence, .68 for Last-TP, .65 for Global-TP, and .82 for HTC (which uses the agent's own log-probabilities). The paper states it does not exceed HTC and does not expect it to, calling a black-box proxy approaching its white-box ceiling the credible outcome.
• Alignment is essential. An untrained draft is barely above verbalized confidence (.61 AUROC on SWE-Bench Verified), while distillation lifts SU to .77; SFT and teacher-forced (TF) distillation land within 1 point of each other.
• Phase separation is worth +5–6 AUROC. Dropping it (the SU-Uniform ablation) yields .71 AUROC on SWE-Bench Verified, confirming that pooling phases degrades the action-span signal.
• The veto gate reduces errors and cost. At the operating threshold, execution error rate falls from 21% to 15% on SWE-Bench Verified and from 14% to 6% on DA-Code. Average tokens per task fall 14–19%. True-veto recall is 0.72 and 0.69; false-veto rate is 0.14 and 0.16.
• Task success is not improved. On SWE-Bench Verified the resolved rate moves slightly down from 49% to 44%, and on DA-Code it is essentially unchanged (64% to 63%). The authors attribute this to replans not being guaranteed to recover the task and to false vetoes blocking actions that would have succeeded.
• Calibration is weaker than discrimination. Distillation improves the Brier score from .1265 to .1012 and the Brier skill score from .019 to .215 over a constant base-rate reference, but the authors describe SU's raw output as a failure-likelihood score rather than a calibrated probability, and flag closing the reliability gap as required work before any policy reads the probability value itself.
• Cross-agent transfer works. A Qwen-distilled draft reaches .69 AUROC on Claude 3.5 Sonnet, above the untrained baseline of .60. Training on the open-weight corpus alone reaches .69 on Claude, within a few points of the mixed-corpus result of .75.
• Phase structure is verified empirically. Over 500 SWE-rebench OpenHands trajectories (250 success, 250 failure), reasoning entropy is high and collapses sharply at the transition to the action span, with the failure signal present in both phases with opposite sign.

Methodology in Plain English

The setup treats the large agent model as a black box that emits a trajectory of alternating reasoning spans (free-form text) and action spans (structured tool calls), with component labels read off the harness's message structure (OpenHands). Each step's action is labeled by an external verifiable objective: in this study, code execution success (exit code 0).

The key move inverts speculative decoding. Instead of a small draft model proposing tokens to accelerate a large one, the small draft model (Qwen3-4B) is run in teacher-forcing mode over the agent's already-produced tokens. For each token, the draft's next-token distribution is evaluated at the token the agent actually produced. A single forward pass yields three token-level signals with no cost to the agent pipeline: speculative surprisal (negative log-probability of the observed token under the draft), speculative gap (difference between the draft's top-token log-probability and the observed token's log-probability), and speculative entropy (entropy of the draft's distribution). The intuition is that if the draft tracks the agent reasonably well, high surprisal implies the agent would also have faced low acceptance there, making it a usable proxy for the agent's uncertainty.

Features are computed separately for reasoning and action tokens within a short lookback window of the three most recent steps. For each of the two phases and each of the three signals, eight statistics are computed — mean, variance, max, min, skewness, trend, and the mean over the first and last 10% of tokens — giving 48 features, plus two span lengths, for a 50-dimensional feature vector. A sparse L1-regularized logistic regression (following the HTC framework) maps these features to a failure-likelihood score. The gate compares the predicted success probability against a threshold; below it, the action is blocked and a cheaper replan is triggered, with an abstain option after a maximum number of retries.

Training data are 10,000 SWE-rebench OpenHands trajectories (327,098 calls, 84.7% tool-call success) generated by Qwen3-Coder-480B, with a held-out split of 1,000 trajectories (32,585 calls, 84.7% success), plus 500 SWE-Bench Verified and 500 DA-Code trajectories used only for out-of-distribution evaluation. Trajectories average roughly 65 agent turns and roughly 54 thousands tokens of context, totaling about 0.54B context tokens for training. Draft training used AdamW, a learning rate of 1e-5 with cosine schedule and 5% warmup, weight decay 0.1, gradient clipping 1.0, 2 epochs, effective batch size 32 sequences, max sequence length 32768 tokens, YaRN context extension, assistant-token-only loss masking, bf16 precision, and full fine-tuning. The per-step draft pass costs approximately 100 ms on a single A100; draft SFT/TF training cost roughly 50 GPU-hours on 4×A100, and calibrator fitting was under 1 minute on 64× CPU.

Why This Matters

Impact on research. The paper reframes uncertainty estimation for agents as a black-box trajectory-scoring problem rather than a white-box probing or sampling problem, and shows that the reasoning/action phase split matters — treating a trajectory as a homogeneous token stream actively cancels the failure signal. It also positions a draft model as an evaluator rather than an accelerator, an inversion of speculative decoding's usual role.

Real-world applications:

• Gating code execution in software engineering agents, replacing execute-fail-retry cycles with cheaper replans.
• SQL agents, where the verifiable objective would check query results.
• Web agents, where the objective would check a target state.
• Tool-orchestration agents, where the objective would check the right call with the right arguments.
• Routing low-confidence steps to larger models, human reviewers, or extra test-time compute (such as resampling).

Industry relevance. The method requires no changes to the agent and adds one small forward pass per step, which the authors argue is a small fraction of the agent model's latency. Because it works with fully closed-source agents — demonstrated on Claude 3.5 Sonnet — it fits production deployments where the most capable agents are API-served, and it addresses the two axes practitioners are accountable for: error rate and compute spend.

Future Directions

1. Stronger downstream policies. The authors state that raising task success would require a policy stronger than the simple replan studied here, leaving this as a promising direction.
2. Streaming veto. A gate that watches the action span's entropy token by token and interrupts the moment it diverges, catching failure mid-generation before the block is finished.
3. Transfer beyond code. The method requires only trajectories with identifiable reasoning and action spans plus an external binary oracle, but the authors explicitly make no quantitative claim for SQL, web, or tool-orchestration settings and call confirming transfer the natural next step.
4. Post-hoc recalibration. Applying temperature scaling or isotonic regression, which is order-preserving and would leave AUROC and the gate unchanged while reducing ECE, before SU's output is consumed by any policy that reads the probability value itself rather than thresholding it.

Target Audience

Researchers and engineers working on LLM agent reliability, uncertainty quantification, and inference efficiency, particularly those deploying software engineering agents behind closed APIs. It is also relevant to practitioners with limited white-box access to their models, and to teams serving multiple agents behind a single interface who need a draft that transfers across agents. Readers without background in speculative decoding, calibration metrics, and agentic scaffolds will find the method sections demanding.

Authors’ abstract

LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.