LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents

AI summary

Overview

Research area: AI agents and efficient inference — specifically weak-to-strong escalation (W2SE) and uncertainty estimation for computer-use agents (CUAs) that operate graphical user interfaces.

Technical level: Advanced. The paper assumes familiarity with hidden-state representations, Gaussian discriminant analysis / Mahalanobis distance, AUROC evaluation, threshold calibration, and budget-constrained sequential decision problems.

Scope: A training-free framework that uses a weak agent's own latent uncertainty to decide when to hand a task to a stronger black-box agent, and to keep monitoring that strong agent after handoff — evaluated on OSWorld with four open-weight backbones and Claude Opus 5 as the strong agent.

What This Paper Is About

Running a frontier model at every step of a long-horizon computer-use task is expensive, but a small open-weight model alone often fails even simple tasks. The paper studies weak-to-strong escalation: let the weak agent act by default and hand control to a strong agent only when needed — which requires a cheap, per-step signal for when to escalate, and a way to keep supervising the strong agent afterward even though its API exposes no internals.

Key Contributions

1. Problem formalization. The authors cast weak-to-strong escalation as a budget-constrained sequential stopping problem: maximize expected task success subject to the probability of escalation being at most a target rate $p_{\mathrm{esc}}$, and show that a single latent uncertainty signal can drive both escalation and post-handoff oversight.
2. Training-free uncertainty score. They propose a reference-bank method that compares a weak model's final-layer hidden state against its own past successful and failed trajectories, scoring a state as more uncertain when it lies closer to failures under a covariance-aware (Mahalanobis) metric — no extra forward pass, no dedicated monitor training, no external knowledge base or reference model.
3. Post-handoff "Shadow" oversight. After handoff, the white-box weak agent switches from actor to observer, replaying the strong agent's executed steps in its own action format every three steps and extracting a shadow representation to decide whether to roll back the pending action and re-decide with higher reasoning effort.
4. Empirical and theoretical support. On OSWorld with four open-weight backbones, LEASH beats random escalation at matched escalation rates and recovers most of the weak-to-strong performance gap; a proposition shows that, under a Gaussian shared-covariance model, the initial gate escalates the largest possible fraction of the weak agent's failures at any fixed escalation rate.

Main Findings

• Consistent gains over standalone weak agents. Success improves from 42.37% to 67.18% (EvoCUA-8B, 45.4% escalation), 37.40% to 73.66% (Qwen3.5-35B-A3B, 64.1%), 45.04% to 66.03% (Nemotron-3-Nano-Omni 30B-A3B, 43.1%), and 51.91% to 69.85% (Qwen3.8-27B, 51.1%).
• Better than random escalation at matched rates. LEASH outperforms random task escalation by 6.55–11.94 percentage points across the four backbones (e.g., Qwen3.5-35B-A3B: 73.66% vs. 61.72% at 64.4% random escalation; EvoCUA-8B: 67.18% vs. 59.98%).
• Matches always-on frontier model at lower cost. With Qwen3.5-35B-A3B, LEASH reaches 73.66% success at $0.4117 per task, versus always-on Claude Opus 5 at 72.90% and $0.5094 — a 19.2% inference-cost reduction. At matched inference cost, Figure 1 reports 9.1 percentage points higher task success than random routing.
• Recovers most of the escalation gap. Following the performance-gap metric of Burns et al. (2023), LEASH recovers 75–102% of the gap between the weak agent alone and always-on Opus, versus 37–69% for random escalation, while costing 69–81% as much as always-on Opus.
• Beats always-on mid-tier model. All four LEASH configurations outperform always-on Claude Sonnet 4.5 (59.16%, $0.4038 per task), and three do so at lower estimated cost.
• Outperforms prior escalation methods. With EvoCUA-8B: LEASH 67.18% at 45.4% escalation and $0.3627, versus StepWise at 58.40% (49.6%, $0.2824) and Progress Advantage at 62.98% (46.9%, $0.3533).
• Uncertainty signal quality is moderate and backbone-dependent. Initial-stage AUROC for pooled LDA is 0.749 (Nemotron-3) and 0.600 (Qwen3.8-27B); runtime-stage AUROC is 0.609 and 0.575 respectively — highest among the compared estimators at runtime for both backbones, but not uniformly best at the initial stage (the probability-based baseline reaches 0.671 for Qwen3.8-27B).
• Ablation isolates each layer. On Qwen3.5-35B-A3B: weak agent alone 37.40%; adding the Manager 41.15% ($0.1372); adding escalation 67.94% ($0.4050); full LEASH 73.66% ($0.4117). Shadow oversight thus adds 5.72 points for a small cost increase.
• Shadow oversight fires on genuinely hard cases. Among 31 shadow-triggered tasks, success is 64.5% with Shadow versus 48.4% without, while always-on Opus achieves only 51.6% on the same tasks. On 137 escalated-but-not-triggered tasks, success is 74.5% with Shadow, 69.3% without, and Opus-only 74.5%.
• Escalation timing matters. On an escalation-prone subset, LEASH reaches 64.6% at $0.67 per task versus 60.4% at $0.79 under shuffled escalation timing — a 4.2-point gain with 15% lower cost.
• The reference bank scales favorably. Increasing bank size from 40 to 99 raises initial-stage AUROC from 0.701 to 0.770 (Qwen3.5-35B-A3B) and from 0.633 to 0.696 (EvoCUA-8B).

Methodology in Plain English

The framework has three layers, all driven by one signal: how similar the weak model's internal state is to states it has previously produced on tasks that failed.

Building the reference bank. The authors run the weak agent on a small held-out set of OSWorld tasks without any escalation and record the model's final-layer hidden state at two places: the last token of the task query (before any action is decoded) and the last token of each generated action. Each stored state gets a label. Query states get the task outcome (Pass/Fail). Action states get the outcome of the subgoal segment they belong to, where subgoals are produced and checked by a "Manager" agent (Claude Opus 5) that decomposes the task up front and verifies progress at subgoal boundaries.

Scoring uncertainty. For each channel (query and action), they fit a simple shared-covariance Gaussian discriminant — means per outcome plus a pooled covariance with regularization, requiring only means and a singular value decomposition, no gradient training. A new hidden state is standardized and its squared Mahalanobis distance to the Pass centroid is subtracted from its distance to the Fail centroid. A larger score means the state sits closer to past failures. The regularization strength is chosen per backbone and channel by task-grouped cross-validated AUROC.

Deciding when to escalate. An initial gate escalates when the query score exceeds a threshold; a runtime gate escalates after the first action whose score exceeds its own threshold. Thresholds are set as empirical quantiles on the reference tasks so that a target fraction is escalated at initialization and a target overall escalation rate is met — the runtime gate only sees tasks the initial gate retained, so its target fraction is $(p_{\mathrm{esc}}-p_q)/(1-p_q)$. The runtime threshold uses each reference trajectory's maximum step score, since one bad step is enough to hand off. Once escalated, the strong agent gets the instruction, current screenshot, and recent weak-agent steps, and keeps control to the end.

Keeping the strong agent on a leash. Since the black-box strong model exposes no internals, the weak model is repurposed as an observer: every three steps it is fed the strong agent's executed steps in its own action format plus the current observation, generates a candidate action, and produces a shadow hidden state scored with the same action-channel estimator. If the score exceeds a threshold set at the 90th percentile of reference per-step scores, the strong agent's pending action is discarded and it is asked to reassess and re-decide with higher reasoning effort. At most six such interventions are allowed per task, and intervention stops once two have changed the strong agent's proposed action.

Why This Matters

Impact on research. The paper reframes escalation as a resource-allocation problem and shows that a single cheap, training-free representation-level signal can serve routing, runtime handoff, and oversight of a model whose internals are hidden. It complements weak-to-strong generalization (training-time supervision transfer) with an inference-time counterpart, and gives a theoretical argument — under Gaussian assumptions — for why distance-to-failure ranking is the right thing to threshold at a fixed budget. It also proposes reusing activations already computed during decoding, sidestepping the cost of extra monitoring agents, repeated sampling, or a separate trained monitor.

Potential real-world applications:

• Enterprise workflow automation, where agents drive spreadsheets, browsers, and internal tools across hundreds of steps and per-task API cost dominates deployment budgets.
• Software QA and GUI regression testing, where most interactions are routine and only a minority need a frontier model.
• Customer-support and back-office agents that must operate third-party interfaces behind black-box APIs the operator does not control.
• Cost-tiered agent products that keep a cheap open-weight model in the driver's seat while metering access to a premium model.

Industry relevance. The result that a 35B-class open-weight agent plus selective escalation matches always-on frontier performance at 19.2% lower cost is directly actionable for teams balancing accuracy against inference spend, and the framework is drop-in: it needs no training phase and no access to the strong model's internals.

Future Directions

• Beyond GUI automation. The authors state that generality beyond computer-use agents on OSWorld remains to be validated; testing on other agentic domains is open.
• Escalation benefit rather than failure risk. The score estimates the weak agent's failure probability, not the expected gain from escalation; Proposition 1 needs the escalation gain to be nondecreasing in the score, which can fail when both agents struggle on the same task.
• Oversight under distribution shift. Shadow monitoring operates on strong-agent trajectories that differ from the weak agent's own reference distribution, and the effect of interventions is not causally isolated in the current evaluation.
• Extending the theory and the signal. The analysis covers only the initial gate — the runtime gate would need an optimal-stopping formulation — and the uncertainty estimator is modular, so stronger signals could be substituted as they become available.

Target Audience

Researchers and engineers working on agentic systems, inference-cost optimization, model routing and cascading, and uncertainty estimation for LLM/VLM agents. It is most useful to practitioners deploying computer-use agents under real cost constraints, and to readers interested in weak-to-strong oversight and monitoring of black-box models. Readers without background in representation-space statistics or sequential decision-making will find the method sections dense, though the framing and results are accessible.

Authors’ abstract

Computer-use agents increasingly tackle long-horizon tasks, where running a frontier model at every step is expensive and a small open-weight model alone often fails. Weak-to-strong escalation offers a middle ground where a weak agent acts by default and hands control to a strong one only when needed. The core challenge is deciding when to escalate, which requires an uncertainty signal cheap enough to check at every step. However, even after handoff, the strong agent can still get lost, but a black-box API that the strong agent is usually served exposes no internals to monitor. To this end, we propose LEASH, a training-free framework that unleashes the strong agent only when the weak agent's latent uncertainty signals it is needed, and keeps it on a leash afterward. LEASH has three layers: (1) initial task uncertainty for early routing, (2) step-level uncertainty for runtime escalation, and, after handoff, (3) a shadow observer for continued oversight of the strong black-box agent. Here, we use latent uncertainty as a fuel for all three layers. Specifically, we compare the weak model's hidden states with those from its past successful and failed trajectories, and a state that looks more like the failures signals higher uncertainty. During weak-agent execution, our uncertainty estimator reuses activations already produced during action generation, so it needs no extra forward pass or dedicated monitor training. Experiments on OSWorld across four contemporary model backbones show that LEASH consistently improves over standalone weak agents and outperforms random escalation by 6.55--11.94 points at matched escalation rates. With Qwen3.5-35B-A3B, LEASH achieves 73.66% task success at $0.4117 per task, matching always-on Claude Opus 5 (72.90%) while reducing inference cost by 19.2%.