Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success

AI summary

Overview

Research area: AI agent systems — specifically the control logic (scaffolding) wrapped around reasoning models that run long tool-use loops, and the evaluation of cheap "decision models" used as controllers.

Technical level: Intermediate. The paper is readable without deep ML background, but assumes familiarity with LLM agents, tool-calling loops, cost accounting per token, and standard evaluation statistics (confidence intervals, AUROC, calibration).

Scope (one sentence): A pre-specified, placebo-controlled benchmark study on Terminal-Bench 2.1 testing whether a cheap typed decision model (Jev 1.13) consulted at fixed decision points makes a Claude Sonnet 5.5 terminal agent cheaper without reducing its pass rate.

What This Paper Is About

Agent scaffolds increasingly hand small decisions — whether to stop, verify work, or change approach — to cheaper models instead of the expensive reasoning model. The paper asks whether one such typed decision model, Jev 1.13, actually saves money for a Claude Sonnet 5.5 terminal agent without hurting task success. It separates two things that are usually conflated: whether the controller's judgments carry real information about how a run will end, and whether acting on that information produces savings.

Key Contributions

1. A pre-specified, placebo-controlled design that holds a single agent loop fixed and varies only the controller across five arms: none (native), deterministic rules, a small LLM (Claude Haiku 5.5), Jev 1.13, and a placebo that samples Jev's own development-phase choice frequencies at the same moments.
2. A confirmatory negative result: Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02–1.36) at an identical 73.0% pass rate, so the hypothesis that Jev lowers cost is rejected.
3. Head-to-head baselines showing Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo (ratio 1.22), with no detectable pass-rate difference between any arms.
4. A mechanistic account of the failure, combining offline judgment quality (Jev ranked finish requests best at AUROC 0.78 but was badly miscalibrated, assigning mean finish probability 0.50 when 87% of those runs passed) with logged decisions showing where the extra cost originated.

Main Findings

• Cost increase without success gain: Jev cost 19% more per attempt than no controller (ratio 1.19, 95% CI 1.02–1.36, p = 0.026) at an identical pass rate of 73.0% in both arms; the pass-rate difference was +0.0 pp (95% CI −11.1 to +12.7).
• Guardrail at the edge of resolution: The one-sided 95% lower bound on the pass-rate difference was −9.5 pp against a pre-specified −10 pp margin, which the authors read as no evidence of a large drop rather than as noninferiority.
• All informed controllers cost more; the placebo did not: Against native, rules cost ×1.24 (95% CI 1.12–1.38), the small LLM ×1.47 (1.16–1.86) and Jev ×1.19, while the placebo was ×0.98 (0.78–1.16).
• Placebo matched pass rates for less: Jev was 22% more expensive than the placebo (ratio 1.22, 95% CI 1.08–1.39, p = 0.001) with a pass-rate difference of +1.6 pp (95% CI −4.8 to +9.5). The authors conclude that Jev's choice content, not the act of intervening, added the cost.
• Jev beat the small LLM on cost: ratio 0.81 (95% CI 0.68–0.95, p = 0.019) with a pass-rate difference of −3.2 pp; Jev matched the rules arm (ratio 0.96, 95% CI 0.83–1.11, p = 0.623).
• Good ranking, bad calibration: On Q-finish states (n = 53), Jev's AUROC was 0.78 versus 0.62 for rules and 0.60 for the small LLM, but its mean probability of finish was 0.50 when 87% of those runs passed, and its argmax accuracy (0.55) fell below the base rate (0.87). It chose verify for 29 of 53 finish requests.
• Even its least confident group mostly passed: Runs where Jev assigned finish probability ≥ 0.5 passed 96% of the time (23 of 24); runs below 0.5 still passed 79% (23 of 29).
• Abandon was never used: No controller chose abandon in any of the 315 trials. On Q-progress (n = 72), Jev's abandon probability barely separated failing from passing runs (AUROC 0.55).
• Cost tracked verification frequency: Placebo produced 0.33 verification messages per attempt (cost ratio 0.98), Jev 0.51 (1.19), rules 0.92 (1.24) and the small LLM 1.30 (1.47). Of Jev's extra $0.029 per attempt, $0.011 was spent after the first verification message; the rest came from longer trajectories overall (16.0 steps versus 13.3).
• Broad but not uniform: Jev was more expensive on 15 of 21 tasks; removing any single task leaves the ratio between 1.16 and 1.25. Three of the six cheaper tasks were within 3% of native, and the three clear reductions were hard tasks whose runs mostly ended at the Harbor timeout or budget cap, where there is usually no finish request to verify.
• Failure modes were uncatchable: Most native failures (10 of 17) ended at a timeout or cap rather than a finish request, and no controller proposed to abandon them; all 15 runs of make-doom-for-mips failed at the Harbor timeout (11) or the $1.00 cap (4).
• Negligible controller overhead: Jev averaged 0.52 s per decision and 0.11% of spend; the small LLM averaged 1.46 s and 0.47%. No controller call timed out or returned a malformed answer.
• Stratified view: Within strata, the Jev-versus-native pass-rate difference was +7.1 pp (easy), −22.2 pp (mid) and −8.3 pp (hard), with cost ratios of 1.29, 1.48 and 0.76; the mid and hard strata are too small for inference.

Methodology in Plain English

The researchers built one fixed agent loop and changed only the decision-making component inside it. The loop runs a Claude Sonnet 5.5 agent in a Daytona sandbox via Harbor, giving it two tools (bash and finish), truncating tool output to the first 4,000 and last 8,000 characters, and never compacting the context.

Four deterministic triggers are detected and logged in every arm, including the arm with no controller: repeated failure (two or more consecutive failing commands), no progress (a repeated command or repeated output), a finish request, and a checkpoint every 10 steps. Only one trigger fires per step, and a trigger type cannot refire within three steps. At progress triggers the controller picks among continue, investigate, recover or abandon; at finish triggers it picks between finish and verify. The non-trivial actions append one of three fixed advisory messages that never contain commands, paths or test information. The controller sees only a compact state derived from the transcript — never tests, verifier output or reference solutions — and guards override every controller, for example blocking abandon before 40% of the step budget is used.

The five arms were: native (triggers logged but no controller), rules, Claude Haiku 5.5 with reasoning disabled, Jev 1.13 (pinned snapshot jev-1.13-20260917, priced at $0.042 per million input tokens, about $0.00008 per decision), and a placebo sampling from Jev's development-phase frequencies (continue 0.625, investigate 0.125, recover 0.25, abandon 0; finish 0.652, verify 0.348).

Terminal-Bench 2.1 has 89 tasks; the authors excluded two whose agent timeouts exceed 60 minutes and one that Sonnet 5.5 refuses on its first turn. Thirty tasks were run twice with native under generous caps (100 steps, $3) and stratified as easy, mid or hard, yielding 8 development and 21 held-out tasks (14 easy, 3 mid, 4 hard). Evaluation used 21 tasks × 5 arms × 3 attempts = 315 trials at 60 steps and $1.00 per trial, with arm order randomized within each task–attempt block. Comparisons are paired by task with 10,000 task resamples for percentile 95% intervals and Holm-adjusted p-values for the secondary comparisons; total model spend was $72.29, of which $59.46 was the evaluation. Separately, every trigger state logged by native was replayed offline to each controller without acting on the answers, to score judgment quality against the trial's grader outcome.

Why This Matters

Research impact. Evaluations of judgment models typically measure only whether the model's predictions are informative. This paper shows that informative predictions can fail to convert into useful interventions, and it supplies a design — placebo-controlled, cost as the primary outcome, rules and small-LLM baselines — that separates signal quality from policy quality. It also documents two harness pitfalls: a naive "did the agent verify?" feature misses agents that check work by running their own programs, and safety refusals return an empty reply that a loop can mistake for a finish request.

Real-world applications:

• Agent platform engineering: teams adding stop/verify/retry logic to coding or terminal agents can use these results to justify testing any cheap controller against a placebo before deploying it.
• Cost budgeting for automated pipelines: a 19% increase in cost per attempt applies to every task an agent runs at scale, so the finding directly informs spend forecasts.
• Benchmark and evaluation harness design: the trigger logging, compact-state construction and offline replay method are reusable for auditing controller behavior.
• Guardrail design: the result that abandon was never selected, and that hopeless runs simply ran to the timeout or budget cap, is directly relevant to designing stopping policies.

Industry relevance. Cheap typed decision models are marketed partly on price and speed, and this paper confirms they are fast and nearly free per call (0.52 s and 0.11% of spend for Jev) — the cost comes from the policy, not the inference. That reframes where vendors and practitioners should invest effort: in mapping probabilities to actions rather than in making the classifier cheaper.

Future Directions

• Base-rate-aware thresholds: act on calibrated probabilities rather than argmax, since even Jev's least confident finish-request group passed 79% of the time; the authors note such a threshold would intervene far less.
• Give controllers stopping actions they will actually use: no controller chose abandon in 315 trials, and the abandon guard was never the obstacle, so hopeless runs need a stopping mechanism that fires in practice.
• Pre-specified follow-up on new tasks: the authors call for a replication acting on calibrated probabilities with base-rate-aware thresholds, specified in advance on fresh tasks.
• Open-weight replication: a pre-specified replication on an open-weight reasoning model was planned as C5 but not run, because OpenRouter hosts serve different quantizations and the model could not be held fixed across arms.

Target Audience

Researchers and engineers working on LLM agent scaffolds, agent evaluation methodology, and cost optimization for multi-step tool-use systems. It is also useful for practitioners deciding whether to insert a cheap model into an agent loop as a controller, and for benchmark designers interested in placebo controls and cost-as-primary-outcome study design. Readers primarily interested in model capability rather than system-level policy will find the framing less directly applicable.

Authors’ abstract

Agent scaffolds increasingly delegate small decisions (whether to stop, verify, or change approach) to cheaper models. We ask whether a typed decision model, Jev 1.13, consulted only at deterministic decision points, makes a Claude Sonnet 5.5 terminal agent cheaper without making it worse. In a pre-specified study on Terminal-Bench 2.1, we hold one agent loop fixed and vary only the controller: none, deterministic rules, a small LLM (Claude Haiku 5.5) answering the same questions, Jev, and a placebo that samples Jev's development-phase choice frequencies at the same moments. On 21 held-out tasks (315 trials), Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at an identical pass rate (73.0% vs. 73.0%). Jev was cheaper than the small-LLM controller (ratio 0.81) but 22% more expensive than the placebo, and no pass-rate difference between arms was detectable. Offline, Jev ranked finish requests better than the alternatives (AUROC 0.78 vs. 0.62 for rules and 0.60 for the small LLM) but was poorly calibrated: its mean probability that a run was complete was 0.50 when 87% of those runs passed, so it asked the agent to verify work that was already correct. No controller ever chose to abandon a run. Informative decision signals did not become useful interventions; we discuss what that would require.