JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

AI summary

Overview

Research area: Artificial intelligence — specifically LLM agent harnesses, small "decision models," and cost/latency-efficient agent control.

Technical level: Intermediate. The paper is readable without deep mathematical background, but it assumes familiarity with LLM agents, tool calling, benchmarking, and basic supervised fine-tuning / RL terminology.

Scope (one sentence): The paper asks whether an agent harness can safely hand most of its routine multiple-choice decisions (which tool, which arguments, whether to stop) to a fast, cheap decision model, and answers with a new three-level benchmark (AgentDecisionBench), a threshold-calibrated harness (JevAgentHarness), and a trained 4B decision model (AgentDecision-4B).

What This Paper Is About

Modern LLM agents are powerful but slow and expensive because their harnesses call a large model at nearly every step, and most of those calls end in a multiple-choice decision such as which of many available tools to invoke next. The paper investigates whether a small decision model — a model that scores a listed set of options and returns a probability for each, rather than generating text — can take over those routine choices without degrading task success. The goal is to make agentic tasks faster and cheaper while keeping success within 3 points of an LLM-only agent (5 points on τ²-bench).

Key Contributions

1. AgentDecisionBench, a three-level benchmark for evaluating decision models inside agents: L1 (isolated multiple-choice questions), L2 (step-level scoring on recorded agent runs, where answers are checked but not executed), and L3 (task-level control of a running agent, where every choice changes subsequent state). It is built on public benchmarks plus DecisionGym, a new simulated customer-service environment with 68 tools (plus finish) that maintains a hidden answer key at every reachable state.

2. JevAgentHarness, a harness layer that poses each routine choice as a typed multiple-choice question, uses the decision model's answer only when its confidence clears a threshold fitted per decision point on the agent's own shadow-mode trajectories, and otherwise defers to the LLM. It supports eight decision points and five integration modes (Replace, Assist, Guard, Cascade, Trigger), with all 40 point–mode combinations implemented and adapters for existing harnesses.

3. AgentDecision-4B, an open 4B decision model (Qwen3-4B with LoRA) post-trained on states the agent actually visits — including states reached after its own errors — using Jev's answers as labels, first via supervised fine-tuning and then GRPO inside the agent loop.

4. A pre-registered two-stage evaluation protocol that tests each delegation alone (Stage 1) and then the passing combination together on new tasks (Stage 2), with non-inferiority tests and Holm correction across eleven registered tests.

Main Findings

• Isolated accuracy does not predict agent performance. StartLux-9B was the most accurate model on BFCL tool choice (92.2%) yet solved only 69.5% of agent tasks. AgentDecision-4B was less accurate on BFCL (89.3%) but solved 91.5% of tasks, versus 82.0% for Jev. Step-level accuracy ranked models nearly like task success (Kendall correlation 0.84), while the spreads across levels were 5.1 points at L1, 19.7 at L2 and 57.5 at L3.

• A fitted threshold is what protects success. Delegating tool choices without a fitted confidence threshold lowered success by 19.0 points. With thresholds fitted on shadow-mode trajectories (1.00 for tools and 0.99 for arguments with Jev and Gemini 3.5 Flash), Jev made 27.2% of tool choices and 20.8% of argument choices and cut cost by 14.2% without changing the outcome of any of the 500 tasks. A fixed threshold at 0.5 instead lost 21.5 points.

• Thresholding must be per decision point and per model. With the weaker Flash-Lite as agent LLM, the fitted threshold left every tool choice to the LLM, so the cascade saved nothing. A shortlist ("Assist") mode lost 7.8 points, and a stop guard kept success but cost 6.2% more.

• Training on the agent's own states matters more than scale. SFT on 15,694 recorded decision states (30.9% of them following a mistake) raised step-level accuracy on shifted states from 52.9% to 98.4%. Qwen3-4B's task success on shifted tasks rose from 13.5% to 96.0%, and StartLux-4B's from 49.0% to 98.0% using the same recipe.

• Deployment savings were large on structured tasks and modest on coding. With AgentDecision-4B on its own GPU, the DecisionGym agent ran 86.3% cheaper and 2.5× faster without losing 3 or more points. On ALFWorld, a 4B model trained on that environment made the agent 89.0% cheaper and roughly twice as fast, with success rising from 91.4% to 95.0%. For coding agents, where actions are free text, cost fell only 7.0–10.5% and success varied by harness (for example, SWE-agent fell from 63.0% to 57.0%, while the pi terminal agent rose from 42.7% to 50.6%).

• GRPO added cost but no measurable gain. SFT took 2.2 GPU-hours; GRPO added 4.5 GPU-hours and $26.53 of LLM calls without an improvement. Run without SFT, GRPO left the untrained model at 11.5% on shifted tasks versus 13.5% before.

• Training on public data trades task-level gains for isolated accuracy. Further training on public tool-use data raised isolated accuracy from 78.8% to 85.1%, but task-level success on shifted tasks fell from 96.0% to 91.5%.

• Three recurring failure modes. All 34 of Jev's failed tasks began with an irreversible wrong tool choice, usually cancelling where the agent should have asked. On hard DecisionGym tasks with a cancellation/shipping rule the model never saw, AgentDecision-4B got only 81.6% of those decisions right and the agent solved 3.0% of tasks versus 88.0% for the LLM alone; retraining on the rule raised held-out accuracy from 76.5% to 95.3%. Modes that still call the LLM, such as guards, save little.

• The full combination transfers only with the trained model. With Jev, the chosen combination (tool and argument cascades plus a reflection trigger) lost 2.2 points and failed non-inferiority; with AgentDecision-4B substituted for Jev, it passed at 81.1% lower cost on 500 new tasks.

Methodology in Plain English

The researchers first formalize eight places in an agent loop where a choice recurs: which tool to call, which arguments to pass, which action to take in tool-free settings, whether to write memory, whether to reflect after a failure, whether to stop, whether to route the step to a cheaper LLM, and which stored notes to read back. For each, they define how a decision model's answer can be combined with the LLM's — deciding alone, cascading on confidence, shortlisting, guarding, or triggering an optional step.

They derive a simple rule for when delegation is worthwhile: hand over only if the decision model's confidence is at least the LLM's accuracy on that choice minus the ratio of the saved call price to the cost of a wrong choice. Because they treat each solved task as worth $1, they can estimate that threshold empirically. To do this, they run the agent on 150 separate tasks per agent LLM with the decision model answering every question silently, then fit a threshold per decision point from the resulting trajectories. If no threshold qualifies, the LLM keeps that decision.

For evaluation, they build three nested tests. L1 turns existing datasets (1,114 tool-choice and 1,949 argument items from BFCL v4, and 928 When2Call items) into multiple-choice questions. L2 scores the model at each of 5,314 states from 200 new DecisionGym tasks without letting a mistake propagate. L3 puts the decision model in full control of tool choice in a live agent. DecisionGym supplies the crucial ingredient for L2 and L3: a scripted plan per task, so an answer key exists at every reachable state, including states reached after detours, and the key is hidden from all models.

Training follows the same logic. They record 587 training tasks and 15,694 decision states, deliberately including states after errors by replaying known mistakes. Targets blend the correct answer with Jev's probability distribution when Jev ranks the correct answer first, and SFT uses cross-entropy plus a Brier term to keep probabilities calibrated. GRPO then optimizes task success, penalizing LLM calls per episode and irreversible wrong calls, with the SFT model as a KL anchor.

Finally, they run a two-stage, pre-registered experiment: test each delegation alone and keep only those that are cheaper and non-inferior (within 3 points, 5 on τ²-bench), then test the passing combination together on fresh tasks.

Why This Matters

Impact on research. The paper shows that a benchmark score on fixed questions is a poor predictor of whether a model is useful inside an agent — the ranking changed substantially across the three levels, and the spread between models grew from 5.1 points at L1 to 57.5 points at L3. It also argues that decision models should be trained on the distribution an agent actually induces, including its own mistakes, which is a different data regime from the successful demonstrations typically used.

Real-world applications:

• Customer-service agents that must choose among dozens of near-duplicate tools (cancel, look up, ask the customer) under domain policies, where per-decision latency directly affects response time.
• Coding and terminal agents (mini-SWE-agent, SWE-agent, pi, Codex CLI-style harnesses) that could route steps to cheaper models, though the paper's own savings there were small (7.0–10.5%).
• Household and text-game agents such as ALFWorld, where a 4B model trained on that environment cut cost by 89.0%.
• Cost control for large-scale agent evaluation and deployment, motivated by the roughly $40,000 cost of evaluating agents for one public leaderboard.

Industry relevance. The harness is designed as an adapter layer over existing harnesses (hooks for mini-SWE-agent, pi, Codex CLI and Claude Code), and it degrades gracefully — if an adapter or the decision model fails, the step runs as before. The reported median latency of 31 ms per AgentDecision-4B answer on one L4 GPU versus 809 ms for a Flash call frames the trade-off in directly operational terms. The authors also caution that lower cost can make a fast model's failures easy to overlook: with the cheaper Flash-Lite as agent LLM, the stop guard accepted 40.7% of the LLM's own wrong attempts to end a task, so a decision-model guard should never be the only control on an irreversible action.

Future Directions

• Refitting as a first-class operational concern. Thresholds and the trained model are fitted to one agent's runs, so a new agent, new tools or a new task family requires refitting; the authors suggest tracking the share of answers above the threshold in deployment to detect drift.
• Closing the new-rule gap. The hard-task failure (3.0% success versus 88.0% for the LLM alone when a cancellation/shipping rule was unseen) shows that retraining or refitting is needed when domain rules change, not just when tools change.
• Richer delegation points for free-text actions. In coding agents the decision model can only choose which LLM writes a step, and it could barely distinguish file-reading steps from others; extending typed decisions to free-text actions is an open problem.
• Comparisons and robustness not yet done. The paper did not compare against learned routers such as RouteLLM on the same tasks, did not test adversarial observations or policy injection, and used only English tasks with mostly Gemini agent LLMs.

Target Audience

Researchers and engineers working on LLM agents, agent harnesses, and inference cost optimization will get the most from this paper, particularly those interested in model cascades, deferral and confidence calibration. It is also useful for practitioners who deploy agents in latency- or cost-sensitive settings and need to know which decisions can safely be delegated. Readers looking for a general introduction to agent architectures should treat the benchmark and harness sections as the entry point and skim the formal delegation theory in the appendix.

Authors’ abstract

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.