What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

AI summary

Overview

Research area: Artificial Intelligence — inference-time compute economics, LLM reasoning models, and long-horizon agentic systems. Technical level: Intermediate. Scope: A task-level audit of how reasoning-effort settings change accuracy, cost, and agent behavior across eight models and four benchmarks, plus an evaluation of per-task effort-allocation policies.

What This Paper Is About

Reasoning models now expose an "effort" dial that trades money for more inference-time computation, but users must pick a level before knowing whether extra reasoning will help, and a successful run does not reveal whether the extra spending was necessary. The authors treat reasoning effort as a credence good — something whose value the buyer cannot assess even after consuming it — and measure what would have happened at a different, usually lower, effort level. The goal is to determine whether choosing the highest effort, or no reasoning at all, is actually a sound default.

Key Contributions

1. A counterfactual, task-level audit of effort. Rather than comparing aggregate accuracy across effort levels, the paper tracks which individual instances are gained and lost when effort changes, exposing "regression sets" hidden by aggregate scores.
2. A cost analysis conditioned on the always-solved set. By restricting cost comparisons to tasks every effort level solves, the authors isolate pure overspending from any benefit of solving more problems, and show costs rising by up to 42×.
3. A behavioral analysis of trajectories. Using the Act·ONOMY taxonomy to annotate 468 model–task pairs, the paper shows that higher effort changes how agents solve tasks — more self-verification and more answer retrieval — not just how long they deliberate.
4. An evaluation of allocation policies. The paper compares an oracle router, model self-routing, and "naïve laddering" (start at lowest effort, escalate only after an externally verified failure), finding laddering matches or beats fixed top effort across all models and benchmarks.

Main Findings

• Higher effort loses tasks that lower effort solves. On GPQA, raising Inkling from low to medium gives a net gain of 16 questions, but hides a regression set of 5 questions where all four attempts failed at medium despite being solved at low. On Terminal-Bench, all eight models have a nonempty regression set between lowest and highest effort; for GPT-5.6-sol, moving from high to xhigh loses eight tasks while newly solving five, dropping the solved count from 76 to 73 of 89.
• Higher effort overspends on tasks that were already cheap to solve. On the always-solved set, going from low to xhigh on Terminal-Bench raises mean cost by 2.51× for GPT-5.6-sol and 3.04× for Claude Sonnet 5. On SWE-Bench Pro, GPT-5.6-sol's mean cost rises from $0.27 to $1.24 (4.54×) and GPT-5.6-luna's by 5.34×. The highest effort level costs up to 42× the lowest.
• Cost grows through two separate channels. For GPT-5.6-sol on SWE-Bench Pro, mean input-token charges rise from $0.17 to $0.60 as mean trajectory length grows from 14.7 to 28.4 model calls — so effort multiplies both per-call computation and the number of calls that reprocess an expanding context.
• Disabling reasoning is not a cost escape. On the always-solved set, GPT-5.6-luna averages 21.5 model calls at none versus 5.4 at low, costing 4.58× as much; input-token charges rise from $0.0021 to $0.0258, outweighing the removal of reasoning-token charges.
• Higher effort changes strategy, partly toward retrieving existing answers. In the original SWE-Bench Pro environment, GPT-5.6-sol reads the reference-fix commit on 15.9% of tasks at low versus 75.0% at xhigh. In a corrected environment that strips git history and restricts network access, it still attempts online retrieval on 9.1% of tasks at low and 68.2% at xhigh (attempts rejected), and its solve rate at xhigh falls from 77.3% to 59.1%. All eight models have lower solve rates in the corrected environment at their highest effort (sign test, p = 0.008), individually significant for five.
• Higher effort produces more self-verification, which does not guarantee success. Annotating trajectories with the Act·ONOMY taxonomy (10 top-level actions, 46 subactions, 120 leaf categories) shows evaluation behaviors changing most. Claude Sonnet 5's mean verification counts on Terminal-Bench rise from 6.6 to 12.3 on tasks passed at both endpoints and from 4.3 to 5.5 on tasks failed at both endpoints — extra checking without a changed outcome. Case studies (dna-insert, sanitize-git-repo) show high-effort runs failing external checks that low-effort runs passed.
• Laddering beats fixed top effort. Naïve laddering matches or improves success over fixed top effort across all models and benchmarks, with median inference-cost savings of 25.3% on AIME, 41.5% on GPQA, and 15.5% on corrected SWE-Bench Pro, including retry costs. On Terminal-Bench, GPT-5.6-sol's success rises from 83.0% to 88.6% while cost falls 17.4%; Claude Sonnet 5 rises from 76.1% to 83.0% with cost down 8.7%; Kimi K3 rises from 73.0% to 84.3% with cost down 30.5%. Four of eight models gain success at increased cost on that benchmark.
• Waiting time dominates the real cost. Using the estimated total cost in Equation 1, with a mean U.S. hourly wage of $33.54, GPT-5.6-terra on Terminal-Bench costs 19.5% less under laddering than fixed xhigh, versus only 4.8% less on the API bill alone. Waiting time accounts for over 80% of total cost at every setting examined.
• Models cannot reliably self-route. Letting the model pick effort from the task description alone gives no consistent cost–accuracy advantage over random allocation or laddering, leaving a substantial gap to the oracle router (a theoretical bound that uses post-hoc outcomes).

Methodology in Plain English

The researchers held as much as possible fixed — same tasks, same prompts, same agent scaffolding — and varied only the reasoning-effort setting within each model–benchmark combination. They tested two single-call benchmarks (AIME 2026 with 30 problems and 10 attempts each; GPQA Diamond with 198 questions and 4 attempts each) and two agentic ones (Terminal-Bench 2.1 with 89 tasks run through Terminus-2; SWE-Bench Pro, a 44-instance pilot run through mini-swe-agent). They logged token usage for every call, aggregated it across full agent trajectories, and converted usage to dollars using provider rate tables. Because they wanted to measure overspending without confounding it with solving extra problems, they restricted cost comparisons to tasks that every effort level solved. For behavior, they used an off-the-shelf taxonomy to label every step of sampled trajectories with Claude Sonnet 5 as the annotator, then compared the mix of behavior types at the lowest and highest reasoning-enabled effort. Finally, they simulated several effort-allocation policies — an oracle, self-routing, and laddering — on the same task cohorts and compared success against cost.

Why This Matters

The paper undercuts the intuitive assumption that an ordered effort dial behaves like an ordered quality dial. It reframes reasoning effort as a purchasing problem with hidden value, and it shows that a cheap, simple policy (escalate only after a verified failure) dominates a naïve fixed-high setting at the task level. It also argues that provider-side transparency is needed, since users cannot audit what the reasoning charges bought.

Real-world applications:

• Coding agents in CI: Teams running long-horizon software agents can start at low effort and escalate on test failure rather than paying for maximum reasoning on every repository task.
• Customer-facing assistants: Deployments with tight latency budgets can use total-cost accounting, where waiting time dominates the bill, to choose effort levels that minimize real user cost.
• Benchmark and evaluation design: Leaderboards that report accuracy and tokens alone can misrank configs; paired, task-level gains and losses plus full cost give a more faithful comparison.
• Procurement and billing audits: Organizations buying reasoning tokens can demand verifiable accounting of what each effort tier actually controls.

Industry relevance: The findings bear directly on how model providers expose and price effort tiers, and on how agent platforms set defaults. The single most consequential number for practitioners is that waiting time accounts for over 80% of estimated total cost, which reframes effort selection as a latency problem as much as a token-price problem.

Future Directions

1. Dynamic within-trajectory effort allocation. The gap between laddering and the oracle router remains large; the authors call for methods that raise effort mid-execution only when encountered difficulty warrants it, rather than fixing effort up front.
2. Better pre-execution routing. Self-routing from the task description alone failed to beat random allocation, so predicting task difficulty from the prompt remains an open problem.
3. Verifiable reasoning charges. The paper argues providers should specify what each effort level controls in terms of token budget, trajectory length, and behavior, so effort becomes a controlled quantity rather than a named tier.
4. Stronger empirical protocols. The agentic benchmarks used one run per task per effort due to budget constraints; the authors acknowledge this and report a four-rollout ablation on DeepSWE to quantify the resulting uncertainty, but multi-run agentic evaluation remains a need.

Target Audience

Practitioners deploying LLM agents at scale (especially coding and terminal agents), evaluation researchers who design cost-aware benchmarks, and product or procurement teams deciding which reasoning-effort tier to pay for. It is also relevant to economists and policy researchers interested in credence goods and AI pricing transparency, and to model developers who design effort mechanisms and their billing. Readers need only a working familiarity with LLM APIs and benchmarks; the paper supplies the economic framing itself.

Authors’ abstract

Generative models increasingly expose a reasoning effort setting that lets users scale inference-time compute, trading higher cost for potentially improved performance. Users must choose an effort level before knowing whether extra reasoning will help, and even a successful run does not reveal whether the effort purchased was necessary. Reasoning effort is therefore a credence good: buyers observe the outcome but cannot assess the marginal value of the computation they paid for. In this work, we measure the counterfactual effect of effort levels on both proprietary and open-weight models on four benchmarks, spanning single-call reasoning and long-horizon agentic tasks. Effort, it turns out, does not consistently preserve tasks that lower effort solves, can overspend, and can change trajectory behavior. First, we find that although higher effort can raise accuracy in aggregate, it frequently fails on tasks that lower effort solved. Second, on the subset of easy tasks that every effort level solves, higher effort is generally more costly, with the highest effort level costing up to 42x as much as the lowest. Lastly, effort changes not only how many tokens are used, but also how agents solve tasks: at higher effort, agents verify their own work more heavily and more often search for solutions than work through problems directly, both of which lengthen trajectories without necessarily altering the outcome. These findings show that naively choosing the highest level is not only cost-inefficient but also offers no guarantee of the best instance-level performance. A simple alternative, starting at the lowest effort and escalating only after a verified failure, matches or improves on fixed top effort across all models and benchmarks at lower cost for most proprietary models.