TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents

AI summary

Overview

Research area: Cost efficiency for LLM-based coding agents (agent harnesses and black-box control), intersecting software engineering benchmarks and token-economics.

Technical level: Intermediate. The method requires no model training and is conceptually simple, but understanding the results requires familiarity with agent harnesses (Claude Code, Codex, GitHub Copilot CLI), prompt-prefix caching economics, and benchmarks such as SWE-Bench.

Scope: The paper evaluates a plug-in external controller, TokenEconomizer, that reduces cumulative token cost across sequences of coding tasks by learning repository-specific policies for instruction injection, tool pruning, and failure-loop intervention.

What This Paper Is About

Frontier coding agents are usually evaluated and optimized one task at a time, but developers use them repeatedly inside the same repository, where the same operational facts get rediscovered and the same overheads get re-billed on every turn. The authors show that no single fixed cost-control plugin (output compression, semantic navigation, repository maps) saves tokens consistently across repositories, because the effects are environment-dependent. Their goal is a black-box controller that watches completed executions in a given repository and adapts three decisions — what operational knowledge to inject, which unused tools to prune, and when to interrupt a stuck trajectory — so that cumulative token cost falls without hurting task success.

Key Contributions

1. A sequence-level formulation of coding-agent cost. The authors reframe cost optimization over a chronological stream of tasks within one environment and define a standardized protocol for cumulative, cache-aware, end-to-end accounting that charges every controller call to the task that triggered it.
2. A trace-based diagnosis of where cost is removable. Profiling 60 SWE-Bench tasks across 6 repositories identifies three recurring sources of headroom: repeated operational work, loaded-but-unused tools, and disproportionately expensive failed trajectories.
3. TokenEconomizer, an experience-driven external controller. It applies three environment-agnostic meta-strategies (micro-skill activation, tool pruning, failure-loop intervention) that learn repository-specific policies from completed traces, acting only through the prompt prefix, tool catalog, and event stream — no weight updates or agent-loop modification.
4. A multi-harness, multi-benchmark evaluation. Five agent–model settings across three commercial agents and four benchmarks totaling 579 tasks, showing cost reductions in every measured repository–setting cell with comparable task success.

Main Findings

• Fixed strategies do not transfer. RTK (command-output compression), Serena (semantic code navigation), and Aider's RepoMap (structural context injection) each lower cost in some repositories and raise it in others. All three reduce cost on Django but increase it on SymPy; on scikit-learn only RTK helps, while on Sphinx only RepoMap does.
• Operational knowledge repeats. Actions recurring in at least half of a repository's tasks account for 12–28% of the bytes its tool calls send and receive. The share of calls matching an action from an earlier task rises from 14% for the second and third tasks to 30% once at least 5 tasks precede the current one; against another repository's history only 7% of calls match.
• Most tools go unused. The agent is shown 23 tool schemas on every turn, yet no repository invokes more than 7; 16 tools are never called and 3 account for 98% of all calls.
• Failures are disproportionately expensive. The 10 failed executions are 17% of tasks but 32% of total spend. Turn limits cannot identify them: Matplotlib failures run 45 turns versus 20 for successes, while Sphinx failures run 36 versus 31. 45 trajectories repeat an earlier failure signature, and 51% of the 60 tasks' spend falls after a trajectory's first repetition — but 36 of the 50 successful runs also repeat a signature, so repetition alone does not mark failure.
• Main SWE-Bench result. TokenEconomizer reduces mean cache-aware cost by 18.4–30.9% across the five settings. Pass@1 improves in 4 of 5 settings and falls by 1.0 percentage point only on Claude Code with Opus 4.8. By contrast, lowered reasoning effort reduces Pass@1 by 1.8–10.3 percentage points.
• Consistency across cells. Cost falls in all 30 repository–setting cells, by 5.2–40.1%. TokenEconomizer+RTK also reduces cost in all 30; TokenEconomizer+RepoMap in 29 of 30. RTK alone increases cost in 12 cells and RepoMap alone in 15.
• Savings accumulate. All repositories except xarray initially cost more under TokenEconomizer, but all become cheaper by the sixth task, ending with cumulative savings of 16.9–32.0%.
• Ablations (Claude Code). With Sonnet 5: micro-skills −11.8%, tool pruning −13.2%, failure-loop guard −7.1%, combined −23.6%. With Opus 4.8: micro-skills −14.3%, tool pruning −17.7%, failure-loop guard −5.4%, combined −30.9%. Pass@1 stays within 3 points of default for every variant, and no single meta-strategy reproduces the full stack's saving.
• Combining with fixed strategies. On Opus 4.8, adding RTK raises the saving from 30.9% to 33.2% with Pass@1 above default; RepoMap or RTK also lifts Pass@1 to 75.1 and 74.8 where TokenEconomizer alone is 1.0 point below default.
• Generalization. On Web-Bench, cost falls 21.2% (Claude Code, Pass@2 69.3→73.3), 16.0% (Codex, 64.0→66.5), and 18.7% (Copilot CLI, 60.8→59.3). On CredData, 19.6%, 16.5%, and 18.3%. On ReposVul, reductions are smaller at 7.5%, 6.4%, and 9.3%, attributed to shorter trajectories.
• Pareto frontier. TokenEconomizer variants account for every point on the cost–quality Pareto frontier at or above the default pass rate.

Methodology in Plain English

The controller sits outside the agent and treats it as a black box. Before each task it reads the task description plus a running repository state built from earlier completed tasks, then sets three things.

First, micro-skills: a separate LLM periodically summarizes recent trajectories into short operational facts (e.g., which test runner works, where sources live). Each candidate skill comes with regex or predicate detectors that check, on every later trajectory, whether the skill was exercised, whether it was confirmed or contradicted, and whether the task actually needed it. Skills are activated only if confirmed across tasks editing non-overlapping files and if their need rate stays above a threshold; a decaying support score with asymmetric penalties retires a skill once contradictions outweigh confirmations, and its identifier is never reactivated.

Second, tool pruning: core shell and file tools are always kept; other tools are removed after a fixed number of consecutive tasks leave them uncalled, or when an LLM judge — shown tool descriptions and a task description with repository links masked — deems them irrelevant. Periodic re-audits restore the full catalog so wrongly pruned tools can return, and a tool called during re-audit is never masked again.

Third, failure-loop intervention: a turn budget is set from a high percentile of successful prior trajectories times a margin. A cheap rule-based check (repeated failure signature, dense errors, exceeding the budget) triggers an LLM judge that distinguishes an unproductive loop from a hard-but-progressing task; a confident stuck verdict redirects first and terminates only on a repeat. A forced stop at a multiple of the budget acts as a backstop when a strict stall rule holds. Cost is measured as mean cache-aware billed cost, with all controller calls charged to the triggering task, and the same configuration serves all harnesses and benchmarks.

Why This Matters

Research impact. The paper argues that per-task token efficiency misses the dominant cost structure of real agent use, and it provides a reusable evaluation protocol (cumulative, cache-aware, controller overhead included) plus evidence that environment-adaptive control beats fixed plugins. Its broader claim is that this pattern — cheap trace signal, conservative action, built-in recovery — is a general design template for controlling black-box agents.

Real-world applications (implications of the method):

• Enterprise coding-agent deployments where token spend scales with repository-level usage rather than isolated tasks.
• Continuous web-development workflows, where the paper reports 16.0–21.2% savings on sequential Web-Bench streams that inherit prior workspace changes.
• Security auditing at scale, where repeated audit prompts across new repositories still yielded 16.5–19.6% savings on CredData and 6.4–9.3% on ReposVul.
• Tool-plug-in vendors and harness maintainers, since the controller interoperates with existing plugins such as RTK and RepoMap rather than replacing them.

Industry relevance. All three target agents are commercial products (Claude Code, Codex, GitHub Copilot CLI), savings hold without per-environment tuning, and the controller requires no model access, retraining, or agent-loop modification — so it can track agents as they evolve. The authors also note the parameters act as predictable knobs trading savings against risk, letting users move along the cost–quality frontier.

Future Directions

• Evaluate on real interactive sessions. The authors note their SWE-Bench streams are related but independent fixes starting from fresh checkouts, spanning 1.9 to 8.5 years of repository history, rather than actual user sessions; continuous sessions should offer more reusable experience and are called a natural next test.
• Extend to more minimal harnesses. The paper covers three agents but not harnesses such as Pi; supporting them requires an adapter rather than new meta-strategies, and this remains untested.
• Apply the same template to other exposed controls. The authors suggest learning when to change reasoning effort or model, when to compact context, and how to reuse environment setup across tasks.
• Calibrate decisions with non-benchmark signals. The failure-loop budget currently uses the benchmark grader for success; the paper raises using signals such as user feedback instead, and also suggests sharing micro-skills across related repositories or using longer histories.

Target Audience

Researchers and engineers working on coding agents and agent harnesses, especially those interested in inference cost, context management, and continual/cross-task learning. It is also relevant to practitioners deploying commercial coding agents at scale and to benchmark designers who want cost accounting that reflects repeated use of a repository. Readers focused on model-level efficiency (quantization, distillation, routing) will find less direct relevance, since the paper explicitly holds the model and reasoning effort fixed.

Authors’ abstract

Advances in frontier coding agents have sharply increased the token costs of AI-assisted software engineering. Their token efficiency is typically evaluated on isolated tasks, whereas real-world developers use them to continuously build, update, and maintain large-scale projects. This mismatch motivates us to evaluate agents over sequences of realistic coding tasks, where we find that agents exhibit recurring environment-dependent token costs that existing methods struggle to reduce. To minimize these costs, we introduce TokenEconomizer, an external controller that cuts environment-specific token costs over a sequence of tasks. These cuts come from determining (i) which operational micro-skills to inject, (ii) which persistently irrelevant tools to prune, and (iii) when to intervene in unproductive failure loops. Across Claude Code, Codex, and GitHub Copilot CLI paired with frontier models, TokenEconomizer reduces the average token cost by 18.4-30.9% with comparable task success. Savings hold in every repository under every tested agent-model setting and generalize to diverse domains such as sequential web development and security auditing, where costs fall by 6.4-21.2% depending on task structure. Our work establishes adaptive black-box cost control as a promising direction for reducing token costs of coding agents in realistic settings.