Total Cost of Agency: Exact Attribution of Memory Injection Cost in Multi-Agent LLM Workflows

AI summary

Overview

Research area: cost accounting and observability for multi-agent large language model (LLM) workflows, with a focus on agent memory and retrieval. Technical level: Intermediate. The paper combines a formal cost decomposition with an empirical benchmark; it is readable without deep systems background but assumes familiarity with LLM APIs, tokenization and agent workflow graphs. Scope: The paper defines and measures the "Total Cost of Agency" (TCA), isolating the portion of multi-agent workflow cost attributable to memory context injected into node prompts, and characterizes how that cost grows with workflow depth and responds to retrieval-capacity settings.

What This Paper Is About

Every node in a multi-agent LLM workflow retrieves context from memory and injects it into its prompt, and those injected tokens are billed as input tokens at the same per-token price as the system prompt and the user query. Production observability tools such as LangSmith, Arize Phoenix and Weights & Biases Weave report total token counts and cost per call, but do not split a node's input tokens into the tokens the node generated versus the tokens it was handed, so this component of the bill is invisible to the teams paying it. The paper's goal is to formalize that component, measure it exactly rather than estimate it from word- or character-count proxies, and characterize its behavior across workflow depth and retrieval capacity on a real-API benchmark.

Key Contributions

1. The Total Cost of Agency (TCA) decomposition. A formal decomposition of multi-agent workflow cost into five components — base prompt, inference, memory injection, miss penalty and context-accumulation — with explicit statements of which components accrue billable cost under which deployment conditions. Three components are billed on every node in every configuration (base prompt, inference, memory injection); the miss penalty is non-zero only when the durable tier is a billed remote service, and the accumulation term is conceptual rather than separately measurable.
2. An exact attribution method for injected tokens. A two-pass, non-billable token count that measures injected tokens directly using the provider's own tokenizer, as the difference between the tokenized prompt with memory and the tokenized prompt without it. The method is exact by construction, free (a non-billable call that performs no inference), low latency (roughly ten milliseconds per node against inference latencies of eighty to three thousand milliseconds), and yields the base-prompt cost as a by-product.
3. An empirical characterization across workflow categories and depths. Results from a 200-task enterprise benchmark spanning five workflow categories and five depths, executed against real model APIs, covering injection cost share, growth with depth, and sensitivity to retrieval window capacity.
4. Full reporting of negative and null results. The paper states explicitly that its graph-rewriting transforms are approximately cost-neutral in isolation, that two of the five decomposition terms are zero by construction in this harness, that total cost is dominated by model tier assignment (held fixed and treated as orthogonal prior work), and that prompt caching was not evaluated.

Main Findings

• Memory injection is a substantial, separately attributable cost line. On the unoptimized baseline at the mid tier, inference accounts for $0.020408 per task and memory injection for $0.003220. Injection is therefore 13.6 percent of the variable cost a compile-time optimizer can act on, and 11.8 percent of the full billed cost (the paper also describes this second figure as approximately 12 percent). The 13.6 percent figure rests on direct measurement; the 11.8 percent figure is a derivation accurate to roughly a point.
• The base prompt is comparable in size to injection. The profiler's observation-weighted mean is 304.4 base tokens per node at the mid tier over 1,896 observations; multiplied by the benchmark's mean of four nodes per task this gives 1,233 tokens, or $0.003699, which is 13.5 percent of the $0.027327 total per task. Base prompt contributes 1,233 billed input tokens against injection's 1,073.
• Injection share rises steeply with workflow depth. The share rises monotonically from 8.4 percent at depth two to 27.6 percent at depth six on the mid tier, and tracks nearly the same curve on the small tier, reaching 28.0 percent. At depth one the term is a structural zero: a source node has no upstream output to inject, and across 1,112 observations of the extraction node class at the two active tiers the recorded injected-token count was exactly zero in every case.
• Injected tokens grow linearly, not quadratically, over the measured range. Absolute injected tokens per task are 143 at depth two, rising through 324, 466 and 602 to 754 at depth six. A linear fit over depths two through six gives R² = 0.9974 with a slope of 150 tokens per additional level of depth. A quadratic fit over the same points reaches R² = 0.9987 with a leading coefficient of −4.57, so the fitted curve is concave and the data are inconsistent with convex growth in this range. Including the structural-zero depth-one point changes the linear fit to R² = 0.9984.
• The retrieval window is a live lever at fixed tier. Holding everything else fixed and reducing warm-window capacity from its default of 32 entries to 2 lowers mean injected tokens per task from 1,074 to 766, a 28.7 percent reduction, and lowers cost per task from $0.007436 to $0.006939, a 6.7 percent reduction. Cutting 28.7 percent of injected tokens removes $0.000308 per task, or 4.1 percent of the total; the paper does not claim the remaining $0.000189 as an injection effect because a smaller window also shortens node outputs, which was not instrumented.
• The accuracy axis is treated as a guard, not a result. End-to-end accuracy falls from 0.600 to 0.570 across the capacity change, which the paper reads as within the 0.575–0.645 band the ablation conditions span at a single seed rather than as a measured accuracy cost. The three-seed headline conditions give 95% intervals of 0.535–0.629 and 0.583–0.670.
• Graph-rewriting transforms are approximately cost-neutral in isolation. Conditions A through G at the small tier lie within one percent of each other, spanning $0.00740 to $0.00744 per task. Condition F (shared namespace promotion alone) is the only condition below the unoptimized baseline, by $0.00002; condition G (all three transforms) is the most expensive of the rewrite conditions rather than the cheapest.
• The full system is more expensive than the baseline at the cheapest tier. Condition H costs $0.00998 against the baseline's $0.00742, because the small tier is the price floor and the tier assigner can only escalate a minority of accuracy-sensitive nodes to the mid tier. The paper notes that a dollar-level change in that row conflates a change in price with a change in volume, since injected tokens are billed at the input price of the tier a node lands on.
• Two of the five decomposition terms are zero by construction. Warm-tier lookups do miss — the instrumentation records a fallback on every depth-one node, and on more nodes once the window is small — but the durable store is an in-process key-value store with no per-token billing, so a fallback costs latency and no dollars. A deployment whose durable tier is a billed remote service would populate the term.
• Total cost is dominated by model tier assignment, not memory management. The paper states that memory injection is a second-order lever by comparison, which is why tier assignment is held fixed throughout.
• Tier-independence of the injection share depends on a price-ratio assumption. Because the providers in the study price input and output tokens in the same ratio at both tiers, the share curves nearly coincide and are close to tier-independent. The absolute dollar cost of the same injected context still differs by the ratio of the two input prices.

Methodology in Plain English

The researchers built a benchmark of 200 enterprise workflow tasks split across five categories (billing reconciliation, software asset management, identity and access management, cross-domain reconciliation and policy checking) and five workflow depths (two through six). The design is fully balanced: 40 tasks per category and 40 per depth, with eight tasks in every (category, depth) cell, so per-category and per-depth comparisons are not confounded by composition. Workflows are composed from six specialist node classes — extraction, which appears in all 200 tasks, plus query generation, policy checking, billing reconciliation, IAM audit and cross-domain reconciliation — and execute against three real SQLite databases seeded with synthetic but realistic enterprise data (two hundred invoices, several hundred purchase orders and payments, and thousands of entitlement and access-log records).

To measure injection cost, each node's prompt is assembled twice: once from the base components alone and once with retrieved memory included. Both are tokenized with the provider's own tokenizer, and the difference is attributed to injection. Neither count is billed and together they add roughly ten milliseconds per node. The node's output is then written back to the warm tier, becoming part of the next node's injection — the mechanism that makes carried context accumulate with depth. Integration requires no change to workflow structure: the two passes are performed by a decorator around the prompt-assembly step, which records per-node counts alongside the tier and memory strategy in use.

Grading is execution-grounded and deterministic rather than delegated to a model-based rubric. A query-generation node is graded correct only if its generated query executes successfully against the task's database, returns a non-empty error-free result, and the combined query-plus-answer text attains a recall of at least 0.6 over the task's annotated required topics; every other node class is graded by the required-topic recall criterion alone. End-to-end task accuracy is the verdict of the deepest node. All experiments use models across a small tier (Claude Haiku, priced at $1.00 per million input tokens and $5.00 per million output tokens) and a mid tier (Claude Sonnet, $3.00 and $15.00), with the frontier tier disabled by budget configuration to isolate the memory injection effect from frontier escalation. Headline conditions run across three random seeds (42, 7 and 99); ablation and capacity conditions run at seed 42.

Why This Matters

The paper identifies a cost component that standard production observability tooling does not expose as a separate number, and shows it is exactly measurable at no billing cost and modest latency. It also demonstrates that the size of this component is a structural property of workflow depth rather than of any single model tier, which means it scales with architecture decisions teams make before any model is called.

Real-world applications:

• Enterprise workflow cost governance. Teams running multi-node agent pipelines for billing, compliance or IT operations can attribute the context portion of their bill and reason about it separately from model selection.
• Memory and retrieval system design. Engineers choosing retrieval window sizes, eviction policies or memory strategies get a token-level measurement of the tradeoff, rather than a proxy estimate.
• Observability and platform tooling. The two-pass method suggests a concrete instrumentation point — a decorator around prompt assembly — that any provider exposing a tokenizer can support, applicable to any framework that assembles prompts explicitly before dispatch.
• Workflow architecture planning. Because injection share rises with workflow depth (from a structural zero at depth one to 27.6 percent at depth six), architects can anticipate how carrying context across more nodes changes the cost profile of a design.

Industry relevance: the paper notes that a frontier model can cost twenty to twenty-five times what a small model costs per token, and that real agent workflows string several model invocations together in a dependency chain. It argues that the cost of a multi-step workflow is dominated not only by which model each step uses but by how much context each step must carry, and that the second quantity is absent from standard instrumentation.

Future Directions

• Measuring prompt caching. The paper states that caching is the most plausible mitigation but was not evaluated, and that measuring the hit rate on injected memory and the residual cost after caching is the first thing the authors would run next. It gives four structural reasons injected memory is an unfavorable caching candidate here: a hit requires a byte-identical prefix while injected memory consists of upstream outputs generated within the current task instance (so the first injection is necessarily a write, never a read); a change at the system level invalidates the message level beneath it, so memory injected at one node cannot be read from cache at another unless prompts are restructured so shared memory precedes node-specific instruction; minimum cacheable prefix lengths can exceed the per-task injected context at the small tier; and content written to cache and read zero times costs more than not caching at all, so in a fan-out graph where each node carries distinct context, naive caching raises cost. Providers discount cached input tokens substantially: a cache read is billed at 0.1 times the base input price, against 1.25 times for a five-minute write and 2 times for a one-hour write.
• Tracing the cost–accuracy curve for retrieval capacity. The paper names a specific experiment: a sweep over K_opt in {1, 2, 4, 8, 16, 32, 64} at three seeds on the small tier under the memory-only condition, reporting injected tokens, cost and accuracy as mean and standard deviation at each setting — 21 runs of 200 tasks. This would locate the point at which tightening the window turns unfavorable, which the current two-setting experiment cannot establish.
• Extending to dynamic and looping workflows. The benchmark comprises fixed-topology DAGs; ReAct-style reasoning loops and dynamically branched workflows are not evaluated. The paper states an expectation, derived from the same analysis rather than measured, that because a loop at step t injects the observations of steps 1 through t−1, and agent loops routinely run to iteration counts exceeding the depth of the deepest workflow, injection would be a larger fraction of cost in that setting.
• Testing transferability across providers and frameworks. All results use one model provider and one agent framework. The paper expects the accounting to transfer because input-token billing is universal and the attribution method requires only explicit prompt assembly before dispatch, but whether the magnitudes do is untested; it also notes that any share-of-cost figure must be recomputed per provider because tier-independence depends on a price-ratio assumption.

Target Audience

This paper benefits most engineers and researchers working on LLM agent infrastructure, production cost management and memory systems: platform and MLOps teams operating multi-agent workflows, practitioners choosing retrieval or memory strategies, and researchers studying cost-aware routing and cascades who want to know which cost lever their work does and does not cover. It is also relevant to anyone building or evaluating LLM observability tooling, since its central claim is that existing tools report input tokens as one quantity and do not expose the generated-versus-handed split. Readers evaluating the paper's claims should note its stated boundaries: prompt caching is unmeasured, all figures are for the uncached case, results come from a single provider and framework, the ablation and capacity experiments use a single seed, and the accuracy axis is treated as a guard rather than a result.

Authors’ abstract

Every node in a multi-agent large language model (LLM) workflow retrieves context from memory and injects it into its prompt, where those injected tokens are billed as input tokens at the same per-token price as the system prompt and the user query. Production observability tools report total token cost but do not separate the tokens a node generates from the tokens it is handed, so this component of the bill is invisible to the teams paying it. We introduce the Total Cost of Agency (TCA), a decomposition of multi-agent workflow cost into base prompt, inference, memory injection, miss penalty and context-accumulation components, and an exact attribution method: a two-pass, non-billable token count that measures injected tokens directly rather than estimating them from word-count proxies. On a 200-task enterprise benchmark executed against real model APIs, memory injection accounts for 13.6 percent of the variable cost a compile-time optimizer can act on, about 12 percent of the full billed cost, and its share rises from a structural zero at workflow depth one to 27.6 percent at depth six. Injected tokens grow linearly with depth over the measured range (R^2 = 0.9974, depths two through six); a quadratic fit yields a negative leading coefficient, so the data do not exhibit convex growth at these depths. We show the component is controllable at fixed model tier: reducing the retrieval window capacity from 32 to 2 entries lowers injected tokens by 28.7 percent with an accuracy change within seed-level variation. We report in full that our graph-rewriting transforms are approximately cost-neutral in isolation, that two of the five decomposition terms are zero by construction in this harness, and that total workflow cost is dominated by model tier assignment, which we hold fixed and treat as prior work. Prompt caching is not evaluated; all figures are for the uncached case.