Cache Engineering for Agents

AI summary

Overview

Research area: LLM agent systems and inference serving — specifically KV-cache management for long-horizon agent trajectories.

Technical level: Advanced. The paper assumes familiarity with KV caching, prefill/decode serving, radix-tree prefix caching, and rotary position embeddings (RoPE).

Scope: Proposes "cache engineering" — letting agents directly drop and reposition KV-cache entries instead of rewriting their text history — and presents AKV, a serving system that makes this operation correct and efficient.

What This Paper Is About

LLM agents accumulate long trajectories of reasoning, tool calls, and tool responses, and the KV cache for those trajectories becomes the dominant cost; the paper reports that KV cache reads account for 82.4% of total API cost. The standard fix, "context engineering," trims or summarizes the text history, but editing an early part of the trajectory invalidates every cached state after it and forces expensive re-prefilling of the retained suffix. The authors instead let agents manipulate the cache itself — excluding selected entries from attention and rewriting retained keys' position encodings — while keeping the text trajectory unchanged and reusing all remaining cached states.

Key Contributions

1. A formalization of cache engineering. The paper defines three cache operations — append, drop, and repos — plus a cache operation sequence that deterministically describes the resulting cache state, and it identifies history dependence: identical token prefixes can produce different cache states under different operation orderings.

2. AKV, the first serving system that lets agents control their own caches. It pairs a stateless client interface (a cache field alongside standard messages) with stateful server-side management built on an extended radix tree with drop and reposition nodes, drop-aware eviction, and segment-batch attention for restoring missing states in a single prefill pass.

3. Empirical gains on agent benchmarks. Across three LLMs and four agent benchmarks, a tool-response rolling-drop policy raises average scores by 1.7 points and cuts average cost by 27.7% versus threshold-triggered summarization.

4. Programmable policy search. Because cache operations form an explicit interface, agents can search and refine their own cache policies through recursive self-improvement, and cache management can be exposed to the agent as a callable tool.

Main Findings

• Cost is dominated by cache I/O: KV cache reads make up 82.4% of total API cost in the authors' accounting.

• Trimming can backfire: on the first 100 BCP tasks with GPT-OSS-120B (four trials per task, at most 100 agent rounds), trim-based context engineering that keeps only the latest 12 tool-response rounds costs 1.85× as much as the full-context baseline, because of re-prefill.

• Summarization trades accuracy for cost: summary-based context engineering at a 48K threshold reduces cost by about 7% but lowers accuracy by 5 percentage points.

• Better scores and lower cost: AKV with tool-response rolling drop (retaining the latest 12 tool responses) achieves the highest average success rate for all three models, improving average success rate by 1.3–2.1% and 0.9–8.7% over the two baselines, while cutting cost by 27.7% versus Standard and 21.9% versus Text Engineering, averaged over three models and four benchmarks. Per-model averages: MiniMax-M2.7 53.7 vs 52.4 (Standard) and 52.8 (Text Engineering), at 4.4 cents vs 5.0 and 5.7; GPT-OSS-120B 53.1 vs 51.0 and 48.3, at 7.3 cents vs 11.2 and 10.2; MiroThinker-1.7-Mini 45.8 vs 44.1 and 37.1, at 14.8 cents vs 20.4 and 18.1.

• Accuracy loss tracks summarization thresholds: on the first 100 BCP tasks, all three methods are comparable below 48K logical tokens; Text Engineering is lowest in the 48–96K range and Standard is lowest above 96K, matching their respective thresholds.

• Serving speedups: on Top-80 SWE-Verified at each method's peak-throughput concurrency, AKV (concurrency 14) achieves 1.36× and 1.51× speedups in mean and P95 request latency over Standard (concurrency 12), with input and output throughput 1.53× and 1.18× those of Standard. The abstract states "up to 1.2× the output throughput and a 1.5× speedup in P95 request latency."

• Gains widen under load: at concurrency 14 and 16, Standard's mean request latency rises to 30.16 s and 36.50 s and P95 to 108.86 s and 229.90 s, while AKV stays at 9.09 s and 10.32 s mean and 28.49 s and 30.63 s P95.

• Repositioning matters for accuracy: adding repos raises BCP pass rate from 55% to 62% in the ablation, allowing exploration beyond the position limit that would otherwise halt execution. Dropping reasoning cache, alone or with tool responses, yields lower pass rates than dropping tool-response cache alone on both BCP and SWE.

• Recursive self-improvement helps: RSI with MiniMax-M2.7, refining policies on 20 BCP tasks and testing on 100 held-out tasks, produces policies at iterations 3 and 6 that reduce per-task cost by 15.6% and 17.8% relative to Standard, and the abstract reports a 7.1% cost reduction over the expert-designed policy.

• Cache-as-tool (CAT) raises scores: on a 100-task BCP subset, CAT scores 61.0% for GPT-OSS-120B and 64.0% for MiniMax-M2.7, versus 52.1% and 56.0% for Standard; GPT-OSS-120B cost falls from 13.4 to 12.1 cents, while MiniMax-M2.7 rises from 5.4 to 5.9 cents as it takes more turns.

• Where the speed comes from: the decode speedup is attributed mainly to reduced full-attention time, since dropping older tool-response cache shortens the active context and reduces both KV reads and attention computation per decode step.

Methodology in Plain English

The authors start from an observation about attention: old tokens get less attention than recent ones, and tool responses get less attention than assistant reasoning at comparable positions. So instead of deleting text, they let the agent delete cache.

Three operations do the work. append is ordinary inference. drop removes chosen cache entries from all future attention but leaves the surviving entries untouched and does not shift the position counter. repos rewrites the position encodings baked into retained keys — for RoPE, by rotating each key from its old position to its new one — and resets the counter for future appends, preventing position indices from growing past the model's limit even when the cache is small.

This creates a systems problem: the same token sequence can now yield different cache states depending on when drops happened, so ordinary prefix matching can silently reuse the wrong state. The AKV server solves this by extending the radix tree with drop nodes and reposition nodes that record operation history, by evicting cache as soon as no in-flight request needs it, and by restoring evicted states with segment-batch attention — splitting queries at operation boundaries so each segment attends only to what it should see, all inside one forward pass while the FFN still runs over the full sequence.

Experiments compare AKV against two summarization baselines (Standard, with model-specific 96K/192K thresholds, and Text Engineering at 48K) on MiniMax-M2.7, GPT-OSS-120B, and MiroThinker-1.7-Mini across BrowseComp-Plus, GAIA-Text, SWE-Verified, and xbench-DeepSearch-2510, measuring pass rate and per-task cost in cents as well as latency, TTFT, TPOT, and throughput on eight NVIDIA A800-80GB GPUs.

Why This Matters

Research impact. The paper reframes context management for agents as a cache-level rather than text-level problem, and it shows that prefix caching's core assumption — the token prefix fully determines cached state — breaks once agents can edit their own caches. That gives serving-system researchers a new correctness criterion to design against.

Real-world applications:

• Deep-research agents that run hundreds of search-and-read rounds and would otherwise blow through context limits.
• Software-engineering agents on repositories, where SWE-Verified results show both accuracy and cost gains.
• Any long-running tool-using assistant where per-task API cost is the binding constraint on deployment.
• Agent harnesses that want to expose memory management to the model itself via a cache-as-tool call.

Industry relevance. Inference providers bill heavily for cache reads, and the reported 82.4% share makes cache behavior a first-order business concern. AKV's stateless API design — clients submit ordinary messages plus a cache field — means it can be dropped into existing serving workflows without clients tracking server-side residency, which is the practical requirement for adoption.

Future Directions

• Exploring richer cache engineering policies beyond rolling drop, including policies the agent invents at runtime rather than ones designed by researchers.
• Training agents directly under this scheme, rather than prompting existing models to use cache operations — the paper lists this as future work.
• Extending serving support to linear attention models, where the cache has different structure than the KV cache assumed here.
• Understanding when to retain or drop non-tool content: the ablations show dropping reasoning cache hurts accuracy, and the trade-offs across task types are not fully mapped.
• Generalizing policy search: RSI was demonstrated on 20 training tasks with MiniMax-M2.7 and evaluated on 100 held-out tasks, leaving open how well discovered policies transfer across models and benchmarks.

Target Audience

Researchers and engineers working on LLM inference serving, KV-cache optimization, or agent infrastructure. It will be most useful to those building agent harnesses or serving stacks who already understand prefill/decode bottlenecks and prefix caching and want a concrete mechanism — and a correctness argument — for letting agents manage their own memory. Readers new to KV caching will find the formal notation and systems sections demanding.

Authors’ abstract

Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.