Evolving Infra: Agent-Adaptive LLM Serving at Scale

AI summary

Overview

Research area: Systems for LLM inference — specifically prefill–decode (PD) disaggregated serving infrastructure for agentic workloads (cs.DC).

Technical level: Advanced. The paper assumes familiarity with PD disaggregation, KV caching, continuous batching, roofline modeling, and MoE/speculative decoding.

Scope: The paper presents Evolving Infra, a serving framework that decouples compute allocation from KV cache management so that a cluster can continuously re-shape itself around changing agent workloads, and evaluates it against NVIDIA Dynamo on coding, math, and computer-use agents.

What This Paper Is About

Agentic workloads (coding agents, math agents, computer-use agents) put very different and constantly shifting demands on prefill versus decode compute, and their harnesses fork, spawn sub-agents, and compact context in ways that determine which cached KV will actually be reused. Existing PD-disaggregated systems adapt poorly: switching a node's phase is expensive or strands its cached KV, planners rely on coarse signals like queue depth or full input length, and memory pools evict by recency rather than by whether an active branch still needs the cache. The goal is a serving system that adapts both compute allocation and KV cache management to agent behavior at runtime.

Key Contributions

1. Decoupled compute allocation and KV cache management. A symmetric engine that serves both prefill and decode from one scheduling loop, so a node's phase can be reassigned at runtime without a restart, combined with a shared host-memory KV pool (pooling L2 across all nodes over RDMA) that stays readable by every node in either phase.
2. Agent-adaptive compute allocation. Online-fitted iteration-time models for prefill and decode (no offline profiling), where the prefill model explicitly accounts for KV reuse, feeding a planner that sizes the PD split from average per-request work over a window rather than momentary queue signals, and that switches only when a candidate split's gain is statistically persistent.
3. Agent-adaptive KV cache management. A cluster-wide KV directory (prefix tree) that tracks, per block per node, how many active branches reference it, updating counts as branches fork, compact, finish, and as KV moves between nodes; eviction prioritizes zero-count blocks, and routing balances predicted TTFT because fetches are asynchronous and eviction is agent-aware.
4. A quantified diagnosis of existing systems. Measurements showing phase-switch cost, planner misestimation, and KV-reuse misjudgment in Dynamo and a reimplementation of Arrow on Claude Code traces.

Main Findings

• Phase switches are the dominant cost in existing systems. On Claude Code traces, Dynamo takes 681 s per switch on average, and Arrow loses 55.5% of available KV reuse over 34 in-place switches. In contrast, an Evolving Infra node switches in 0.7 s and loses 0.3% of KV reuse.
• Static splits cannot track workload composition. Best splits range from 2P7D to 6P3D across agents; as composition shifts from Putnam to OSWorld, the best split per 10-minute window moves from 3P6D to 7P2D, and the best static split (5P4D) loses up to 24% of throughput in the Putnam and OSWorld periods.
• Recency does not predict agent KV reuse. In Claude Code, 98% of the prompt after a 359 s sub-agent call matches earlier KV, but only 14% after a 12 s compaction. LRU eviction therefore keeps compacted histories and drops KV of branches waiting on tools or sub-agents; these evictions cause 39% of the KV reuse that Dynamo loses.
• Online planners misestimate in opposite directions. Dynamo and Arrow deliver 49% and 65% less throughput than the best static split (4.9 req/s). Arrow drifts from 6P3D to 8P1D because it counts cached prefixes as prefill work and reacts to sub-agent bursts; Dynamo drifts past 4P5D to 2P7D following decode KV utilization.
• Routing trade-offs are severe without agent-aware transfer. At the best static split (4P5D), P95 TTFT reaches 27 s (cache-aware) and 34 s (sticky sessions) from overloading prefix-holding nodes, 65 s with the load-aware Dynamo KV router that recomputes prefixes, and 95 s when a shared memory pool loads fetched KV only at prefill start.
• The iteration-time models are accurate. Outside the fitted settings, mean error is 4.2% for single requests and 6.3% for batches of 2–8 requests, and below 7% overall. With 8 or more running requests, context length changes decode iteration time by only 2–7%.
• End-to-end gains. On coding, math, and computer-use tasks with GLM-5.2 and Kimi K2.6: up to 2.7× Dynamo's throughput at stationary load (and up to 38% over the best static split), up to 4.1× lower mean E2E latency and 14× lower P95 TTFT versus Dynamo. Under shifting composition from math to computer-use, 2.1× Dynamo's throughput and 28% over the best static split.

Methodology in Plain English

The researchers first profile production-style serving on agent traces to isolate why adaptation fails, then build infrastructure that removes the reasons.

The key structural move is to stop tying a GPU node's role to its cache. Every node runs the same engine that can do both prefill and decode, and all cached KV lives in a pool made of every node's host memory, addressed through a cluster-wide directory. Because the cache is not owned by a node's engine, flipping a node's role costs nothing in lost cache — it can be done in under a second.

For deciding the split, they fit simple formulas describing how long one iteration takes on each node. Prefill time is modeled from uncached tokens, query–key pairs (which captures the quadratic attention cost against the cached prefix), and tokens restored from host memory. Decode time is modeled from a fixed weight-reading cost, the fraction of MoE experts touched, and the number of running requests. Coefficients are fit online by nonnegative least squares during a warmup where every node runs both phases, then frozen. The planner plugs the average work of a request over a window into these formulas, computes the throughput of each candidate split as the minimum of total prefill and total decode capacity, and only switches if a candidate's gain exceeds twice its standard error across window slices — a filter that ignores short bursts.

For cache management, the router tags each request with its agent branch ID and a version that increments on context rewrite. It then maintains per-block, per-node reference counts of active branches, adjusting them on appends, compaction, branch completion, and on same-phase versus cross-phase KV transfers. Blocks with a count of zero are evicted first; positive-count blocks are evicted LRU only afterward. Since eviction is agent-aware and fetches are completed asynchronously before prefill starts, the router can freely send each request to whichever prefill node has the shortest predicted TTFT.

Why This Matters

Impact on research. The paper reframes PD disaggregation as a joint compute-and-memory adaptation problem rather than two independently tuned knobs, and it introduces agent context management (forks, sub-agents, compaction, branch lifetimes) as a first-class signal for cache policy — a shift away from LRU and access recency that has implications for any cache-hierarchy design serving stateful sessions.

Real-world applications:

• Coding agents and IDE assistants, where long-lived branches, sub-agents, and compaction are routine and prefix reuse dominates cost.
• Computer-use agents, where each step appends a screenshot and the workload is strongly prefill-heavy.
• Math and research agents, where long reasoning traces make the workload decode-heavy and the optimal split different from the above.
• Multi-tenant agent platforms with mixed agent populations whose composition follows workday rhythms, where a single static deployment is known to lose up to 24% throughput.

Industry relevance. The direct comparison is against NVIDIA Dynamo, the production framework, on GB300 hardware with GLM-5.2 and Kimi K2.6 — the exact stack operators deploy. Reported gains of up to 4.1× mean E2E latency and 14× P95 TTFT translate into either fewer GPUs for the same SLA or materially better tail latency, and the design builds on SGLang, Mooncake RDMA transport, and HiCache rather than replacing them.

Future Directions

• Extending the reference-counting model beyond append-only contexts. The paper contrasts its design with agent-aware systems (Continuum, ThunderAgent) that assume contexts only grow; the natural next question is how counts should behave for branching patterns and harness behaviors not covered by the fork/compact/finish/version primitives.
• Generalizing the iteration-time models. The decode model deliberately omits the KV-context term because context length moved iteration time by only 2–7% at 8+ running requests; whether this holds for other models, attention variants, or much longer contexts is unstated.
• Planner behavior under genuinely adversarial workloads. The statistical threshold (mean gain > 2 standard errors) is evaluated on shifting math-to-computer-use composition, but sensitivity to window length, slice count, and burst frequency is not characterized in the excerpt.
• Multi-tenant and fairness-aware adaptation. The planner optimizes aggregate throughput (the slower phase bounds the cluster); how it would trade off per-agent priority, SLOs, or isolation across tenants is not addressed.

Target Audience

Systems researchers and infrastructure engineers working on LLM serving — particularly those operating or designing PD-disaggregated deployments, KV cache hierarchies, or schedulers for agent platforms. It is also valuable for agent-framework developers, since the paper makes explicit which harness primitives (fork, sub-agent spawn, compaction, branch completion) the serving layer needs to be told about to manage cache correctly. Readers without background in inference engines, batching, and KV caching will find the infrastructure sections dense.

Authors’ abstract

Agentic workloads are a major source of LLM inference demand. Existing prefill--decode (PD) disaggregated serving systems offer limited flexibility for runtime adaptation of compute allocation and KV cache management, and rely on coarse signals that only partly capture the diverse and dynamic agent workload. These limitations lead to load imbalance between prefill and decode nodes and eviction of KV caches that agents will reuse. We present Evolving Infra, an agent-adaptive…