Serving LLMs: Prefill, Decode, KV Cache, and Batching

An LLM request is not one workload with one bottleneck. Optimising the wrong part of it can leave user-visible latency exactly where it was. Production traffic says so. Pratyush Patel and six coauthors measured real inference traffic and opened their abstract with the finding: “Based on our extensive characterization, we find that there are two main phases during an LLM inference request: a compute-intensive prompt computation, and a memory-intensive token generation, each with distinct latency, throughput, memory, and power characteristics.” Two phases, four ways of differing. That is the whole design problem.

The consequence was measured, not asserted. Token generation underutilises compute so badly that the authors moved it onto separate, cheaper, power-capped machines. They called the system Splitwise, and published it in 2024. Against a Baseline-H100 cluster it delivered 1.4x higher throughput at 20 percent lower cost, or 2.35x more throughput at the same cost and power budgets. Microsoft's own evaluation ran on a DGX-H100 variant with all GPUs power-capped to half the maximum power, on Llama-2-70B and BLOOM-176B code and conversation traces.

A request therefore passes through phases with different bottlenecks. Admission and queueing come first: scheduling, quotas, routing and available accelerator capacity decide when work starts. Then prefill, the compute-intensive prompt computation that builds the attention state. Then first-token emission, when the client sees something. Then the decode loop, the memory-intensive phase that generates tokens sequentially while reusing that state. Tool or retrieval pauses interrupt generation and create further prefill and decode phases. Completion and cleanup release or persist caches, slots, traces and billing records. If prompt computation and token generation differ in latency, throughput, memory and power, a fix aimed at one of them is not a fix for the other.

A service can improve one metric while making another worse, and one paper puts both numbers on the same model. Reiner Pope and nine coauthors closed their abstract with the pair: “Finally, we achieve a low-batch-size latency of 29ms per token during generation (using int8 weight quantization) and a 76% MFU during large-batch-size processing of input tokens, while supporting a long 2048-token context length on the PaLM 540B parameter model.” Read that as two settings, not one. The 29 ms per token needs a low batch size; the 76% model FLOPS utilization needs a large one. Both were measured on PaLM 540B running on TPU v4 slices, and the work appeared at MLSys in 2023. The same paper showed that multiquery attention's lower KV memory enables up to 32x larger context lengths, which is a memory result, not a latency one. Latency and utilisation sit on a frontier. There is no single configuration that wins both.

The thresholds a serving team is held to are equally specific and equally workload-dependent. MLCommons tightened them in MLPerf Inference v5.0, with results published on 2 April 2025. Thomas Atta-Fosu and four coauthors explained the new Llama 2 Chat 70B Interactive bar for MLCommons: “To prioritize reliability under peak demand, MLPerf adopts a stricter 99th percentile threshold of 25 tokens/second (TPOT of 40ms), ensuring consistent responsiveness even during high-load scenarios. Additionally, we set a 99th percentile TTFT limit of 450ms to minimize initial latency, aligning with user expectations for near-instantaneous query response.” The original Llama 2 70B server scenario, introduced in v4.0, allowed 2 s TTFT and 200 ms TPOT. NVIDIA's blog described the new constraints as “a 5x shorter TPOT and 4.4x lower TTFT”. The 40 ms figure came from an analysis of user data from platforms including ChatGPT and Perplexity AI in late 2024. The bar was set by what people will sit through.

The same two metrics carry different bars for different traffic. The newly added Llama 3.1 405B benchmark has mean input and output lengths of 9,400 and 680 tokens. It is allowed a 99th-percentile TTFT of 6 seconds and a TPOT of 175 ms. Time to first token is set by queueing and prefill, and governs perceived responsiveness. Time per output token reflects decode, and shapes the reading experience. Throughput measures useful work completed per resource and time. It improves with batching and utilisation, sometimes at the cost of queue delay. All three are measured at tail percentiles, because that is where the benchmark's promise lives.

During autoregressive decoding, attention needs information derived from previous tokens. Caching key and value tensors avoids recomputing that history at every step.

Cache size grows with sequence length, batch occupancy, layers, heads and representation precision. Long contexts and many concurrent requests can therefore exhaust memory even when model weights fit comfortably. The obvious remedy is to throw old entries away. Guangxuan Xiao and four coauthors showed at ICLR in 2024 that it fails: naive window attention, which keeps only the most recent KV entries, collapses once the text exceeds the cache size. Retaining the KV of a few initial "attention sink" tokens restores stable decoding. That is enough to let Llama-2, MPT, Falcon and Pythia stream up to 4 million tokens with no fine-tuning.

The alternative to truncation is recomputation, and that has a price too. The closing sentence of their abstract reports it: “In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup.” So the cache is not an optimisation bolted onto decoding. It is the thing decoding runs on. Both ways of shrinking it, dropping entries and recomputing them, cost either correctness or a factor of up to 22.2x.

A production queue mixes prompt lengths, output lengths, priorities and cancellation behavior. One such queue has been described by the people who run it. Ruoyu Qin and eight coauthors, at Moonshot AI and Tsinghua University, documented the scheduler behind Moonshot AI's Kimi chatbot at the USENIX FAST conference in February 2025. The system is called Mooncake. It separates prefill and decoding clusters, pools idle CPU, DRAM, SSD and NIC capacity into a disaggregated KVCache, and adds a prediction-based early-rejection policy for overload. Their abstract states the payoff: “In tests using real traces, Mooncake increases the effective request capacity by 59%~498% when compared to baseline methods, all while complying with SLOs.” The deployment spans thousands of nodes and processes over 100 billion tokens daily. There, it let Kimi serve 115% and 107% more requests on NVIDIA A800 and H800 clusters respectively. The preprint version reports 75% more requests under real workloads, and up to a 525% throughput increase in simulation.

Start with the user-visible SLO and the dominant resource constraint. Every result in this lesson was earned in that order. The Splitwise team characterised the two phases before splitting them across machines, and Mooncake's 59%~498% capacity gain was measured on real traces against SLOs, not on a microbenchmark.

First, instrument the request lifecycle: queue, prefill, first token, decode, tools, cancellation, completion. Then a change can be attributed to a phase. Second, segment workloads. Interactive, long-context, batch, multimodal and tool-heavy traffic do not share a bar. MLPerf Inference v5.0 holds Llama 2 70B Interactive to a 99th-percentile 450 ms TTFT and 40 ms TPOT; it allows the Llama 3.1 405B benchmark 6 seconds and 175 ms. Third, profile compute and memory: utilisation, cache occupancy, bandwidth, fragmentation and communication. Remember that the memory-intensive token-generation phase can underutilise compute while saturating something else.

Fourth, test one intervention under matched load. Quantization is the clearest case for going one at a time. The same GPTQ conversion delivered around 3.25x on an NVIDIA A100 and 4.5x on an NVIDIA A6000. Prefix caching's up-to-6.4x applies to workloads that actually share a prefix. Fifth, re-run quality gates on rare tasks, structured output, long context and safety behavior. Reduced precision was reported as negligible accuracy degradation against the uncompressed baseline, and that is a claim your own tasks either reproduce or do not. Sixth, canary under real traffic and watch tail latency, errors, cost and capacity. The numbers that decided Splitwise were throughput at a given cost and power budget. The numbers that decided Mooncake were served requests at an unchanged SLO.

Strong offline numbers can become impractical once long prompts, large outputs, concurrency and review latency are included. Serving design is what connects model capability to user experience and operating cost. In every measured system here, the winning move was structural rather than local. Splitwise separated a compute-intensive prompt computation from a memory-intensive token generation. Mooncake pooled idle CPU, DRAM, SSD and NIC capacity into a disaggregated KVCache. RadixAttention reused a prefix that is genuinely shared. StreamingLLM kept a few initial tokens so a window would not collapse. The reported gains belong to those decisions, on those traces, at those SLOs: 1.4x throughput at 20 percent lower cost, 59%~498% more effective request capacity, up to 6.4x, up to 22.2x.

The next lesson covers monitoring and change management. In production, prompts, sources, tools, traffic and model versions evolve together, so a healthy endpoint can still deliver degraded outcomes.