The Pulse
Study Finds Remote KV Cache Can Worsen LLM Latency
A study of a GLM-5.3 deployment found that a remote KV cache raised 16,000-token prompt p99 latency and that a synchronous, hit-heavy replay produced a growing queue. Sharding the cache across ranks, adding asynchronous loading and fixing c
AI.info Team · Written with AI ·
A remote key-value (KV) cache left median response time unchanged but pushed the 99th-percentile time-to-first-token for 16,000-token prompts from 3.96 seconds to 17.3 seconds in a production deployment studied by Muhammad Ibrahim. In a paper submitted to arXiv on October 15, Ibrahim traces the slowdown to a remote store that could not deliver cached attention data fast enough to beat recomputing it on the GPUs.
The study examines GLM-5.3, an FP8 mixture-of-experts model, running on eight-GPU NVIDIA B300 nodes. After the deployment’s remote cache tier caused the tail-latency increase, the system was rolled back to its local cache tier, the paper says. A replay of production request traffic on an isolated engine let Ibrahim measure the cache path and test changes without touching the live service.
When a cache hit takes longer than recomputing
KV caches preserve attention data from earlier prompts so a serving system can reuse a shared prefix instead of processing it again. The paper studies LMCache backed by Valkey, a remote key-value store, in a setup where eight GPUs split prefill work across a prompt and must stay synchronized. If one rank stalls while loading data, the other ranks wait with it.
In the replay, a remote hit for a 16,000-token prompt took 0.88 seconds; recomputing the prompt took 0.31 seconds. Ibrahim’s cost model estimates that, for this model and hardware, a remote tier needs to supply more than 2.53 gigabytes per second for the data transfer to beat recomputation. A single Valkey instance delivered at most 1.71 GB/s in the measured bandwidth test.
The paper also estimates a fixed per-hit cost of about 0.33 seconds for scheduling, lookups and coordination among GPU ranks. That overhead means a remote hit on prompts shorter than roughly 17,000 tokens cannot beat recomputation on median latency at any bandwidth in this implementation. The paper presents the threshold as a model-based prediction, not a result directly measured across prompt lengths.
Synchronous loading turned delay into a queue
In a hit-heavy test of 450 prompts, 98.5% of prompt tokens came from the remote tier. With one Valkey instance and synchronous loading, the p99 latency reached 83.32 seconds, while GPU utilization was 7%. Median latency climbed from 3.52 seconds in the first third of the replay to 52.30 seconds in the last, as requests accumulated faster than the engine could clear them.
Sharding the stored KV data across the eight context-parallel ranks spread the cache work across servers, one per rank. The change raised measured GET bandwidth 6.3-fold, to 10.76 GB/s. Adding asynchronous loading and fixes for three client bugs brought p99 to 0.67 seconds on the same hit-heavy request set, with 98.5% of prompt tokens still served from the remote tier.
“The broader lesson is that on current GPUs recompute is a demanding baseline.”
Muhammad Ibrahim, paper author
The faster tail still came with a median trade-off
The improved cache configuration did not make every request faster. Its median latency was 0.40 seconds, compared with 0.31 seconds for recomputation on the same prompts, although its p99 was lower than recomputation’s 1.28 seconds. Cache reuse also avoids GPU compute, which may matter when prefill engines are saturated; the study’s results therefore distinguish latency from compute use rather than treating a cache hit as an automatic speedup.
The sharded store alone increased prefill throughput by 14% in a separate test of eight concurrent prompts of about 200,000 tokens. Combined with an upstream FlashInfer kernel fix, throughput rose 34%; that larger gain is not attributable to cache sharding alone.
One engine and one prompt-length range limit the claim
The paper’s latency tests focus on prompts between 14,715 and 17,889 tokens, and the isolated replay uses a single engine as the Valkey client. Each test condition ran once, Ibrahim reports, and the study does not measure contention among multiple engines sharing remote-store instances. The findings describe a specific GLM-5.3, B300 and LMCache-Valkey configuration—not every remote KV cache.
For that configuration, the central result is precise: the original single-server cache path was slower than recomputation and produced a growing queue; rank-based sharding, asynchronous loading and client fixes reduced the hit-heavy test’s p99 from 83.32 seconds to 0.67 seconds, while its median remained slower than recomputation.