The Pulse
AWS Says Bidirectional KV Transfer Cuts Agent TTFT by 3.3x
AWS says bidirectional key-value cache transfer in vLLM reduced time to first token by up to 3.3x in tests with long, multi-turn conversations. The approach lets prefill nodes retrieve cached context from decode nodes, but requires a statef
AI.info Team · Written with AI · fact-checked ·
“As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT).”
Sunita Nadampalli, Principal Engineer and AI/ML expert at AWS
AWS published a vLLM implementation for reusing key-value (KV) cache across turns on October 19, reporting up to 3.3x lower time to first token for one tested model. The technique changes the usual flow in disaggregated inference: instead of sending cache only from prefill nodes to decode nodes, it lets prefill nodes retrieve previously computed cache from decode nodes when a conversation continues.
That matters for agents that repeatedly send system instructions, conversation history, tool results and earlier model responses as part of each new prompt. When a prefill node lacks those cached representations, it must calculate them again before generation can begin. AWS says the repeated work can raise latency and consume accelerator capacity as context grows.
Decode nodes return cache for the next turn
In conventional prefill-decode serving, the prefill node processes a prompt and builds its KV cache; the decode node reads that cache and generates a response. With the new flow, the decode node retains eligible cache blocks after generation and sends their location metadata through the response stream. A stateful router associates that metadata with the conversation, then passes it to the prefill node on the next request.
The prefill node can then pull reusable blocks from decode using a NIXL read over remote direct memory access, and compute only the uncached portion of the prompt. The router stores block-location metadata, not the cache tensors themselves. AWS says vLLM has connector-side support, but deployments also need a KV-cache-aware router; the company points to an example proxy in the vLLM project.
Up to 3.3x lower TTFT in AWS tests
AWS tested one prefill node and one decode node on Amazon EC2 P5en instances connected with Elastic Fabric Adapter. It used Qwen3-32B and zai-org/GLM-5-FP8, and disabled prefix caching on the prefill node to model cache eviction or a request landing on a different prefill node.
At the longest prompt lengths tested, AWS measured a 1.6x TTFT reduction for Qwen3-32B and a 3.3x reduction for GLM-5-FP8. Without transfer, TTFT rose from 158 milliseconds at 2,000 tokens to 778 milliseconds at 24,000 tokens for Qwen3-32B. For GLM-5-FP8, it rose from 500 milliseconds at 2,000 tokens to 2,334 milliseconds at 20,000 tokens. With transfer enabled, the reported figures reached 460 milliseconds for Qwen3-32B and 702 milliseconds for GLM-5-FP8 at those longest prompt lengths.
Cache reuse has a time limit and a cost
Retrieving cache is not free. AWS measured KV-transfer times of about 30 to 130 milliseconds for Qwen3-32B and 29 to 110 milliseconds for GLM-5-FP8, but says the prefill computation avoided outweighed that overhead in the tested runs. The company cautions that results depend on the model, workload, hardware and network configuration.
The feature keeps decode-side cache blocks for a default of 480 seconds; turns arriving after that period use normal prefill. A configurable recompute threshold defaults to 64 tokens, allowing the prefill node to recompute a short prefix when transferring it would take longer. AWS says the setup uses vLLM v0.21.0 or later with NixlConnector, the bidirectional-transfer option enabled on both node types, and a router that tracks conversation cache metadata. Those requirements make the reported gains a serving-architecture result, not a drop-in speedup for every vLLM deployment.