The Pulse
vLLM Adds Push and Reverse KV Transfer for Agent Workloads
vLLM’s bidirectional KV transfer cut time to first token by up to 3.3 times in a long-context GLM-5-FP8 test, while a separate push connector improved TTFT by up to 3 times for Qwen3-32B. The two approaches target different inference worklo
AI.info Team · Written with AI · fact-checked ·
GLM-5-FP8 saw up to 3.3 times lower TTFT
In vLLM’s multi-turn tests, bidirectional key-value (KV) transfer cut time to first token by up to 3.3 times for GLM-5-FP8 as conversation prompts grew. A separate test of the project’s push-based connector reported up to 3 times better TTFT and up to 30% better time per output token for Qwen3-32B. Both results came from one-prefill, one-decode deployments on Amazon EC2 P5en instances connected with AWS Elastic Fabric Adapter, and the authors caution that results depend on workload and configuration.
The two features address different delays in disaggregated inference, where one model-serving instance processes a prompt and another generates the response. AWS published a detailed account of push-based transfer on October 19, 2026; vLLM’s project blog described bidirectional transfer for multi-turn agent workloads on October 12. The improvements target overhead between those separate compute stages, not a change to the models themselves.
Push mode moves setup off the decode path
In the conventional pull flow, the prefill instance builds the prompt’s KV cache, then a router passes block information to the decode instance. Decode allocates destination blocks, prepares the transfer, and reads the cache before generation can begin. Those sequential steps add work to the path to the first token.
The push connector changes that order. Decode registers its destination blocks in advance while prefill is still computing; prefill then starts the transfer when the KV blocks are ready. Transfer setup runs on the prefill side and a background thread, keeping those tasks off decode’s latency-sensitive generation loop.
AWS says it first built and validated the feature on AWS Trainium clusters, then contributed it to vLLM as an accelerator-agnostic capability. Its published performance results, however, use P5en GPU instances and the Qwen3-32B model. They compare push with pull in that specific test setup; they are not a general guarantee across models, hardware or request patterns.
Bidirectional transfer reuses an agent’s earlier turns
For an agent, each new request may include earlier prompts, answers and tool results. In ordinary prefill-to-decode transfer, the decode instance holds the KV blocks from prior output, but the prefill instance cannot retrieve them on a later turn. It may therefore recompute context that the system already processed, especially if its own local cache has evicted those blocks.
Bidirectional transfer lets prefill read eligible KV blocks back from decode before processing the new portion of the prompt. A KV-cache-aware router tracks block-location metadata between turns and associates it with a conversation ID; the KV tensors move directly between the serving instances.
The router carries metadata only; KV tensors move peer-to-peer.
Sunita Nadampalli, AWS
In the reported tests, Qwen3-32B TTFT rose from 122 milliseconds at a 2,000-token prompt to 460 milliseconds at 24,000 tokens with reuse enabled. Without it, the same comparison rose from 158 to 778 milliseconds. For GLM-5-FP8, the corresponding enabled results were 358 milliseconds at 2,000 tokens and 702 milliseconds at 20,000; the recompute-only path reached 2,334 milliseconds at 20,000 tokens.
The two modes make different trade-offs
Push and bidirectional transfer are not simply two stages of one configuration. AWS says push suits deployments that can select a decode instance early and prioritize lower TTFT. Pull remains the better fit when a workload needs bidirectional reuse or delays decode-instance selection until after prefill for load-aware routing.
Bidirectional reuse also has a memory cost: decode retains eligible blocks for a default 480 seconds after generation, and that fixed timer is not renewed. A second setting, the recompute threshold, defaults to 64 tokens; below it, prefill recomputes locally rather than paying the cost of a remote transfer. Operators need a stateful, KV-aware router to connect the turns, and the project’s instructions specify vLLM version 0.21.0 or later for the described setup.
For teams serving agents, the practical choice depends on whether the bottleneck is transfer setup before generation or repeated prefill work across a conversation. The published benchmark gains come from a defined two-instance AWS configuration; the articles leave deployment-specific performance, cache-memory pressure and routing behavior to be measured in each system.