# Bidirectional KV Transfer Cuts Agent TTFT by 3.3x in AWS Tests

Published 2026-10-20 · AI.info Team · AI.info

> In AWS tests with long, multi-turn conversations, bidirectional key-value cache transfer in vLLM reduced time to first token by up to 3.3x. The approach lets prefill nodes retrieve cached context from decode nodes, but requires a stateful router and trades some transfer time for less recomputation.

> “As the conversation grows, this redundant work consumes more accelerator capacity and increases time to first token (TTFT).”
>
> [Sunita Nadampalli, Principal Engineer and AI/ML expert at AWS](https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm/)

AWS published a vLLM implementation for reusing key-value (KV) cache across turns on October 19, reporting up to 3.3x lower time to first token for one tested model. The technique changes the usual flow in disaggregated inference: instead of sending cache only from prefill nodes to decode nodes, it lets prefill nodes retrieve previously computed cache from decode nodes when a conversation continues.

That matters for agents that repeatedly send system instructions, conversation history, tool results and earlier model responses as part of each new prompt. When a prefill node lacks those cached representations, it must calculate them again before generation can begin. The repeated work can raise latency and consume accelerator capacity as context grows, AWS says.

## Decode nodes return cache for the next turn

In conventional prefill-decode serving, the prefill node processes a prompt and builds its KV cache; the decode node reads that cache and generates a response. With the new flow, the decode node retains eligible cache blocks after generation and sends their location metadata through the response stream. A stateful router associates that metadata with the conversation, then passes it to the prefill node on the next request.

The prefill node can then pull reusable blocks from decode using a NIXL read over remote direct memory access, and compute only the uncached portion of the prompt. The router stores block-location metadata, not the cache tensors themselves. vLLM has connector-side support, but deployments also need a KV-cache-aware router; AWS points to an example proxy in the vLLM project.

## Up to 3.3x lower TTFT in AWS tests

AWS tested one prefill node and one decode node on Amazon EC2 P5en instances connected with Elastic Fabric Adapter. It used Qwen3-32B and zai-org/GLM-5-FP8, and disabled prefix caching on the prefill node to model cache eviction or a request landing on a different prefill node.

At the longest prompt lengths tested, AWS measured a 1.6x TTFT reduction for Qwen3-32B and a 3.3x reduction for GLM-5-FP8. Without transfer, TTFT rose from 158 milliseconds at 2,000 tokens to 778 milliseconds at 24,000 tokens for Qwen3-32B. For GLM-5-FP8, it rose from 500 milliseconds at 2,000 tokens to 2,334 milliseconds at 20,000 tokens. With transfer enabled, the reported figures reached 460 milliseconds for Qwen3-32B and 702 milliseconds for GLM-5-FP8 at those longest prompt lengths.

## Cache reuse has a time limit and a cost

Retrieving cache is not free. AWS measured KV-transfer times of about 30 to 130 milliseconds for Qwen3-32B and 29 to 110 milliseconds for GLM-5-FP8, but the prefill computation avoided outweighed that overhead in the tested runs, the company says. The company cautions that results depend on the model, workload, hardware and network configuration.

The feature keeps decode-side cache blocks for a default of 480 seconds; turns arriving after that period use normal prefill. A configurable recompute threshold defaults to 64 tokens, allowing the prefill node to recompute a short prefix when transferring it would take longer. The setup uses vLLM v0.21.0 or later with NixlConnector, the bidirectional-transfer option enabled on both node types, and a router that tracks conversation cache metadata. Those requirements make the reported gains a serving-architecture result, not a drop-in speedup for every vLLM deployment.

## Sources

1. Original source · Blog post [AWS](https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm/) aws.amazon.com

## Companies in this story

- [Amazon](https://ai.info/companies/amazon)
- [Amazon Web Services](https://ai.info/companies/amazon-web-services)

## Related stories

- [AutoTuneBench Recasts LLM Inference Speed Claims](https://ai.info/news/autotunebench-recasts-llm-inference-speed-claims)
- [DeepSeek V4.1-Flash Cuts Agent Costs With Causal Encoder-Decoder](https://ai.info/news/deepseek-v41-flash-low-cost-agents)
- [Harvard Publishes Agent Traces Spanning 209 Billion Input Tokens](https://ai.info/news/harvard-freeinference-agent-traces-209-billion-input-tokens)
- [vLLM 0.33 Drops CUDA 12.9 Builds](https://ai.info/news/vllm-033-drops-cuda-129-builds)
- [Three High-Severity vLLM Flaws Expose Serving Systems to Remote Attacks](https://ai.info/news/vllm-high-severity-flaws-serving-systems)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/aws-bidirectional-kv-transfer-agent-ttft-vllm
