Skip to content
AI.info

The Pulse

SageMaker adds cache-aware routing for real-time AI endpoints

Amazon SageMaker AI can route model requests using reported cache contents and instance load. AWS says the option is available at no additional charge and requires a supported vLLM container.

SageMaker adds cache-aware routing for real-time AI endpoints

Routing by cache contents

Amazon SageMaker AI can now route real-time model requests using the cached prompt data held by each serving instance and how busy that instance is. The feature, made public by Amazon Web Services on November 16, is intended for workloads such as multi-turn conversations and requests with long shared prompts.

The option, called KV cache-aware routing, directs a request toward an instance with a matching cached prompt prefix, while weighing the length of that match against the instance’s load. It is available in all AWS Regions where SageMaker AI is available, at no additional charge.

A key-value, or KV, cache holds work a model has already done on a prompt, so the work can be reused when the same beginning appears again. That repeated text can include a system prompt, a document or earlier turns in a conversation.

How the router chooses

The new strategy uses cache events reported by the model container to maintain an index of which instances hold cached content. When a request arrives, the endpoint tokenizes its prompt and checks the index for matching cached tokens.

The index contains truncated, one-way hashes mapped to the relevant instance or inference component copy; it does not store prompts, token IDs, request payloads or KV tensors. If a lookup is slow, finds no useful match or finds that all matching instances are busy, the endpoint falls back to load-aware routing.

SageMaker’s existing prefix-aware strategy instead infers where cached content is likely to be from the start of the request. It cannot see when an instance has discarded a cache entry, and a growing conversation can be sent to different instances while its text remains shorter than the configured prefix length.

The cache-aware option matches tokens rather than a fixed number of bytes, allowing it to match a growing conversation history. It also considers instance load rather than always sending a request to the longest match.

Benchmark and overhead

AWS says its benchmarks found that, compared with random routing, the new strategy reduced median time to first token by up to 97 percent and increased throughput by up to 234 percent for long prompts under load. Time to first token measures how long it takes for a response to begin.

The tests used Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances. They ran for 30 minutes with 20 or 100 simulated users conducting multi-turn chats and document question-and-answer sessions, using prompts of up to about 64,000 tokens.

AWS reported that cache hit rates rose from 3–48 percent to 80–93 percent in those benchmarks. The company says results depend on the workload and that the gain was smallest with short prompts, where less processing can be saved.

The cache lookup adds a few milliseconds for short prompts and longer for long prompts, mostly because of tokenization, according to AWS. For text-only requests using standard options, the computed tokens are passed to the model so the prompt is not tokenized twice.

Requirements and setup

To enable the strategy, customers set RoutingStrategy to KV_AWARE in the endpoint configuration and add the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model container. Clients continue to call the endpoint through the same APIs.

It requires the AWS vLLM Deep Learning Container version 0.33.0 or later and at least two instances or inference component copies. The option supports single-model endpoints and inference components, including LoRA adapters, on GPU instances.

Cache hit-rate metrics are available, but fallbacks to load-aware routing are not reported in the SageMaker AI console or metrics at launch. With only one instance, every request goes to that instance and the routing setting has no effect.

Sources

  1. Amazon Web Services
  2. Amazon Web Services

Companies in this story

Related stories

Explore

More articles