The Pulse
AWS Adds GPU-Aware Routing to SageMaker HyperPod
AWS has introduced a Kubernetes-native Inference Gateway for SageMaker HyperPod that routes LLM requests using live GPU signals. The gateway targets lower first-token latency and higher throughput without requiring changes to model servers
AI.info Team ·
Round-robin routing meets uneven GPU fleets
AWS is adding GPU-aware request routing to Amazon SageMaker HyperPod, taking aim at a limitation of ordinary Kubernetes load balancing: round-robin distribution treats every model-serving pod as equivalent even when the underlying GPUs, queues, cache states, and available memory differ.
The new Amazon SageMaker HyperPod Inference Gateway sends requests through a Kubernetes-native routing layer that reads model information from the request and selects a serving pod using live infrastructure and model-server signals. AWS says the system can reduce first-token latency by up to 82% while requiring no changes to model servers or client applications.
The announcement, published by AWS on September 18, 2026, follows the release of version v2.0.0-eksbuild.2 of the SageMaker HyperPod Inference Amazon EKS add-on. AWS’s release notes date that add-on release to September 10 and identify the gateway as its main new capability.
Three routing layers replace blind distribution
The gateway uses a body-based router to read the model field in an inference request. It can also resolve a LoRA adapter to its base model, then set routing headers that connect the request to the correct Kubernetes HTTPRoute and model-specific inference pool.
Inside that pool, an endpoint picker scores candidate pods. The scoring system can consider queue depth, KV-cache utilization, prefix-cache affinity, and whether a requested LoRA adapter is already resident in GPU memory. AWS’s documentation describes each scheduler as running its own endpoint picker, allowing different models to use separate routing policies.
The design separates deployment from traffic management. SageMaker HyperPod’s Inference Operator continues to deploy and orchestrate model-serving workloads, while the gateway manages request routing in front of those workloads. AWS says the gateway does not depend on a specific model server or orchestration layer and can work with OpenAI-compatible servers such as vLLM, SGLang, and TGI.
AWS reports the largest gains on mixed hardware
AWS tested the gateway against a Kubernetes round-robin baseline using four models ranging from 8 billion to 235 billion parameters. The company evaluated mixed GPU generations, bursty traffic, shared prompt prefixes, and a uniform fleet under steady traffic.
In the mixed-GPU test with Llama 3.1 8B, AWS reports a 97% reduction in both 95th-percentile and 99th-percentile time to first token, with throughput rising 8%. On Qwen3 32B, the same class of test produced a 98% reduction in 95th-percentile first-token latency, a 97% reduction at the 99th percentile, and a 50% throughput increase.
Bursty workloads also showed large latency differences. AWS reports a 94% reduction in 95th-percentile first-token latency and a 98% reduction at the 99th percentile for Llama 3.1 70B. For Qwen3 235B, the 99th-percentile result improved by 89%, while average throughput remained comparable to round-robin.
The gateway delivered smaller gains when traffic ran across a uniform fleet under steady conditions. AWS reports comparable results in that test, meaning the system’s benefit depends on the conditions that create uneven utilization rather than on routing alone.
Cache and adapter awareness target repeat work
AWS is also using routing to preserve computation that would otherwise be repeated. Requests with shared prompt prefixes can be directed to pods that already hold the prefix in cache, reducing the work required for multi-turn conversations and document question-answering workloads.
LoRA adapter routing addresses a related problem. When a serving pod already has the requested adapter loaded in GPU memory, the gateway can send the request there instead of forcing another pod to load it. If no pod has the adapter resident, the endpoint picker sends the request to a pod with available capacity.
Those policies make the gateway different from a simple CPU or network load balancer. The routing decision depends not only on which model a request names, but also on the current state of the model-serving process and the data it has already prepared.
One add-on, with a security setting customers must configure
AWS delivers the gateway through the SageMaker HyperPod Inference EKS add-on rather than as a separate installation. Administrators define routing through an InferenceGatewayConfig custom resource and enable gateway routing on an InferenceEndpointConfig resource. AWS says the setup uses no sidecars, service mesh, or application-code changes.
The operational boundary is less automatic than the routing itself. AWS documentation states that gateway endpoints have no request-level authentication or authorization by default. Unless customers configure JWT authentication in the gateway resource, access is restricted by the surrounding VPC and network controls.
The per-cluster gateway is available in AWS Regions where the SageMaker HyperPod inference add-on is available. AWS lists a global inference router, canary traffic splitting, and priority-based flow control as planned additions, but those capabilities are not part of the September 18 release.
AWS’s announcement describes the performance tests and configuration model, while the SageMaker documentation details the gateway’s routing components and authentication requirements.