The Pulse
AWS routes HyperPod inference across clusters through one endpoint
Amazon SageMaker HyperPod has a global gateway for routing inference requests across registered clusters in one or more AWS Regions. The gateway selects healthy clusters using configurable weights and automatically moves traffic away from failed clusters.

A new Amazon SageMaker HyperPod gateway routes AI inference requests across registered clusters through a single endpoint, including clusters in different AWS Regions. The Global Inference Gateway selects a healthy cluster for each request and automatically moves traffic away from a failed cluster, Amazon Web Services said in a technical post published November 20.
The service adds a layer above HyperPod’s existing per-cluster Inference Gateway, which continues to choose the model pod within the selected cluster. AWS operates the global routing fleet and the private connections to clusters, while customers choose which clusters to publish and group behind a gateway.
From cluster endpoints to one gateway
In an example in the AWS post, a team serving four models across five HyperPod clusters in two Regions would otherwise track 20 endpoints, one for each model on each cluster. The existing Inference Gateway reduces that to five endpoints, but leaves clients to choose a cluster; the global gateway provides one endpoint across the registered clusters.
For each request, the new gateway scores healthy clusters using available key-value cache and GPU memory, queue depth, regional proximity and recent errors. A metrics publisher on each cluster supplies its load signals.
The default weights are 0.20 for cache and GPU memory headroom, 0.25 for queue depth, 0.40 for regional proximity and 0.15 for error rate. Each custom weight can range from 0.0 to 1.0; customers who specify weights must set all four, and the values must add up to 1.0. A weight of zero turns off that signal.
In beta tests, Bagelcode used a single endpoint to route requests across two AWS Regions, said Jinhyung Park, the company’s VP of AI Platform. Park said the gateway “kept serving them when one cluster could no longer serve the model.”
Publishing and connecting clusters
To make a cluster reachable, a customer enables global routing in its existing InferenceGatewayConfig. The HyperPod Inference controller then publishes the cluster privately from the customer’s account. Customers create the global gateway by calling CreateGlobalInferenceGateway with the clusters they want it to use.
When a cluster is published, the controller creates a Network Load Balancer in front of its Inference Gateway, publishes a virtual private cloud endpoint service and uploads the listener certificate to the customer’s TLS bucket. A gateway creation call fails if it lists a cluster that has not been published.
Clients send OpenAI-compatible requests signed with AWS Signature Version 4, known as SigV4, to the gateway’s regional endpoint. After authentication and cluster selection, AWS PrivateLink carries the request to the cluster, where the existing gateway routes it to a model pod.
Access and operations
The service checks SigV4 signatures, AWS Identity and Access Management permissions and rate limits before requests reach a cluster. Callers need permission to invoke the gateway, and streaming responses have a separate permission action.
The gateway does not log or store prompts or completions. Customers can change routing weights after creation, add a prepared cluster or detach one; when a cluster is added, the gateway validates it and begins routing to it once it is healthy.
Sources
- Original source · Blog post Amazon Web Services aws.amazon.com
Companies in this story
Related stories
- Salesforce uses SageMaker routing to shift model requests across regions
- AWS makes snapshot-based AgentCore V2 the default for new runtimes in select regions
- AWS customers can now deploy PolyAI's voice-dialog model through SageMaker JumpStart
- SageMaker adds cache-aware routing for real-time AI endpoints
- Hugging Face switches language-model routing to OpenRouter