The Pulse
Salesforce uses SageMaker routing to shift model requests across regions
Salesforce has integrated Amazon SageMaker AI's cross-region inference router into its internal model-serving gateway. The router can move requests to other deployed endpoints when local destinations show errors, while requests with residency requirements remain in-region.

Salesforce is using Amazon SageMaker AI to route model requests between deployed endpoints in different regions, replacing failover logic in its internal gateway. An Amazon Web Services post published on Nov. 18 describes how the system shifts traffic when local destinations show errors while keeping requests with data residency requirements in their designated region.
The Agentforce Foundation team serves models for products including Agentforce and the Salesforce Trust Layer across more than 30 global regions, according to AWS. The models include both those trained in-house and open-source models.
From custom failover to a router
Salesforce provisions capacity for customer traffic across hundreds of AWS accounts. AWS says overlapping customer peaks and short-notice expansions can leave some regions with idle instances while others run short of capacity.
Before the change, the company's internal gateway redirected traffic that local endpoints could not handle, using health checks, retries and region-switching logic. AWS says the earlier gateway lacked capacity signals from local and remote endpoints, so it could not tell when to shift traffic or whether a destination had room.
The earlier route also sent cross-region traffic out of the virtual private cloud through network gateways and over public AWS networking. In design reviews, the team raised the risk that moving traffic during an incident could overload another region.
How requests move
The SageMaker AI inference router gives an application one invocation path to endpoints across regions and accounts. A routing target identifies the endpoints and inference components that can serve a particular model; an endpoint in another account must grant the named router permission to invoke it.
The router favors a destination in its own region while that destination serves requests successfully. Platform faults, model-container 5xx errors, throttling and timeouts reduce a destination's health; persistent errors cause the router to send a growing share of new requests to healthy destinations. When a destination fails a request without running the model, such as when it throttles the request, the router retries it once elsewhere.
AWS says the router typically starts shifting traffic within about a minute of persistent errors. For destinations with little traffic, the shift can take several minutes; traffic returns over several minutes after recovery, more slowly than it left. Invalid input and model latency do not trigger a shift.
The router uses capacity already deployed at the endpoints. AWS says it does not deploy models, copy artifacts or add instances; each endpoint's own autoscaling adds instances as traffic grows.
Residency and visibility
Salesforce placed the router call in its gateway after the data residency check and before response handling. Requests tagged with a data residency requirement use an in-region routing target whose only destination is the local endpoint. Other requests go to the router with the same body, content type and accept type the gateway previously sent.
Salesforce logs the resource that served each request and the routing reason alongside request and trace identifiers, and adds those fields as application performance monitoring tags. The team combines them with router metrics when investigating a traffic shift.
In a configuration test, Salesforce created a router in US West (Oregon) and confirmed that requests reached a local destination. The team then registered an endpoint in US East (Ohio) owned by a different account, removed the local destination and confirmed that requests reached the remote endpoint across the account and region boundary.
Sources
- Original source · Blog post Amazon Web Services aws.amazon.com
Companies in this story
Related stories
- AWS customers can now deploy PolyAI's voice-dialog model through SageMaker JumpStart
- OpenTSLM time-series model becomes available in SageMaker JumpStart
- SageMaker adds cache-aware routing for real-time AI endpoints
- Salesforce puts CRM reasoning inside new Koa model
- Salesforce releases open-weight speech transcription model