Skip to content
AI.info

The Pulse

Google Cloud outlines an Envoy stack for routing AI agent traffic

Google Cloud has outlined an Envoy-based approach to routing model requests, limiting token use and managing AI agents’ tool calls. The company describes how the proxy can work with Agent Substrate to handle suspended agents and restrict outbound traffic.

Google Cloud outlines an Envoy stack for routing AI agent traffic

Google Cloud has outlined an Envoy-based stack for routing AI model requests, tracking token budgets and controlling agents’ access to tools. In a blog post, the company described how the open-source proxy and related components can handle traffic that conventional request-based networking does not distinguish.

Envoy is an edge and service proxy designed for cloud-native applications. Google Cloud’s account brings together body parsing, token limits, model and tool routing, and external security checks as the main parts of an agent-focused stack.

Reading requests and counting tokens

Google Cloud identifies four challenges in putting traditional Layer 7 proxies in front of autonomous agents: information buried in request bodies, limits based on request counts, stateless load balancing and tool execution without inline inspection. The company says agents choose which services to contact, consume tokens over multiple turns and may need their sessions directed back to a server that retains their context.

The Envoy ecosystem includes body parsers that normalize requests across model families, inspect the payload for the target model and translate the request into the provider’s format. Google Cloud says an application can switch from a commercial model to a self-hosted model through an Envoy configuration update without changing the application code.

The proxy also parses server-sent event response streams to capture token counts from usage information, according to the post. The described token controls set per-minute and per-day limits by tenant, project or model rather than relying solely on request counts.

Envoy estimates input tokens from request bytes and records actual usage extracted from model responses in a quota database. Google Cloud says it can reject subsequent requests that exceed quota with an HTTP 429 response before they reach the inference cluster. It also describes tracking token consumption and message counts per session or agent to throttle loops that exceed their assigned budgets.

Routing models and tools

For model routing, Google Cloud describes using the Kubernetes Gateway API Inference Extensions and Endpoint Picker to send prompts with identical system instructions to a GPU worker holding the relevant cache. The company says this reduces latency. It also says a request can be retried or redirected to a secondary model pool if the primary model fails or returns a 429 status.

For Model Context Protocol (MCP) tool calls, Envoy can route using request bodies and HTTP headers. The post describes using the Mcp-Method and Mcp-Name headers under the July 28, 2026 MCP specification, allowing routing without parsing the body. Backends reject mismatches between those headers and the body with an HTTP 400 / -32020 HeaderMismatch error, according to Google Cloud.

Envoy can connect through external processing to policy engines including Google Cloud’s Model Armor, ShieldGemma or local classifiers. The company describes those callouts as a way to detect prompt injections and sensitive-data leaks during traffic processing.

Handling idle agents

The company also describes pairing Envoy with Agent Substrate, a runtime for agent environments. Agent Substrate uses lightweight isolation runtimes that can suspend and resume, including Cloud Hypervisor, microVMs and gVisor application sandboxes.

When traffic reaches a suspended agent, Envoy buffers the initial packet while signaling the runtime to resume the instance. Google Cloud says the end-to-end resumption completes in milliseconds. Envoy can also act as an inbound and outbound gateway, routing incoming traffic and restricting outgoing calls to verified domains.

Sources

  1. Original source · Blog post Google Cloud cloud.google.com

Companies in this story

Related stories

Explore

More articles