The Pulse
Google Cloud introduces open AI stack for Kubernetes
Google Cloud is introducing and expanding an open-source AI stack for Kubernetes and Google Kubernetes Engine. The post describes tools for model training, distributed inference, Ray workloads and agent-code isolation, and Google Cloud repo
AI.info Team · Written with AI · fact-checked ·
A unified stack
Google Cloud is introducing and expanding an open-source AI stack for Kubernetes and Google Kubernetes Engine (GKE), presenting the work at KubeCon + CloudNativeCon North America 2026. The announcement is dated November 10, 2026.
The listed components include OpenRL, llm-d 1.0, Ray on GKE with KubeRay, Agent Substrate, gVisor and TorchTPU. OpenRL, llm-d, KubeRay, Agent Substrate and gVisor are CNCF Sandbox projects; TorchTPU is a backend for PyTorch on Cloud TPUs and GPUs.
Training, inference and scheduling
OpenRL is a Kubernetes-native post-training service for reinforcement learning, contributed to the CNCF Sandbox by Google. OpenRL lets researchers write Python training code around four primitives: forward_backward, optim_step, sample and save_state.
OpenRL interleaves training steps and rollout inference on the same accelerators, rather than requiring separate static clusters for each. The service sends parameter changes to rollout workers instead of transferring full model weights after each optimization step.
For serving, llm-d 1.0 is a distributed inference control plane that works with engines such as vLLM and SGLang. Its prefix-cache-aware routing is intended to send multi-turn requests to replicas with matching key-value cache context, according to the post.
Google Cloud says its prefix-cache-aware routing more than doubled cache hit rates in production deployments, from 35% to 70%. It also reports a 70-85% reduction in median time-to-first-token on multi-turn agent workloads.
llm-d can move cache data from accelerator memory to host memory and then to local or cloud storage when accelerator memory approaches capacity. KubeRay is a Kubernetes operator for managing Ray clusters, jobs and services, with Kueue integration for shared GPU and TPU capacity.
Isolation for agent workloads
For agent execution, Agent Substrate provides workflow orchestration while gVisor isolates untrusted code at the system-call level. gVisor is a user-space kernel that intercepts and virtualizes system calls.
These sandboxes can start in milliseconds per agent turn and can serve both OpenRL rollout generation and production agent workflows. Google Cloud frames isolation as a way to run model-generated code in shared clusters without exposing neighboring workloads to that code.
Benchmark results
Google Cloud reports testing Gemma 4 with Harvey.ai's open-source Legal Agent Benchmark, which covers 24 legal practice areas. The evaluation included matters ranging from mergers and acquisitions due diligence to contract disputes, according to Google Cloud.
In that evaluation, standard supervised fine-tuning reached around 50% accuracy on multi-step reasoning rubrics, while reinforcement learning with OpenRL raised accuracy to 70%, according to Google Cloud. The training pipeline shared GPU and TPU capacity with concurrent LoRA and inference jobs on GKE without manual partition management, the post says.