The Pulse
IBM-Backed llm-d 1.0 Targets Production-Ready Model Serving
IBM says llm-d 1.0 is designed to help organizations run AI inference on their own infrastructure while managing traffic from people and autonomous agents. In a test of an agentic coding workload, its prefix-aware router captured 97% of the
AI.info Team · Written with AI · fact-checked ·
A coding assistant user expects a quick answer; an autonomous agent may send a burst of requests, pause to use a tool, then return with more. IBM-backed llm-d 1.0, announced October 20 at PyTorch Conference North America, is designed to manage both kinds of traffic on shared infrastructure. Its release puts the emphasis on what happens after deployment: keeping service responsive as workloads shift, users compete for resources and operators update software.
People and agents compete for the same GPUs
IBM says llm-d 1.0 adds support for multi-tenant quality of service, zero-downtime operations and agent-aware serving. Flow control and request prioritization are intended to stop bursts from agent workloads overwhelming services that need low latency. The framework can also keep requests from a multi-turn session routed to infrastructure holding the relevant cached context, rather than repeatedly sending them to a replica that must recompute it.
That distinction matters because agents often revisit the same code, documents and instructions across multiple model calls. Prefix-cache-aware routing steers requests toward replicas that may already hold that context; KV-cache offloading moves cached information among GPU, CPU and storage resources when GPU memory is tight. IBM describes these techniques as part of a larger serving stack, not new features invented for version 1.0.
Version 1.0 focuses on operating the stack
The project’s earlier work includes batch inference optimization, prefill/decode disaggregation—which separates prompt processing from text generation—and wide expert parallelism for distributing mixture-of-experts models across accelerators. IBM says the new release integrates those approaches with operational controls for production use. That includes a stable API and the ability to keep service running through software updates, hardware problems and changing traffic, rather than relying on a single performance benchmark.
“llm-d is ready for day 2 operations,” said Carlos Costa, Distinguished Engineer at IBM Research, llm-d co-founder and lead maintainer.
IBM describes “day 2” as the work of operating an AI service after its initial deployment, including balancing competing priorities and maintaining reliability. The release is intended for organizations running their own infrastructure, with a vendor-neutral design that supports different models, accelerators and clouds.
IBM cites cache reuse and lower serving costs
IBM says a recent test of an agentic coding workload captured 97% of the available cache reuse with llm-d’s prefix-aware router. In a separate demonstration, IBM and Red Hat ran 3,000 software-engineering agents concurrently on a self-hosted deployment. IBM says that setup cost five to 10 times less than the actual price of serving the same open model as a service.
Those figures are examples IBM provides for particular workloads, not a general cost guarantee for every deployment. The research post does not specify the test hardware or a detailed methodology for either comparison. The results nevertheless illustrate the release’s central argument: routing requests to reuse context and managing shared compute can affect the cost of running agents, not just how quickly a model generates an answer.
llm-d connects model servers with Kubernetes
llm-d works with inference engines including vLLM and SGLang, adding distributed routing and operational features around the servers that run models. The Cloud Native Computing Foundation hosts the project as a sandbox, while IBM, Red Hat and Google serve as lead maintainers. That positioning makes llm-d a layer between model-serving software and Kubernetes-based infrastructure, where organizations can coordinate replicas and traffic across a deployment.
For teams considering self-hosted inference, version 1.0’s claim is narrower than a promise of automatic savings: it offers a more operationally mature way to manage varied workloads on shared infrastructure. The practical test is whether those controls preserve responsiveness and cache reuse under each organization’s own traffic, hardware and service priorities.