# Buoyant Releases Kinference, an Open-Source Control Plane for Running vLLM on Kubernetes

Published 2026-11-05 · AI.info Team · AI.info

> Buoyant has released Kinference, an open-source project that runs on vLLM and adds token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing and inference policies.

## Kinference is designed as a control plane layer for an inference platform

The Kinference team announced Kinference on Nov. 4, 2026, as a new open source project for building scalable inference platforms with [vLLM](https://vllm-project.github.io/2026/11/04/introducing-kinference.html). Kinference was created at Buoyant, the Kinference team says, after watching customers struggle to run open weight models in their environments. Kinference uses vLLM for underlying inference and provides token rate limiting, tenant usage and cost tracking, intelligent load balancing and inference policies. The project is designed as a control plane layer, and Kinference can be paired with an existing stack by pointing inference clients to its HTTP endpoint.

## Kinference provides inference-specific load balancing

Clients can send OpenAI- or Anthropic-compatible inference traffic to that endpoint. In addition to policies and a routing layer, Kinference’s inference-specific load balancing takes into account signals like GPU queue sizes, historical performance and the contents of the KV cache on the GPU. KV-cache awareness lets Kinference send requests to the vLLM worker that will do the least work for that specific request, the Kinference team says. In public Kinference benchmarks against standard Kubernetes load balancing, Kinference cut median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45% of requests on one workload type.

For its capacity benchmark, the Kinference team measured the requests per minute each endpoint could serve within a latency SLO. Kinference served 7.4× the goodput on shared-prefix and about 3.5× the goodput on multiturn. It was able to attain the chosen SLO at concurrency 32, the highest measured, on the shared-prefix workload, and at concurrency 8 on the multiturn workload; the kube-proxy approach was not able to attain the SLO at any concurrency level.

## Inference policies allow platform owners to route or reject requests

Kinference policies let platform owners route or reject requests based on tenant, requested model, the state of the underlying GPU pool, semantic classification of prompts and many other factors. One example classifies prompt complexity and routes simpler requests to cheaper models unless, taking cache usage into account, it will be faster to send the request to an expensive model. Another policy example bursts excess requests to cloud provider endpoints like [OpenAI](https://ai.info/companies/openai), [Anthropic](https://ai.info/companies/anthropic) and [Databricks](https://ai.info/companies/databricks) when queue depth or cache pressure indicate GPUs are saturated.

A further example prevents prompts tagged as containing PII from going to a provider, while setting “onEmpty: Reject” to fail closed. Policies can go through human or agentic review, be committed to git, linted and vetted, and be applied or rolled back independently.

## Almost every metric is available per tenant

Kinference reports input and output token rates, histograms of input and output token sizes, and TTFT, ITL and E2E latency histograms. Its metrics also include uncached vs cached prompt tokens for cost estimation, model usage and GPU saturation. Almost every metric is available per tenant, so platform owners can track tenant usage and build specific SLOs around performance for specific tenants.

Kinference is fully open source under Apache v2 and is available on [GitHub](https://ai.info/companies/github) today. Its installation guide starts with Gateway API CRDs, then installs the gateway CRDs and components and Kinference using Helm. Feedback, issues and benchmark reports from other deployments are very welcome, the Kinference team says.

## Sources

1. Original source [vLLM Blog](https://vllm-project.github.io/2026/11/04/introducing-kinference.html) vllm-project.github.io

## Related stories

- [vLLM 0.33 Drops CUDA 12.9 Builds](https://ai.info/news/vllm-033-drops-cuda-129-builds)
- [Agent Substrate targets denser Kubernetes workloads](https://ai.info/news/nvidia-agent-substrate-kubernetes-agents)
- [Google plans Kubernetes Python client upgrades for AI workloads](https://ai.info/news/google-kubernetes-python-client-ai-ml-modernization)
- [vLLM details opt-in FlyDSL MonoKernels for faster decode on AMD GPUs](https://ai.info/news/vllm-flydsl-monokernels-agent-inference)
- [Google Cloud introduces open AI stack for Kubernetes](https://ai.info/news/google-cloud-open-ai-stack-kubernetes)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/vllm-project-introduces-kinference-inference-control-plane
