The Pulse
Buoyant Releases Kinference, an Open-Source Control Plane for Running vLLM on Kubernetes
Buoyant has released Kinference, an open-source project that runs on vLLM and adds token rate limiting, per-tenant usage and cost tracking, KV-cache-aware load balancing and inference policies.
AI.info Team · Written with AI · fact-checked ·
Kinference is designed as a control plane layer for an inference platform
The Kinference team announced Kinference on Nov. 4, 2026, as a new open source project for building scalable inference platforms with vLLM. The Kinference team says it created the project at Buoyant after watching customers struggle to run open weight models in their environments. Kinference uses vLLM for underlying inference and provides token rate limiting, tenant usage and cost tracking, intelligent load balancing and inference policies. The Kinference team says it designed the project as a control plane layer, and Kinference can be paired with an existing stack by pointing inference clients to its HTTP endpoint.
Kinference provides inference-specific load balancing
Clients can send OpenAI- or Anthropic-compatible inference traffic to that endpoint. In addition to policies and a routing layer, Kinference’s inference-specific load balancing takes into account signals like GPU queue sizes, historical performance and the contents of the KV cache on the GPU. The Kinference team says KV-cache awareness lets Kinference send requests to the vLLM worker that will do the least work for that specific request. In public Kinference benchmarks against standard Kubernetes load balancing, Kinference cut median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45% of requests on one workload type.
For its capacity benchmark, the Kinference team measured the requests per minute each endpoint could serve within a latency SLO. Kinference served 7.4× the goodput on shared-prefix and about 3.5× the goodput on multiturn. It was able to attain the chosen SLO at concurrency 32, the highest measured, on the shared-prefix workload, and at concurrency 8 on the multiturn workload; the kube-proxy approach was not able to attain the SLO at any concurrency level.
Inference policies allow platform owners to route or reject requests
Kinference policies let platform owners route or reject requests based on tenant, requested model, the state of the underlying GPU pool, semantic classification of prompts and many other factors. One example classifies prompt complexity and routes simpler requests to cheaper models unless, taking cache usage into account, it will be faster to send the request to an expensive model. Another policy example bursts excess requests to cloud provider endpoints like OpenAI, Anthropic and Databricks when queue depth or cache pressure indicate GPUs are saturated.
A further example prevents prompts tagged as containing PII from going to a provider, while setting “onEmpty: Reject” to fail closed. The Kinference team says policies can go through human or agentic review, be committed to git, linted and vetted, and be applied or rolled back independently.
Almost every metric is available per tenant
Kinference reports input and output token rates, histograms of input and output token sizes, and TTFT, ITL and E2E latency histograms. Its metrics also include uncached vs cached prompt tokens for cost estimation, model usage and GPU saturation. Almost every metric is available per tenant, so platform owners can track tenant usage and build specific SLOs around performance for specific tenants.
The Kinference team says Kinference is fully open source under Apache v2 and is available on GitHub today. Its installation guide starts with Gateway API CRDs, then installs the gateway CRDs and components and Kinference using Helm. The Kinference team says feedback, issues and benchmark reports from other deployments are very welcome.