The Pulse
vLLM 0.29.0 Makes Model Runner V2 the Default
vLLM 0.29.0 routes all supported models through Model Runner V2 by default after 594 commits from 277 contributors. The release also expands speculative decoding, CUDA graph profiling, model support and deployment options while targeting Mo
AI.info Team ·
594 commits move vLLM’s runtime to a new default
vLLM 0.29.0 changes the execution path for every supported model: Model Runner V2 is now the default, completing a rollout that began with pooling models and later expanded to dense architectures. The release, published September 9, contains 594 commits from 277 contributors, including 91 people making their first contribution to the project.
Model Runner V2, or MRV2, replaces the older runner’s default position with a design that keeps more request bookkeeping on the GPU, treats asynchronous execution as a primary constraint and separates model-specific behavior from the common serving path. Users do not need to change vLLM’s public APIs to adopt it.
“I co-created and now co-lead the vLLM project, a widely-adopted open-source inference engine for LLMs,” Woosuk Kwon, co-founder and chief technology officer of Inferact, writes on his website. Kwon is listed among the contributors acknowledged in vLLM’s technical introduction to Model Runner V2.
Why vLLM is retiring the older runner
vLLM’s own Model Runner V2 documentation describes the project as a response to technical debt that accumulated as asynchronous scheduling, speculative decoding and support for new model architectures were added to the earlier implementation. The new runner divides responsibilities into smaller components, including model state, input preparation, sampling and speculative-decoding paths.
The change also moves several operations that previously depended on CPU-side tensor work onto the GPU through Triton kernels. vLLM says MRV2 maintains persistent request state separately from the tensors prepared for each execution step, allowing request ordering to change without rebuilding the same state structure around every batch.
That architecture matters most in high-concurrency serving, where small CPU operations can interrupt GPU work. In the project’s March technical post, vLLM reported a 56.2% throughput increase on Qwen3-0.6B running on one GB200 when input preparation moved to the GPU, rising from roughly 16,000 to 25,000 output tokens per second. The same post reported 6.3% lower time per output token for speculative decoding on four GB200 GPUs running GLM-4.7-FP8 with one speculative token.
MRV1 remains a fallback, not a peer path
vLLM is not removing Model Runner V1 in 0.29.0. The release notes say the project is considering MRV1 deprecated and is targeting its removal for v0.32. The older runner still handles a small number of ROCm models and configurations that MRV2 does not yet support.
Users who configure sequence parallelism, dual-batch overlap, elastic expert parallelism, custom logits processors or certain speculative-decoding methods may still be routed to MRV1. vLLM says it expects to close those gaps within two to three weeks, but the release does not guarantee that every configuration will switch immediately.
The distinction gives operators a transition period while making MRV2 the normal path for new deployments. It also changes the maintenance direction of the project: the release notes say vLLM does not intend to accept further MRV1-specific improvements or optimizations.
CUDA graphs and sampling receive major updates
MRV2 gains CUDA graph memory profiling for automatic KV-cache sizing, along with batch-sharded sampling that reduces per-step logits memory by a factor of the tensor-parallel size. The release also adds prompt-embedding support, hidden-state extraction for speculation, padded full CUDA-graph dispatch for uniform decoding under speculative decoding and skipped data-parallel synchronization before EAGLE and MTP draft prefill.
Those changes target the sections of an inference server that determine how much work reaches the GPU per step and how much memory remains available for concurrent requests. They do not change the OpenAI-compatible serving interface, but they can affect startup behavior, memory allocation and the set of model features that select a particular runner.
Model Runner V2 also continues vLLM’s work on persistent batching, speculative decoding, tiered KV-cache offload and disaggregated execution. The release lists disk-based secondary-tier offload, queue admission control, new prefix-cache controls and additional CUDA-graph paths among the engine-core changes.
New checkpoints arrive with the same release
vLLM 0.29.0 adds support for several model families and checkpoints. The list includes Hy4-preview; Tencent’s 770B model with 49B active parameters, Gated DeepSeek Sparse Attention and native MTP; Qwen3.8-Flash-Next with BF16, FP8 and NVFP4 support; GraniteSWA and GraniteMoeSWA; NemotronH_Omni_Reasoning_V3; and Kimi K3 NVFP4 checkpoints.
The release also adds bidirectional attention for DeepSeek-backbone embedding models, FP8 ModernBERT support and additional MTP coverage for Nemotron vision-language models. Kimi K3 now runs on ROCm with the V2 runner, extending the new default beyond CUDA deployments.
Installers cover CUDA, ROCm, CPU and XPU
vLLM publishes Python wheels for CUDA 12.9 and CUDA 13.0, along with CPU and XPU packages. The project also provides Docker images for CUDA 13.0, CUDA 12.9, Ubuntu 24.04, ROCm, CPU and Intel XPU deployments.
The default CUDA image is available as vllm/vllm-openai:v0.29.0, with separate tags for CUDA 12.9 and Ubuntu 24.04. ROCm users can install the release from vLLM’s wheel index or use vllm/vllm-openai-rocm:v0.29.0.
For operators, the immediate task is less about changing an API than checking assumptions around unsupported runner features, ROCm model coverage and memory behavior. vLLM 0.29.0 makes MRV2 the standard execution path; the remaining compatibility list determines when that standard becomes the only path.