Skip to content
AI.info

The Pulse

vLLM details opt-in FlyDSL MonoKernels for faster decode on AMD GPUs

A vLLM post explains how the opt-in MonoKernel path combines model operations on AMD Instinct GPUs and where it falls back to standard execution. For DeepSeek-V4.1-Flash it reports concurrency-1 P90 interactivity rising from 318.1 to 429.1 tokens per second per user.

vLLM details opt-in FlyDSL MonoKernels for faster decode on AMD GPUs

Turning on a specialized vLLM kernel path raised DeepSeek-V4.1-Flash’s concurrency-1 P90 interactivity from 318.1 to 429.1 tokens per second per user on an AMD MI355X node, the vLLM project said in a Nov. 18 post. The post described how its opt-in FlyDSL MonoKernels combine parts of model execution to reduce the delay between generated tokens on AMD Instinct GPUs.

The comparison ran on the same node at tensor parallel size two, with stock vLLM as the baseline. Across the concurrency sweep, the post reports total throughput 1.04 to 1.16 times the stock figure; it describes decode interactivity, not the speed of an entire agent task.

Combining work within a launch

Generating a response involves a chain of GPU kernels for operations including attention and expert computation. At small batch sizes, individual kernels may do little work while still incurring startup and completion costs. The post says those costs can make up a substantial part of decode latency, even when HIP graphs reduce CPU launch overhead.

A MonoKernel combines dependent stages within a single GPU kernel launch, with a schedule designed by its author. A model can use several such kernels; the name does not mean the entire model runs in one kernel.

Thread blocks can continue to subsequent stages when their required inputs are ready.

Different boundaries for different models

The initial DeepSeek-V4.1-Flash implementation divides its standard decoder-layer path between kernels named K1 and K2. K1 prepares attention inputs and updates the key-value cache; K2 completes attention and the feed-forward work, including tensor-parallel reductions inside the kernel. Those in-kernel reductions do not eliminate communication or synchronization between ranks.

In the initial MiniMax-M3 implementation, the fused boundary is narrower: it covers the feed-forward portion and surrounding communication, while attention remains on the standard path. The choice depends on a model’s dependencies, communication pattern and GPU resource limits.

FlyDSL is AMD’s Python domain-specific language and MLIR compiler for ROCm kernels. Kernel authors choose the schedule and synchronization; FlyDSL compiles that design rather than automatically fusing or reordering the stages.

Opt-in execution and testing

The path is off by default and can be enabled with --monokernel. Unsupported deployments fail at startup with their reasons listed. Prefill, mixed batches and steps too wide for the specialized kernels use standard layers instead, while eligible speculative verification steps can use the mono path.

The scheduler, cache manager, sampling, speculative decoding and HIP graphs remain in place. The kernels use vLLM’s loaded weights and cache and return the same tensors at the layer boundary, allowing specialized and standard layers to be mixed within a step.

Source

  1. vLLM Blog

Related stories

Explore

More articles