Skip to content
AI.info

The Pulse

FlashInfer on ROCm routes AMD GPU kernels automatically

The FlashInfer release selects between AMD's AITER library and in-tree HIP kernels for supported attention calls. It also adds FP8 prefill, fused mixture-of-experts operations and other features, but must be built from source.

FlashInfer on ROCm routes AMD GPU kernels automatically

FlashInfer on ROCm now selects between AMD's AITER library and in-tree HIP kernels for supported attention calls, using a default automatic backend. The release, described in an AMD blog post on November 20, also adds FP8 prefill, fused mixture-of-experts operations and AITER decode under CUDA graph capture.

FlashInfer is an open-source library of attention and large language model serving kernels. The ROCm release supports AMD Instinct MI300X, MI325X, MI350X and MI355X GPUs. Its decode, prefill and multi-head latent attention (MLA) wrappers retain the signatures and argument order of upstream FlashInfer v0.6.18.

How routing works

Single, batch paged and batch ragged prefill, as well as batch decode, have both AITER and in-tree HIP backends. For attention calls, the automatic setting first checks whether AITER can handle the data types, key-value cache layout, masks and head dimensions, then applies architecture-specific routing rules.

AMD says those rules follow its testing and benchmarking on CDNA3 and CDNA4 GPUs. The in-tree path handles calls that AITER cannot serve, including an FP8 key-value cache read by a 16-bit query and batch decode using tensor cores.

MLA and fused mixture-of-experts operations use AITER only. Normalization, activation, rotary position embedding and key-value cache append operations use in-tree kernels by default, with AITER available as an opt-in for some entry points.

After planning a batch prefill or batch decode call, the wrapper reports the selected backend and, for most AITER declines, the reason for the fallback. AITER attention kernels are launched from FlashInfer's C++ code; MLA remains an exception that runs AITER through Python.

New operations and limits

The release also brings cascade attention, MLA and standalone rotary position embedding operations that were in progress in the preceding prefill-attention release. POD-Attention overlaps prefill and decode in one kernel.

The fused mixture-of-experts operation supports bf16 and fp16, as well as FP8 expert weights. Each expert weight must first be pre-shuffled, following quantization for FP8 weights.

The in-tree FP8 key-value cache kernels use FNUZ encodings on both supported architectures and refuse OCP encodings. On gfx950, a cache written in float8_e4m3fn for AITER FP8 prefill cannot also be read by decode: the in-tree decode rejects that encoding, while AITER decode accepts only 16-bit key-value caches.

Some upstream modules are not supported on ROCm because they wrap CUDA-only libraries or other CUDA-specific components. Other operations, including flashinfer.topk and the selective state update in flashinfer.mamba, do not have a ROCm implementation.

Building and compiling

The supported configuration uses ROCm 10.1.0, PyTorch 2.12.0, Python 3.12, Ubuntu 24.04 and amd-aiter 0.1.21.post2 built from source. No wheel or prebuilt image is published for the release, so FlashInfer on ROCm must be built from the repository.

Selected attention, cache-append and fused mixture-of-experts operations can compile under torch.compile when the custom-operations flag is set before importing FlashInfer. AMD cautions that PyTorch 2.12 can silently miscompile an operation that writes into a view of a larger tensor, and specifies separately allocated key and value caches for append_paged_kv_cache.

Sources

  1. Original source · Blog post AMD ROCm Blogs rocm.blogs.amd.com

Companies in this story

Related stories

Explore

More articles