# FlashInfer on ROCm routes AMD GPU kernels automatically

Published 2026-11-20 · AI.info Team · AI.info

> The FlashInfer release selects between AMD's AITER library and in-tree HIP kernels for supported attention calls. It also adds FP8 prefill, fused mixture-of-experts operations and other features, but must be built from source.

FlashInfer on [ROCm](https://rocm.blogs.amd.com/artificial-intelligence/flashinfer-release3/README.html) now selects between AMD's AITER library and in-tree HIP kernels for supported attention calls, using a default automatic backend. The release, described in an AMD blog post on November 20, also adds FP8 prefill, fused mixture-of-experts operations and AITER decode under CUDA graph capture.

FlashInfer is an open-source library of attention and large language model serving kernels. The ROCm release supports AMD Instinct MI300X, MI325X, MI350X and MI355X GPUs. Its decode, prefill and multi-head latent attention (MLA) wrappers retain the signatures and argument order of upstream FlashInfer v0.6.18.

## How routing works

Single, batch paged and batch ragged prefill, as well as batch decode, have both AITER and in-tree HIP backends. For attention calls, the automatic setting first checks whether AITER can handle the data types, key-value cache layout, masks and head dimensions, then applies architecture-specific routing rules.

AMD says those rules follow its testing and benchmarking on CDNA3 and CDNA4 GPUs. The in-tree path handles calls that AITER cannot serve, including an FP8 key-value cache read by a 16-bit query and batch decode using tensor cores.

MLA and fused mixture-of-experts operations use AITER only. Normalization, activation, rotary position embedding and key-value cache append operations use in-tree kernels by default, with AITER available as an opt-in for some entry points.

After planning a batch prefill or batch decode call, the wrapper reports the selected backend and, for most AITER declines, the reason for the fallback. AITER attention kernels are launched from FlashInfer's C++ code; MLA remains an exception that runs AITER through Python.

## New operations and limits

The release also brings cascade attention, MLA and standalone rotary position embedding operations that were in progress in the preceding prefill-attention release. POD-Attention overlaps prefill and decode in one kernel.

The fused mixture-of-experts operation supports bf16 and fp16, as well as FP8 expert weights. Each expert weight must first be pre-shuffled, following quantization for FP8 weights.

The in-tree FP8 key-value cache kernels use FNUZ encodings on both supported architectures and refuse OCP encodings. On gfx950, a cache written in float8_e4m3fn for AITER FP8 prefill cannot also be read by decode: the in-tree decode rejects that encoding, while AITER decode accepts only 16-bit key-value caches.

Some upstream modules are not supported on ROCm because they wrap CUDA-only libraries or other CUDA-specific components. Other operations, including flashinfer.topk and the selective state update in flashinfer.mamba, do not have a ROCm implementation.

## Building and compiling

The supported configuration uses ROCm 10.1.0, PyTorch 2.12.0, Python 3.12, Ubuntu 24.04 and amd-aiter 0.1.21.post2 built from source. No wheel or prebuilt image is published for the release, so FlashInfer on ROCm must be built from the repository.

Selected attention, cache-append and fused mixture-of-experts operations can compile under torch.compile when the custom-operations flag is set before importing FlashInfer. AMD cautions that PyTorch 2.12 can silently miscompile an operation that writes into a view of a larger tensor, and specifies separately allocated key and value caches for append_paged_kv_cache.

## Sources

1. Original source · Blog post [AMD ROCm Blogs](https://rocm.blogs.amd.com/artificial-intelligence/flashinfer-release3/README.html) rocm.blogs.amd.com

## Companies in this story

- [Silo AI](https://ai.info/companies/silo-ai)

## Related stories

- [AMD details GPU-initiated DMA transfers for multi-GPU workloads](https://ai.info/news/amd-gpu-initiated-dma-multi-gpu)
- [AMD reports faster inference glue kernels using AITER config pins](https://ai.info/news/amd-aiter-inference-glue-kernel-optimization)
- [vLLM details opt-in FlyDSL MonoKernels for faster decode on AMD GPUs](https://ai.info/news/vllm-flydsl-monokernels-agent-inference)
- [Cerebras puts split AI inference into production with AMD GPUs](https://ai.info/news/cerebras-disaggregated-inference-amd-production)
- [AMD reports token-cache reuse in coding-agent benchmark](https://ai.info/news/amd-atom-token-cache-coding-agent-benchmark)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/flashinfer-rocm-automatic-kernel-routing-amd-gpus
