Skip to content
AI.info

The Pulse

AMD reports faster inference glue kernels using AITER config pins

AMD measured faster glue kernels in two unnamed production models using configurations selected offline and packaged with their kernel text. The company says the gains were measured against TorchInductor’s single default configuration, with a smaller advantage over open-list autotuning.

AMD reports faster inference glue kernels using AITER config pins

Offline-selected configurations made inference glue kernels 1.388 times faster on one production model and 2.846 times faster on another than TorchInductor’s single default configuration, AMD reported in a technical post. Against open-list autotuning, the measured kernel speedup on the second model was 1.15 times.

Source rocm.blogs.amd.com

The measurements cover two unnamed production models, called Model A and Model B, with 143 and 25 glue kernels respectively. The reported speedups concern those kernels, not overall request latency.

Pinning configurations to kernel text

The work focuses on the small kernels around larger operations such as matrix multiplication and attention. These glue kernels handle tasks including casts, concatenations, normalizations, activations and gathers; TorchInductor generates them in Triton.

In the approach described by AMD, configuration candidates are tested offline rather than timed at startup, and a configuration that passes checks and improves the trace is written into the package. AITER is a library of Triton kernels for AMD Instinct GPUs; the tuned package can include both a replacement kernel body and a selected configuration.

The conversion leaves the graph intact: each fusion still calls a function with the same pointers and grid. Kernels are matched to pins using a fingerprint derived from the SHA-256 hash of the Triton function text, with the numeric suffix removed and the result truncated to 20 hexadecimal characters. If two pins with the same fingerprint disagree, a matching kernel name determines which is applied.

Separating body changes from tuning

The tests separated a kernel-body swap from a configuration change. The baseline retained the TorchInductor body and used a single default glue configuration; the AITER stage swapped the body while keeping that default. A tuned stage then changed the configuration, and a control applied the tuned configurations to the compiler body.

For Model A, the body swap covered three SiLU kernels out of 143. For Model B, 24 of 25 files matched the compiler text, while the remaining file differed only in constants that the build reinitializes.

AMD says its Model A sweep tested 5,220 candidates, of which 1,329 failed the bitwise check; none failed to compile. The search selected new configurations for 67 kernels and left the other 76 at baseline. Of the 67 selected configurations, 12 matched a configuration emitted by the compiler heuristic for that kernel, while 55 did not.

Checks and limits

Each kernel was screened in three rounds of five repetitions, with the three fastest passing candidates confirmed over seven rounds of ten, interleaved with the baseline. A candidate was retained if its median was at least 0.5 percent faster and its maximum across the confirmation rounds was below the baseline’s minimum.

For Model B, 17 kernels received a new configuration, six retained the previous one because no candidate cleared the 0.5 percent threshold, and two had no passing candidate. A fast sigmoid allowed one unit in the last place of bf16 error was not kept because it made glue slower on both models.

The company also reported variability in three Model A open-list autotuning compiles: 1618.9, 1769.8 and 1863.5 microseconds, a spread of 245 microseconds. Those runs had not been screened for another process on the GPU.

The authors say the approach is suited to fixed shapes and does not help when the hot path remains an untuned matrix multiplication, the shape changes every step, or the time is a launch the kernel body cannot fuse away. Request latency was measured separately, with the profiler off.

Source

  1. AMD ROCm Blogs

Companies in this story

Related stories

Explore

More articles