Is Finer Better? The Limits of Microscaling Formats in Large Language Models

AI summary

Overview

Research area: Low-precision numerics and quantization for large language models (microscaling data formats for efficient training and inference). Technical level: Advanced (assumes familiarity with floating-point formats, block quantization, and perplexity-based evaluation). Scope: The paper identifies and explains a counterintuitive failure mode of FP4 microscaling when block sizes shrink, traces it to the quantization of the shared scales, and proposes an FP8 unsigned E5M3 scale format as a hardware-friendly fix.

What This Paper Is About

Microscaling formats let several tensor elements share one scale factor, and the field has been moving toward ever smaller blocks in the expectation that finer granularity always reduces quantization error. This paper shows that expectation breaks down: beyond a certain point, shrinking the block size actually increases error and hurts model quality, an effect the authors call "perplexity inversion." Their goal is to explain where this behavior comes from and to offer a fix that does not require expensive global rescaling of weights and activations.

Key Contributions

1. Discovery and analysis of a quantization anomaly in microscaling formats in which decreasing block size paradoxically increases quantization error.
2. A theoretical framework that decouples the sources of quantization error and explains their interaction with LLM tensor statistics, pinpointing the origin of the anomaly to the quantization of the microscaling scales, which hinders representation of low-magnitude blocks. The framework is stated to extend beyond FP4 elements with FP8 unsigned E4M3 scales.
3. A hardware-friendly design proposal to support FP8 unsigned E5M3 scales, which mitigates the issue at minimal hardware cost.
4. Demonstration that FP4 microscaling with UE5M3 scales, without per-tensor scaling, matches or exceeds UE4M3 scales with per-tensor scaling across several models and benchmarks.

Main Findings

• The expected trend holds only with unquantized scales: When block scales are kept in BF16 (not quantized) and elements are FP4, the perplexity gap decreases monotonically as block size is reduced from 256 to 8 for every model tested.
• FP8 UE4M3 scales break the trend: With FP8 E4M3 scales, granite-3.3-8b shows a clear upswing at block size 16; llama-3.1-8b and mixtral-8x7b-instruct show inversion at block size 8; llama-2-7b is strictly monotonic with no inversion down to block size 8. Inversion can still appear at block sizes 2 and 4 even when absent at block size 8.
• Per-block error routinely worsens with finer blocks: In 2D density plots comparing block size 8 versus 16 for a granite-3.3-8b Query weight tensor, about 25% of data points sit above the diagonal, meaning larger error at the smaller block size. Weights in granite-3.3-8b never exceed magnitude 1.0, so outlier truncation is not the cause. The pattern holds across weight tensors and models, including models with no perplexity inversion at block size 8.
• A narrow-distribution threshold: Per-tensor MSE curves for block size 8 and 16 overlap and cross over at σ ≈ 2 × 10^-2. Below that threshold, block size 8 has higher MSE than block size 16. Granite-3.3-8b has most weights below the threshold and shows pronounced inversion; llama-2-7b has a large fraction above it and shows none.
• Scale quantization is the driver: With non-quantized scales, MSE versus σ remains monotonic and block size 16 error is consistently larger than block size 8 error.
• Normal distributions reproduce model behavior: Random tensors drawn from a Normal distribution with μ = 0 and variable σ closely match pretrained model data, including mamba-codestral-7b, which is especially narrow. For σ < 0.5, MSE rises dramatically as σ decreases before dropping again toward zero. Different ideal distributions show qualitatively similar trends with some scattering around the Normal curve.
• Theory matches experiment: The non-quantized-scale model agrees with experimental data at χ² ≈ 2 × 10^-9; the FP8 UE4M3 scale model agrees at χ² ≈ 4 × 10^-8; INT4 microscaling modeling agrees at χ² ≈ 1.3 × 10^-6.
• Three error components: For large σ, the error of elements with x_i ≠ x_max dominates the total MSE. In narrower distributions, the error on representing each block's maximum (zero when scales are unquantized) becomes more important and can dominate; its weight grows as block size shrinks. At the lowest σ, error is entirely dominated by rounding all block elements to zero.
• A cheap fix: FP8 UE5M3 uses the unused bit in the unsigned scale as an extra exponent, dropping the minimum non-zero representable absolute scale value from 2^-9 (UE4M3) to 2^-17 (UE5M3) while preserving precision. An alternative UE4M4 repurposing is less hardware friendly and less robust.
• Benchmark results at block size 8: For granite-3.3-8b, UE5M3 gives Wiki perplexity 5.04 versus 7.43 for UE4M3 and 5.39 for UE4M3-S, with GSM8K 56.17 and MMLU 57.51 versus 44.88 and 55.23 for UE4M3-S. For bamba-9b-v2, UE4M3 without scaling collapses to Wiki 21.25 and GSM8K 2.65, while UE5M3 reaches 6.53 and 39.42. For llama-3.1-8b and nemotron-nano-9b-v2, UE5M3 is roughly on par with UE4M3-S. Per-tensor scaling for UE4M3-S was computed dynamically, so those results reflect the best accuracy achievable with that format.

Methodology in Plain English

The authors first measure how much perplexity degrades when both weights and activations of all linear layers except the last one are quantized with FP4 microscaling, sweeping the block size while also measuring states such as nan, inf, or None. Perplexity is measured on next-token prediction over the Wikitext2 test split with sequence length 2048 tokens; attention matmuls are not quantized. They vary two things independently: whether block scales are quantized at all, and whether they are FP8 UE4M3 or BF16.

To localize the problem beyond model-level metrics, they quantize individual weight tensors twice with different block sizes and compute per-block and per-tensor mean squared error, plotting MSE against each tensor's standard deviation. They then build a statistical model: weights are treated as draws from a Normal distribution with mean zero, each block is normalized by its maximum, and the resulting scaled values follow a truncated Gaussian. The MSE is computed analytically per quantization bin, integrated over the distribution of the block maximum, and decomposed into separate terms. Adding scale quantization requires accounting for the probability mass of each possible scale value and for the case where an entire block rounds to zero. Finally, they implement UE5M3 in the design of a systolic array processing engine with a microarchitecture similar to Agrawal et al. (2021) and estimate hardware overhead.

Why This Matters

This work changes the default assumption that finer quantization granularity is always safer. It shows that the shared scale, not the element format, can become the limiting factor, and that models with narrow weight distributions are the most exposed. It gives practitioners a diagnostic threshold (σ ≈ 2 × 10^-2) and a theory that can be reused to evaluate other formats, block sizes, and element precisions before committing to them in training or inference pipelines.

• Quantized LLM deployment: Choosing block sizes and scale formats for FP4 inference without silently degrading model quality.
• AI accelerator design: Deciding whether to implement the extra exponent bit in scale handling, since the change touches scale fusion and activation quantization logic.
• Model compression pipelines: Avoiding global per-tensor scaling, whose dynamic absmax computation or pre-calibrated estimates add cost and error.
• Hardware format standardization: Informing discussions around microscaling specifications such as those from the Open Compute Project and vendor formats like MXFP4 and NVFP4.

Industry relevance is direct: microscaling formats are already supported in commercial AI accelerators, and the memory math is unforgiving (a 4-bit-element format with 16-bit scales at block size N costs 1/2 + 2/N bytes per element, with every halving of block size increasing storage by a factor of 4/(N+4)); multiplication complexity also grows as M²·K for scale mantissa width M and partial-sum width K, which is why 8-bit scales became the de-facto standard.

Future Directions

• Extending the theoretical framework to sub-4-bit element formats, sub-8-bit scales, and even smaller block sizes, which the authors explicitly identify as a use case.
• Determining when per-tensor or per-channel scaling is preferable to UE5M3, given the outlier sensitivity and activation calibration problems described.
• Further hardware validation of the UE5M3 overhead estimates beyond the systolic array design modeled here.
• Investigating why models differ in their susceptibility to inversion (for example, why llama-2-7b shows none down to block size 8) and whether distribution-shaping or calibration during training can prevent it.

Target Audience

Researchers and engineers working on low-precision numerics, quantization for LLMs, and AI accelerator or datapath design. It is most useful to readers already comfortable with floating-point formats, block quantization, and perplexity-based evaluation; the statistical derivations in the appendices make it dense for beginners, though the experimental results and the UE5M3 proposal are accessible on their own.

Authors’ abstract

Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.