What Matters for NVFP4 Training? A Comparative Study of Low-Precision Pretraining Recipes

AI summary

Overview

• Research area: Low-precision training of large language models, specifically pretraining in the NVIDIA NVFP4 4-bit floating-point format on Blackwell GPUs.
• Technical level: Advanced — the paper assumes familiarity with quantized training, microscaling block formats, matrix-multiply (GEMM) structure in Transformers, stochastic rounding, and Random Hadamard Transforms.
• Scope (one sentence): A unified, controlled comparison of competing NVFP4 pretraining recipes on dense 8B and 30B-A3B mixture-of-experts models trained for up to 1T tokens, isolating which techniques actually recover accuracy relative to BF16.

What This Paper Is About

Training directly in 4-bit floating point promises large throughput and energy gains, but quantization noise can quietly degrade model quality, and prior research groups disagree about which recipe ingredients are essential. Many proposed techniques have only been validated on small models over short token horizons, so it is unclear which trends survive at scale. This paper asks a practical question — which ingredients are non-negotiable and which are merely useful — by re-implementing the major existing recipes in one training stack and ablating them on medium-scale models run for up to 1T tokens.

Key Contributions

1. First detailed, unified ablation of NVFP4 training techniques. The authors instantiate the NVIDIA '25 NVFP4 recipe, a Quartet-II-style backward pass (one-dimensional weight quantization with backward re-quantization, Random Hadamard Transforms on both backward GEMMs, and MS-EDEN), and Four Over Six scale selection in a single framework, and measure how each addition changes the loss gap to BF16.
2. Evidence that selective high-precision layers are required at scale. Keeping the last 15% of layers in BF16 is shown to be necessary for accuracy recovery, even with an otherwise optimized recipe; without it, training diverges past 300B tokens on the Nemotron-H 8B model.
3. A finding that backward-pass de-biasing is not uniformly valuable. MS-EDEN applied to all backward tensors gives no clear benefit over applying it only to gradients; applying MS-EDEN to gradients and Four Over Six to the remaining backward tensors (X.T, W.T) yields the lowest observed loss gap among tested configurations when all layers are quantized.
4. "Mosaic NVFP4," a combined recipe. It stacks one-dimensional weight quantization with backward re-quantization, dimension-128 tiled Random Hadamard Transforms on both backward GEMMs with resampled sign vectors, MS-EDEN on gradients, Four Over Six on all four forward and backward weight/input tensors, and selective high-precision layers. The authors describe it as, to their knowledge, the strongest FP4 training result demonstrated at this scale in loss gap to BF16, and note it remains compatible with efficient kernels.

Main Findings

• Every stacked technique helps, and the total gain is 0.36 pp. Starting from the NVIDIA '25 NVFP4 recipe and stacking four cumulative updates on Nemotron 3 Nano (trained to 400B tokens in this ablation), each modification improves training loss, giving a 0.36 percentage-point total improvement once fully stacked. The largest single gain comes from replacing stochastic rounding with MS-EDEN and adding Four Over Six on the backward tensors, which the authors attribute largely to MS-EDEN achieving unbiasedness at lower quantization-error cost than stochastic rounding.
• High-precision layers are not optional. On Nemotron-H 8B, the Mosaic NVFP4 recipe diverges past 300B tokens when all layers are quantized. The authors interpret this as a representation failure that the error-reduction techniques studied cannot address, even though a close variant of the recipe converged on small models over short horizons without high-precision layers.
• Backward-pass techniques are hard to separate in a stable regime. With the last 15% of layers kept in BF16, the different backward-quantization schemes show no clear separation on Nemotron 3 Nano. With all layers quantized, separating them is possible but the gap is small, suggesting bias correction matters most for the gradient tensors (G and G.T).
• Final loss gaps: under 1.2% for dense 8B, under 0.81% for the MoE. On the dense Nemotron-H 8B model trained to 1T tokens, Mosaic NVFP4 finishes with less than a 1.2% relative training-loss gap to BF16 and improves on the prior gap by 0.24 percentage points. On the 30B-A3B Nemotron 3 Nano, it finishes with less than a 0.81% relative training-loss gap and improves by 0.36 percentage points, with sustained improvement starting after the first 75B tokens.
• Recipe rankings do not stabilize until well past 150B tokens. Weaker recipes often look better early in training, which the authors attribute to randomly sampled sign vectors in the Hadamard transforms, training noise, or per-recipe convergence behavior. Short ablations can detect catastrophic instability but are not sufficient to decide whether a method preserves accuracy at scale.
• Improvements are larger than run-to-run noise. Across three repeated 1T-token runs of the NVIDIA '25 NVFP4 recipe, variability is approximately 0.06% in training loss and 0.04% in validation loss for Nemotron 3 Nano, and 0.027% and 0.064% respectively for Nemotron-H 8B.
• Extra stochasticity gives a further 0.028 pp. Resampling the stochastic-rounding pattern for corrected scale factors rather than reusing a fixed pattern, and independently randomizing the DGRAD transform sign vector per data-parallel replica rather than once per global batch, yielded an additional 0.028 pp improvement in loss gap on the 30B-A3B model under a different optimizer configuration. The reported results do not include these extra sources of stochasticity, so the authors call their numbers likely conservative.
• Downstream results are mixed. On Nemotron 3 Nano, MMLU (5-shot) moves from 67.5 to 68.4 (reported difference +1.3% against a 0.9% noise reference) and MMLU-Pro (5-shot CoT) from 45.0 to 47.1 (+4.7% against a 6.3% noise reference), while Math Average goes from 79.4 to 79.2 and Code Average from 57.1 to 56.6. On Nemotron-H 8B, MMLU moves from 61.0 to 60.3, MMLU-Pro from 29.7 to 27.6 (+/–7.1% against a 9.8% noise reference), Math Average from 51.3 to 52.3, and RACE from 77.7 to 78.8 (+1.4% against a 1.1% noise reference). Noise references are the relative range across three repeated runs for individual tasks, and a score-weighted average of those for aggregate rows.

Methodology in Plain English

The team rebuilt all the competing NVFP4 recipes inside one common training stack, on the same GPU architecture, so that comparisons reflect recipe choices rather than implementation differences.

They describe a Transformer's computation in terms of three matrix multiplications and the tensors each one quantizes: the forward product of inputs and weights (X and W), the gradient computation that contracts the output gradient with the transposed weight (G and W.T), and the weight-gradient computation that contracts the transposed gradient with the transposed input (G.T and X.T). A GEMM can use 4-bit tensor cores only when both of its operands are quantized along their last dimension, so each of these six tensors is a separate design decision.

Against that backdrop they vary five things: whether weights are quantized in one or two dimensions; whether backward-pass weights are re-quantized from the dequantized forward weights rather than from the high-precision weights; how large the Random Hadamard Transform tiles are and whether the sign vectors are static or resampled during training; how gradient bias is handled (stochastic rounding versus MS-EDEN, which combines random rotations with a per-block corrective rescaling); and where Four Over Six adaptive block-scale selection is applied.

They run these variants on two models: the dense 8B Nemotron-H hybrid Mamba-Transformer (24 Mamba-2 layers, 4 attention layers, 24 FFN layers, 8 KV heads), trained on the Nemotron-H data distribution with diverse data for the first 600B tokens and high-quality data for the remainder of 1T tokens; and the 30B-A3B Nemotron 3 Nano MoE (3.2B active out of 31.6B total parameters, 23 Mamba-2, 6 attention and 23 MoE layers, 128 routed experts with top-6 routing and 2 shared experts), trained on the first 1T tokens of the Phase 1 mixture from a 25T-token curriculum. Both use AdamW with a Warmup-Stable-Decay schedule and sequence length 8192; annealing covers the final 20% of the horizon for Nemotron 3 Nano and the final 15% for Nemotron-H 8B. Batch sizes are 3072 and 768, learning rates 1e-3 falling to 1e-5 and 8e-4 falling to 8e-6, and weight decay is 0.1 for both.

Why This Matters

Impact on research. The paper reframes the FP4 training problem from "which algorithm wins" to "which ingredients are load-bearing," and shows that short-horizon ablations can rank recipes incorrectly. That is a methodological caution for anyone benchmarking low-precision methods, and it points to a specific unsolved failure — representational collapse without high-precision layers — that current error-reduction techniques do not fix.

Real-world applications:

• Cheaper foundation-model pretraining: 4-bit tensor-core matmuls are one of the most direct levers for reducing the cost of the dense matrix multiplications that dominate pretraining.
• Energy-efficient training runs: the abstract frames FP4 training as promising improvements in both throughput and energy efficiency at large scale.
• Practical recipe selection for model builders: the result that MS-EDEN on gradients without additional treatment of X.T and W.T is "likely without significant cost to model quality in the stable regime" offers a cheaper configuration for teams that already keep high-precision layers.
• Production-scale hybrid architectures: validation spans both dense and MoE hybrid Mamba-Transformer designs, which are increasingly common in deployed model families.

Industry relevance. The work comes from NVIDIA researchers together with ISTA and Red Hat AI, targets NVIDIA Blackwell hardware and the Nemotron model family, and explicitly checks compatibility with efficient kernels — making it a directly actionable reference for organizations planning large NVFP4 pretraining runs.

Future Directions

• Scale beyond medium models. The authors state they have not established whether the conclusions hold for larger production runs, other modalities, post-training, or other FP4 formats.
• Diagnose the high-precision-layer requirement. Since error-reduction techniques cannot compensate for fully quantized models at scale, the underlying representation failure remains unexplained and is an open target.
• Exploit additional stochasticity. Resampling rounding patterns for corrected scale factors and randomizing the DGRAD sign vector per data-parallel replica gave a further 0.028 pp on the 30B-A3B model under a different optimizer; the authors leave studying these extra sources of randomness to future work.
• Extend error reduction to the remaining backward tensors. Because de-biasing helps gradients more than X.T and W.T, the paper suggests there is room for complementary error-reduction techniques on those remaining tensors.

Target Audience

Machine learning systems researchers and engineers working on low-precision or quantized training; teams planning large-scale pretraining on NVIDIA Blackwell hardware; practitioners choosing among published FP4 recipes (NVIDIA '25 NVFP4, TetraJet/TetraJet-v2, Four Over Six, Quartet/Quartet II, MS-EDEN); and methodologists interested in why short-horizon low-precision ablations fail to predict long-run accuracy.

Authors’ abstract

Training large language models directly in 4-bit floating-point (FP) formats promises substantial improvements in throughput and energy efficiency. While some recipe design choices have been validated at scale, many promising approaches remain untested beyond small models and short token horizons, leaving open the question of which trends will hold. We present a systematic comparison of recent NVFP4 recipes at medium model scale, evaluating dense 8B and 30B-A3B MoE models trained for up to 1T tokens, with a focus on which ingredients are necessary for accuracy recovery. We propose a final recipe grounded in the principles behind established stable NVFP4 training at scale, incorporating state-of-the-art techniques such as unbiased gradient estimation with lower quantization error than stochastic rounding. To the best of our knowledge, this is the strongest FP4 training result demonstrated at this scale to date in loss gap to BF16. Through ablation studies, we find that: (i) each technique in the optimized recipe measurably improves loss trajectory, (ii) selective high-precision layers are necessary for recovering accuracy at scale, (iii) not all tensors in the backward pass benefit equally from de-biasing, leaving room to apply complementary error-reduction techniques to the remaining tensors.