Numerical Precision, Mixed Precision, and Loss Scaling

Reduced formats sacrifice bits differently, and each row of this comparison gives up a different one. FP32 keeps a broad exponent range and the longer mantissa; it is the common reference format. FP16 buys established hardware acceleration with a much narrower exponent range, and pays for it in overflow and underflow. bfloat16 keeps an FP32-like exponent range with fewer mantissa bits, which is why it often avoids explicit loss scaling. FP8 offers greater efficiency in return for a tighter recipe, explicit scaling and specific hardware. That last row is the one usually left as a gesture, so it is worth making concrete.

There is a specification behind it, and it defines two formats rather than one. E4M3 has an exponent bias of 7 and a maximum normal value of 448. E5M2 has a bias of 15 and a maximum normal value of 57,344. Micikevicius and fourteen co-authors from NVIDIA, Arm and Intel published it in 2022, and it says which belongs where: “The recommended use of FP8 encodings is E4M3 for weight and activation tensors, and E5M2 for gradient tensors.” Forward tensors get the mantissa. Gradients get the range.

E4M3 buys its extra binade — 18 instead of 17 — by breaking with IEEE 754 on special values. It represents no infinities, and it keeps only one mantissa bit-pattern for NaNs. The hardware line on that card is not decorative either, because the same nominal type is not the same set of numbers on every vendor's silicon. AMD's HIP documentation implements both of the OCP encodings — “E4M3 Format: Sign: 1 bit, Exponent: 4 bits, Mantissa: 3 bits” and “E5M2 Format: Sign: 1 bit, Exponent: 5 bits, Mantissa: 2 bits” — and an incompatible FNUZ (finite-and-NaN-only) variant. That one ships with a warning: “Note: On a gfx94x GPU, the type will default to the fnuz type.” gfx94x is CDNA3 / MI300. "FP8" in a config file is not yet a number format. It is a format plus a vendor.

Autocast policies are built around exactly these differences, and the policy is published rather than folklore. PyTorch lists 51 operations under the heading “CUDA Ops that can autocast to float32” — among them softmax, log_softmax, layer_norm, group_norm, sum, prod, cumsum, cumprod, norm, exp, log and cross_entropy. NVIDIA's mixed-precision guide draws the same line with its AMP lists: an “AllowList: Convolutions, Fully-connected layers” against a “DenyList: Large reductions, Cross entropy loss, L1 Loss, Exponential” and an “InferList: Element-wise operations (add, multiply by a constant)”. The reasoning is written down too: “Furthermore, values computed by large reductions should be left in FP32. Examples of this include statistics (mean and variance) computed by batch-normalization, SoftMax.” None of this is a matter of taste. It is a list you can open.

The five families below are the ones those two lists actually name, so you can check the policy in your own framework instead of trusting a bullet.

Hardware support and kernel behavior still need measurement, and the two reduced formats fail in opposite directions. FP16 carries more mantissa detail than bfloat16 and much less exponent range: established acceleration as the strength, dynamic loss scaling as the usual need, overflow and underflow as the risk, and the skipped-step rate as the thing to watch. bfloat16 keeps the wide exponent range at lower mantissa precision: easier dynamic range, usually no loss scaling, rounding and accumulation as the risk, hardware throughput as the thing to watch. An FP32 reference costs memory and bandwidth and buys a stable comparison. The caveat is that different kernels remain different kernels, so the check is a short matched run rather than an assumption.

That "usually no loss scaling" is not bought by the format alone. Google's Cloud TPU documentation says where the rest of it comes from: “By default, TPUs perform matrix multiplication operations with bfloat16 values and accumulations with IEEE float32 values.” The product is narrow. The running total is not. The same page also deletes a whole region of the number line: “the bfloat16 on Cloud TPU does not support subnormals, so all subnormals are flushed to zero during the conversion”, with round-to-nearest-even rounding and overflow to inf.

Intel specified the identical design for its FMA unit in a 2018 white paper on bfloat16 hardware numerics: “There is no need to support denormals; FP32, and therefore also BF16, offer more than enough range for deep learning training tasks”, “FP32 accumulation after the multiply is essential to achieve sufficient numerical behavior on an application level”, and “The BF16*BF16 multiplication is performed without loss of precision; its result is passed to a general FP32 accumulator”. Two vendors, one arrangement, arrived at independently.

This is also why "rounding and accumulation" is the right risk to write on the bfloat16 card, rather than a hedge. The range problem was moved into an FP32 accumulator and the subnormals were removed outright. What is left to go wrong is what the short mantissa does to values on the way in and on the way out.

Underflow rounds small values toward zero, silently removing information. Overflow produces infinities that can spread into moments, parameters, and future activations.

Reduced mantissa precision also changes how values round and how they add up. A run can stay finite and still drift from an FP32 reference through accumulated numerical differences. NVIDIA's own documentation gives FP16 its bounds in a list of example magnitudes: “Maximum normalized 65,504”, “Minimum normalized 2-14 = ~6.10e-5”, “Minimum denormal 2-24 = ~5.96e-8”, with the loss-scaling section repeating “65,504 (the maximum value representable in FP16)”. Those three numbers are the whole map. Overflow above the first, full precision down to the second, degraded precision in the denormals down to the third, and zero below it.

The bottom of that map is where small gradients go to die. Micikevicius and co-authors put the consequence plainly: “any value whose magnitude is smaller than 2−24 becomes zero in FP16”. They also measured how much of a real gradient distribution lives down there — “approximately 5% of weight gradient values have exponents smaller than −24” in their Mandarin speech model. Not a hypothetical tail: one gradient in twenty, deleted before the optimizer sees it. NVIDIA describes the same behaviour from its side, saying such values “are otherwise lost to 0s”.

Loss scaling does not make FP16 more precise. It slides that distribution up the magnitude axis until it stops falling off the bottom of it.

Dynamic loss scaling may skip a parameter update when overflow is detected. The scheduler, EMA, gradient accumulation counter, and logging must agree about whether a real optimizer step occurred. PyTorch's canonical training-loop comment states the branch: “If these gradients do not contain infs or NaNs, optimizer.step() is then called, otherwise, optimizer.step() is skipped.” Advancing some states during a skipped update creates silent divergence from the intended algorithm. Test this path deliberately.

The constants that decide how often the branch is taken are a policy, not an algorithm, and they differ between the stacks you are likely to use. PyTorch's GradScaler defaults are init_scale=65536.0, growth_factor=2.0, backoff_factor=0.5 and growth_interval=2000: start at 2^16, halve on overflow, double after 2,000 consecutive clean iterations. NVIDIA documents that same policy in words — “N = 2000, increasing scaling factor by 2, decreasing scaling factor by 0.5” — and adds that “as long as one skips updates infrequently the training schedule does not have to be adjusted to reach the same accuracy as FP32 training”.

Microsoft's DeepSpeed fp16 block starts from the same ceiling, initial_scale_power=16, the same 2^16 = 65,536. Then it diverges. Its loss_scale_window is 1000, half PyTorch's growth interval, so it probes upward twice as eagerly. It also exposes hysteresis=2, consecutive_hysteresis=false and min_loss_scale=1, three settings PyTorch does not offer at all. Porting a recipe between the two changes the skip rate even when the starting scale is identical. "The loss scaler" is not a thing you can reason about generically.

How rare is rare enough? PyTorch quantifies it: “Since step skipping occurs rarely (every several hundred iterations) this should not impede convergence.” That is the number to compare your logged skip rate against. A run skipping every few steps is not doing dynamic scaling. It is failing.

A fixed scale can also be enough, and the factor required can be small. Micikevicius and co-authors report a Multibox SSD detector that “diverges when gradients are not scaled, but scaling them by a factor of 8 (increasing the exponents by 3) is sufficient to match the accuracy achieved with FP32 training”. Across the networks NVIDIA trained, constant scaling factors “ranged from 8 to 32K” — three exponents at one end, fifteen at the other.

Frameworks automate much of this sequence, but the semantics still matter. One step runs: (1) autocast the selected matrix and convolution work into a reduced format while keeping sensitive operations safer; (2) scale the loss for FP16 so small backward values stay representable; (3) backpropagate, producing scaled gradients through the recorded graph; (4) unscale and inspect, restoring intended gradient units and detecting infinities or NaNs; (5) clip and update — apply guardrails, write into FP32 master state, and skip invalid steps; (6) adjust the dynamic loss scale from the overflow history.

Step 5 is where the master copy sits, and it is normally asserted without a number. There are two measured reasons for it, both in the mixed-precision paper by Micikevicius and co-authors. The first is the familiar one: an update smaller than 2^-24 becomes zero in FP16 before it can reach the weight at all. The second is worse, because the update is perfectly representable and vanishes anyway — inside the addition. “This can happen when the magnitude of a normalized weight value is at least 2048 times larger that of the weight update.” 2048 is 2^11, one more than FP16's 10 mantissa bits. Line the two operands up to add them, and the smaller one shifts clean off the end of the mantissa. The weight is unchanged, no flag is raised, and every value in the run is finite.

The cost of removing the master copy was measured rather than feared. On their Mandarin speech model, trained on roughly 800 hours of speech for 20 epochs, updating FP16 weights directly instead of an FP32 master copy produced an 80% relative accuracy loss.

The mechanism is not one team's account of its own experiment. Wang and colleagues at IBM described the same right-shift truncation in 2018, and gave it a name: “When we use our FP16 format for accumulations, this truncation happens when the magnitude differs larger than the swamping threshold 2mantissa+1”. Swamping, with a threshold at 2^(mantissa+1) — which for FP16's ten mantissa bits is exactly the 2048 the other paper reports.

Measure wall-clock throughput, peak memory, invalid-value rates, skipped steps, and validation quality against an FP32 or higher-precision reference. Short equivalence tests should use identical seeds and data order where practical.

PyTorch publishes exactly such a comparison, and it is worth reading for its protocol and not only for its result. On Ampere and later devices, “TF32 tensor cores are designed to achieve better performance on matmul and convolutions on torch.float32 tensors by rounding input data to have 10 bits of mantissa, and accumulating results with FP32 precision, maintaining FP32 dynamic range.” NVIDIA's A100 whitepaper specifies the format the same way — “TF32 includes an 8-bit exponent (same as FP32), 10-bit mantissa (same precision as FP16) and 1 sign-bit” — and lists Peak FP32 at 19.5 TFLOPS against Peak TF32 Tensor Core at 156 TFLOPS, with separate Peak FP16 Tensor Core and Peak BF16 Tensor Core rows of 312 TFLOPS each.

The reference run itself is a 10240×10240 matmul on an A100, scored against a float64 computation of the same product. With TF32 the product “takes 0.016s on GA100”, and its maximum absolute error against that reference is 0.1747, a relative error of 0.0022. With TF32 disabled, the same product “takes 0.11s on GA100” for a maximum error of 0.0031, a relative error of 0.000039. Two timings, two errors, one controlled input. That is what "validate speed and behavior together" produces when someone actually does it.

Bitwise identity is rarely expected across kernels. The goal is bounded, understood deviation under a reproducible protocol.

The A100 matmul is what a bound looks like once it is written down instead of assumed. PyTorch's own summary of the two runs is that “the speed is ~7x faster on A100, and that relative error compared to double precision is approximately 2 orders of magnitude larger.” Nothing failed. No NaN, no Inf, no overflow, no skipped step. The relative error simply moved from 0.000039 to 0.0022, and the wall clock from 0.11 s to 0.016 s. A run that reported only finiteness would have called both configurations healthy.

That is the shape of the decision a precision policy actually is. Rounding the mantissa to 10 bits is not a deployment detail bolted onto a fixed algorithm. It changes the function being computed, by an amount that has to be measured against a higher-precision reference and then accepted on purpose. The 156 TFLOPS against 19.5 is the reason anyone would. The two orders of magnitude is the price on the tag.