NaNs, Divergence, Saturation, and Silent Instability

Once an infinity enters a squared-gradient accumulator, many later values may become invalid. The error that finally surfaces can be far from the line of code that caused it.

Capture the earliest non-finite activation, loss component, gradient, or parameter. Preserve the triggering batch and full state before recovery changes the evidence.

This is not a counsel of perfection. Both major frameworks ship a first-failure mode. It stops at the operation that produced the invalid value, not at the crash.

The docstring of torch.autograd.detect_anomaly records PyTorch's default: “If check_nan is True, any backward computation that generates "nan" value will raise an error. Default True.” JAX puts the same idea behind a flag. Turning on jax_debug_nans “will cause computations to error-out immediately on production of a NaN”, because “Switching this option on adds a NaN check to every floating point type value produced by XLA.”

Both are slow, and both say so. The PyTorch documentation warns that the mode is for debugging only, since the extra tests slow execution. That is why it is off in an ordinary run. It is not a reason to leave it off in a run that has already failed.

The exact path depends on operation order and optimizer state. The OPT-175B team logged one such chain instead of inferring it, reporting “a correlation between loss divergence, our dynamic loss scalar crashing to 0, and the l2-norm of the activations of the final layer spiking”. Three instruments, one failure. The activation norm, the scaler, and the loss each recorded a different stage of the same event.

A run can degrade long before it crashes, and the two published extremes below are both finite for most of their duration.

Divergence that never leaves the representable range still ends a run. The BLOOM team spent months on one before changing the number format rather than the code at the failing line. A 104-billion-parameter run was, in Stas Bekman's 2022 account of the training, “a complete failure”, with an ever-diverging lm-loss. His conclusion is one sentence long: “Training huge LLM models in FP16 is a no-no.”

The replacement works because of range, not precision — “The key to BF16 format is that it has the same exponent as FP32 and thus doesn't suffer from overflow”. Google Cloud's TPU documentation gives the same fact from outside the project: “The dynamic range of bfloat16 and float32 are equivalent. However, bfloat16 uses half of the memory space.” The GLM-130B team recorded the choice independently: “In BLOOM-176B, the BF16 format is used instead of FP16, due to its wide range of values on NVIDIA Ampere GPUs (i.e., A100).”

Silent degradation is the other end, and it can be counted. The same mixed-precision paper sampled the activation gradients of the Multibox SSD network. Its histogram caption reads: “2% of the values are in the [2-34, 2-32) range, 2% of values are in the [2-24, 2-23) range, and 67% of values are zero”. Two thirds of the sampled signal was already gone, with no NaN anywhere.

Accumulation loses more of it. IBM researchers training networks in 8-bit floating point described what happens when a small addend meets a large one: “it is possible that this smaller number may be truncated entirely after addition due to limited mantissa bits”. They called it swamping, and it limited reduced-precision training until chunk-based accumulation and stochastic rounding were introduced. A flat gradient histogram, not an exception, is the evidence here.

Stable implementations use transformed formulas and guarded domains.

Half precision has a hard ceiling. It is 65,504. Micikevicius and his co-authors set the loss-scaling factor against that ceiling, advising “choosing a factor so that its product with the maximum absolute gradient value is below 65,504 (the maximum value representable in FP16)”. The Multibox SSD detector “failed to train in FP16 without loss-scaling”; a “loss-scaling factor of 8 recovers the relevant gradient values and mixed-precision training matches FP32 mAP”. Overflow instead fills the weight gradients with infinities and NaNs. Those, the paper warns, “irreversibly damage the weights after an update”.

The distance to that ceiling shrinks with scale, and two teams measured the approach. In the ViT-22B work an 8-billion-parameter model's loss began rising within 2,000 steps, and the paper says why: “Without normalization, attention logits quickly grow to over 50000 in magnitude, resulting in one-hot attention weights after the softmax, and subsequently unstable training losses and gradients.” Over 50,000, against a representable maximum of 65,504. The GLM-130B team met the same pressure in a different architecture: “Second, the attention scores grow so large that they exceed FP16's range, as the model scales up.”

Minimize the run while preserving the trigger. Steps 1 and 4 are shipped features with defined semantics, not manual instrumentation you have to build.

The docstring of torch.autograd.detect_anomaly states what enabling it buys: “Running the forward pass with detection enabled will allow the backward pass to print the traceback of the forward operation that created the failing backward function.” That is the jump from the last symptom to the first cause, performed by the framework. JAX does the isolation differently but to the same end. Under @jax.jit, jax_debug_nans re-runs the function “in de-optimized op-by-op mode”, so the check lands on the individual operation that produced the NaN rather than on the fused computation that reported it.