Vanishing, Exploding, Saturated, and Dead Gradients

“The loss is not improving” can hide very different mechanisms. The main ones leave different traces. A vanishing signature shows earlier layers receiving extremely small gradients: slow feature learning, common in long saturated paths, easy to mistake for frozen parameters. Confirm it by checking depth-wise norm decay. An exploding signature shows gradient magnitude growing to unstable values: large parameter jumps, loss spikes or divergence, common in recurrent or deep products. It is often the step before NaN values appear. A blocked or dead signature shows a local derivative that is exactly zero across many examples — dead ReLU regions, detached graph paths, hard clipping or masks. Nothing upstream receives a learning signal at all.

The exploding signature has been recorded at scale, in a run with the standard guardrail switched on the whole way through. Google trained PaLM, a 540B-parameter model, using “global norm gradient clipping with a value of 1.0 for all models”. The paper then reports what happened anyway: “For the largest model, we observed spikes in the loss roughly 20 times during training, despite the fact that gradient clipping was enabled.” The remedy was operational rather than numerical. The team restarted “from a checkpoint roughly 100 steps before the spike started” and skipped “roughly 200–500 data batches”. An ablation then showed that those same batches did not spike from a different checkpoint. The cause was the combination of data and parameter state, not bad data.

Meta looked at divergences across its own 7B, 30B, 65B and 546B-parameter runs and pointed somewhere else entirely: at the optimizer. Adam enters a state where the update vector is large and essentially uncorrelated with the direction of descent. Roughly twenty spikes under a threshold of 1.0 is the reason a signature is not a diagnosis. The same visible symptom had a data-and-state explanation in one case and an optimizer-state one in the other.

Backpropagation multiplies local Jacobian effects along sequential paths. Repeated contraction can erase sensitivity; repeated expansion can make it unstable. The size of that erasure is measurable, and it has been measured.

Fix the initialization standard deviation at 0.01 and follow the gradient back through a ten-layer network. Eight layers later it arrives at 1/(1.7 × 10⁴) of the size the derivation says it should have. Kaiming He and three colleagues ran that calculation in 2015, tracing conv10 back to conv2 of the VGG team's “model B”. One constant, applied once per layer, costs that much over eight layers. Nothing in the forward code is invalid and nothing in the loss looks unusual. The defect lives only in the profile of the gradient across depth.

Deep enough, the same error stops being a slowdown and becomes a wall. At 30 weight layers — 27 conv and 3 fc — the earlier Glorot/Xavier scheme stops learning entirely while the rectifier-aware initialization converges. He and his co-authors put it plainly: “Our initialization is able to make the extremely deep model converge. On the contrary, the “Xavier” method completely stalls the learning, and the gradients are diminishing as monitored in the experiments.” Both schemes are one line of code today. PyTorch ships xavier_uniform_ with bound a = gain × √(6/(fan_in+fan_out)) and kaiming_uniform_ with bound gain × √(3/fan_mode), each citing the paper it came from. At that depth, the difference between those two lines is the difference between a network that learns and one that does not.

Branches, normalization, gating, and residual connections alter the path geometry. The simple product intuition identifies risk without fully describing modern networks.

Saturating activations have small derivatives at extreme inputs, yet their outputs may still vary slightly. A dead ReLU stays on the negative side and emits zero, with the usual zero local derivative. Both can starve upstream learning. The difference between them is not cosmetic, and it has been written down precisely.

The mechanism is one line of the batch normalization paper of 2015. For the sigmoid g, Ioffe and Szegedy write: “As |x| increases, g′(x) tends to zero. This means that for all dimensions of x=Wu+b except those with small absolute values, the gradient flowing down to u will vanish and the model will train slowly.” Train slowly — not stop. The derivative tends toward zero without arriving there.

What that residue leaves room for had already been measured in 2010. Glorot and Bengio found the logistic sigmoid “unsuited for deep networks with random initialization because of its mean value, which can drive especially the top hidden layer into saturation”. Then they reported the escape: “Surprisingly, we find that saturated units can move out of saturation by themselves, albeit slowly, and explaining the plateaus sometimes seen when training neural networks.”

That is the distinction the remedies turn on, and it is a distinction about recoverability. A plateau that eventually breaks is the signature of saturation: a small non-zero derivative still moves the unit, slowly, under its own power. A ReLU sitting on the negative side has an exactly zero derivative and no such route back. Nothing in its own gradient can move it. The repair has to come from outside it — the scale that pushed it there, a bias shift, or an update size that overshot. Waiting works for the first case and never works for the second.

The first non-finite tensor is more informative than the final corrupted loss.

In half precision the boundaries are published numbers, so the search has somewhere to start. FP16's maximum normalized value is 65,504. Its minimum denormal is 2⁻²⁴, roughly 5.96e-8. Below that, values become zero. Above 65,504 they become an infinity, which turns into a NaN at the first subtraction or zero multiply. Both edges come from NVIDIA's mixed-precision training guide.

PyTorch's AMP documentation describes the same trap from the training side. Gradient values with small magnitudes “will flush to zero (‘underflow’), so the update for the corresponding parameters will be lost”. That is what gradient scaling exists to prevent. NVIDIA's guide adds the measured version. Shift the gradient histogram by 15 exponents — a scale factor of 32K — and it “would recover all but 0.1% of values lost to 0”. Dynamic scaling backs that factor off again whenever an overflow appears.

The procedure follows from those edges. Assert finite inputs, activations, losses, gradients and parameters. Reproduce one step on a fixed batch with deterministic behavior. Binary-search the modules to locate the earliest operation producing a non-finite value. Inspect scale and domain at that operation — divisions, logarithms, exponentials, norms, mixed precision. Then repair the source, before adding protective clipping on top of it.

One scalar loss cannot reveal how learning signals travel. A contraction of 1/(1.7 × 10⁴) is a shape across depth, not a value at any single layer. A gradient that has shattered into white noise still reports a healthy norm. Both are invisible to the number the training log prints.

A fix should correspond to the observed failure rather than to the most familiar technique. Five families do different jobs. Scale control: initialization, normalization, learning rate, input standardization. Path design: residual connections, gating, shorter dependency routes, architectural changes. Gradient control: clipping, loss scaling, stable numerical formulations. Activation choice: non-saturating or leaky responses when evidence supports the change. Graph repair: reconnecting detached tensors, and removing unintended zero-gradient operations.

Path design has the cleanest measured before-and-after, and He and colleagues printed the failure as a training curve rather than a generalization gap. The caption to their first figure reads: “Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks. The deeper network has higher training error, and thus test error.” A 56-layer plain network was worse than a 20-layer one on the very data it was fitting. No amount of regularization explains that. Once the path changed, the same reformulation trained networks up to 152 layers, and an ensemble of six reached “3.57% error on the ImageNet test set” — a figure the challenge organisers record independently, listing MSRA's winning ILSVRC 2015 entry at a classification error of 0.03567. Srivastava and colleagues had shown the same year that gated skip paths let “hundreds of layers” train “directly through simple gradient descent”.

But a remedy can work for a reason its authors got wrong, and the field's most-used normalization fix is the example. Batch normalization arrived in 2015 with numbers attached: it “achieves the same accuracy with 14 times fewer training steps”, and an ensemble using it reached “4.82% top-5 test error” on ImageNet. Ioffe and Szegedy attributed the gain to reducing “internal covariate shift”. The technique and its explanation were adopted together. Three years later Santurkar and three colleagues measured the explanation and rejected it: “In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother.” The accuracy numbers never stopped being true. The mechanism attached to them did. Choose a remedy from evidence about your own failure, not from the story that travels with the remedy.