Computational Graphs and the Gradient Signal

This graph shows why a parameter receives only the signal connected to it. Cut one edge and the rest of the program still runs. It still reports a falling loss, and it still trains nothing that mattered.

SimSiam is the measured case. Chen and He held the architecture and every hyper-parameter fixed and removed one stop-gradient. ImageNet linear-evaluation top-1 accuracy moved from 67.7±0.1% to 0.1%, chance level, while the training loss fell to −1, its minimum possible value. Nothing in the optimizer changed. Only the graph did.

A debugger should preserve these distinctions, because the three columns have different causes and different repairs. The rightmost one is misread most often, and two published measurements show what it actually looks like.

Start with precision. A histogram of activation gradients, collected across all layers while the Multibox SSD detector trained in FP32, found 67% of the values already at zero. Much of what remained sat in the band [2^-27, 2^-24), below anything half precision can represent. Micikevicius and his co-authors, at ICLR 2018, state the consequence in one sentence: “Note that much of the FP16 representable range was left unused, while many values were below the minimum representable range and became zeros.” The same network diverged in half precision with no loss scaling. Multiplying the loss by 8 before backpropagation gave 77.1% mAP against a 76.9% FP32 baseline. PyTorch's Automatic Mixed Precision documentation states the mechanism in one clause: such values “flush to zero ("underflow"), so the update for the corresponding parameters will be lost”. A tensor of exact zeros in that regime is not a saturated derivative. It is a number that did not fit.

Depth compounds the same loss without any help from precision. He and his co-authors trained a 30-layer model — 27 conv layers and 3 fc layers. At a fixed initialization std of 0.01 they computed that the gradient std propagated from conv10 back to conv2 is 1/(1.7 × 10^4) of the value their derivation prescribes. What that costs is in their ICCV 2015 paper: “Our initialization is able to make the extremely deep model converge. On the contrary, the “Xavier” method completely stalls the learning, and the gradients are diminishing as monitored in the experiments.” Eight layers of attenuation is the distance between a model that trains and one that never starts. On a norm plot it appears only as the third column of this comparison: small, finite, easy to dismiss.

Training usually differentiates one reduced objective with respect to millions of parameters. Reverse-mode automatic differentiation reuses intermediate dependencies efficiently for that direction. The system computes derivatives of the executed program. A conditional branch, detached tensor, in-place mutation, or train/eval mode can therefore change the graph and its gradients.

The efficiency has a number on it. A survey by Baydin and co-authors, in the Journal of Machine Learning Research, states that for f : Rn → Rm “the time it takes to calculate the m×n Jacobian by the forward mode is n c ops(f), whereas the same computation can be done via reverse mode in m c ops(f), where c is a constant guaranteed to be c < 6 and typically c ∼ [2, 3]”. A single scalar loss makes m equal to one. So the gradient of a billion parameters costs a few forward passes, and nothing in that bound depends on n. The cheapness is the reason nobody inspects the graph until it breaks.

These failures often survive shape checks and forward unit tests. Four of the five below have been isolated in published experiments where the graph was the only thing that changed.

Use small deterministic cases that show clearly which parameter answers for which output. Step 5 is the one that gets skipped, and Batch Normalization is the reason not to skip it. Training normalizes with mini-batch statistics, inference with population statistics. The two modes are two functions, not one function in different clothes. Ioffe and Szegedy reported that the technique matched the prior state of the art with 14 times fewer training steps, and that an ensemble reached 4.82% top-5 test error. That much performance riding on a mode flag means a diagnostic pass taken in the wrong mode is not a noisy measurement of your model. It is a clean measurement of a different one.

A defective component can be traced backward through suppliers and assembly stages. A shared supplier can contribute to several products, while an unused supplier receives no responsibility.

An audit like that ends in fault. Backpropagation ends in a number for how much a small change in one parameter would have moved the loss. That is neither blame nor proof of cause. What it does explain is branching, accumulation, and disconnected paths — the tied embedding that answers for two uses at once, and the detached encoder that answers for none.

A single global gradient norm can hide dead layers and one dominating tensor. Track distributions by module, parameter type, and task contribution.

Two per-layer statistics can disagree about the same network. Glorot and Bengio monitored activations and gradients layer by layer after standard initialization. The variance of the back-propagated gradients shrank as it moved toward the input. The variance of the weight gradients stayed roughly constant across layers. They wrote: “What was initially really surprising is that even when the back-propagated gradients become smaller (standard initialization), the variance of the weights gradients is roughly constant across layers, as shown on Figure 8.” The caption of that figure puts it flatly: “Even though with standard initialization the back-propagated gradients get smaller, the weight gradients do not!” Two honest diagnostics, one network, opposite verdicts. A single averaged number would have reported neither.

Useful diagnostics include gradient-to-parameter norm ratios, update-to-parameter ratios, activation statistics, and the fraction of missing or non-finite values. Read them per module. The ratio that identifies a stalled model in that 30-layer network is the one taken between two named conv layers, not the one taken over the whole net.

One default catches nearly everyone once. PyTorch's backward accumulates gradients in the leaves rather than replacing them. Its documentation for torch.Tensor.backward warns that you “might need to zero .grad attributes or set them to None before calling it”. A forgotten zero_grad turns the training loop into a running sum. The curve then looks like a learning-rate problem, and it is not one. That is the pattern this whole lesson repeats: an execution fact about the graph, wearing the costume of an optimizer failure.