Gradients: Local Sensitivity in Parameter Space

The parameter vector is a location on a high-dimensional surface whose height is the loss.

Everything this lesson measures — the slope along one coordinate, the gradient vector assembled from all of them, the descent direction — is read at that single point. The rest of the surface is never consulted.

A partial derivative asks a narrow question. If this one parameter moves a little and the others hold still, how does the loss change? The answer is a local approximation around the current values.

Parameters interact, so separate partial derivatives do not tell the whole story over a large step. Curvature decides how fast the local approximation goes wrong, and curvature is not spread evenly over the surface. Exploding gradients are walls of high curvature. Pascanu and colleagues made that case in 2013, on the reasoning that “when the first derivative explodes, so does the second derivative”. Their fix became the standard one: whenever ‖ĝ‖ ≥ threshold, rescale the gradient to threshold·ĝ/‖ĝ‖. Seven years later four researchers at MIT stopped arguing the point and measured it. They plotted the local gradient-Lipschitz constant against the gradient norm along the training trajectories of AWD-LSTM on Penn Treebank and ResNet20 on CIFAR-10. Their abstract reports what came out: “Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient.” The first-order forecast is worst exactly where the arrow is longest. That is why clipping exists at all.

What that curvature usually looks like has also been measured. The obstacle in high-dimensional non-convex optimization is “the proliferation of saddle points, not local minima”. Dauphin and colleagues argued that in 2014, from random matrix theory and from their own experiments. They then searched directly for critical points of a trained network with a Newton method. The search confirmed the theoretical prediction about a critical point’s index. The index is the fraction of negative eigenvalues of the Hessian there, and it “tightly and positively correlates with its error level”. A place where every partial derivative reads zero is, at high error and high dimension, almost never a floor.

A second group reached the same floor from the opposite direction, with no Newton search anywhere in it. The loss of a multilayer ReLU network can be mapped onto the Hamiltonian of a spherical spin glass, which is what Choromanska and colleagues did in 2015. The physics comes with an inheritance: “Results from random matrix theory applied to spherical spin glasses have shown that these functions have a combinatorially large number of saddle points.” They concluded that the low critical values form a band above the global minimum, and that the number of local minima outside that band diminishes exponentially with network size. The minima found inside it are “highly degenerate, with many eigenvalues of the Hessian near zero”. Two different kinds of argument, one answer. A vanishing partial derivative says even less than it appears to.

For a small proposed change d, the dot product between gradient and d approximates the loss change, so aligned motion raises loss while opposite motion lowers it locally.

A direction perpendicular to the gradient has zero first-order change. Second-order curvature may still matter.

A zero derivative can also mean the function has gone flat. It need not mean that nothing depends on the coordinate. The smallest demonstration fits on one line, and Sundararajan and colleagues open their 2017 paper on integrated gradients with it: “a one variable, one ReLU network, f(x) = 1 − ReLU(1−x)”. Take the baseline at x = 0 and the input at x = 2: “the function changes from 0 to 1, but because f becomes flat at x = 1, the gradient method gives attribution of 0 to x”. The output has moved the whole distance. The derivative at the input is exactly zero. Their example is about an input coordinate, but the mechanism is the one that also silences a parameter derivative wherever the composed function has saturated. The dot product is telling the truth about an infinitesimal step, and nothing about a finite one.

Large and small gradients need context before diagnosis.

The small-magnitude case has been photographed. A four-hidden-layer sigmoid network, measured layer by layer by Glorot and Bengio in 2010, did this at the top of the stack: “The top hidden layer quickly saturates at 0 (slowing down all learning), but then slowly desaturates around epoch 100.” For roughly a hundred epochs that layer’s derivatives were near zero. The network was not finished. It was stuck. Their depth-five model “never escaped this regime during training”. Ioffe and Szegedy, at Google, named the mechanism in one line: for a unit driven into saturation, “the gradient flowing down to u will vanish and the model will train slowly”. Convergence and saturation produce the same reading on the same instrument. Only the loss curve and the activation statistics separate them.

The large-magnitude column is the other half of the same lesson. A long gradient is not a licence to take a long step. That is precisely where the MIT group found the local smoothness constant growing with the norm. The clipping threshold is an admission that the first-order picture has already broken down.

Finite differences provide a debugging reference. Use them on small models and selected parameters.

The last step below says to try several values of epsilon, and there is a reason no single value is correct. Two errors run in opposite directions. Truncation error falls as the step h shrinks, and the other one rises. Baydin and colleagues set out the squeeze in a 2018 survey in the Journal of Machine Learning Research, in a section called “What AD Is Not”: “However, as h is decreased, round-off error increases and becomes dominant (Figure 3).” Switching from a one-sided to a central difference does not escape the squeeze. It only “moves the truncation error from first-order to second-order in h”. The same survey prices the method. A Jacobian of f : R^n → R^m by central differences “requires 2mn evaluations”, against reverse-mode automatic differentiation’s m·c·ops(f) with “c < 6 and typically c ~ [2, 3]”. This is a debugging tool, not a training method.

PyTorch 2.14 ships the trade-off as defaults. torch.autograd.gradcheck uses eps=1e-06, atol=1e-05, rtol=0.001, and the documentation says why: “The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.” Run the check in float64, on a handful of coordinates, and read the relative error rather than the absolute difference.

Gradient values are shaped by parameterization as well as behavior.

Two of the traps below have been measured by people who went looking for them. In both cases the number is larger than the intuition that the detail is a technicality.