Backpropagation, Worked From Output to Input

A network may have millions of parameters but only one scalar batch loss, and training needs the derivative of that loss with respect to every parameter.

The procedure arrived with its name already attached. Nature published it in 1986, in a letter from Rumelhart, Hinton and Williams whose abstract opens by naming the thing: “We describe a new learning procedure, back-propagation, for networks of neurone-like units.” Writing up the 2024 physics prize, the Nobel Committee for Physics called that letter “a key advance” of the 1980s.

Computing each derivative with a separate forward perturbation would cost roughly one function evaluation per parameter. Reverse mode costs a small constant number of them, whatever the parameter count is. Griewank pinned that constant down in 2012: OPS{∇f(x)} ≤ ω·OPS{f(x)} for the scalar case, and “if one considers only polynomial operations and counts the number of multiplications, the complexity ratio is exactly ω = 3”. A 2018 survey in the Journal of Machine Learning Research puts the same constant at “c < 6 and typically c ∼ [2, 3]”. Griewank draws the conclusion: “In other words, as Wolfe observed, gradients can ‘always’ be computed at a small multiple of the cost of computing the underlying function, irrespective of n the number of independent variables, which may be huge.” That last clause is the whole reason training at this scale is possible. The price of the gradient does not grow with n. A million parameters and a thousand cost about the same per step.

The direction is a documented engineering decision, not a matter of taste. JAX’s autodiff cookbook states that jacfwd “uses forward-mode automatic differentiation, which is more efficient for ‘tall’ Jacobian matrices (more outputs than inputs), while jacrev uses reverse-mode, which is more efficient for ‘wide’ Jacobian matrices (more inputs than outputs)”. PyTorch says the same from the other side. Its torch.autograd.functional.jacobian takes a strategy argument. That argument defaults to ‘reverse-mode’. The note reads: “If func has more outputs than inputs, ‘forward-mode’ tends to be more performant. Otherwise, prefer to use ‘reverse-mode’.” Training is the extreme wide case. There are millions of inputs and one scalar out. That is why the default is the default.

Backpropagation separates value computation from sensitivity computation.

The retain step is the one that costs money, and PyTorch documents the default without euphemism: “Activation checkpointing is a technique that trades compute for memory. By default, tensors computed during the forward pass are kept alive until they are used in gradient computations in the backward pass.” That is the price of reverse mode. Every intermediate the backward rules will need stays resident from the moment it is produced until the moment it is consumed.

The trade can be run in the other direction, and the size of it has been measured. Four authors showed in 2016 that an n-layer network can be trained in O(√n) memory, at the cost of one extra forward pass per mini-batch. On a 1,000-layer residual network on ImageNet that took the memory from 48 GB to 7 GB. Pushed further, O(log n) memory is achievable for O(n log n) extra forward computation. So the five steps below are not equally negotiable. Seeding, traversal and accumulation are fixed by the algorithm. “Retain context” is a dial, and an engineer sets it anywhere between 48 GB and 7 GB by choosing how much of the forward pass to recompute instead of remember.

For Y=XW+b the incoming gradient has the shape of Y, so matrix products produce the gradients that match X and W, while a reduction across the batch produces the bias gradient.

Shape reasoning is a powerful check: every gradient must match the value with respect to which it is taken.

Matching shapes is necessary and nowhere near sufficient. Both frameworks ship a numerical check for this exact moment. PyTorch’s torch.autograd.gradcheck compares an analytical backward rule against finite differences. Its defaults are eps=1e-06, atol=1e-05 and rtol=0.001. It warns that “The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.” JAX’s jax.test_util.check_grads does the same for forward and reverse mode, up to a chosen derivative order. It states that “Gradients are only checked in a single randomly chosen direction, which ensures that the finite difference calculation does not become prohibitively expensive even for large input/output spaces.” Run one of them in double precision on a two-layer toy first. Do that before trusting a hand-written backward rule on anything larger.

An elementwise activation contributes one local derivative per coordinate, and the size of that derivative decides whether anything upstream learns.

Sigmoid units can kill a network from the top down. Glorot and Bengio watched it happen in 2010, in a network of depth five: “the top hidden layer quickly saturates at 0 (slowing down all learning)”. Their model never escaped that regime for the rest of training. The local derivative there is not merely small. It stays small, so the upstream gradient is multiplied down to nothing every step.

Someone then put a stopwatch on the alternative. A 2012 paper states the finding directly: “In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0, x).” Its four-layer convolutional network with ReLUs reached 25% training error on CIFAR-10 six times faster than the same network with tanh units. Same data, same architecture, same backward algorithm. The only difference was the local derivative each coordinate contributed.

Differentiation direction determines which derivatives are computed efficiently, and the distinction is older than the neural networks it is now taught with.

Reverse mode was in print long before anyone pointed it at a neural network. The Nobel Committee for Physics wrote in 2024 that the authors of the 1986 letter “reinvented a scheme for this, which had previously been applied to related problems by others”, and pointed back to Werbos in 1982 and to Linnainmaa’s 1970 master’s thesis. A 2018 survey in the Journal of Machine Learning Research reaches the same attribution independently, in its section on the origins of AD and backpropagation: “Prior to Werbos, the work by Linnainmaa (1970, 1976) is often cited as the first published description of the reverse mode.”

So the two columns below are not a deep-learning convention with a 1986 birthday. They are a numerical-analysis distinction with a 1970 publication date, which the 1986 Nature letter applied to networks of neurone-like units.

Validate gradients from the smallest unit outward — and do it in double precision, because the alternative has been measured.

ReLU has no derivative at 0, and PyTorch says so. Its autograd notes list relu at 0 among the functions that are not differentiable, and state that where a function is convex (at least locally) autograd uses “a subgradient of minimum norm”. For ReLU that convention fixes ReLU'(0)=0. It is a library choice, not a theorem. Four researchers tested in 2021 what the choice does, and their abstract reports: “We observe considerable variations of backpropagation outputs which occur around half of the time in 32 bits precision. The effect disappears with double precision, while it is systematic at 16 bits.”

Read that against the check you are about to run. Levels 1 to 3 compare an analytical derivative with a finite difference. At 32-bit precision you would be making that comparison inside a regime where a documented convention already moves the answer about half the time, and at 16 bits it moves it every time. Double precision is where the two numbers are allowed to disagree only because your rule is wrong.

One caution about that same paper, because it is a good example of how to read one. Its headline ImageNet accuracy result was withdrawn by its own authors. Their 2023 erratum traces it to a bug in the backward implementation of their modified ReLU, and reports that after the fix “the revised experiments showed no substantial effects”. The precision findings above are unaffected, being “completely similar to [1, Figure 2], up to non-significant variations”. Cite the part that survived the correction, not the part that did not.