Neural Networks as Differentiable Programs

Ordinary software stores instructions chosen directly by a programmer. A neural network stores a computational structure plus many adjustable numbers called parameters.

Training does not invent arbitrary code. It searches parameter values that make the fixed computation behave better on examples. An objective supplies the measurement.

That the structure comes first is a theorem rather than a slogan, and the theorem has a number in it. Four units decide the question. Lu and four co-authors stated it in 2017: “We show a universal approximation theorem for width-bounded ReLU networks: width-(n + 4) ReLU networks, where n is the input dimension, are universal approximators. Moreover, except for a measure zero set, all functions cannot be approximated by width-n ReLU networks, which exhibits a phase transition.” Above the line, the parameters can be set to reach anything. Below it, except on a set of measure zero, no parameter setting reaches the target at all. No quantity of data or training changes that.

The width-bounded case was proved independently in 2020 by Kidger and Lyons. Take any nonaffine continuous activation with a continuous nonzero derivative at some point — ReLU included. Networks of arbitrary depth and width n+m+2 are then dense in C(K;R^m) for compact K ⊆ R^n, with n the number of input neurons and m the number of output neurons. Two groups, two proofs, the same shape of conclusion. A choice made before the first example arrives fixes which functions are reachable.

The word “network” hides several distinct objects. Keeping them separate prevents most beginner confusion.

The winning entry in the 2012 ImageNet challenge puts numbers on all four at once. Krizhevsky and two co-authors described it in a single sentence: “The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.” Read that sentence as a parts list. The eight layers, the max-pooling and the 1000-way softmax are the architecture, chosen by hand and fixed before training began. The 60 million numbers are the parameters, and they are what training changed. The 650,000 unit outputs are activations, recomputed for every image and stored for none of them. The softmax and the loss attached to it are where the objective enters.

The objective's verdict on that parameter setting is on the public record. The challenge organisers list team SuperVision at a five-guess error of 0.15315 and the runner-up ISI submission at 0.26172 — 15.3% against 26.2%, in the same table, on the same test set.

Architecture alone does not determine behavior. Data, initialization, objective, optimization, and stopping choices all shape the learned function.

The clearest documented case of one architecture serving many behaviours is BERT. Devlin and three co-authors reported it in 2019: “As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.” Eleven natural language processing tasks, one pre-trained parameter set, no substantial change to the shape of the computation. The scores were GLUE 80.5, MultiNLI 86.7%, SQuAD v1.1 test F1 93.2, SQuAD v2.0 test F1 83.1. They are not only the authors' own. Stanford's independently maintained SQuAD 2.0 leaderboard carries the matching row: “Nov 08, 2018 | BERT (single model) | Google AI Language | 80.005 | 83.061”.

The converse experiment is sharper still, because it holds the architecture fixed and corrupts only the data. Zhang and four co-authors reported that “Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data.” The same networks still fit when the real images are replaced by unstructured random noise. Arpit and ten co-authors examined the phenomenon independently in 2017, and concluded that “training data itself plays an important role in determining the degree of memorization”. Same graph of operations, same optimizer, same loss. What the labels were decided whether the result was a classifier or a lookup table.

A differentiable operation has a usable local sensitivity. A small input change implies a describable output change. Chain those sensitivities together across every step of the computation, and training can estimate how each parameter affected the final loss.

Neural networks also contain piecewise or nondifferentiable points. At those points the frameworks do not compute a derivative — they apply a convention. ReLU at zero is the standard case, and a 2021 paper by Bertoin and three co-authors opens by naming both the freedom and the choice: “A priori, any value in [0, 1] bears a variational sense as it corresponds to a subgradient [27]. Yet in most libraries s = 0 is chosen; it is the case for TensorFlow [2], PyTorch [26] or Jax [10].” One of the three states the convention in its own source code. The comment in TensorFlow's ReluGrad functor reads “NOTE: When the activation is exactly zero, we do not propagate the associated gradient value. This allows the output of the Relu to be used, as well as its input.”

How much that convention costs is itself a lesson in checking a number before repeating it. The published paper reported backward passes whose outputs differ “around half of the time in 32 bits precision”, and an ImageNet test-accuracy gain of “more than 10 points” for ReLU'(0)=0 over ReLU'(0)=1. The authors withdrew both figures themselves. Their erratum, in 2023, followed a bug that three other researchers found in the backward implementation of their modified ReLU.

With the bug fixed, the bifurcation zone survives: 43% of parameters at 32-bit precision, 100% at 16-bit, 0% at 64-bit. But the L2 magnitude of the difference falls by a factor of about 10^5. No effect on training loss or test accuracy appears for reasonable values of ReLU'(0) between 0 and 1. An effect appears only for unrealistically large values, around 1000 and above. The ImageNet ResNet50 experiment showed no qualitative change and is not reported. The convention is real and it is visible in the arithmetic. The dramatic reading of it did not survive someone else running the code.

How cheap is that credit assignment? A 2018 survey by Baydin and three co-authors puts a figure on it. Take a function of many inputs and one output. For that case, “only one application of the reverse mode is sufficient to compute the full gradient”, “compared with the n passes of the forward mode needed for populating the same”. The survey also bounds the overhead. Automatic differentiation “guarantees that the amount of arithmetic goes up by no more than a small constant factor”. That asymmetry is why gradient-trained networks are practical at all. Some of them carry hundreds of millions of parameters. The reason is not any resemblance to biology.

Engineers switch views depending on the question. No single view is sufficient for design, debugging, and explanation.

The representation view is the oldest of the three, and it arrived on a date. Back-propagation was published in Nature in the issue of 9 October 1986. The procedure repeatedly adjusts connection weights to minimise the difference between actual and desired output, and the abstract states the representation claim outright: “As a result of the weight adjustments, internal ‘hidden’ units which are not part of the input or output come to represent important features of the task domain, and the regularities in the task are captured by the interactions of these units.”

Notice what the founding sentence claims and what it does not. It claims that hidden values come to carry features of the task. That is the representation view, in its original wording. It does not claim that any one of them carries a single human-readable meaning, and the object it names is a unit. The Royal Swedish Academy of Sciences' 2024 Physics scientific background records the same paper as the pivotal step: “A key advance was the demonstration by David Rumelhart, Hinton and Ronald Williams in 1986 of how architectures with one or more hidden layers could be trained for classification using an algorithm known as backpropagation”.

Use this trace whenever you meet an unfamiliar neural model. Every step asks for something a reader can go and check, in the way that eight layers, 60 million parameters and a five-guess error of 0.15315 can be checked.