Activation Functions: Where Nonlinearity Enters

Compose two affine transformations and the result is still affine. So a stack of dense layers with no nonlinear response between them can be rewritten as a single dense layer. Ten layers, one layer's worth of function.

That is not a rule of thumb. The exact condition was proved, and it was proved as an equivalence. Leshno and colleagues settled it in 1993. Their abstract says that “a standard multilayer feedforward network with a locally bounded piecewise continuous activation function can approximate any continuous function to any degree of accuracy if and only if the network's activation function is not a polynomial”. If and only if. The theorem does not recommend nonlinearity; it draws the boundary. An affine activation is a polynomial of degree one, and it falls on the wrong side. Kidger and Lyons restate the same classical result in 2020: “Then N^ρ_n is dense in C(K) if and only if ρ is nonpolynomial.”

Nonlinearity lets intermediate layers carve, gate, and recombine regions of the input space. Depth then changes the family of functions available to training.

In 2014 four researchers counted what that change is worth. A rectifier network cuts its input space into regions, and on each region the whole network is affine. A stack of linear maps with nothing between them has exactly one such region. Now take a rectifier network with n₀ inputs and L hidden layers of width n ≥ n₀. Their bound gives it at least (n/n₀) raised to the power (L−1)n₀, times n to the power n₀. So “the number of linear regions of deep models grows exponentially in L and polynomially in n, which is much faster than that of shallow models with nL hidden units”. Same units, same arithmetic per layer. A different family of functions.

Each activation changes both forward values and backward sensitivities.

The last of the five is usually described rather than defined. It has a definition. Hendrycks and Gimpel introduced the GELU in 2016 as xΦ(x) — the input multiplied by the standard Gaussian CDF evaluated at that input. PyTorch's torch.nn.GELU documents that exact form, and next to it the tanh approximation 0.5*x*(1+Tanh(sqrt(2/π)*(x+0.044715*x³))). The smooth gate has a closed form you can read in the API reference. It is not a metaphor.

Their own margins are worth reading before treating it as an upgrade. On a 9-layer CIFAR-10 CNN, median error was 7.89% for GELU against 8.16% for ReLU and 8.41% for ELU. On a 40-4 wide ResNet for CIFAR-100, 20.74% against 21.77% and 22.98%. Fractions of a percentage point, not a new regime. The claim the two authors make for the whole family is broader and more modest at once: “In all, the activation choice has remained a necessary architecture decision for neural networks lest the network be a deep linear classifier.”

No hidden activation wins every problem. Depth, normalization, initialization and hardware all affect the choice.

The ReLU column's advantage was measured before it was assumed. The measurement sits in a figure caption from 2012: “A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line).” Same architecture, same target error, one sixth of the run. Stanford's CS231n notes reproduce the number independently: “It was found to greatly accelerate (e.g. a factor of 6 in Krizhevsky et al.) the convergence of stochastic gradient descent compared to the sigmoid/tanh functions.” The team behind that caption entered the ImageNet competition the same year, and the ILSVRC-2012 results table records what happened. Their entry, SuperVision, took 0.15315 top-5 error. The best submission that was not theirs, ISI, took 0.26172.

The dead negative region in the same column is equally concrete. A 2020 paper states it without hedging: “In this paper, we rigorously prove that a deep ReLU network will eventually die in probability as the depth goes to infinite.” Its authors bound the probability that a network is born dead under any symmetric initialization, then turn the bound into a design rule — width N = log2(L/δ) holds that probability below δ at depth L. CS231n gives the version you meet on a real training run: “you may find that as much as 40% of your network can be 'dead' (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high”. Neither the speed nor the failure is a matter of taste. Both have numbers attached.

A small table reveals how response range changes the signal.

The one entry usually left as a guess is the negative-side slope. It does not have to be guessed. It can be learned, and then read off. He and colleagues did that with PReLU in 2015, and their paper reports what the network chose: “First, the first conv layer (conv1) has coefficients (0.681 and 0.596) significantly greater than 0.” Both coefficients are close to a linear unit. The first layer kept most of its negative side rather than discarding it. Deeper layers went the other way, learning much smaller coefficients — 0.008 to 0.21 in the conv3 block, 0.031 to 0.075 in the fully connected layers. The negative half of the input matters most while the signal is still raw, and less once the features are abstract.

The accounting is small, and worth stating honestly. Swapping every ReLU for a PReLU in that 14-layer ablation model cut ImageNet top-1 error from 33.82% to 32.64%, a gain of 1.2%. The full multi-model system, PReLU together with the matching initialization, reached 4.94% top-5 test error. By the authors' account that was the first result to pass the 5.1% human error rate Karpathy measured by hand-labelling ILSVRC images.

Hidden activations shape internal computation. Output transformations encode the task interface: one logit, several independent logits, or a normalized categorical distribution.

Applying sigmoid or softmax too early can duplicate work that a loss implementation handles more stably, and framework documentation usually specifies whether the loss expects logits.

Both major frameworks say so outright, in the API reference rather than a tutorial. PyTorch documents torch.nn.CrossEntropyLoss as a criterion whose “input is expected to contain the unnormalized logits for each class (which do not need to be positive or sum to 1, in general)”. TensorFlow puts a warning on tf.nn.softmax_cross_entropy_with_logits: “This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results.” Two libraries, one contract. A softmax added in front of either for tidiness does not make the loss safer. It makes it wrong.

Choose deliberately, then confirm the choice with diagnostics.

The last step — change the activation only with evidence — is worth pricing before you spend a week of compute on it. The published evidence is thin by design. Shazeer ran the controlled comparison in 2020, matching every feed-forward variant for parameters and computation. Heldout log-perplexity after 524,288 steps came out at 1.677 for the ReLU baseline, 1.679 for GELU, 1.683 for Swish, 1.636 for SwiGLU and 1.633 for GEGLU. The entire spread is a few hundredths. The paper declines to account for it: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.”

Four years later the OLMo paper wrote “Like LLaMA, PaLM, and others we use the SwiGLU activation function (Shazeer, 2020) instead of ReLU”, and the activation row of its model-comparison table still splits — OLMo-7B, LLaMA2-7B, OpenLM-7B and PaLM-8B on SwiGLU, Falcon-7B on GeLU. That is the size of the external evidence on offer: hundredths of log-perplexity, no mechanism, and a field that has not converged. Measurements from your own pre-activations, emitted values and gradients are stronger than any of it.