Initialization and Signal Propagation

Initialization decides what the optimizer is handed: the distribution of activations, and the local derivatives that come with them. The systems that train at scale treat this as a decision about depth, not a default.

GPT-2 states the rule in one sentence: “We scale the weights of residual layers at initialization by a factor of 1/√N where N is the number of residual layers.” OpenAI published that in 2019, calling it a modified initialization “which accounts for the accumulation on the residual path with model depth”.

NVIDIA's Megatron-LM discloses its own recipe for the same accumulation. It does not arrive at the same constant. “We start by initializing our weights W with a simple normal distribution W ∼ N(0, 0.02). We then scale weights immediately before residual layers by 1/√2N…” There N counts the transformer layers of self-attention and MLP blocks.

Two organisations, the same instinct — divide by the square root of the depth. No agreement on whether the divisor is √N or √(2N). Depth is in the formula either way.

Exact formulas vary by distribution, mode, and framework convention. The two libraries most people use publish their conventions plainly enough to set side by side.

Xavier, or Glorot, balances variance using fan-in and fan-out assumptions, and pairs conventionally with tanh or linear-like activations. Glorot and Bengio proposed it in 2010, in a paper whose abstract closes “we propose a new initialization scheme that brings substantially faster convergence”. PyTorch ships it as torch.nn.init.xavier_uniform_, with bound a = gain × sqrt(6/(fan_in + fan_out)). Keras publishes the identical recipe as two classes: GlorotUniform at limit = sqrt(6 / (fan_in + fan_out)), and GlorotNormal at stddev = sqrt(2 / (fan_in + fan_out)). Goal: stable forward and backward scale. Caution: the assumptions are approximate.

He, or Kaiming, accounts for the variance a rectifying activation discards, and pairs with the ReLU family. Keras states it without an intermediate gain term: “It draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / fan_in) where fan_in is the number of input units in the weight tensor.” PyTorch reaches the same place through a published lookup table instead. torch.nn.init.calculate_gain gives 1 for Linear/Identity, 1 for Conv1D, Conv2D and Conv3D, 1 for Sigmoid, 5/3 for Tanh, √2 for ReLU, √(2/(1+negative_slope²)) for Leaky ReLU and 3/4 for SELU.

That is the whole content of the warning that gain depends on the nonlinearity. It is not a vague caution but a table. Using tanh's 5/3 where ReLU's √2 belongs is a scale error of a fixed, computable size, repeated at every layer. Note also that the two libraries disagree about which fan to divide by — sqrt(2 / fan_in) for He, sqrt(6 / (fan_in + fan_out)) for Glorot. So "the same" initialization can mean two different distributions, depending on which mode a framework selects.

Architecture-specific scaling adjusts residual branches, gates, heads, or depth explicitly, and is where the very deep recipes live: a zero or small final branch scale, or a divisor in the depth itself, as in the 1/√N and 1/√(2N) rules above. Goal: stable composition. Caution: it is recipe-dependent, and the recipes do not agree with one another.

Before the downbeat the players sit at distinct, moderate volumes, which is what leaves the conductor an ensemble to shape. Identical or deafening starts leave little that anyone can usefully correct.

An orchestra recovers from a bad first bar. A deep stack multiplies its opening levels layer after layer. Diversity and usable dynamic range at step zero decide what the optimizer is able to correct later. This is why the same two initialization recipes are interchangeable in a 14-layer network — 33.90/13.44 top-1/top-5 error against 33.82/13.34 — and decide whether a 30-layer one learns at all. The recipe did not become more powerful with depth. The multiplication did.

A single synthetic or real batch can reveal early signal failure. Every check below runs before any long training job.

Run a forward pass, recording per-layer mean, variance, saturation, sparsity, and output scale. Compute the loss, confirming that the initial value matches target cardinality and reduction expectations. Run backward, recording gradient norms, missing gradients, and layerwise attenuation or growth. Simulate one update, measuring update-to-parameter ratios and checking numerical validity.

Then repeat across seeds, to separate a robust regime from one fortunate initialization. Here the size of the effect has been measured rather than assumed. A 2021 MLSys study took five deep-learning case studies — CIFAR10 with VGG11, PascalVOC segmentation with an FCN on a ResNet18 backbone, GLUE SST-2 and RTE with BERT, and peptide-MHC I binding with a shallow MLP. It then worked through the sources of variance one at a time: “we randomized the seeds 200 times, while keeping all other sources fixed to initial values”. Its verdict places the initialization seed exactly: “In contrast, model initialization generally is less than 50% of the variance of bootstrap, on par with the visit order of stochastic gradient descent.”

Read both halves of that sentence. Bounded — under half the variance you would get by resampling the data. But on par with the data-visit order, which nobody would report a single draw of either.

A loss that is invalid, flat, or extremely sensitive from step one sends you to the workflow below.

1. Verify shapes and reductions, to rule out broadcast errors and unexpected loss scale. 2. Plot activation distributions, looking for saturation, dead outputs, exploding variance, and constant channels. 3. Trace gradients by depth, checking attenuation, growth, and missing paths after a single backward pass. 4. Compare initialization recipes, changing only the relevant module or scaling rule. Start by reading what the framework was already doing: PyTorch's nn.Linear obtains its default by calling init.kaiming_uniform_(self.weight, a=math.sqrt(5)), a line its own source annotates “Setting a=sqrt(5) in kaiming_uniform is the same as initializing with uniform(-1/sqrt(in_features), 1/sqrt(in_features))”. 5. Retest several seeds, requiring the healthy regime to persist beyond one random draw.

Step 4 is the one most often skipped in the wrong direction. The optimizer gets swapped, the learning rate gets halved, and nobody has yet established which distribution the weights were drawn from.

Normalization can widen the stable initialization range, while residual connections shorten gradient paths. Neither makes arbitrary scale harmless. At the limit, neither turns out to be strictly required either.

An ICML paper from 2018 removed both. It trained vanilla convolutional networks ten thousand layers deep, with no residual connections and no batch normalization, using only a delta-orthogonal initialization derived from a mean field theory of signal propagation and the condition of dynamical isometry. The abstract says it plainly: “In this work, we demonstrate that it is possible to train vanilla CNNs with ten thousand layers or more simply by using an appropriate initialization scheme.”

Take the direction of that result carefully. It is not advice to delete your normalization layers. It shows that the work those layers do at step zero — keeping signal norms from collapsing or exploding through composition — is work the initialization can, in principle, do by itself. That is why the three choices have to be designed together, rather than defended one at a time.

If hidden units begin with identical incoming and outgoing weights, they compute the same value and receive the same gradient. They cannot spontaneously divide responsibilities.

Random initialization breaks that symmetry. Its scale must still preserve useful signal rather than injecting arbitrary large noise. The randomness is there to make units different; the scale is there to keep them informative. Two requirements, set by different considerations.

The most reliable recipe treats architecture, activation, initialization, normalization, and learning rate as a coupled system, and validates their early statistics together.

Most layers, meanwhile, never meet either named recipe at all. The two major libraries do not even agree on the default. PyTorch documents nn.Linear's weights as “are initialized from U(−√k, √k), where k = 1/in_features”, a range the implementation obtains by calling init.kaiming_uniform_(self.weight, a=math.sqrt(5)) — which is neither the Glorot bound nor the plain He one. Keras 3's Dense layer ships kernel_initializer="glorot_uniform", the Glorot recipe that PyTorch's nn.Linear does not use. Same layer, two ecosystems, two distributions, no announcement in either case.

Read the defaults before claiming that a network uses He initialization.