Initialization and the Scale of Signals

Initialization sets the starting function. No example has changed a parameter yet. Poor scale can make activations vanish, saturate, or explode across depth.

A first-batch diagnostic should inspect activation means, standard deviations, zero fractions, and gradient norms. Waiting for several failed epochs wastes evidence available immediately.

Every failure in this lesson was decided before the first update. A 30-layer rectifier network that never starts learning. A detector that diverges on its first attempt. A 1.82-point spread in final accuracy produced by nothing but the random seed. None of them is a bug in the training loop. Each is a choice about the scale or the distribution of the initial parameters.

Initialization rules choose weight scale from the number of incoming and sometimes outgoing connections. The rule now shipped in every framework was derived by measuring this chain as it broke.

The standard draw at the time was W ~ U[-1/sqrt(n), 1/sqrt(n)], scaled from the fan-in alone. In 2010 Glorot and Bengio replaced it with what they called the normalized initialization, W ~ U[-sqrt(6)/sqrt(n_j+n_{j+1}), +sqrt(6)/sqrt(n_j+n_{j+1})]. The scale now comes from the fan-in and the fan-out together. Then they measured the per-layer quantity the stages below multiply: the average singular value of each layer's Jacobian at initialization. “With our normalized initialization, this ratio is around 0.8 whereas with the standard initialization, it drops down to 0.5.”

That is the whole of the depth problem in two numbers. A per-layer factor near 0.8 is survivable for a long way down the stack. A factor of 0.5 is a halving repeated once per layer, in both directions.

The names are less important than the assumptions they encode — and each family is a published result with a measurement attached, not a preference.

Xavier / Glorot is the 2010 rule above, and it ships today unchanged. PyTorch's xavier_uniform_ draws from a symmetric interval with a = gain * sqrt(6/(fan_in+fan_out)), and Keras's GlorotUniform uses limit = sqrt(6 / (fan_in + fan_out)). The paper's own table records what the change bought on a 5-hidden-layer tanh network: test error from 27.15% to 15.60% on Shapeset-3x2, and from 55.9% to 52.92% on CIFAR-10.

He / Kaiming is the rectifier-aware rule from the 30-layer experiment above. Its mode is a choice of which direction to protect, and PyTorch states the trade in its own documentation for kaiming_uniform_: “Choosing 'fan_in' preserves the magnitude of the variance of the weights in the forward pass. Choosing 'fan_out' preserves the magnitudes in the backwards pass.”

Orthogonal initialization is not folklore either. Saxe and two co-authors derived it for deep linear networks in 2013, and their abstract states: “We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times.” A 2017 paper then computed the full singular-value spectrum of the input-output Jacobian and found the limit of the idea. ReLU networks cannot achieve dynamical isometry at all, while “sigmoidal networks can achieve isometry, but only with orthogonal weight initialization” — and isometric networks learn “orders of magnitude faster”. PyTorch ships the draw as torch.nn.init.orthogonal_, which will “Fill the input Tensor with a (semi) orthogonal matrix”, citing Saxe (2013). It is a structural constraint with a known scope, not a universal replacement.

Not every trainable tensor should use the same random distribution. For two of these, the published record is a training run that failed until the initial value was changed.

Fixing a random seed makes one initialization reproducible under a controlled software and hardware stack. Reproducing a number is not the same as trusting it. A fixed seed does not show that a result is robust to other initial conditions.

One experiment put a figure on that. A ResNet-9 was trained on CIFAR-10 under the first 10,000 seeds, with nothing else changed. David Picard reported the spread in 2021: “The interesting values of this test are the minimum and maximum accuracy among all run (obtained at the end of training), which is this case go from 89.01% to 90.83%, that is, a 1.82% difference.” Table 1 of the same paper gives 500 longer runs at 90.70 ± 0.20, with a minimum of 90.14 and a maximum of 91.41. A 1.82-point improvement is publishable in many venues. Here it is available for free, from the seed alone.

The seed is not even the largest source. A study of five deep-learning tasks, presented at MLSys 2021 by Bouthillier and 16 co-authors, concluded that “most evaluations focus on the effect of random weight initialization, which actually contribute a small part of the variance”. Their first recommendation is the opposite of fixing it: “randomize as many sources of variations as possible in the performance estimation”. Compare several seeds for unstable or high-stakes experiments, and report the selection rule so favorable runs are not mistaken for typical behavior.

And a fixed seed does not by itself buy back the run. PyTorch opens its own page on reproducibility with a disclaimer: “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.” NVIDIA names the cuDNN kernels that are nondeterministic on identical hardware — cudnnConvolutionBackwardFilter with ALGO_0 or ALGO_3, cudnnConvolutionBackwardData with ALGO_0, cudnnPoolingBackward with CUDNN_POOLING_MAX, cudnnSpatialTfSamplerBackward and cudnnCTCLoss. They use atomic operations that introduce truly random floating-point rounding errors. NVIDIA also states flatly that “Across different architectures, no cuDNN routines guarantee bitwise reproducibility.” Most of the nondeterministic list is the backward pass, which is the half the seed was supposed to pin down.

Test the untrained network before launching a long run.

Step 1 is not a formality. Every rule here depends on fan-in and fan-out, and those are defined against a tensor layout that may not be the one in your head. PyTorch says so in a Note under kaiming_uniform_ in torch.nn.init: “Be aware that fan_in and fan_out are calculated assuming that the weight matrix is used in a transposed manner, (i.e., x @ w.T in Linear layers, where w.shape = [fan_out, fan_in]). This is important for correct initialization.” The frameworks also disagree about what you get if you never call an initializer at all. Keras's Dense defaults to kernel_initializer="glorot_uniform" with bias_initializer="zeros". PyTorch's nn.Linear draws both weight and bias from U(-sqrt(k), sqrt(k)) with k = 1/in_features.

Step 5 is the one the seed section quantifies: 10,000 seeds of one ResNet-9 spanned 89.01% to 90.83% with nothing else changed. A single run tells you about a starting point, not about a design.