Normalization and Train–Evaluation Behavior

Most normalization layers standardize selected coordinates and then apply trainable affine parameters. Four steps recur across the whole family. Choose which values share a mean and variance estimate. Subtract the estimated mean. Divide by a standard deviation, with a numerical epsilon in the denominator. Then apply a learnable coordinate-wise gain γ and a learnable coordinate-wise offset β.

Everything that distinguishes one normalization layer from another lives in the first of those four steps, and in where the statistics come from at inference. The arithmetic afterwards is shared. The epsilon is not decoration either. It is a documented constant, and it has a different default in every specification a model passes through. The deployment checklist at the end of this lesson gives the three numbers.

The same formula can behave differently because its averaging axes change.

Batch normalization estimates the mean and variance across examples — and usually across spatial positions too — for each channel. During training, every example's output therefore depends on the other examples it happened to travel with. At evaluation the layer falls back on stored running statistics. One rule at training, a different rule at inference, in the same layer, selected by a mode flag.

Layer normalization estimates them across features within a single example or token. The property its comparison card asserts is one its authors stated themselves, in 2016: “Unlike batch normalization, layer normalization performs exactly the same computation at training and test times.” PyTorch encodes the same property in the implementation rather than in prose. Its torch.nn.LayerNorm documentation says the layer “uses statistics computed from input data in both training and evaluation modes”, computes them over the last D dimensions of normalized_shape, and defaults eps to 1e-5. There are no running buffers at all. Nothing can go stale because nothing is stored.

Group normalization sits between the two: channel groups within each example, no cross-example dependence, a group count that has to be chosen, and channel structure retained differently from a per-token statistic. Wu and He built it for exactly the regime where the batch dimension stops being trustworthy. That regime is the subject of the next two sections.

Batch statistics can be noisy or systematically different from deployment.

The penalty has been measured twice, by different people, on different benchmarks. Wu and He, presenting group normalization in 2018, state that “BN’s error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation”, and they put a number on it: “on ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN”.

The second measurement rules out the obvious alternative explanation. Ioffe isolated the same failure in 2017, on an Inception-style ImageNet model. A baseline that normalized over minibatches of 32 reached 78.3% top-1 validation accuracy after 130k steps. He then broke the same gradient batch into micro-batches of 4, for normalization purposes only. He is explicit about what was and was not changed: “In other words, the gradient was still aggregated over 1600 examples per step, but the normalization involved groups of 4 examples rather than 32 as in the baseline.” The gradient signal was identical. Accuracy fell to 74.2%, and reaching even that took 210k steps rather than 130k. All that changed was which examples shared a mean.

Non-i.i.d. batches cost more. Sampling batches of 16 random labels with 2 images each dropped the same model to 67% test accuracy, against the baseline’s 78.3%. One diagnostic detail is worth carrying. Evaluating that same model in training mode, with matching batch composition, recovered 76.5%. The weights had learned something usable. What was broken was the agreement between the statistics used at training and the statistics used at inference. Batch size and batch composition belong to the definition of the model, not only to its schedule.

Dataset preprocessing uses statistics estimated from training data to transform input features consistently. Layer normalization recomputes statistics inside the network, for each current representation.

The two operations solve different problems and can coexist. Internal normalization does not excuse leakage in external preprocessing: a normalization layer inside the model has no way of knowing that the scaler fitted outside it saw the validation set. The distinction is visible in what each one stores. A fitted preprocessing transform holds numbers estimated once from a particular dataset, and those numbers are exactly what leakage contaminates. torch.nn.LayerNorm holds no running buffers at all and recomputes from the tensor in front of it. One is a fitted artifact with provenance to audit. The other is a function of the current activation.

Normalization changes activation scale, parameterization, effective step sizes, and the geometry seen by optimization. Its benefits should not be reduced to one slogan.

The original account named a cause. Batch normalization arrived in 2015, and Ioffe and Szegedy called the problem internal covariate shift. They reported that “applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin”. They also gave a hard benchmark number: “Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters.” That figure did not have to be taken on the authors' word. The ResNet paper tabulated it independently, listing “BN-inception [16] 4.82” in its table of ImageNet ensemble test-set error rates.

Then the explanation was displaced. Twice, in one year, in one set of proceedings. Santurkar and colleagues at MIT wrote at NeurIPS 2018: “we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother.” In the same proceedings, Bjorck and colleagues at Cornell, working empirically and independently, landed somewhere else again: “We conduct several experiments, and show that BN primarily enables training with larger learning rates, which is the cause for faster convergence and better generalization.” Two groups, one year, one technique, two different non-covariate-shift accounts. The technique kept working while its explanation was replaced, and then replaced by two candidates at once. That is the ordinary course of an empirical field, not a scandal.

That normalization changes the geometry seen by optimization is not only an interpretation. In one case it is a proof with a practical consequence. Xiong and colleagues showed at ICML 2020 why the original Transformer needs a learning-rate warm-up stage: “Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large.” In the Pre-LN arrangement the gradients are well behaved at initialization and the warm-up can be dropped. Nguyen and Salazar had already reported the same practical consequence in 2019, where pre-norm residual connections enabled warmup-free training. Moving one normalization layer removes an entire stage of the training schedule.

So document what a layer does: which axes, which mode, which stored state. Why someone said it helps is the weaker record. It has already been rewritten once.

Validate normalization under the exact inference regime. Record the normalized axes, confirm the mode transitions on fixed inputs, inspect the stored state, test the batch regimes, and recheck after domain shift.

Two of those five steps have published numbers behind them. Step 3, inspecting stored state and precision, has three specifications to check against, and they do not agree. PyTorch's torch.nn.BatchNorm2d defaults to eps 1e-5 and momentum 0.1, and states plainly what it keeps: “Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.” Keras's BatchNormalization defaults to epsilon 0.001 and momentum 0.99. The ONNX BatchNormalization operator specifies epsilon 1e-05, momentum 0.9, and a training_mode attribute defaulting to 0, with the running update written out as running_mean = input_mean * momentum + current_mean * (1 - momentum). The mode switch is part of the interchange standard itself, not merely a framework convention. So the same exported layer can be reconstructed with a different smoothing constant, and a different epsilon in the denominator, from the one it was trained with. Momentum 0.1 and momentum 0.99 are not even the same quantity read the same way.

Step 5, rechecking after domain shift, has a measured cost. Replacing the batch-normalization statistics estimated on the training set with statistics of the corrupted images improved robustness across 25 different models, and helped even when adapting to a single sample. Schneider and colleagues put a number on it at NeurIPS 2020: “Using the corrected statistics, ResNet-50 reaches 62.2% mCE on ImageNet-C compared to 76.7% without adaptation.” That improvement was bought by re-estimating numbers the model was already carrying. Tent, an ICLR 2021 spotlight from Wang and colleagues, re-estimates normalization statistics at test time by a different route and reports state-of-the-art ImageNet-C error. Deciding “whether statistics remain valid” after deployment is therefore not a formality. It is worth more than most architecture changes.