Capacity, Regularization, Dropout, and Weight Decay

The question of what a large network can fit was settled by deleting the signal. Zhang and colleagues replaced the CIFAR-10 labels with a completely random labelling. Then they trained state-of-the-art convolutional networks with stochastic gradient descent until training error reached zero. Their 2016 abstract states it plainly: “Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise.”

Two things follow. The same architectures that generalize on the real labels will memorize a random one to perfection, so the ability is always present. And switching explicit penalties on or off does not remove the ability. Whether training actually lands on a memorizing solution depends on optimization, initialization, data order, augmentation, and where the run is stopped. It does not depend on whether a regularizer is enabled.

An independent team then showed that memorization is ordered rather than uniform. Arpit and colleagues re-ran the noise-versus-real-data comparison in 2017. Networks learn simple patterns before they memorize, and regularizers act selectively on the two regimes: “for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data”.

Parameter count is therefore an incomplete measure of effective capacity. Architecture, norms, margins, and the training procedure also shape the learned solution.

One label can cover several mechanisms. Weight decay penalizes or directly shrinks parameter magnitude during updates. It interacts with the optimizer's definition, injects no input noise, and is often applied selectively. Dropout randomly masks units or connections during training. That creates stochastic subnetworks, changes the train-mode computation, and requires rescaling or disabling at evaluation. It can also disrupt some normalized architectures. Data augmentation transforms training examples while preserving intended labels. The invariances are encoded through the data itself. Where the domain knowledge is right, coverage expands; where it is wrong, the result is invalid examples.

How much any of them buys is an empirical question, not a property of the name. Srivastava and colleagues measured it for dropout in 2014. Permutation-invariant MNIST error falls from about 1.60% to 1.35%. SVHN convolutional error falls from 3.95% with no dropout, to 3.02% with dropout on the fully connected layers only, to 2.55% with it on the convolutional layers too.

A year earlier, Wan and colleagues had run the same No-Drop/Dropout contrast on their own architectures. MNIST went from 1.62% to 1.28%, CIFAR-10 from 23.5% to 19.7%. SVHN went from 2.26% to 2.25% — one hundredth of a percentage point. Their explanation of the null result is one sentence: “Due to the large training set size both Dropout and DropConnect achieve nearly the same performance as No-Drop.” The mechanism did not change between the two datasets. The amount of training data did.

Validation performance may improve and then deteriorate while training loss continues downward. Stopping at a selected checkpoint limits how far optimization follows training-specific patterns.

The validation set becomes part of model selection, so repeated decisions can overfit it. Preserve a final test set and record the stopping rule.

A heavily re-used held-out set has a measurable price. Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets from scratch in 2019, following the original collection procedures, then re-scored the existing models against the new sets: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.”

The useful part of that paper is what the authors refused to conclude. They attribute the drops to harder images in the newly collected sets rather than to adaptive overfitting of the old ones. The raw accuracy gap cannot make that distinction for you. Worth imitating, before you blame your own selection procedure for a number that moved. Beyer and colleagues took the other route into the same question in 2020, asking “whether the community has started to overfit to the idiosyncrasies of its labeling procedure”.

During training, random masks remove part of the current computation. Sample a mask over activations or connections. Apply the framework's scaling convention to preserve expected magnitude. Compute a noisy prediction through the surviving subnetwork. Backpropagate, so that only the active paths receive gradients for that sample. Then, at evaluation, disable masking and use the matched inference scaling.

The procedure is specific enough to have been granted as a patent. US 9,406,017 B2 was filed on 30 August 2013 and granted by the USPTO on 2 August 2016 to Hinton and three co-inventors, assignee Google. Its abstract defines the train-time half: “For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability.” The JMLR paper describes the test-time half as approximating the ensemble “by simply using a single unthinned network that has smaller weights”.

That last step is where the cost of a mistake becomes measurable. Normalization layers keep their own running statistics of exactly the quantity dropout perturbs. Ioffe and Szegedy reported in 2015 that batch normalization “acts as a regularizer, in some cases eliminating the need for Dropout”. Li and colleagues then quantified what happens when the two are stacked in the wrong order. Dropout at rate 0.5 placed before the BN layers raised DenseNet-BC error on CIFAR-10 from a 4.72% baseline to 8.70%, and on CIFAR-100 from 22.58% to 31.45%. Nothing about the data or the objective changed. As they put it: “The inconsistency of that variance (we name this scheme as "variance shift") causes the unstable numerical behavior in inference that leads to more erroneous predictions finally, when applying Dropout before BN.”

More regularization is not a universal response to weak validation. The problem is often upstream of the model. Northcutt and colleagues estimate at least 3.3% label errors on average across ten heavily used test sets, and at least 6% of the ImageNet validation set.

The consequence is not a uniform shaving of everyone's accuracy. It is a reordering. On corrected ImageNet labels, ResNet-18 overtakes ResNet-50 once the prevalence of originally mislabelled test examples rises by just 6%. On CIFAR-10, VGG-11 overtakes VGG-19 at 5%. “Surprisingly, we find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data.”

Beyer and colleagues re-annotated the ImageNet validation set independently in 2020 and found the reported accuracy gains “substantially smaller” under the corrected labels. A tuning run that reads a benchmark like that as ground truth is being told, by the labels rather than by the model, that its bigger network is better.

Separate the hypotheses instead of changing several controls at once. Establish a data baseline with frozen splits, preprocessing and evaluation. Vary model capacity under similar training budgets. Add one regularizer at a time, changing only decay, dropout, augmentation or stopping. Inspect train–validation gaps through curves, slices and calibration rather than a single endpoint. Then recheck that latency, robustness and subgroup behavior remain acceptable.

The capacity step is the one that punishes a two-point comparison, because the curve it samples is not monotonic. Belkin and colleagues documented a risk curve in which test error rises to a peak at the interpolation threshold and then falls again as capacity keeps growing. It appeared in PNAS in 2019: “This "double-descent" curve subsumes the textbook U-shaped bias–variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance.”

Nakkiran and colleagues reproduced the effect in modern deep networks in 2019. They found it not only across model size but also as a function of training epochs and of training-set size: “as we increase model size, performance first gets worse and then gets better”. Two widths sampled on either side of that peak can report the opposite of the true ordering. A longer run or a larger training set can move where the peak sits. So sweep capacity at several points, and hold epochs and data fixed while you do.