Regularization, Early Stopping, and Capacity Control

Many parameter settings can achieve similar training loss. Regularization gives the learning procedure a reason to prefer some of them: smaller coefficients, smoother functions, fewer branches, stable representations, or behavior unchanged by valid transformations.

The aim is not to make the model weak. It is to reduce reliance on detail that lacks enough evidence to be trusted.

That the training fits really are interchangeable has been measured. A 1,649,402-parameter Inception was trained on CIFAR-10 in several configurations, and reached 100.0% training accuracy in every one of them. What moved was the test accuracy: 89.05% with random crops and weight decay, 85.75% with neither. On ImageNet the same architecture went from 77.84% top-1 and 93.92% top-5 with augmentation, dropout and weight decay to 59.80% and 80.38% with all three switched off. The regularizers were not stopping the network from fitting the training set. It fitted the training set perfectly either way. They were selecting which perfect fit it settled on. The five authors who ran the experiment, writing in 2016, state the consequence flatly: “Explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error.” The same network also reached 100.0% training accuracy on CIFAR-10 after the labels were randomised. On test it then scored 9.78%. A perfect fit with nothing in it to transfer.

Dropout is the clearest demonstration that this is a preference rather than a punishment. Randomly deleting units during training, reported in 2014, took the best convolutional network on Street View House Numbers from 3.95% error to 3.02%, with dropout on the fully connected layers. Applied to every layer, it reached 2.55%. On CIFAR-10 the same progression ran 14.98% to 14.32% to 12.61%. On CIFAR-100 it ran 43.48% to 37.20%. The networks were not made smaller. They were made unable to depend on any particular unit being present.

The word regularization covers several mechanisms with distinct effects. Each acts at a different point in the pipeline, and each of the five below is demonstrated later in this lesson by a measured case rather than by assertion.

The names matter less at this stage than the behavior they encourage.

Two of the claims in this comparison are not folklore. The exact-zero behaviour of the L1 penalty is design intent, and it is stated by the person who introduced the method. Robert Tibshirani, presenting the lasso in 1996: “Because of the nature of this constraint it tends to produce some coefficients that are exactly 0 and hence gives interpretable models.” Driving coefficients to exactly zero while keeping ridge-like stability was the point of the constraint, not a lucky side effect.

The caution that a zero coefficient does not prove irrelevance is likewise a theorem rather than an opinion. Martin J. Wainwright analysed the same estimator in 2006 and found a sharp threshold. Recovering the true sparsity pattern holds above it and fails below it. The threshold relates the number of observations to the problem dimension and the number of non-zero coefficients. Below it the zeros report the size of the sample, not the irrelevance of the features.

A flexible model may first learn broad patterns and later specialize to training detail.

Stopping time is a regularizer in the formal sense, not a convenience. Two independent groups established that. The 2014 one opens its abstract with the definition: “Early stopping is a form of regularization based on choosing when to stop running an iterative algorithm.” Its data-dependent stopping rule uses no hold-out or cross-validation data at all, attains minimax-optimal rates for Sobolev and other kernel classes, and is tightly connected to the solution path of kernel ridge regression — the stopping index doing the work a ridge penalty would do. A 2007 paper had reached the same equivalence from the other side. It derived the stopping rule by solving an explicit bias–variance trade-off, and obtained the minimax optimal rate O(m^-1/2) in some cases.

The stages below are the usual ordering, not a law. A 2022 experiment trained a network on division modulo 97 with 50% of the table held out. Training accuracy became close to perfect in under 10^3 optimization steps. Validation accuracy needed close to 10^6 steps to reach the same level, with very little evidence of any generalization before 10^5. The abstract states it: “In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting.” A validation-plateau rule firing at 10^4 steps would have kept a checkpoint two orders of magnitude before the model learned the task. The phase transition was independently reproduced and reverse-engineered on modular addition at ICLR 2023. A stopping rule is a bet that the validation curve being watched is the ordinary one.

Regularization should be tuned along with capacity and evaluated beyond one aggregate score. Step 4 is not decoration: the section on the cost of control below reports a case in which the aggregate improved while one small subgroup paid for the improvement.

These situations require root-cause work before another penalty setting.

The first and the fifth of them are not house preference. Three regulators — the FDA, Health Canada and the UK's MHRA — jointly issued ten guiding principles for machine learning in medical devices on 27 October 2021. Principle 4 is titled “Training Data Sets Are Independent of Test Sets”, and reads in full: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” Site is named there explicitly, alongside patient and data acquisition. Principle 6 requires that “Model design is suited to the available data and supports the active mitigation of known risks, like overfitting, performance degradation, and security risks”. A regularization sweep run on a split that violates principle 4 is tuning a number that measures nothing. Three regulators have put that in writing.

Stronger regularization usually increases bias while reducing variance. That trade can improve validation performance. It may also erase rare legitimate patterns.

Report which slices lose performance, not only whether the aggregate improves; the “simpler” solution is preferable only when its mistakes are more acceptable for the intended decision.

Who pays has been measured. A 2019 experiment trained a ResNet18 on 29,500 lighter-skin images from Diversity in Faces plus 500 darker-skin images from UTKFace. The control it studied is one of the five levers above: noise added during training, together with clipping. That control cost accuracy far more on the 500-image subgroup than on the majority. The same effect turned up in sentiment analysis of African-American English tweets at epsilon = 3.87 and 8.99, and in iNaturalist species classification at epsilon = 4.67. The authors state the pattern themselves: “Furthermore, accuracy of DP models tends to decrease more on classes that already have lower accuracy in the original, non-DP model, i.e., "the poor become poorer."” A second group confirmed the same disparity at KDD in 2021 and proposed DPSGD-F to remove it. The direction of the damage is predictable. Constraint falls hardest where the evidence was thinnest, which is precisely where an aggregate score is least able to register it.

The trade is not even monotone. Deep double descent was posted in December 2019 and published in the Journal of Statistical Mechanics in 2021. Its six authors worked on CIFAR-10, CIFAR-100 and two machine-translation datasets. As model size grows, they showed, test performance “first gets worse and then gets better”. The same pattern appears “as a function of the number of training epochs”. There are also “certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance”. A sweep of three settings along one axis can read such a curve backwards.