Regularization as a Portfolio

Methods with the same goal can alter different mechanisms. A small Inception of 1,649,402 parameters reached 100.0% training accuracy on CIFAR-10 in every configuration its authors tested — explicit regularizers on, explicit regularizers off. So the branch you reach for below matters more than how hard you pull it. Two branches can also collide badly enough to cost 8.87 accuracy points, as the next section shows.

These interactions should be tested rather than assumed. The measured cases below are not warnings about hypothetical portfolios: each is a published number produced by adding a second regularizer to a network that was already working.

Add methods one at a time, in an order that keeps you able to say what each one did.

Step 4 is the one most often skipped. A regularizer's coefficient is not its only setting. Its position in the schedule is another.

Timing has been measured. ResNet-18 and All-CNN were trained on CIFAR-10 and CIFAR-100 with SGD, momentum 0.9 and an exponentially decaying learning rate. Golatkar and colleagues then moved weight decay and data augmentation around in time. Switching both off after the first ~100 epochs left final accuracy essentially unchanged. Their first finding reads: “Applying weight decay or data augmentation beyond the initial transient of training does not improve generalization”.

Switching weight decay on late did damage. Discussing their Figure 1: “delaying WD by 50 epochs causes a 40% increase in test error, from 5% regularizing all along, to 7%”. Both runs converge in the same loss landscape. Both use the same coefficient. Only the timing differs.

The 2019 paper is called Time Matters in Regularizing Deep Networks, and its abstract draws the conclusion: “This suggests that what matters for training deep networks is not just whether or how, but when to regularize”. An ablation that only turns methods on and off will not see any of this.

A model can overfit through excessive capacity, noisy labels, shortcut features, narrow data coverage, or checkpoint selection. No single regularizer addresses every cause. And the whole explicit portfolio buys less than its reputation suggests.

There is a number for that. A small Inception of 1,649,402 parameters was trained on CIFAR-10 in four configurations. All four reached 100.0% training accuracy. Their test figures were 89.05%, 89.31%, 86.03% and 85.75%. Turning on both explicit regularizers — random cropping plus weight decay — moved test accuracy from 85.75% to 89.05%. Then the same team relabelled CIFAR-10 at random. The same architecture still reached 100.0% training accuracy on pure noise, at 9.78% test accuracy. Zhang and colleagues drew the conclusion in 2017: “Explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error.”

An independent group at MILA/Montréal re-ran the question on their own CNNs, on CIFAR-10 and on random labels, across dropout, input dropout, Gaussian noise and weight decay. Their finding: “Our findings extend their claim and indicate that explicit regularizations can substantially limit the speed of memorization of noise data without significantly impacting learning on real data”. Explicit regularization slows memorization. It does not prevent it.

So begin with evidence from learning curves and error slices. Add the smallest change that targets the failure you observed, then check what else it moved. And notice how far apart the standard settings are.

Dropout was published in 2014, by Srivastava, Hinton and colleagues. Each unit is retained with a probability p. That p can be chosen on a validation set, or it “can simply be set at 0.5, which seems to be close to optimal for a wide range of networks and tasks”. Not everywhere, though: “for the input units, however, the optimal probability of retention is usually closer to 1 than to 0.5”.

Label smoothing carries a much smaller number. Szegedy and colleagues applied it on ImageNet “with K = 1000 classes”, taking “u(k) = 1/1000 and ε = 0.1”, and reported “a consistent improvement of about 0.2% absolute both for top-1 error and the top-5 error”. Half the units removed in one method. A tenth of the target mass redistributed in the other. Two regularizers, two very different magnitudes, and no reason to expect one coefficient's intuition to transfer to the other.

Their names should not be treated as equivalent strength knobs. One name can even cover two different operations. Adding an L2 term to the loss and decaying the weights directly are the same thing under plain SGD. Under Adam they are not.

Loshchilov and Hutter open their abstract with the distinction: “L2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam.” Then they size it: “we show that Adam generalizes substantially better with decoupled weight decay than with L2 regularization, achieving 15% relative improvement in test error”. The result holds across CIFAR-10 and ImageNet32x32, across training budgets from 100 to 1800 epochs, and across three learning-rate schedules — fixed, drop-step and cosine annealing.

The decoupled form is the one PyTorch ships as torch.optim.AdamW, documented as an optimizer “where weight decay does not accumulate in the momentum nor variance”. That is what the weight-decay row's parameterization risk means in practice. The same nominal coefficient is a different intervention depending on the optimizer it is wired into.

Weight averaging is the cheapest entry on this list. Izmailov and colleagues averaged points along the SGD trajectory, called it Stochastic Weight Averaging, and reported gains on CIFAR-10, CIFAR-100 and ImageNet. Their own summary is blunt: SWA “is extremely easy to implement, improves generalization, and has almost no computational overhead”. Cost is not the reason to leave averaging out of a portfolio. Though, as the interactions section showed, the averaged weights still need their BatchNorm statistics recomputed before they can be scored.

Examine calibration, abstention, rare classes, robustness, representation quality, and downstream transfer where relevant. A method can improve one metric by erasing distinctions another task needs. And that trade has been measured on a single setting of a single coefficient.

Label smoothing cut expected calibration error for Inception-v4 on ImageNet from 0.071 to 0.035 at α = 0.1. For ResNet-56 on CIFAR-100 it cut the same measure from 0.150 to 0.024 at α = 0.05. Then the same smoothing degraded the same network in a different role. Müller and colleagues list it among their 2019 contributions: “We show that label smoothing impairs distillation, i.e., when teacher models are trained with label smoothing, student models perform worse.” The smoothed logits lose the between-class similarity information a student needs. A validation-accuracy column would have shown none of this, in either direction.

Report training cost and sensitivity as well. A fragile portfolio that requires a broad sweep may not be the practical winner.