AdamW, Weight Decay, and Parameter Groups

Two procedures answer to the word regularization. Somebody put them head to head on named architectures at a named learning rate and measured the gap instead of assuming it.

An L2 penalty changes the objective by adding a norm term, so its gradient enters the optimizer transformation. Decoupled decay shrinks parameters separately from the adaptive gradient step.

The boundary was drawn precisely in 2019. The two are “equivalent for standard stochastic gradient descent (when rescaled by the learning rate)”, Loshchilov and Hutter wrote, but not for adaptive gradient algorithms — and the abstract names Adam. Their fix was “a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function”.

The distinction is visible in the printed algorithms, not only in the prose. PyTorch 2.13’s torch.optim.Adam folds the coefficient into the gradient ahead of the moment updates: if λ ≠ 0, g_t ← g_t + λθ_{t−1}. It documents the argument as “weight decay (L2 penalty) (default: 0)”. torch.optim.AdamW, on the same documentation set, applies θ_t ← θ_{t−1} − γλθ_{t−1} instead. The page describes that algorithm as one “where weight decay does not accumulate in the momentum nor variance”, and points the reader to the 2019 paper for the details. The name on the argument is identical in both.

This conceptual sequence separates data-gradient adaptation from parameter shrinkage. The shrinkage line is exactly where implementations diverge.

PyTorch 2.13’s AdamW prints θ_t ← θ_{t−1} − γλθ_{t−1}, so the current learning rate γ multiplies the decay at every step. The original paper does not put the base step size α there at all. Its Algorithm 2 sets ηt ← SetScheduleMultiplier(t) and lets the decay enter as −ηtλθ_{t−1}; the authors “introduce a scaling factor ηt delivered by a user-defined procedure SetScheduleMultiplier(t)”. Read the sequence and you know which of the two quantities is scaling the shrinkage in your own run.

Make the policy explicit, then test its consequences. Step 1 is not busywork. The same keyword on the same optimizer name compiles to two different update rules in two mainstream libraries.

PyTorch 2.13 ships the choice as an argument on plain Adam: “decoupled_weight_decay (bool, optional) – if True, this optimizer is equivalent to AdamW and the algorithm will not accumulate weight decay in the momentum nor variance. (default: False)”. The default there is the coupled penalty, folded into the gradient. Keras 3 made the opposite choice for everything it ships. Its keras.optimizers.Adam carries a weight_decay argument of its own, and Keras 3.15.1 applies it in the base class as variable.assign(variable - variable * wd * lr), before each optimizer’s own update step runs. Decoupled, scaled by the learning rate, for every optimizer in the library.

Steps 2 through 5 only mean something once step 1 has said which of those two rules you are running.

In common AdamW implementations the shrinkage applied each step is scaled by the current learning rate. Schedule, batch size and total steps therefore all change the cumulative decay.

The authors state that horizon dependence in the paper and supply a correction for it. Different weight decay factors are optimal for different computational budgets, measured in batch passes. So an appendix on normalized weight decay replaces λ with λ = λ_norm · sqrt(b / (BT)). Here b is the batch size, B the total number of training points, T the total number of epochs. λ_norm is the decay that would apply if only one batch pass were allowed. That is what let a constant setting be reused across short and long runs in AdamWR — 7 settings of normalized weight decay for AdamW, against 12 settings of L2 for Adam, over 1800-epoch runs.

A second group reached the same joint dependence from the other side. Kosson and colleagues, in 2024, give AdamW’s equilibrium weight norm as sqrt(ηC/2λ), set by the learning rate and the decay together. Under a cosine decay schedule they observe that “the weights fall back out of equilibrium as the weights can not decay fast enough to keep up with the shifting equilibrium norm (which decreases with the learning rate schedule)”.

PaLM took the coupling to its conclusion in 2023 and used no constant at all. Its training setup applied a dynamic weight decay of lr^2.0, where lr is the current learning rate.

These conventions are not starting hypotheses. In the mainstream trainers they are hard-coded defaults that you inherit unless you override them.

Hugging Face’s Trainer builds exactly two groups. In Transformers 5.16.1, get_decay_parameter_names sets forbidden_name_patterns = [r"bias", r"layernorm", r"rmsnorm", r"(?:^|\.)norm(?:$|\.)", r"_norm(?:$|\.)"]. create_optimizer then splits the model into a weight_decay group and a weight_decay=0.0 group, and that second group also absorbs every parameter of an nn.LayerNorm module. The documentation states the same thing in one line: “The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights.” Keras 3 exposes the intent as a public API, exclude_from_weight_decay(var_list=None, var_names=None) on its base optimizer. PyTorch supports the pattern by accepting an “iterable of parameters or named_parameters to optimize or iterable of dicts defining parameter groups”, and its state_dict example carries per-group entries with 'weight_decay': 0 and 'weight_decay': 0.5.

The reason the normalization-adjacent parameters get a different policy is mechanical. A weight sitting in front of a normalization layer is not regularized by an L2 penalty at all. Van Laarhoven put it in his 2017 abstract: “However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate.” Hoffer and colleagues made the consequence visible in 2018, on VGG11 and CIFAR-10: “We show empirically that the accuracy gained by using WD can be achieved without it, only by adjusting the learning rate.” The whole accuracy gain attributed to weight decay came back with no weight decay at all, purely by correcting the learning rate for the growing weight norm.

Matrix and convolution weights remain the usual decay targets. Embeddings and other special tensors have no comparable published rule quoted here, so that policy stays a decision to record and to test.

Weight decay can improve validation behavior. But the mechanism the groups who went looking actually found is not the one the name suggests. With batch normalization, Hoffer and colleagues report, decay “improves optimization only”, by holding the weight norm in a small range.

A 2024 paper generalizes that, and its abstract does not hedge: “Overall, we present a unifying perspective from ResNets on vision tasks to LLMs: weight decay is never useful as an explicit regularizer but instead changes the training dynamics in a desirable way.” D’Angelo and colleagues reproduced AdamW’s entire loss curve on GPT-2-124M and OpenWebText, context length 1024, by matching only its effective learning rate η_t/‖w_t‖ with a non-standard schedule. At learning rate 0.001, all three random seeds trained with weight decay 0 diverged late in bfloat16 training. λ=0.1 and λ=0.3 stayed stable and reached lower final training loss. In one-pass pretraining, that is what the coefficient is buying.

An independent derivation lands in the same place. Kosson and colleagues show decay balancing “the average rotation—a proxy for the effective learning rate”, with AdamW’s equilibrium norm derived as sqrt(ηC/2λ). Their conclusion: “we hypothesize that balanced equilibrium rotation is the main benefit of AdamW over Adam+ℓ2”. Treat decay as one instrument whose value must be demonstrated under the intended training system. Treat a smaller norm as evidence about the trajectory, not about the learned function.