Warmup, Decay, Restarts, and Schedule Design

The clock and the transition rules are part of the optimizer configuration. Each phase below is a decision somebody has priced, not a law of optimization. Before drawing any conclusion about which schedule was better, Shallue and colleagues tuned the learning rate, the momentum and the learning-rate decay schedule independently at every batch size, across 35 workloads. The rest of this lesson quotes the runs where each phase was actually measured. Batch size, budget and error bars attached.

A warmup is useful only when its mechanism matches the failure. The first two cases below are not conjectures. Both were measured at a fixed peak rate by groups who published their error bars. In one of them the wrong warmup shape came out worse than no warmup at all.

They are policies, not universal laws. Each of the four families below has published numbers behind the belief it encodes.

Piecewise and cosine have been compared under matched tuning. Shallue and colleagues swept ResNet-50 on ImageNet with Nesterov momentum, batch size 1,024 and 150,000 steps, across six decay functions — constant, linear, cosine, exponential polynomial, inverse exponential polynomial and exponential — plus piecewise linear schedules with 1, 3 and 5 decay events. No family won: “it was possible to get good results with several of the schedules we tried”. They adopted linear decay only because it “performed at least as well as all other schedules we tried, while also being the simplest”.

Cosine's tie to a horizon fixed in advance has a measured cost. A 2024 paper opens its abstract with the complaint: “In this work, we argue that scale and training research has been needlessly complex due to reliance on the cosine schedule, which prevents training across different lengths for the same model size.” Hägele and colleagues replaced cosine decay with a constant learning rate plus a linear cooldown over the last 20% of steps. On a 210M-parameter model trained on SlimPajama that matched cosine almost exactly. A (1−sqrt) cooldown over only 5% of steps nearly matched cosine on a 20-billion-token run. On how much cooldown is enough, they report that “the benefits of extended cooldown periods plateau at around 20%”. The refinement phase can be a fifth of the run rather than the shape of the whole run.

Plateau response is not a vague idea. It is a shipped decision rule with published defaults, and the two dominant frameworks chose the same ones. Keras 3 states the premise — “Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates.” — and its ReduceLROnPlateau callback defaults to monitor="val_loss", factor=0.1, patience=10, min_delta=0.0001 and cooldown=0. PyTorch 2.8's ReduceLROnPlateau — “Reduce learning rate when a metric has stopped improving.” — defaults to mode='min', factor=0.1, patience=10, threshold=0.0001 and cooldown=0. Those four numbers are the entire policy: how much to cut, how long to wait, what counts as an improvement, and how long to ignore the metric afterwards. Note what patience=10 counts. Not steps, not epochs: evaluation events. Change the validation cadence and you have changed the schedule without touching the schedule.

Cyclical and restart policies come with an accuracy-per-iteration figure. On CIFAR-10 a fixed-schedule baseline reached 81.4% test accuracy after 70,000 iterations. The triangular2 policy reached the same 81.4% after 25,000. In the same table a plain decay policy at 25,000 iterations gave 78.5%, and an exponential policy at 70,000 gave 79.1%. Leslie N. Smith published those runs in 2015, together with the LR range test and the triangular policies. The three policies he named — triangular, triangular2 and exp_range — are the three modes PyTorch ships to this day, in a scheduler documented as one that “Sets the learning rate of each parameter group according to cyclical learning rate policy (CLR).”

A long warmup may postpone divergence until expensive compute has already been spent. It can also consume a large fraction of a short fine-tuning run.

Test the peak region, inspect early update ratios, and compare against a genuinely lower-rate baseline. Warmup should have a diagnosed purpose.

The original Transformer paper turned warmup into a formula. Vaswani and colleagues describe “increasing the learning rate linearly for the first warmup_steps training steps, and decreasing it thereafter proportionally to the inverse square root of the step number”, and then fix the constant: “We used warmup_steps = 4000.” The same section opens with the optimizer the ramp sits inside — “We used the Adam optimizer [20] with β1 = 0.9, β2 = 0.98 and ϵ = 10−9”. The ramp, the decay and the moment estimates were chosen together, in one arrangement.

What a diagnosis looks like in practice is the Pre-LN result. Xiong and colleagues did not tune the warmup. They located the cause — large expected gradients near the output layer of a Post-LN Transformer at initialization — moved the layer normalization, and removed the warm-up stage entirely. The reverse move confirms the diagnosis rather than contradicting it. Where the cause is still present, Liu and colleagues report that “removing the warmup stage in Transformer training results in more severe consequences such as model divergence”. The same deletion is safe in one architecture and fatal in the other. That is what it means for warmup to have a purpose. You can say which failure it is holding back, and you can name the change that would make it unnecessary.

An epoch changes meaning when dataset size, sampling, accumulation, or worker count changes. Step count changes meaning when batch size changes.

For language training, tokens often provide a clearer budget. Other tasks may use examples, frames, environment interactions, or optimizer updates. Record the chosen clock explicitly.

The clock and the budget have to be the same number. Hoffmann and twenty-one co-authors tied one to the other. They find that “setting the cosine cycle length too much longer than the target number of training steps results in sub-optimally trained models”, and give the rule in tokens: “when training a model over D tokens, we should pick a cosine cycle length that decays 10× over approximately D tokens”. A schedule stretched past the run leaves the model mid-decay.

The sharpest demonstration that optimizer steps are a poor clock is a paper in which the schedule and the batch are interchangeable. Samuel L. Smith and colleagues open their abstract with the substitution: “It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training.” It holds for SGD, SGD with momentum, Nesterov momentum and Adam, reaching equivalent test accuracies after the same number of epochs but with far fewer parameter updates. The conference version closes with “We train ResNet-50 on ImageNet to 76.1% validation accuracy in under 30 minutes.”; the earlier version of the abstract closes with “We train Inception-ResNet-V2 on ImageNet to 77% validation accuracy in under 2500 parameter updates, efficiently utilizing training batches of 65536 images”. Under 2,500 updates is not a small amount of training. It is a normal amount of training counted on the wrong clock.

Start with the evidence horizon, then choose phases. Step 2's “short range test” is not folklore. It has an author and a procedure. Leslie N. Smith set it out: “It is a “LR range test”; run your model for several epochs while letting the learning rate increase linearly between low and high LR values.” On the CIFAR-10 network in that paper the test suggested bounds of 0.001 and 0.006, and those bounds are what the cyclical runs were then built on. The point of the step is that the peak region is measured under the batch regime you will actually train in, before any warmup or decay is layered on top of it.

Compare equal data and compute budgets, the same checkpoint-selection rule, and equivalent validation access. One schedule may appear better because it receives more effective exploration or more selection opportunities.

The largest controlled test of that suspicion is a public database of 71,638,836 loss measurements, over 168,160 individual models, across 35 workloads. Shallue and colleagues released it in 2019, having tuned the learning rate, the momentum and the learning-rate decay schedule independently at every batch size. The finding is stated in the abstract: “Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes.” They were not resolving the argument about which setting wins. They were showing that most of the argument was about unequal tuning and unequal budgets. Once those were held equal in their own schedule sweep, “it was possible to get good results with several of the schedules we tried”. A schedule comparison that is not tuned equally at both settings is reporting the tuning.

Plot performance against examples, tokens, wall-clock time, and energy when those budgets matter. A schedule can win on one axis and lose on another. Loshchilov and Hutter argued restarts on the same terms. Their SGDR decays the rate within each run and then restarts it warm — “Within the i-th run, we decay the learning rate with a cosine annealing for each batch” — and on CIFAR-10 and CIFAR-100 they “demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively”. Those two numbers mean something only because both arms saw the same budget. A restart schedule handed extra epochs proves nothing at all.