Learning Rate: Stability, Speed, and Scale

A direction indicates how the local objective changes. The learning rate determines how far the optimizer moves after transforming that direction. Move too little and the run wastes compute and may leave useful regions unreachable. Move too much and it overshoots, amplifies noise, or enters numerical regimes where later diagnostics become meaningless.

One hyper-parameter is worth more attention than the rest. Bengio said so bluntly in 2012, of the initial learning rate: “This is often the single most important hyper-parameter and one should always make sure that it has been tuned (up to approximately a factor of 2)”. He then says where the good value sits, and the phrasing matters: “The optimal learning rate is usually close to (by a factor of 2) the largest learning rate that does not cause divergence of the training criterion”.

Read that second sentence as an instruction about search rather than about size. The value you want is defined by its distance from a failure boundary. You locate a boundary by approaching it. Starting small does not avoid the boundary, it only hides where it is.

The learning rate is one factor in a longer computation. Each stage before and after it can change what a given number does.

First, the raw gradient. Backpropagation produces local sensitivity under the current batch and the current loss reduction. Second, the optimizer transform: momentum, normalization, preconditioning, clipping and decay modify that direction before anything is scaled. Third, the learning-rate scale. A scalar, or a per-parameter-group value, sets the update magnitude on the transformed direction. Fourth, the parameter change, applied under finite precision and possibly under distributed synchronization. Fifth, the new operating point. Activations, losses and future gradients all change once the step lands. That is why the safety of a rate is a property of a moment in a run rather than of the run as a whole.

A rate copied from elsewhere arrives at stage three of somebody else's pipeline. Stages one, two and four are yours.

Pretrained layers, new heads, biases, normalization parameters, and embeddings can have different scales or adaptation needs. Parameter groups make those choices explicit. Instead of one scalar standing in for every tensor in the model, the update magnitude is stated separately for sets of parameters that are known to behave differently.

Making the choice explicit is not the same as making it correct. Each group is a claim that this part of the model needs a different step size than that one. A claim of that kind is testable against the group's own update statistics.

Use a short disposable run with strict numerical guards.

The search has a published procedure. Smith named it an “LR range test” in 2015, and the recipe is to “run your model for several epochs while letting the learning rate increase linearly between low and high LR values”. The accuracy curve then shows where progress starts and where it collapses. Smith adds that this “test is enormously valuable whenever you are facing a new architecture or dataset”. It costs a few epochs, and the run is disposable.

What does the resulting boundary actually look like? One has been mapped, on the BIG Transformer trained on CzEng 1.0 on a single GPU. Popel and Bojar found a wide flat region and then a cliff: learning rates from 0.05 to 0.25 made almost no difference to BLEU, 0.01 converged notably more slowly, and 0.30 stopped working altogether. “Setting the learning rate too high (0.30, not shown in the figure) results in diverged training, which means in this case that the learning curve starts growing as usual, but at one moment drops down almost to zero and stays there forever.” Note the shape. Five times the rate, from 0.05 to 0.25, buys nothing. A further step of 0.05 destroys the run irrecoverably. The penalty for being under the boundary is small and the penalty for crossing it is total. That is the whole argument for margin, stated in numbers.

The same paper shows the boundary is not only a property of the rate. At the default rate 0.20 with 16,000 warm-up steps, there is a cliff in batch size too: “In our case, the sharp difference is between batch size 1450, which trains well, and 1400, which drops off after two hours of training, recovering only slowly.” A 50-unit change in a different knob moved a run from healthy to broken. The learning rate stayed exactly where it was.

The procedure, then, has five steps. One: start from a verified setup. Confirm data, loss, gradients and tiny-batch fitting before the sweep, so that the sweep is measuring the rate and nothing else. Two: increase the rate gradually, exploring several orders of magnitude without spending a full training budget. Three: record multiple signals — loss, gradient norms, update-to-parameter ratios, invalid values and validation snapshots. A single loss trace cannot distinguish the failure modes. Four: identify the failure boundary, noting where progress becomes erratic rather than simply choosing the largest number that survived. Five: select with margin, at a lower operating rate that remains stable across batches and seeds. The plateau above is why this costs so little. Inside a flat region, a more conservative rate is nearly free.

The same steering angle is a small correction in a parking lot and a spin on a highway. The consequence changes because operating scale changes, not because the wheel turned differently.

A training run changes speed constantly, through warmup, through decay, and from one module to the next. Step size must match local sensitivity at the moment it is applied. A rate that was safe an hour ago is not automatically safe now.

In the Transformer that variation is not an informal idea but a formula. Vaswani et al. did not use Adam's published defaults at all in 2017. They set β1 = 0.9, β2 = 0.98 and ε = 10−9, and varied the rate over training as lrate = d_model^−0.5 · min(step_num^−0.5, step_num · warmup_steps^−1.5). The authors describe the curve this produces: “This corresponds to increasing the learning rate linearly for the first warmup_steps training steps, and decreasing it thereafter proportionally to the inverse square root of the step number. We used warmup_steps = 4000.” Base models were trained for 100,000 steps, about 12 hours; big models for 300,000 steps, about 3.5 days, on 8 NVIDIA P100 GPUs.

Now the useful part. That same schedule, reimplemented in Tensor2Tensor and run by a different group, does not use the same warmup at all. Popel and Bojar report that “The learning_rate_warmup_steps parameter configures a linear_warmup_rsqrt_decay schedule and it is set to 16 000 by default (for the BIG model)”, alongside a default learning rate of 0.20. That is four times the warmup of the original paper, for the identical schedule. The number describes a run, its hardware and its batch. It does not describe the architecture it is applied to.

No single signature is conclusive, but combinations are informative. One combination has been isolated experimentally, and it is worth walking through, because it shows how much of a rate problem is not the rate.

Facebook AI Research trained ImageNet in an hour in 2017, and stated the rule that got them there: “Linear Scaling Rule: When the minibatch size is multiplied by k, multiply the learning rate by k.” Goyal et al. then trained ResNet-50 on ImageNet at minibatch 8k with that scaled rate, three times, changing only how the rate is approached. With no warmup, top-1 error was 24.84% ±0.37. With constant warmup it got worse: 25.88% ±0.56. With a 5-epoch gradual warmup — “we start from a learning rate of η and increment it by a constant amount at each iteration such that it reaches η̂ = kη after 5 epochs” — the same run reached 23.74% ±0.09, statistically indistinguishable from the 23.60% ±0.12 of the small-batch kn=256, η=0.1 baseline. The 8k run took 1 hour on 256 GPUs.

Read the three 8k rows together. The peak learning rate is identical in all of them. The only difference is the ramp, and the ramp is worth 2.1 points of top-1 error — the distance between a result that matches the baseline and one that does not. Any diagnostic that reports only the peak rate would have found these three runs identical.

The rule itself has limits the same paper marks, and an independent group has priced them. He, Zhang, Zhang, Zhang, Xie and Li, at Amazon Web Services, applied η = 0.1 × b/256 and measured the shortfall: “Increasing batch size from 256 to 1024 by linear scaling learning rate alone leads to a 0.9% decrease of the top-1 accuracy”. Their numbers show 75.87 falling to 75.17 for linear scaling alone, recovered to 75.93 once LR warmup is added. Two labs, the same finding: the schedule around the rate carries part of the result.

Use several signals rather than one loss trace.

Too small: updates cannot change behavior within the available budget. The loss declines slowly or flattens, the update-to-parameter norm ratio is extremely tiny, validation barely moves. The trap is that all of this is routinely mistaken for underfitting — a modelling verdict delivered about a run that never took a full step.

Useful range: progress is fast enough while numerical health stays controlled. The loss improves in a sustained way, the norm ratio is stable layer by layer, validation improves before overfitting begins, and the behavior repeats across seeds. Repeatability is the part that distinguishes evidence from luck.

Too large: updates cross useful regions or destabilize activations. The loss oscillates or explodes, the norm ratio spikes, validation is erratic, and the risk is NaN values and irreversible state.

That last column is not a hypothetical for large runs. It is an operations report. PaLM 540B used an Adafactor learning rate of 10−2 for the first 10,000 steps, decayed as 1/√k, and Google's account of the run records the outcome: “For the largest model, we observed spikes in the loss roughly 20 times during training, despite the fact that gradient clipping was enabled.” Clipping was on and the spikes came anyway. The mitigation was procedural rather than numerical: restart from a checkpoint roughly 100 steps before each spike, and skip 200 to 500 data batches.

OPT-175B took the other route on the same symptom. Meta AI states it plainly: “When the loss diverged, we found that lowering the learning rate and restarting from an earlier checkpoint allowed for the job to recover and continue training”. The paper's first figure plots the schedule this produced — not a curve chosen in advance but the record of what the run survived — under the caption “We found that lowering learning rate was helpful for avoiding instabilities”. The same section records at least 35 manual restarts, alongside more than 70 automatic ones, over roughly two months.

Two organisations, two 100B-plus models, one shared admission. At that scale the usable rate was found during the run.

Parameter groups also multiply tuning complexity. Begin with few groups, log each group's update ratio, and add distinctions only when evidence supports them.

The strongest version of the idea is not a heuristic. It is a parametrization with a counterfactual attached. A 40M-parameter proxy, its width shrunk to 256, was tuned and its hyperparameters handed to GPT-3 6.7B — a model 168 times larger — under the Maximal Update Parametrization. Yang et al. state the result in the abstract: “by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost.” That was 2022, at Microsoft and OpenAI.

The control condition is what makes this a lesson about scale rather than a trick. In the same paper, carrying the identical hyperparameters across naively, without µP, produced “training diverged” for both BERT-base and BERT-large. µTransfer instead reached test loss 1.683, against the tuned Megatron BERT-large baseline's 1.731. Same numbers, different widths: fatal without the scaling rule, better than a tuned baseline with it.

An independent organisation reports the same behavior from its own trainings. Three researchers at Cerebras and EleutherAI describe optimal hyperparameters drifting with model width under standard parametrization and staying stable under µP, with hidden-layer learning rates scaled by the width ratio. In their experience the gain “roughly translates to a 2x compute savings to reach the same performance”. Their guide was published in 2024.

So a per-group rate is a hypothesis about how that group's scale differs from another's. µP is what that hypothesis looks like when it has been stated precisely enough to be wrong.