Momentum and Nesterov Dynamics

Classical momentum combines the previous velocity with the current gradient, then updates parameters using the new velocity. The coefficient determines how slowly old directions decay. Persistent gradients reinforce one another, while high-frequency variation is smoothed. The stored state can also continue pushing after a sharp turn in the objective.

The update is old, and its promise is exact. B. T. Polyak published it in 1964 as the heavy-ball method, in the form x_{k+1} = x_k − α∇f(x_k) + β(x_k − x_{k−1}). Tuned optimally, with α = 4/(√L+√m)² and β = ((√κ−1)/(√κ+1))², it converges on quadratics with factor (√L−√m)/(√L+√m). Gradient descent's factor is (L−m)/(L+m). Ghadimi and colleagues at KTH state the payoff plainly: “This convergence factor is always smaller than the one associated with the gradient iterates, and significantly so when the Hessian of the objective function is poorly conditioned.” Lessard and colleagues tabulate the same optimal tuning and note that the bounds are tight.

There is a condition attached, and deep networks do not meet it. Polyak reached the result by a local argument, and, in the KTH group's words, “this local analysis requires twice differentiability of the objective functions”. A deep network is not a twice continuously differentiable strongly convex objective. On the problems in this course the guarantee is gone. Only the measurements remain.

Those measurements arrived for deep learning in 2013, and they came with their own condition. Sutskever and colleagues showed that stochastic gradient descent with momentum trains deep and recurrent nets when it “uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter”. The ceiling of that schedule, µmax, was drawn from {0.999, 0.995, 0.99, 0.9, 0}. The autoencoders ran for 750,000 updates with minibatches of 200. Table 1 gives Nesterov a training squared error of 0.074 at µmax=0.999 on CURVES, against 0.48 with no momentum. On MNIST it gives 0.73 at µmax=0.99, against 2.1. On FACES, 7.7 at µmax=0.999, against 36.4. Classical momentum never beat 0.10 on CURVES at any ceiling. Their own summary of the table: “It also shows that larger values of µmax tend to achieve better performance and that NAG usually outperforms CM, especially when µmax is 0.995 and 0.999.”

The winning number belongs to the schedule, not to the optimizer. Lucas and colleagues re-ran MNIST autoencoders in 2018 “using a set-up similar to that of Sutskever et al.”, but with the coefficient held fixed and drawn from {0.0, 0.9, 0.99, 0.999}. Held fixed, 0.99 was optimal for both variants: best training MSE 2.51±0.06 for classical momentum against 1.52±0.02 for Nesterov. The runs at 0.999 were unstable. That is the value that had won under a slowly increasing schedule. Same coefficient, different schedule, opposite result.

This sequence repeats at every optimizer step. Frameworks disagree about where in it the learning rate enters. PyTorch multiplies the rate in only at the final parameter update: v_{t+1} = µ·v_t + g_{t+1}, then p_{t+1} = p_t − lr·v_{t+1}. Keras folds it into the buffer one step earlier: velocity = momentum * velocity - learning_rate * g, then w = w + velocity. The five steps are the same. What is stored in the buffer is not.

Implementations vary, so verify the exact update rule in your framework. Verify the risk column too. Overshoot after a turn is not a metaphor about carts: heavy-ball run with Polyak's own optimal parameters can fail to converge on a strongly convex function.

The counterexample is one-dimensional and fully explicit. Lessard and colleagues built it in 2016, with m=1 and L=25 and a gradient of 25x for x<1, x+24 for 1≤x<2, and 25x−24 for x≥2. On it, the iteration x_{k+1} = (13/9)x_k − (4/9)x_{k−1} − (1/9)∇f(x_k) falls into an attractive period-3 cycle from any start in 3.07 ≤ x0 ≤ 3.46. The cycle sits at p = 792/1225 ≈ 0.65, q = −2208/1225 ≈ −1.80 and r = 2592/1225 ≈ 2.12, and the iterates never leave it. Their caption to Figure 6: “The iterates tend to a limit cycle, so the Heavy-ball method does not converge for this particular strongly convex function.”

The KTH group hit the same wall independently, on their own example, with gradient 50x+45 / 5x / 50x, µ=5 and L=50, for every start with x0 < −0.8 or x0 > 0.15: “Heavy-ball iterates with optimal step-sizes ... do not converge for the example in (32)”. Goujaud and colleagues generalised it in Mathematical Programming in 2025: for any condition number and any parameters, heavy-ball is either not accelerated, or there exists a smooth strongly convex function and an initialization on which it does not converge. Nesterov-style lookahead is the standard response to exactly this behaviour. On the autoencoder benchmarks above, it was the variant that won.

Optimizer state can outlive the regime that created it, and the cost of that has been measured on three benchmarks at once. Sutskever and colleagues report it in Table 2 of the 2013 paper, “The effect of low-momentum finetuning for NAG”: “We found it beneficial to reduce µ to 0.9 (unless µ is 0, in which case it is unchanged) during the final 1000 parameter updates of the optimization without reducing the learning rate”. The learning rate was left alone. Only the memory length changed, and only for the last 1,000 of 750,000 updates. Nesterov's training squared error fell from 0.096 to 0.074 on CURVES, from 1.20 to 0.73 on MNIST, and from 10.83 to 7.7 on FACES. The coefficient that was correct for the bulk of the run was actively costing accuracy at the end of it.

There is a theoretical counterpart, and Smith relays it. Liu and colleagues show via a diffusion approximation that momentum helps escape saddle points but hurts convergence near optima, so that “a large momentum helps escape saddle points but can hurt the final convergence, implying that momentum should be reduced at the end of training”. Smith's own recommendation is different. He prefers cyclical momentum to a terminal drop, and reports finding “a small improvement with cycling over only decreasing the momentum”. The five transitions below create the same mismatch mid-run.

Start here when a previously stable run becomes erratic after a phase change. Step 4 separates the two diagnoses in this lesson. One is excessive scale after a rate jump, which the momentum correction factor eta_{t+1}/eta_t addresses in the rate-inside convention. The other is genuinely stale direction memory, which the 2013 autoencoder runs fixed by lowering the coefficient with the learning rate left unchanged.