AdaGrad, RMSProp, Adam, and Adaptive Updates

An embedding row updated rarely can have a very different gradient history from a dense projection matrix. Applying the same raw scale to both may create slow or unstable learning.

Adaptive optimizers maintain coordinate-wise statistics and use them to rescale updates. Letting the optimizer do that changes both optimization dynamics and the meaning of the learning rate.

AdaGrad arrived in 2011, in the Journal of Machine Learning Research, from Duchi and colleagues. Its abstract makes an unusual promise: the adaptation “allows us to find needles in haystacks in the form of very predictive but rarely seen features”. Rarely seen features are exactly the embedding rows that a dense projection dwarfs.

Every mechanism in this lesson descends from that sentence. So does every failure mode in it. The statistics that find the needle also decide how far each coordinate may move next, from a history the optimizer has already recorded. The constants that set the length of that history are not standardised across libraries. Nor are they left at their published values by the systems you have heard of.

Adaptive methods often reach low training loss quickly. Whether that turns into better validation behaviour is not vaguely open. It is a published dispute, with named sides, that a reader can go and read.

One side built a counterexample rather than a benchmark table. Its abstract states: “We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half.” On a binary problem, half is chance. The data was separable, so the information needed for zero error was there. The three adaptive methods did not use it. Wilson and colleagues took that to NIPS in 2017, together with several deep models on which the adaptive solutions generalised worse even when their training performance was better. They closed by recommending that “practitioners should reconsider the use of adaptive methods to train neural networks”.

The rebuttal is equally specific. Another group re-ran the comparison and reached the opposite verdict: “Despite conventional wisdom (Wilson et al., 2017; Balles & Hennig, 2017), we find that when carefully tuned, ADAM and other adaptive gradient methods never underperform MOMENTUM or SGD”. Choi and co-authors explain the reversal procedurally rather than mathematically. Wilson et al. “observed a ranking that, at first glance, contradicts the inclusion relationships … because they were not tuning enough of the hyperparameters”.

Both results stand. What is measured when an optimizer is declared better is partly the optimizer and partly the tuning protocol applied to it. The second half is usually the half that goes unreported.

Shoes that add cushioning where the shocks repeat, and take it away where movement stays gentle, spread one stride into different effective pressure across the sole.

All of that cushioning is built from shocks already absorbed. Coordinate-wise rescaling from history is exactly the trade: a direction that has been quiet for a long time gets treated as though it will stay quiet. RMSProp's decay constant is where that history length is set. It has a traceable origin, because RMSProp has no paper.

Its primary source is a lecture slide from 2012, in Hinton's Coursera course “Neural Networks for Machine Learning”. The title slide of Lecture 6e reads “rmsprop: Divide the gradient by a running average of its recent magnitude”. Three slides later, under “rmsprop: A mini-batch version of rprop”, the method is defined: MeanSquare(w, t) = 0.9 MeanSquare(w, t−1) + 0.1 (∂E/∂w(t))². The credit line reads “(Tijmen Tieleman, unpublished)”. The decay is 0.9. That is the number the method was announced with, and it was never peer-reviewed.

Keras 3 still ships rho=0.9, Hinton's exact constant, and names that lecture deck as its only reference — a single line reading “[Hinton, 2012]”. PyTorch ships alpha=0.99. The implied averaging horizon, 1/(1−decay), is 10 steps in one library and 100 in the other. Same name, same config field, a documented ten-fold departure from the method as it was first stated.

The names conceal distinct accumulation rules. The step from AdaGrad's cumulative sum to the exponential average used by RMSProp and Adam is not only a change of memory length. It is where Adam's convergence guarantee broke.

That break has a date and a page. Section 4 of the ICLR 2018 paper that found the error states it plainly: “The proof in the original paper of ADAM erroneously assumes that Γt is positive semi-definite and is hence, incorrect (refer to Appendix D for more details).” Reddi and colleagues did not stop at the flaw in the argument. They also show that “ADAM can fail to converge to an optimal solution even in simple one-dimensional convex settings”. Theorem 1 formalises it with an explicit one-dimensional convex online problem on which Adam has non-zero average regret. It provably does not reach the optimum, on a problem a reader could draw on paper.

Their repair is already in the reader's code. torch.optim.Adam carries the flag `amsgrad=False`, documented by the PyTorch maintainers as “whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`”. The default is off. The three columns below differ in what they remember. The flag is the scar left by what forgetting cost.

Adam combines direction memory with magnitude normalization. Both memories are tensors the size of the parameters themselves, and they are billed accordingly.

The Hugging Face Transformers documentation states the per-parameter price: “Optimizer states such as Adam store two extra tensors per parameter, the momentum and variance, which are both in fp32. That's an additional 8 bytes per parameter.” Mixed-precision weights cost 6 bytes per parameter, and the gradients 4. The optimizer state is the largest of the three.

The ZeRO paper derives the same accounting in 2020. Rajbhandari and colleagues at Microsoft count an fp32 copy of the parameters, the momentum and the variance at 4Ψ bytes each, so “Mixed-precision Adam has K = 12” and 16 bytes per parameter in total. Their worked example: “For a model such as GPT-2 with 1.5 Billion parameters, this leads to a memory requirement of at least 24 GB, which is significantly higher than the meager 3 GB of memory required to hold the fp16 parameters alone.” Eight times the weights, for two exponential averages.

Two “Adam” runs can differ materially through defaults and state handling. The published record shows where: in the second beta (0.999, 0.98, 0.95), in epsilon (10⁻⁹ through 1.0), and in whether `amsgrad` was on. The five steps below are what makes a run legible to someone who was not present for it.

Steps 4 and 5 carry the weight. The moments are the 8 bytes per parameter that outweigh both the mixed-precision weights and the gradients, so their norms are worth watching. Losing them at a checkpoint is expensive. A resume that silently reinitialises them restarts the second moment at zero. That is the exact condition bias correction exists to handle at step one of training, and no longer expects at step one hundred thousand.

These are tendencies rather than guarantees, and each one below is attached to the evidence in this lesson rather than to habit.

Compare time to a validation target, final decision metrics, sensitivity across seeds, and resource cost. That comparison has already been carried out at scale, in public. The answer is not a ranking.

Fifteen popular deep learning optimizers were benchmarked against each other across more than 50,000 individual runs, presented at ICML in 2021. The abstract reports: “While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it.” Schmidt and colleagues also found that evaluating several optimizers at their defaults worked about as well as tuning a single one.

The same structural conclusion comes out of a different set of experiments. Choi and co-authors write that “we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol”, and put it more sharply still: “the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature”. A default-versus-default run does not measure two optimizers. It measures two search spaces that somebody else chose. One of them was Hinton's 0.9 in 2012; the other was rounded to 0.99 by a library.