HVAdam: A Full-Dimension Adaptive Optimizer

AI summary

Overview

Research area: Machine learning optimization algorithms (first-order adaptive optimizers for deep neural networks). Technical level: Intermediate — the core intuition is accessible, but the paper includes formal convergence proofs, projections, and exponential moving averages. Scope: Introduces HVAdam, an optimizer that uses a "hidden vector" capturing stable gradient trends across all dimensions to escape the valley dilemma and accelerate convergence.

What This Paper Is About

Adaptive optimizers like Adam adjust the learning rate separately for each parameter dimension, which works well when updates align with coordinate axes but fails when the optimal descent direction lies off-axis inside a narrow valley. The authors call this the "valley dilemma": gradients are large along the valley walls but small along the valley floor, so per-dimension scaling causes slow, zigzagging progress. HVAdam addresses this by deriving a shared directional vector across all dimensions to guide updates along the true descent path.

Key Contributions

1. Hidden vector construction: A vector that approximates the invariant components of gradients (the stable trend direction) using a projection-based update, allowing parameter steps along non-axis-aligned directions that Adam overlooks.
2. Restart strategy: A cosine-similarity-based mechanism that detects when the hidden vector becomes stale (the loss landscape direction changes) and reinitializes it, generalizing the method to dynamic landscapes.
3. New preconditioning matrix: A hidden-vector-based, wide-range scaling factor that adjusts the per-dimension learning rate using relative noise magnitude, expanding the achievable learning-rate range beyond Adam and AdaBelief.
4. Theory and extensive evaluation: Formal convergence guarantees under online convex and non-convex stochastic optimization, plus benchmarks across image classification, NLP, GANs, and diffusion models against 15 baseline optimizers.

Main Findings

• Valley dilemma resolution: On four representative functions, HVAdam reaches the optimum fastest, converging straight along the hidden vector while SGD, Adam, and AdaBelief zigzag.
• Image classification gains: On CIFAR-10/CIFAR-100 with VGG11, ResNet34, and DenseNet121, HVAdam achieves the highest test accuracy among compared optimizers. On ImageNet with ResNet50 it reaches 77.22% top-1, closing the gap with non-adaptive methods (Lookaround at 77.32%, SGDM at 76.49%, Adam at 72.87%).
• Vision Transformer performance: With ViT-B/16, HVAdam reaches 99.00% on CIFAR-10 and 92.38% on CIFAR-100, exceeding Adam, SWA, Lookahead, and Lookaround.
• GAN stability and quality: HVAdam produces the lowest FID scores on WGAN, WGAN-GP, and SN-GAN. Its WGAN FID is better than other optimizers' WGAN-GP FID, demonstrating notable stability under the notoriously unstable GAN training regime.
• Language modeling: On Penn TreeBank with 1–3 layer LSTMs, HVAdam achieves leading perplexity and the fastest convergence.
• Ablation validates each component: Adding the hidden vector alone (HVAdam₁), then the new preconditioning (HVAdam₂), then the restart strategy (HVAdam) progressively lowers validation perplexity from 85.04 → 84.54 → 83.42 → 83.31.
• Convergence guarantees: The hidden vector converges at rate O(1/T), and the optimizer retains convergence guarantees in convex and non-convex settings.

Methodology in Plain English

The authors start with the observation that when a loss surface forms a narrow valley, the direction we most need to move offers tiny gradients, while the directions we should avoid have large ones. Because Adam scales each coordinate independently, it takes big steps across the valley and small steps along it — the opposite of what's needed.

HVAdam keeps a running estimate called the hidden vector, which represents the stable direction shared by gradients over time. It is updated by projecting the current smoothed gradient onto the previous hidden vector, essentially finding the overlap between "where we've been heading" and "where the evidence now points." When that overlap (measured by cosine similarity) drops too low, the optimizer assumes the landscape has changed and restarts the hidden vector from scratch.

That same hidden vector feeds a redesigned preconditioning term: the optimizer compares the current gradient against the hidden vector to estimate noise, and uses the relative size of that noise to decide how large or cautious the step should be. This yields a much wider range of effective learning rates than Adam's denominator allows. The final update combines a standard Adam-style term with an additional hidden-vector-driven term scaled by a confidence weight.

The authors verify the idea on simple analytic functions, prove convergence under standard assumptions, then apply it to CNNs, Vision Transformers, LSTMs, GANs, and diffusion models.

Why This Matters

This work attacks a fundamental limitation of adaptive optimizers that has quietly shaped years of training practice: their inability to exploit the geometry of the whole loss landscape. If the hidden-vector idea generalizes as the results suggest, it could change how optimizers are designed for problems where conditioning is anisotropic — GANs, reinforcement learning, and transformer pretraining among them.

Real-world applications:

• Generative model training (GANs and diffusion models), where stability and image fidelity are critical for content creation and synthetic data pipelines.
• Medical and scientific imaging, where reliable convergence on CIFAR-style classification tasks translates to diagnostic and analysis systems.
• Natural language systems (LSTM/Transformer-based language modeling, translation) that depend on stable, efficient training.
• Large-scale vision systems such as ImageNet and ViT pipelines used in production perception and recognition.

Industry relevance: Optimizer improvements compound across every deep learning workload. Faster convergence and better final accuracy directly reduce compute cost, training time, and energy use — attractive to any organization training models at scale. GAN stability in particular is a persistent pain point in industry generative-AI development.

Future Directions

• Computational and memory efficiency: The authors acknowledge the algorithm is more complex and memory-hungry than Adam and plan to optimize this for memory-constrained environments.
• Scaling to large models: Testing whether HVAdam's gains hold for LLM-scale training, where optimizer overhead is a major practical constraint.
• Hyperparameter robustness: The threshold (0.1) and learning-rate function were chosen empirically; systematic study of how sensitive results are to these choices remains open.
• Broader domains: Extending validation to reinforcement learning, federated settings, and other non-stationary optimization problems where the landscape changes frequently.

Target Audience

Machine learning researchers and graduate students working on optimization algorithms, deep learning practitioners training GANs or transformers who struggle with convergence issues, and engineers at organizations training large models who care about accuracy-per-compute tradeoffs. Readers should be comfortable with gradient-based optimization concepts and basic calculus notation.

Authors’ abstract

Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance gap: adaptivity in pre-conditioners, which limits the optimizer's ability to adapt to diverse optimization landscapes. To address this, we propose Anon (Adaptivity Non-restricted Optimizer with Novel convergence technique), a novel optimizer with continuously tunable adaptivity , allowing it to interpolate between SGD-like and Adam-like behaviors and even extrapolate beyond both. To ensure convergence across the entire adaptivity spectrum, we introduce incremental delay update (IDU), a novel mechanism that is more flexible than AMSGrad's hard max-tracking strategy and enhances robustness to gradient noise. We theoretically establish convergence guarantees under both convex and non-convex settings. Empirically, Anon consistently outperforms state-of-the-art optimizers on representative image classification, diffusion, and language modeling tasks. These results demonstrate that adaptivity can serve as a valuable tunable design principle, and Anon provides the first unified and reliable framework capable of bridging the gap between classical and modern optimizers and surpassing their advantageous properties.