Hyperball May Not Be a Free Lunch

AI summary

Overview

Research area: Machine learning optimization — specifically, the theory and practice of "Hyperball-style" optimizers (exemplified by MuonH) that constrain matrix-valued parameters to fixed-norm spheres for scale-invariant deep networks.

Technical level: Advanced. The paper combines matrix calculus, angular geometry of parameter space, and large-scale language-model pretraining experiments.

Scope: The paper derives an "angular effective learning rate" for general optimizer updates, uses it to test whether Hyperball's benefits come from suppressing radial (norm-changing) updates, and argues instead that Hyperball acts mainly as an implicit, state-dependent learning-rate scheduler.

What This Paper Is About

Hyperball-style optimizers fix the norms of matrix-valued parameters and normalize update sizes, and they have shown strong results in large-scale training — but nobody has clearly explained why they help. A common explanation is that removing radial (norm-increasing or norm-decreasing) motion stabilizes the effective learning rate. This paper tests that explanation, finds it insufficient, and proposes that Hyperball mainly works by changing the effective step size rather than by producing a better update direction.

Key Contributions

1. Conceptual: The authors derive an angular effective learning rate that explicitly accounts for the angle between the parameter and the optimizer update, and they state the conditions under which the conventional norm-based effective-learning-rate approximation is valid (specifically, parameter–update orthogonality).
2. Theoretical: Through controlled numerical analyses, they show that under the training configurations considered, radial updates have only a limited direct effect on the angular effective learning rate.
3. Practical: They propose a state-dependent learning-rate alignment method that matches the angular effective learning rate of a non-Hyperball optimizer to that of its Hyperball counterpart at every step, isolating the dynamical difference between the two optimizer families.
4. Empirical: In language-model pretraining experiments, they show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair later performance, and they report no direct evidence that MuonH consistently outperforms MuonWD across the entire optimization process.

Main Findings

• Angular effective learning rate depends on four quantities: the learning rate η_t, the parameter norm ‖W_t‖_F, the update norm ‖U_t‖_F, and the cosine similarity c_t between parameter and update. The conventional expression η_t / (α_t ‖W_t‖_F) is recovered only when c_t = 0.
• Radial updates are not the main story: The radial component (the part of the update parallel to the parameter) does not directly rotate the parameter, and the numerical results show that within the scope of these experiments its direct contribution to the one-step angular displacement is not the dominant explanation for MuonH's phase-dependent dynamics.
• Hyperball's effective learning rate is larger early and tracks the schedule: Under a constant base learning rate, MuonWD's effective learning rate decays rapidly as its parameter norm grows, while MuonH maintains an approximately constant effective learning rate after an initial transient caused by the abrupt evolution of c_t. Under a linearly decaying base learning rate, MuonH's effective learning rate remains approximately proportional to that schedule.
• Learning-rate alignment reproduces the dynamics: The validation-loss trajectories of the two optimizer families can be transformed into one another to a substantial degree by changing only the learning-rate schedule, supporting the interpretation that Hyperball's dominant effect is an implicit state-dependent learning-rate schedule rather than an intrinsically superior update direction.
• Aggressive decay is a trade-off, not a free win: Using a more aggressive decay schedule to induce earlier convergence caused MuonH to perform substantially worse during the later stage than standard MuonWD and MuonH with other schedules.
• Reported pretraining targets: MuonH-Z (the minus-square-root schedule proposed by Zhanpeng Zhou in PR #343) reaches the target validation loss of 3.28 in 3175 steps; MuonH-Ours (the authors' power-0.4 schedule released in PR #345) reaches the same target in 3150 steps.
• Scaling of the effective learning rate differs: In the middle and later stages, the effective learning rate of a non-Hyperball-style optimizer satisfies η_eff^φ ∝ sqrt(ηλ), whereas for a Hyperball-style optimizer η_eff^(H,φ) ∝ η.

Methodology in Plain English

The authors start from a simple geometric observation: what actually changes a network's behavior is how much the direction of a weight matrix rotates from one step to the next, not how much its length changes (since scale-invariant networks ignore length). They write down a formula for this angular displacement in terms of the learning rate, the parameter norm, the update norm, and the angle between the parameter and the update. They then split each update into a part that points along the parameter (radial) and a part that points perpendicular to it (tangential), and analyze how much each contributes.

They separate the direct effect of the radial part on the current step's rotation from its indirect effects on future steps through changing the parameter norm. Using differentiation of the log effective learning rate with respect to training time, they quantify the contribution of each factor during training.

Because simply watching two optimizers train cannot isolate causes, they run an intervention experiment: they hold the optimizer fixed (MuonWD) and only change its learning rate at every step so that its angular effective learning rate matches MuonH's, and then do the reverse. If the validation-loss curves swap to resemble each other, the effective learning rate — not the update direction — was doing the work. Finally, they confirm this on language-model pretraining by testing different learning-rate decay schedules (linear, exponential, and polynomial) with MuonH.

Why This Matters

Impact on research: The paper challenges the intuitive justification given for Hyperball-style optimizers — that fixing parameter norms stabilizes training. It reframes them as implicit learning-rate schedulers, which means the optimizer-specific engineering is not a substitute for careful learning-rate scheduling but makes that scheduling more important. It also provides a measurement tool (the angular effective learning rate) for analyzing non-orthogonal optimizer updates.

Real-world applications:

• Large-scale language-model pretraining, where optimizer choice and learning-rate schedule directly determine compute cost and final model quality.
• Hyperparameter search pipelines that currently tune learning-rate schedules and weight decay jointly — the paper's framing suggests Hyperball optimizers remove the weight-decay coupling.
• Training-run planning: knowing that MuonH can look worse early and better later helps teams avoid prematurely abandoning a configuration.
• Diagnostic tooling for training instability, since tracking the parameter–update angle c_t can reveal when the orthogonality assumption behind standard effective-learning-rate reasoning breaks down.

Industry relevance: Compute budgets for pretraining are enormous, and the MuonH-Z and MuonH-Ours numbers (3175 vs 3150 steps to reach validation loss 3.28) reflect the kind of iteration-count savings that translate into real cost reductions. The paper's caution — that aggressively accelerating early training can degrade later performance — is a direct warning against naive optimization of "time to target loss."

Future Directions

• A more complete theory: The numerical sensitivity analysis and the river-valley landscape interpretation are not a full theoretical characterization of the slow-then-fast convergence dynamics of Hyperball-style optimizers.
• Beyond dense language models: Extending the analysis to mixture-of-experts (MoE) architectures to test whether the observed dynamics persist under sparse activation and expert routing.
• Broader optimizer comparison: The current comparison is restricted to Muon, MuonWD, and MuonH; the authors plan to include AdamW and its Hyperball counterpart AdamH.
• Better evaluation: Moving beyond pretraining validation loss to a wider range of downstream tasks and metrics, to see whether the optimization behavior translates into generalization and practical model quality.
• Two-stage schedules: Whether a Hyperball-style optimizer's effective learning rate can become smaller than a conventional optimizer's in later training, and whether this can hinder performance, remains to be investigated.
• Using Hyperball as a testbed: Because its effective learning rate is proportional to the scheduled learning rate, a Hyperball optimizer avoids the confounding coupling between learning rate and weight decay studied in weight-decay research — a property the authors call particularly promising for learning-rate-scheduling research.

Target Audience

This paper is most valuable to optimization researchers and machine-learning engineers working on large-scale model training, especially those studying optimizer design (Muon, MuonH, AdamW), effective-learning-rate theory, scale invariance, or learning-rate scheduling. Practitioners tuning pretraining runs at scale will find the practical warning about aggressive decay schedules directly actionable. Readers need comfort with matrix norms, Frobenius inner products, and cosine-similarity geometry to follow the derivations, though the headline findings are stated in plain terms. It is likely less useful to readers seeking introduction-level material on optimization, since it assumes familiarity with the Muon optimizer family and scale-invariant network arguments.

Limitations noted by the authors: the experiments focus on dense language-model pretraining rather than MoE architectures; the optimizer comparison is limited to Muon, MuonWD, and MuonH (not AdamW/AdamH); evaluation centers on pretraining validation loss and optimization dynamics rather than downstream tasks; and the analysis leans on a scale-invariance assumption that does not strictly hold in practical training, even though the two implied properties (gradient orthogonality to parameters and decreasing gradient magnitude with growing norm) are observed in practice.

Authors’ abstract

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.