Research
Factored One-sided Adam-Moment for Practical and Scalable SOAP
Overview Research area: Machine learning optimization — specifically second-order / Kronecker-factored optimizers for pre-training transformer language models. Technical level: Advanced.

- arXiv
- 2611.16905
- Published
- 2026-11-12
- Authors
- Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han
AI summary
Overview
Research area: Machine learning optimization — specifically second-order / Kronecker-factored optimizers for pre-training transformer language models.
Technical level: Advanced. The paper assumes familiarity with Adam/AdamW, Shampoo, SOAP, K-FAC, Kronecker products, eigendecompositions, and the Hessian/Fisher curvature framework.
Scope: The paper introduces FOAM (Factored One-sided Adam-Moment), a lightweight SOAP variant that keeps SOAP-class validation loss while cutting optimizer state to half of SOAP's and running at Muon-class wall-clock time, validated on GPT-2-style LLM pre-training from 124M to 760M parameters.
What This Paper Is About
Second-order optimizers such as Shampoo and SOAP can beat AdamW in per-step convergence for transformer training, but they cost much more: SOAP needs roughly 4.5× the optimizer state of AdamW and pays for two periodic eigendecompositions, making it far slower per step. The authors ask whether SOAP's quality benefits can be retained at nearly AdamW-level efficiency. Their answer is FOAM, which removes SOAP's output-side Kronecker factor (keeping only the input side) and replaces the dense rotated second-moment buffer with a small row-by-column factored estimator.
Key Contributions
-
A structural justification for input-only preconditioning. Rather than dropping a Kronecker factor for memory reasons (as the one-sided fallback in Distributed Shampoo does), the authors argue from the Kronecker structure of the Hessian — for an affine layer with cross-entropy, H = x xᵀ ⊗ (diag(p) − p pᵀ) — that the off-diagonal curvature structure lives on the input side, so the input side is the one to keep.
-
A factored Adam-moment update with trace normalization. The dense m×n rotated second moment is replaced by r ∈ ℝᵐ and c ∈ ℝⁿ (row and column means of the squared rotated gradient) reconstructed as τ r̂ ĉᵀ with τ = tr(R)/(‖r̂‖₁‖ĉ‖₁). Lemma 1 shows that under the K-FAC factorization E[ggᵀ] = R ⊗ L, the input-rotated covariance is Λ_R ⊗ L, so the estimator is exact when the output-side factor is diagonal.
-
The FOAM algorithm itself, with per-parameter state reduced from 2mn + 2m² + 2n² to mn + 2n² + (m + n), one eigendecomposition instead of two (amortized cost O(n³) instead of O(m³ + n³)), and per-step cost O(mn²) matching Muon.
-
Empirical validation across three scales (124M, 350M, 760M) plus a construction ablation and Gauss-Newton alignment measurements that test the design choices directly.
Main Findings
-
Quality matches strong baselines. SOAP attains the lowest validation loss at every scale, with FOAM behind by only 0.004, 0.008, and 0.005 nats at small, medium, and large — each within about one seed std. FOAM is ahead of Muon, AdaMuon, and AdamW at every scale. At 760M: AdamW 2.847, Muon 2.806, AdaMuon 2.799, SOAP 2.791, FOAM 2.796.
-
Half of SOAP's state. Matrix-parameter optimizer state: at 124M FOAM uses 1.46 GiB vs SOAP's 2.85 GiB; at 350M 5.06 vs 10.12; at 760M 11.87 vs 23.73. Measured peak training memory at small size is +7.4% over AdamW versus +12.9% for SOAP.
-
Muon-class wall-clock. Per-step time at 760M is 1692 ms for FOAM vs 3343 ms for SOAP and 1449 ms for AdamW. Time to reach AdamW's final loss at 760M is 10.53 h for FOAM vs 10.55 h for AdamW, while SOAP needs 2.15× as long (22.66 h). FOAM's time to AdamW's loss is within 7% of AdamW's own at every scale.
-
The cost gap widens with scale. Figure 1 (GPT-2 large, 29.55K steps, wall-clock normalized so AdamW finishes at 1.0×): Muon 1.15×, FOAM 1.17×, SOAP 2.31×. Within AdamW's own budget FOAM is already 0.04 nats below AdamW's final loss, and it reaches SOAP's quality band at 51% of SOAP's wall-clock time.
-
Ablation supports both simplifications. Starting from SOAP (3.295, 2.92 GiB, 0.48 h), input-only preconditioning gives 3.302 at 1.78 GiB and 0.44 h — within 0.01 nats — while output-only preconditioning at the same state loses about 0.02 nats (3.322). Stacking the factored moment yields FOAM at 3.304 with 1.46 GiB. At medium and large, input-only SOAP and FOAM differ by at most 0.01 nats while factoring saves 18% of state.
-
Gauss-Newton alignment favors the input side. Across all 72 matrix parameters of the 12-layer GPT-2 small backbone, input-only keeps 75% of SOAP-full's alignment (0.38), whereas output-only keeps only 0.10 (dropping the input side costs 80%). FOAM tracks input-only within 1–2% at every checkpoint. On the exact layer-wise Gauss-Newton matrix, input-only exceeds output-only in all six module types at both 124M and 350M, by 1.5× to 11.5×.
-
K-FAC violation is modest. The K-FAC assumption is violated by a 15–20% relative error in practice, mostly at the key projection, costing only about 0.002 nats in final validation loss.
-
Factorability holds in SOAP's eigenbasis. In Adam's raw g² basis the second-moment matrix is only 0.54–0.63 rank-1 separable and drifts down during training, while in SOAP's eigenbasis the rotated matrix stays near 0.90 for the bilateral rotation.
-
Hyperparameter robustness inherited. FOAM's bowl shape and width track AdamW and SOAP within 0.05 nats across LR, WD, β₁, and β₂ sweeps on GPT-2 small. AdamW collapses to 3.86 at β₁ = 0.99, where SOAP and FOAM degrade by only 0.02 nats. The paper states FOAM retains robustness across a three-fold learning-rate range.
Methodology in Plain English
The authors start from SOAP, which eigendecomposes two gradient second-moment matrices — L (output side, m×m) and R (input side, n×n) — and then runs Adam in the rotated coordinate system defined by those eigenbases.
Two things are cut. First, they throw away the entire output-side factor L and its rotation, keeping only R. Their reason is that curvature for a transformer's linear layer is dominated by the input-side term (all output rows share the same input outer product), and the output-side curvature for a softmax layer is nearly diagonal because vocabulary-sized class probabilities make off-diagonal terms pᵢp_k negligible relative to diagonal terms pᵢ(1−pᵢ). This saves m² of memory and one of the two eigendecompositions.
Second, where SOAP stores a full m×n second-moment buffer in the rotated space, FOAM stores two small vectors: the row means and column means of the squared rotated gradient. If the rotated covariance truly factorizes (which the authors prove it does exactly under K-FAC with a diagonal output factor), those m+n numbers reconstruct the entire m×n surface, up to a scalar that trace normalization fixes.
The eigenbasis is only refreshed every pf steps, and even then skipped whenever the factor matrix is already nearly diagonal in the current basis, using the adaptive tolerance rule from Purifying Shampoo.
Experiments pre-train GPT-2-style models (RoPE, RMSNorm, untied embeddings, GELU MLP, no bias) at 124M, 350M, and 760M on FineWeb at a 1× Chinchilla budget — 2.49B, 7.10B, and 15.49B tokens over 4,750, 13,550, and 29,550 steps at a fixed 0.5M-token batch — on 8×H100 nodes with bfloat16 mixed precision. Only matrix parameters are preconditioned; embeddings, lm-head, and 1D/norm parameters go to a fixed AdamW group. Learning rates are tuned on a 5×5 grid at small and transferred via a μP (∝1/width) anchor at larger scales. Diagnostics include per-layer Gauss-Newton alignment via Hutchinson probes and a rank-1 separability measure of the second-moment buffer.
Why This Matters
For research: The paper reframes one-sided and factored SOAP variants — previously treated only as cost-reducing ablations without a criterion for when they preserve quality — as principled approximations with a stated exactness condition (K-FAC with diagonal output curvature) and a measurable residual. It also offers a falsifiable structural claim (the input side carries the curvature structure) backed by layer-wise Gauss-Newton measurements rather than only end-task loss.
Real-world applications:
- Pre-training large language models where optimizer state dominates per-device memory and directly limits batch size or model size.
- Fine-tuning or continued pre-training under tight GPU-memory budgets, where SOAP is currently impractical.
- Any Kronecker-factored optimizer deployment in distributed training, where halving state reduces communication and sharding pressure.
- Workloads where wall-clock, not step count, is the binding constraint — e.g., fixed-time research iteration cycles.
Industry relevance: The memory and time savings are measured on 8×H100 nodes with FSDP at the 350M and 760M scales, and the method is released as code (github.com/naver-ai/foam) from NAVER AI Lab. An optimizer that delivers SOAP's loss at AdamW's wall-clock time and +7.4% peak memory removes the main practical objection to adopting second-order methods in production pre-training pipelines.
Future Directions
- Test beyond 760M. The paper's largest scale is 760M; whether the input-only approximation and the rank-1 factorization remain as lossless at multi-billion-parameter scale and longer training is untested here.
- Close the residual K-FAC gap. The 15–20% violation is concentrated at the key projection; a targeted correction there might recover the remaining 0.004–0.008 nats separating FOAM from SOAP.
- Revisit the output-side assumption for non-softmax objectives. The vanishing-off-diagonal argument is specific to the output layer's softmax cross-entropy; the authors note only indirect Gauss-Newton evidence for hidden layers, leaving the mechanism to be established more directly.
- Combine with other SOAP improvements. The paper positions FOAM within a broader family (KL-Shampoo/KL-SOAP, CASPR, SPlus, Purifying Shampoo); stacking FOAM's simplifications with orthogonal refinements is an open direction.
Target Audience
Researchers and engineers working on optimizer design or large-scale LLM pre-training who already understand Adam, Shampoo/SOAP, and Kronecker-factored curvature approximations. It is most useful to practitioners deciding whether second-order optimizers are worth their memory and wall-clock cost, and to theorists interested in when one-sided and factored Fisher approximations are justified rather than merely cheap.
Authors’ abstract
Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored…
The full abstract on arXiv