# Factored One-sided Adam-Moment for Practical and Scalable SOAP

AI.info

> Second-order optimizers such as Shampoo and SOAP can beat AdamW in per-step convergence for transformer training, but they cost much more: SOAP needs roughly 4.5× the optimizer state of AdamW and…

**arXiv**: 2611.16905

**Published**: 2026-11-12

**Authors**: Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han

## AI summary

## Overview

**Research area:** Machine learning optimization — specifically second-order / Kronecker-factored optimizers for pre-training transformer language models.

**Technical level:** Advanced. The paper assumes familiarity with Adam/AdamW, Shampoo, SOAP, K-FAC, Kronecker products, eigendecompositions, and the Hessian/Fisher curvature framework.

**Scope:** The paper introduces FOAM (Factored One-sided Adam-Moment), a lightweight SOAP variant that keeps SOAP-class validation loss while cutting optimizer state to half of SOAP's and running at Muon-class wall-clock time, validated on GPT-2-style LLM pre-training from 124M to 760M parameters.

## What This Paper Is About

Second-order optimizers such as Shampoo and SOAP can beat AdamW in per-step convergence for transformer training, but they cost much more: SOAP needs roughly 4.5× the optimizer state of AdamW and pays for two periodic eigendecompositions, making it far slower per step. The authors ask whether SOAP's quality benefits can be retained at nearly AdamW-level efficiency. Their answer is FOAM, which removes SOAP's output-side Kronecker factor (keeping only the input side) and replaces the dense rotated second-moment buffer with a small row-by-column factored estimator.

## Key Contributions

1. **A structural justification for input-only preconditioning.** Rather than dropping a Kronecker factor for memory reasons (as the one-sided fallback in Distributed Shampoo does), the authors argue from the Kronecker structure of the Hessian — for an affine layer with cross-entropy, H = x xᵀ ⊗ (diag(p) − p pᵀ) — that the off-diagonal curvature structure lives on the input side, so the input side is the one to keep.
2. **A factored Adam-moment update with trace normalization.** The dense m×n rotated second moment is replaced by r ∈ ℝᵐ and c ∈ ℝⁿ (row and column means of the squared rotated gradient) reconstructed as τ r̂ ĉᵀ with τ = tr(R)/(‖r̂‖₁‖ĉ‖₁). Lemma 1 shows that under the K-FAC factorization E[ggᵀ] = R ⊗ L, the input-rotated covariance is Λ_R ⊗ L, so the estimator is exact when the output-side factor is diagonal.
3. **The FOAM algorithm itself**, with per-parameter state reduced from 2mn + 2m² + 2n² to mn + 2n² + (m + n), one eigendecomposition instead of two (amortized cost O(n³) instead of O(m³ + n³)), and per-step cost O(mn²) matching Muon.
4. **Empirical validation across three scales** (124M, 350M, 760M) plus a construction ablation and Gauss-Newton alignment measurements that test the design choices directly.

## Main Findings

- **Quality matches strong baselines.** SOAP attains the lowest validation loss at every scale, with FOAM behind by only **0.004, 0.008, and 0.005 nats** at small, medium, and large — each within about one seed std. FOAM is ahead of Muon, AdaMuon, and AdamW at every scale. At 760M: AdamW 2.847, Muon 2.806, AdaMuon 2.799, SOAP 2.791, FOAM 2.796.
- **Half of SOAP's state.** Matrix-parameter optimizer state: at 124M FOAM uses **1.46 GiB vs SOAP's 2.85 GiB**; at 350M **5.06 vs 10.12**; at 760M **11.87 vs 23.73**. Measured peak training memory at small size is **+7.4% over AdamW** versus **+12.9% for SOAP**.
- **Muon-class wall-clock.** Per-step time at 760M is **1692 ms for FOAM vs 3343 ms for SOAP** and 1449 ms for AdamW. Time to reach AdamW's final loss at 760M is **10.53 h for FOAM vs 10.55 h for AdamW**, while SOAP needs **2.15×** as long (22.66 h). FOAM's time to AdamW's loss is within **7%** of AdamW's own at every scale.
- **The cost gap widens with scale.** Figure 1 (GPT-2 large, 29.55K steps, wall-clock normalized so AdamW finishes at 1.0×): Muon 1.15×, **FOAM 1.17×**, SOAP 2.31×. Within AdamW's own budget FOAM is already 0.04 nats below AdamW's final loss, and it reaches SOAP's quality band at **51%** of SOAP's wall-clock time.
- **Ablation supports both simplifications.** Starting from SOAP (3.295, 2.92 GiB, 0.48 h), input-only preconditioning gives 3.302 at 1.78 GiB and 0.44 h — within 0.01 nats — while output-only preconditioning at the same state loses about **0.02 nats** (3.322). Stacking the factored moment yields FOAM at 3.304 with 1.46 GiB. At medium and large, input-only SOAP and FOAM differ by at most 0.01 nats while factoring saves **18%** of state.
- **Gauss-Newton alignment favors the input side.** Across all 72 matrix parameters of the 12-layer GPT-2 small backbone, input-only keeps **75%** of SOAP-full's alignment (0.38), whereas output-only keeps only **0.10** (dropping the input side costs 80%). FOAM tracks input-only within **1–2%** at every checkpoint. On the exact layer-wise Gauss-Newton matrix, input-only exceeds output-only in all six module types at both 124M and 350M, by **1.5× to 11.5×**.
- **K-FAC violation is modest.** The K-FAC assumption is violated by a **15–20%** relative error in practice, mostly at the key projection, costing only about **0.002 nats** in final validation loss.
- **Factorability holds in SOAP's eigenbasis.** In Adam's raw g² basis the second-moment matrix is only **0.54–0.63** rank-1 separable and drifts down during training, while in SOAP's eigenbasis the rotated matrix stays near **0.90** for the bilateral rotation.
- **Hyperparameter robustness inherited.** FOAM's bowl shape and width track AdamW and SOAP within **0.05 nats** across LR, WD, β₁, and β₂ sweeps on GPT-2 small. AdamW collapses to **3.86** at β₁ = 0.99, where SOAP and FOAM degrade by only **0.02 nats**. The paper states FOAM retains robustness across a three-fold learning-rate range.

## Methodology in Plain English

The authors start from SOAP, which eigendecomposes two gradient second-moment matrices — L (output side, m×m) and R (input side, n×n) — and then runs Adam in the rotated coordinate system defined by those eigenbases.

Two things are cut. First, they throw away the entire output-side factor L and its rotation, keeping only R. Their reason is that curvature for a transformer's linear layer is dominated by the input-side term (all output rows share the same input outer product), and the output-side curvature for a softmax layer is nearly diagonal because vocabulary-sized class probabilities make off-diagonal terms pᵢp_k negligible relative to diagonal terms pᵢ(1−pᵢ). This saves m² of memory and one of the two eigendecompositions.

Second, where SOAP stores a full m×n second-moment buffer in the rotated space, FOAM stores two small vectors: the row means and column means of the squared rotated gradient. If the rotated covariance truly factorizes (which the authors prove it does exactly under K-FAC with a diagonal output factor), those m+n numbers reconstruct the entire m×n surface, up to a scalar that trace normalization fixes.

The eigenbasis is only refreshed every `pf` steps, and even then skipped whenever the factor matrix is already nearly diagonal in the current basis, using the adaptive tolerance rule from Purifying Shampoo.

Experiments pre-train GPT-2-style models (RoPE, RMSNorm, untied embeddings, GELU MLP, no bias) at 124M, 350M, and 760M on FineWeb at a 1× Chinchilla budget — 2.49B, 7.10B, and 15.49B tokens over 4,750, 13,550, and 29,550 steps at a fixed 0.5M-token batch — on 8×H100 nodes with bfloat16 mixed precision. Only matrix parameters are preconditioned; embeddings, lm-head, and 1D/norm parameters go to a fixed AdamW group. Learning rates are tuned on a 5×5 grid at small and transferred via a μP (∝1/width) anchor at larger scales. Diagnostics include per-layer Gauss-Newton alignment via Hutchinson probes and a rank-1 separability measure of the second-moment buffer.

## Why This Matters

**For research:** The paper reframes one-sided and factored SOAP variants — previously treated only as cost-reducing ablations without a criterion for when they preserve quality — as principled approximations with a stated exactness condition (K-FAC with diagonal output curvature) and a measurable residual. It also offers a falsifiable structural claim (the input side carries the curvature structure) backed by layer-wise Gauss-Newton measurements rather than only end-task loss.

**Real-world applications:**

- Pre-training large language models where optimizer state dominates per-device memory and directly limits batch size or model size.
- Fine-tuning or continued pre-training under tight GPU-memory budgets, where SOAP is currently impractical.
- Any Kronecker-factored optimizer deployment in distributed training, where halving state reduces communication and sharding pressure.
- Workloads where wall-clock, not step count, is the binding constraint — e.g., fixed-time research iteration cycles.

**Industry relevance:** The memory and time savings are measured on 8×H100 nodes with FSDP at the 350M and 760M scales, and the method is released as code (github.com/naver-ai/foam) from NAVER AI Lab. An optimizer that delivers SOAP's loss at AdamW's wall-clock time and +7.4% peak memory removes the main practical objection to adopting second-order methods in production pre-training pipelines.

## Future Directions

- **Test beyond 760M.** The paper's largest scale is 760M; whether the input-only approximation and the rank-1 factorization remain as lossless at multi-billion-parameter scale and longer training is untested here.
- **Close the residual K-FAC gap.** The 15–20% violation is concentrated at the key projection; a targeted correction there might recover the remaining 0.004–0.008 nats separating FOAM from SOAP.
- **Revisit the output-side assumption for non-softmax objectives.** The vanishing-off-diagonal argument is specific to the output layer's softmax cross-entropy; the authors note only indirect Gauss-Newton evidence for hidden layers, leaving the mechanism to be established more directly.
- **Combine with other SOAP improvements.** The paper positions FOAM within a broader family (KL-Shampoo/KL-SOAP, CASPR, SPlus, Purifying Shampoo); stacking FOAM's simplifications with orthogonal refinements is an open direction.

## Target Audience

Researchers and engineers working on optimizer design or large-scale LLM pre-training who already understand Adam, Shampoo/SOAP, and Kronecker-factored curvature approximations. It is most useful to practitioners deciding whether second-order optimizers are worth their memory and wall-clock cost, and to theorists interested in when one-sided and factored Fisher approximations are justified rather than merely cheap.

## Authors’ abstract

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored…

[The full abstract on arXiv](https://arxiv.org/abs/2611.16905)

[Read the original paper](https://arxiv.org/abs/2611.16905)

---

Canonical: https://ai.info/paper/factored-one-sided-adam-moment-for-practical-and-scalable-soap-2611.16905
