Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

AI summary

Overview

Research area: optimization algorithms for large language models trained with reinforcement learning with verifiable rewards (RLVR), grounded in the spectral analysis of gradients.

Technical level: Advanced. The paper combines empirical spectral analysis of LLM gradients with randomized numerical linear algebra (randomized block power iteration) and a new optimizer design.

One-sentence scope: The paper shows that RLVR induces a distinctive, evolving low-rank gradient geometry and introduces an optimizer, Ares, that infers its update rank online from the gradient's spectral energy distribution rather than fixing or scheduling it.

What This Paper Is About

Optimizers used in RLVR post-training of LLMs are largely inherited unchanged from supervised fine-tuning, typically AdamW. The authors ask whether RLVR produces a distinct optimization geometry that justifies a different optimizer, and they investigate this by measuring how gradient energy is distributed across singular components during training.

The goal is an optimizer whose update rank is not prescribed externally (as a constant or a preset schedule) but inferred from the gradient's observed spectrum and tracked as it evolves.

Key Contributions

1. Spectral characterization of RLVR gradients. The authors identify a persistent concentration of gradient energy in leading singular components during RLVR, with a declining entropy-based effective rank. They report that this behavior is not observed under supervised fine-tuning (SFT) or under a random-reward control in the same setup.

2. A spectral-driven adaptive optimizer. They propose Ares (Adaptive Rank-Effective Spectral Optimization), which estimates the effective rank from gradient energy distributions and dynamically adjusts its update subspace during RLVR training, rather than fixing the rank or following a preset schedule.

3. Performance validation across settings. They evaluate Ares across model scales, downstream tasks, and RLVR algorithms, and isolate contributing components through ablations.

4. A taxonomy of optimizers on the rank axis. Table 1 classifies representative optimizers by whether they restrict updates to a low-rank subspace, how the rank/subspace is determined, and whether that determination is spectral-driven. Ares is described as the one method in that table that is both low-rank and spectral-driven.

Main Findings

• Gradient energy concentrates during RLVR. Using the top-32 energy ratio of gradients (Qwen3-1.7B), gradient energy progressively concentrates in the leading singular components. The authors state this is characteristic of RLVR rather than of fine-tuning in general: SFT shows no such rise but a slight broadening, and replacing the verifiable reward with a random one, leaving rollouts and configuration intact, removes most of the increase.

• The rank needed for a fixed energy fraction declines. The rank r(ε) required to capture a fixed fraction ε of gradient energy (Qwen3-1.7B, GRPO, AdamW) declines steadily after a brief early transient. Curves are lightly smoothed in the reported figure.

• Ares improves average accuracy at every tested scale. With GRPO on Qwen3-Base models, Ares reaches average accuracy of 27.2, 38.5, and 42.1 on the 1.7B, 4B, and 8B models respectively, improving over AdamW by 5.0, 2.7, and 2.3 points. Ares ranks first on the large majority of the seven individual benchmarks.

• The advantage persists as model size grows. Rather than shrinking with capacity, Ares's lead holds from 1.7B to 8B and it remains the top method at every scale tested.

• Naive low-rank projection is not enough. GaLore, which projects gradients onto a subspace of externally fixed rank, is on par with AdamW and worse at the 4B scale (32.8 vs. 35.8). The authors attribute this to the fact that RLVR spectral concentration evolves during training.

• Weak optimizers fail badly under RLVR. SGD (2.0, 4.6, 7.2 average accuracy at 1.7B, 4B, 8B), RMSprop (8.7, 11.2, 10.4), and Muon (2.3, 5.8, 8.9) are consistently weak across all three scales, collapsing to near-zero or single-digit accuracy, while AdamW (22.2, 35.8, 39.8) and GaLore (22.9, 32.8, 38.8) are more stable.

• Ablations show components are complementary. On Qwen3-1.7B-Base, Ares scores 27.2 average. Removing components gives: w/o Rank Momentum 24.1, w/o Dynamic Rank 26.6, w/o Low Rank 2.7, w/o Spectral Normalize 25.8, w/o Adaptive LR 23.2. The paper reports that the low-rank update is the core, and that dynamic rank and momentum act as one mechanism: a rank estimated from noisy on-policy gradients fluctuates, and momentum smooths the estimate and stabilizes the subspace.

• Overhead is low relative to standard optimizers. The per-step parameter-update time of Ares stays close to standard optimizers and far below an exact spectral method, because randomized subspace iteration extracts only the leading components rather than a full SVD.

• Effective rank declines during RLVR, and is higher for larger models. The tracked effective rank falls steadily as training proceeds, and the 4B curve sits above the 1.7B curve throughout training, both at initialization and after convergence.

• Exponential rank mapping works best. Among three rank adaptation strategies compared (Exp, Linear, Threshold), the exponential mapping used in Ares performs best overall. Threshold is consistently worst. Smaller models are markedly more sensitive to the choice: the strategies span a wide range on Qwen3-1.7B but are nearly indistinguishable among the smooth mappings on Qwen3-4B.

• No memory saving is claimed in this version. The current implementation maintains dense moment tensors as in AdamW, so the authors state the present contribution is update quality rather than optimizer-state memory.

Methodology in Plain English

The authors first run diagnostic experiments. They measure the spectral structure of LLM gradient matrices throughout RLVR using the energy captured by the leading singular components, and compare against SFT and a random-reward control.

Based on the observation that energy concentrates into fewer directions over training, they build Ares around three steps per optimizer update:

1. Factorize cheaply. Rather than computing a full SVD (cost Θ(mn²)), Ares uses randomized block power iteration with a Gaussian test matrix, an oversampling parameter p, and q power iterations to extract only the leading singular subspace. Cost is governed by the target rank k rather than the ambient dimension. The power scheme amplifies the separation between leading and trailing singular values because (AAᵀ)^q A has singular values σᵢ^(2q+1).

2. Estimate the effective rank. The squared singular values are normalized into an energy distribution pᵢ = σᵢ²/Σσⱼ², summarized by Shannon entropy H, and converted to an effective rank r̂ = exp(H). This real-valued estimate is smoothed with a bias-corrected exponential moving average and then rounded and clipped to the range [1, k], so the rank can only decrease from the ceiling k within a preset maximum-rank budget.

3. Update in the low-rank subspace. Ares reconstructs the gradient as the rank-rₜ truncated SVD of the factorization, then applies an AdamW-style update with bias-corrected first and second moments, decoupled weight decay, and a scale factor s. One-dimensional parameters bypass the spectral treatment and take a standard AdamW step.

Experimental setup: Qwen3-Base models at 1.7B, 4B, and 8B, all trained for 300 steps with GRPO under the same configuration. Training data is the DAPO dataset. Evaluation uses seven mathematical reasoning benchmarks: AIME24, AIME25, AIME26, AMC23, MATH500, Minerva Math, and OlympiadBench (OlyBench), with average accuracy over 8 sampled responses per problem. Each step samples a group of 16 responses per prompt with rollout batch size 128, maximum generation length 8192 tokens, and mini-batch size 32. Learning rates are tuned per optimizer over {1, 2, 5} × 10⁻⁶ and {1, 2} × 10⁻⁵, with 5 × 10⁻⁶ used for Ares. The low-rank optimizers use rank 64; Ares initializes the rank at 64 and adapts it online. Ares-specific hyperparameters are oversampling p = 10, power iterations q = 20, rank-EMA smoothing c = 0.9, scale s = 1, shared across all layers and model scales rather than tuned per layer. Baselines are SGD, RMSprop, Adam, AdamW, GaLore, and Muon.

Why This Matters

Impact on research: the paper argues that effective optimization may require designing optimizers around the geometry and dynamics of the training process, rather than inheriting assumptions from supervised training. It reframes rank in RLVR as a quantity characterizing the evolving optimization geometry rather than merely a compression parameter, and it positions Ares against concurrent work that studies low-rank and spectral structure in RLVR.

Real-world applications (implied by the setting, not enumerated as such in the paper):

• Post-training pipelines for mathematical reasoning models, the paper's own evaluation domain.
• Training of reasoning-capable assistants where compute budget and update quality trade off.
• Any RLVR pipeline that must choose an optimizer across multiple model scales without retuning rank per scale.
• Research workflows studying optimization dynamics, where the tracked effective rank could serve as a diagnostic signal.

Industry relevance: RLVR is described as a central component of training strong reasoning models, and the paper reports that optimizer choice materially changes final accuracy, including outright collapse for some optimizers. That makes the optimizer a consequential, not interchangeable, building block in reasoning-model post-training.

Future Directions

1. Realizing memory savings. The paper notes that the low-rank factorization opens a path to maintaining first- and second-moment states in a factored form, but the current implementation does not yet realize this saving; the authors leave it for future work.

2. Extending beyond mathematics. The paper reports gains that "extend beyond mathematics," but the reported benchmark suite consists of mathematical reasoning benchmarks, leaving broader domains as an open direction.

3. Understanding why larger models operate at higher effective rank. The authors suggest greater capacity lets a model explore and exploit a richer set of useful update directions, but this interpretation is presented as a suggestion rather than a settled mechanism.

4. Broadening the rank-search range. Because Ares's tracked rank can only decrease from the ceiling k within a preset maximum-rank budget, it performs adaptive truncation rather than discovering a rank above k, which bounds what the method can express.

Target Audience

Researchers and engineers working on LLM post-training and reinforcement learning with verifiable rewards, and optimizer researchers interested in spectral methods for large-model training. The paper is most useful to those already comfortable with policy gradient methods, singular value decomposition, and adaptive optimizer internals; readers without that background will find the spectral-rank machinery and the RBPI analysis in Appendix C demanding.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training. This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer? We investigate this question through the spectral structure of LLM gradients and find that gradient energy progressively concentrates into a shrinking effective subspace, which is characteristic of RLVR, as it is absent under supervised fine-tuning (SFT) and is largely removed when the verifiable reward is replaced by a random one. Moreover, the rank required to capture a fixed fraction of gradient energy steadily decreases over training, revealing that the effective dimensionality of useful updates itself evolves with training. These findings motivate a new optimization principle: rather than prescribing the update rank externally, an optimizer should adapt its rank to the evolving spectral structure of the gradient. We introduce Ares (Adaptive Rank-Effective Spectral Optimization), which estimates the effective update rank from the gradient's spectral energy distribution, tracks its evolution, and confines the gradient signal driving each update to the resulting low-rank subspace. Across LLMs of different scales, Ares improves accuracy over widely used optimizers and generalizes across RLVR algorithms and tasks. Results suggest that effective optimization may require optimizers to be designed around the geometry and dynamics of the training process. We release the code and data at https://github.com/RUCAIBox/Ares.