The Path Not Taken: RLVR Provably Learns Off the Principals

AI summary

Overview

Research area: Machine learning — post-training of large language models, specifically the optimization dynamics of Reinforcement Learning with Verifiable Rewards (RLVR) versus supervised fine-tuning (SFT).

Technical level: Advanced. The paper combines large-scale empirical checkpoint analysis with formal results (propositions, a theorem, and corollaries) drawn from policy-gradient theory, matrix perturbation theory, and numerical precision analysis.

Scope (one sentence): The paper argues — and empirically validates at the parameter level — that RLVR's apparently sparse weight updates are a surface artifact of a persistent, model-conditioned optimization bias that steers RL updates away from the principal (high-curvature) directions of a pretrained model, explaining this via a "Three-Gate Theory."

What This Paper Is About

RLVR reliably improves the reasoning ability of large language models while apparently changing only a small fraction of parameters, which is a puzzle given its high computational cost and large behavioral gains. This paper asks where that optimization bias comes from and how it shapes parameter evolution, then tests the answer by comparing weight-space changes under RLVR against those under SFT. The central claim is that RLVR and SFT operate in different regions of parameter space: RLVR learns off the principal directions, while SFT targets them.

Key Contributions

1. Observation (Sec. 2). The authors identify a persistent, model-conditioned optimization bias in RLVR fine-tuning that is largely invariant to datasets and RL variants, yet highly consistent across runs for a fixed pretrained model, and show that measured sparsity is a superficial readout of this bias amplified by bfloat16 precision.

2. Theory (Sec. 3). They propose the Three-Gate Theory — Gate I (KL Anchor), Gate II (Model Geometry), and Gate III (Precision) — which mechanistically explains how RL updates are constrained, steered, and filtered into the observed pattern.

3. Evidence (Sec. 4). They provide a parameter-level validation contrasting RL and SFT training dynamics, including reduced spectral drift, smaller principal-subspace rotation, low overlap with principal weights, and basis-rotation interventions that isolate model geometry as the steering mechanism.

4. Insight (Sec. 5). They show that SFT-era sparse and low-rank priors (such as principal-targeted variants) are misaligned with RLVR's off-principal dynamics, motivating geometry-aware, RLVR-native learning algorithms.

Main Findings

• Sparsity is a surface artifact, not zero gradients. Using a numerically robust bfloat16-aware probe (η = 10⁻³, chosen because bfloat16 has only 7 mantissa bits), the authors report SFT sparsity typically between 0.6% and 18.8%, while RL sparsity ranges from 36% to 92% across models. They note absolute levels are lower than earlier reports, motivating re-evaluation on current checkpoints with bf16-aware probes. Increasing the learning rate to scale sub-ULP updates above the representable threshold largely removes the apparent sparsity, which the authors say challenges the interpretation that sparsity stems from zero gradients.

• The bias is consistent across runs. For the 13th block of DS-Qwen-1.5B, pairwise Jaccard overlap across five independent RLVR runs (different datasets and RL algorithms) is 0.580 (Q), 0.580 (K), 0.597 (V), 0.552 (O), 0.585 (MLP-down), 0.578 (MLP-up), and 0.575 (MLP-gate), against random baselines of 0.430, 0.413, 0.467, 0.373, 0.453, 0.443, and 0.437 respectively.

• Updates land in structured stripes. Consensus maps (fraction of runs updating each coordinate) reveal contiguous row/column bands rather than random scatter, with row-wise stripes in Q/K/V projections and column-wise stripes in O projections. Temporal tracking at steps t ∈ {240, 720, 1200} (smoothed with a 3-step window) shows the routing bias emerges early and is reinforced over training. Similar stripe-structured footprints are observed on Llama and Mistral.

• RLVR preserves spectral geometry; SFT distorts it. RLVR checkpoints show consistently small principal-subspace rotation and minimal spectral drift, with singular-value profiles nearly identical to the base model, while SFT induces substantially larger rotations and pronounced drift on the same metrics (principal angles and normalized ℓ₂ spectral shift). This is analyzed on Qwen3-8B, DS-Qwen-1.5B, and Qwen3-14B-Base (SFT and RL applied separately and matched on in-domain math performance).

• RLVR avoids principal weights. RL update masks show sub-random overlap with principal weights (top-α magnitude entries of the rank-k SVD reconstruction of W₀) and super-random overlap with low-magnitude weights. The residual overlap with principal weights is largely accounted for by weights that are both principal and low-magnitude; excluding this intersection drops the overlap significantly.

• Disrupting model geometry destroys the bias. Rotating O/V layers with function-preserving orthogonal rotations and permuting heads across all Q/K/V/O layers in Qwen3-4B-Base (layer 20 with Rotate+Permute, layer 25 with Rotate) collapsed update overlap to random levels in the intervened layers while leaving untouched layers high — causal evidence that pretrained geometry is the source of the bias.

• The signature extends beyond math and code. Off-principal routing, minimal rotation, and spectrum preservation also appear in agentic checkpoints (AgentFlow, VERL-Agent, SkyRL, VERL-Tool on WebSearch, DeepSearch, and SWE) and RLHF models trained with DPO and SimPO, for example agentflow-planner-7b (layer 16, o_proj) and SkyRL-Agent-WebResearch-8B (layer 11, k_proj).

• SFT-era PEFT priors are misaligned with RLVR. Restricting updates to principal weights yields the weakest optimization trajectory and markedly degrades performance (measured by forward-KL drift and accuracy), whereas updating non-principal, low-magnitude weights closely tracks the dense RLVR trajectory. PiSSA, which explicitly targets principal weights, offers no obvious gain over standard LoRA, and enforcing principal-direction updates via learning-rate scaling often destabilizes training and precipitates early collapse.

Methodology in Plain English

The authors start by comparing released checkpoints: pairs of base and fine-tuned models spanning SFT and several RLVR variants (GRPO, DAPO, Reinforcement++) across dense and Mixture-of-Experts families. Because bfloat16 storage cannot represent changes smaller than a magnitude-dependent threshold, they define an "unchanged weight" using a relative tolerance (η = 10⁻³) that is equivalent to bitwise equality, rather than a fixed absolute tolerance.

To find where updates land, they build binary "changed" masks against the base weights for each layer and run, then measure agreement across runs with Jaccard overlap against a Bernoulli baseline, and visualize consensus ratios showing how many runs touched each coordinate. They track the same row-wise and column-wise update ratios over training steps to see when the pattern emerges.

For the theory, they reason about the standard KL-regularized RLVR objective and show that a one-step on-policy update is bounded in policy-space KL divergence, which translates into a bound on weight movement via the Fisher information. They then apply matrix perturbation results (Wedin's sin-Θ theorem) to argue that a norm-bounded weight update rotates principal subspaces only slightly, moves singular values only slightly, and nearly preserves top-k energy. A precision corollary states that a stored weight changes only when the update exceeds roughly half a ULP.

For validation, they compute principal angles between the base and fine-tuned top-k singular subspaces, a normalized ℓ₂ spectral shift, and overlap ratios between RL update masks and two masks built from the base weights: principal weights (top-α magnitude entries of the rank-k reconstruction, used as a computationally tractable proxy for high-curvature directions) and low-magnitude weights. Finally, they run intervention experiments that rotate or permute weight bases to test whether the pretrained geometry causes the observed routing.

Why This Matters

Impact on research. The authors describe their work as, to their knowledge, the first parameter-level characterization of RLVR's learning dynamics and the first parameter-space account linking RL optimization dynamics to weight evolution. It complements concurrent work focused on policy-level or distributional effects and argues that RLVR occupies a distinct, geometry-driven optimization regime from SFT — implying that repurposed SFT-era heuristics are a flawed starting point for RL-native method design.

Real-world applications (areas the findings bear on; the paper does not report deployed systems):

• Post-training pipelines for reasoning models in mathematics and code, where choosing sparse or low-rank fine-tuning strategies determines whether RL gains are preserved.
• Parameter-efficient adaptation of large models for RL, where the results warn that principal-targeted initializations and principal-direction learning-rate scaling can destabilize training.
• Tool-using and multi-turn agent training, since the off-principal signature is observed in agentic checkpoints on WebSearch, DeepSearch, and SWE tasks.
• Preference-based alignment (DPO, SimPO) for instruction following, where the same spectrum-preserving, off-principal pattern appears.

Industry relevance. The work speaks directly to the cost asymmetry between RL and SFT post-training and to engineering choices — mixed precision settings (verl keeps optimizer states and gradient reductions in float32), sparsity measurements, and low-rank adapter design — that practitioners make when scaling RL fine-tuning on large checkpoints.

Future Directions

• Design geometry-aware, RLVR-native learning algorithms. The paper explicitly calls for methods built around RLVR's off-principal dynamics rather than adapted from SFT-era PEFT, and frames this as a step toward a white-box understanding of RLVR training.
• Re-evaluate sparsity claims on current models. The authors note that absolute sparsity levels on recent checkpoints are lower than earlier reports and that bf16-aware probes are needed, implying a need to re-examine existing sparsity findings.
• Better proxies for curvature in long-chain-of-thought models. Directly quantifying curvature in large reasoning models is described as computationally prohibitive, and the paper relies on principal weights as a proxy; more direct or efficient curvature measures remain open.
• Extend the analysis beyond the models studied. The checkpoint suite covers several model families and tasks (math, code, STEM, logic puzzles, instruction following, agents, RLHF), but the theory-to-practice mapping for settings not represented in that suite is not established in the provided content.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for reasoning models, and parameter-efficient fine-tuning; the paper assumes familiarity with policy-gradient objectives, KL regularization, SVD/principal angles, and numerical precision formats. Practitioners choosing between full-parameter RL, sparse fine-tuning, and LoRA variants will find the Sec. 5 case studies most directly actionable, while theorists will be most interested in the Three-Gate derivation.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.