GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR

AI summary

Overview

Research area: Parameter-efficient fine-tuning (PEFT) for large language models, specifically low-rank adaptation methods designed for Reinforcement Learning with Verifiable Rewards (RLVR).

Technical level: Advanced. The paper assumes familiarity with LoRA-style adapters, Singular Value Decomposition, spectral analysis of weight matrices, and reinforcement learning post-training (GRPO).

Scope: The paper proposes GeoRA, a geometry-aware low-rank adaptation method for RLVR, and evaluates it against LoRA, PiSSA, MiLoRA, SparseFT and FullFT on Qwen and Llama models from 1.5B to 32B parameters across mathematical, medical and coding reasoning tasks.

What This Paper Is About

Existing parameter-efficient fine-tuning methods are largely built for supervised fine-tuning (SFT), where updates tend to modify the dominant principal directions of the pre-trained weights. RLVR behaves differently: its useful updates are geometrically biased toward low-magnitude directions that preserve pre-trained structure. This creates a mismatch — SFT-oriented low-rank methods such as PiSSA encode the wrong inductive bias, while sparse fine-tuning methods that better match RLVR dynamics get little real speedup from unstructured sparsity on modern hardware.

GeoRA's goal is to build a low-rank adapter whose initialization is aligned with the actual geometry of RLVR updates, while keeping dense, hardware-friendly computation.

Key Contributions

1. A geometry-aware low-rank adaptation framework for RLVR. GeoRA combines an RL-native structured low-rank parameterization with an explicit frozen residual anchor, targeting both the geometric mismatch of SFT-oriented low-rank methods and the efficiency bottleneck of sparse methods.

2. Evidence that the effective RL update subspace is directional and compressible. The authors show the RLVR update subspace is anisotropic with a heavy-tailed spectrum, and use SVD within a geometry-constrained subspace to initialize the adapters' principal directions, with the frozen residual acting as a structural anchor.

3. A masking-based construction of the geometry-constrained matrix. Two priors — a Spectral Prior selecting the bottom ρ-fraction of entries from the rank-r approximation, and a Euclidean Prior selecting low-magnitude weights in the original weight space — are combined by union to define the adaptation target.

4. Broad empirical validation. Experiments on Qwen and Llama models from 1.5B to 32B parameters show consistent gains over strong low-rank baselines in mathematics, medicine and coding RLVR, with stronger out-of-domain generalization, less forgetting, and lower training cost than FullFT (99.5% fewer trainable parameters).

Main Findings

• In-distribution mathematical RLVR performance. On Qwen3-8B, GeoRA reaches 23.75 on AIME24, 21.67 on AIME25, 78.00 on MATH500 and 12.75 on OlymMATH, the strongest overall column values among the compared methods on those benchmarks. On Llama-3.1-8B it reaches 18.54, 8.75, 61.90 and 8.85 respectively.

• Out-of-distribution retention. GeoRA scores 82.93 on HumanEval, 37.92 on GPQA, 75.96 on MMLU, 53.73 on IFEval and 68.85 on TruthfulQA on Qwen3-8B, generally the strongest or near-strongest among baselines. On Llama-3.1-8B it reaches 70.80, 32.65, 70.95, 78.72 and 61.61.

• Extension to medical and coding RLVR. On medical tasks GeoRA obtains 76.12 (MedQA), 64.31 (MedMCQA) and 80.64 (PubMedQA), averaging 73.69 versus 71.96 for LoRA and 73.33 for FullFT. On coding, GeoRA averages 79.65 (LiveCodeBench 67.75, HumanEval 89.61, MBPP 81.60) versus 79.07 for LoRA and 79.80 for FullFT.

• Training stability and dynamics. On the AIME benchmark (average of 2024 and 2025) with Qwen3-8B, GeoRA stays top-performing throughout training and reaches strong performance substantially earlier than other low-rank baselines. Under an aggressive learning rate of 5×10⁻⁵ on Qwen3-4B, GeoRA keeps the highest reward trajectory without collapse and maintains low KL divergence, whereas PiSSA suffers a catastrophic late-training drop.

• Robustness to hyperparameters. GeoRA maintains high reward across a broad range of learning rates, while PiSSA and MiLoRA degrade rapidly at larger learning rates and LoRA shows a clear drop at the high end of the learning-rate sweep. Similar robustness is reported for rank (4, 8, 16, 32) and sparsity (10%, 20%, 30%, 40%).

• Efficiency. Compared with FullFT (8.00B trainable parameters, 231 s/it, 95.73% VRAM), GeoRA trains 0.04B parameters (−99.5%), 185 s/it (−19.9%) and 68.43% VRAM (−28.5%). SparseFT shrinks parameters by 68.0% but is slower per iteration (256 s/it, +10.8%) and less memory-efficient (81.25% VRAM, −15.1%) than FullFT.

• Ablation on initialization. On Qwen3-4B, GeoRA averages 25.41 across benchmarks with reward 0.88, versus 24.60 for Random-r initialization (reward 0.85) and 23.40 for Tail-r initialization (reward 0.82), showing the benefit comes from aligning initialization with the effective RLVR update subspace rather than from low-rank adaptation alone.

• Ablation on geometric masks. Removing the Spectral Prior gives 24.40 average and removing the Euclidean Prior gives 25.10, both below the full GeoRA result of 25.41, indicating the two priors are complementary.

• The two priors are largely non-overlapping. On Qwen3-8B with ρ=0.2, each mask selects 20.0% of parameters, but their overall intersection is only 4.55% with a Jaccard index of 0.128 (MLP: 4.63%, 0.131; attention: 4.25%, 0.119).

• Spectral structure. Sparse random noise has a spectrum nearly indistinguishable from dense random noise with relatively flat decay, showing sparsity alone does not induce low-rankness. The geometry-constrained matrix W_Geo preserves a heavy-tailed spectrum similar to pre-trained weights, and the FullFT update ΔW_FullFT is similarly compressible.

• Geometric mechanism metrics. For Llama-3.1-8B, GeoRA reports NSS 0.092, head-subspace alignment 0.005 and tail-subspace alignment 0.98, versus PiSSA's 0.395, 0.98 and 0.01. For Qwen3-8B, GeoRA reports 0.096, 0.015 and 0.96 versus PiSSA's 0.418, 0.95 and 0.03.

• Initialization cost is one-off. Standard SVD initialization takes 18.87 minutes at 8B, 25.78 at 14B, 35.23 at 32B and 97.82 at 72B, while randomized SVD takes 0.21, 0.26, 0.47 and 0.72 minutes respectively. For reference, Qwen3-8B training is reported at 838 minutes.

• Smaller-scale results. On Qwen2.5-1.5B the table reports GeoRA at 10.83 (AIME24), 2.50 (AIME25), 54.60 (MATH500) and 5.50 (OlymMATH); on Qwen3-4B at 13.33, 9.17, 73.40 and 5.75.

Methodology in Plain English

The starting observation is that RLVR does not change a model the way supervised fine-tuning does. RLVR tends to nudge the model in low-energy directions that stay close to what it already learned, whereas SFT tends to rewrite the dominant directions. Standard low-rank adapters and their SFT-oriented variants therefore aim at the wrong target.

GeoRA first builds a "geometry-constrained" version of each weight matrix. It does this by masking out the parts of the weights that RLVR should not touch, using two complementary filters: one that keeps entries with small absolute values in the rank-r approximation of the weights (a spectral prior aimed at stability), and one that keeps small-magnitude entries in the original weight matrix (a Euclidean prior aimed at plasticity). The union of these two masks defines the region where adaptation is allowed.

GeoRA then applies SVD to this constrained matrix and keeps the top-r singular components, splitting them across the two adapter matrices A and B. Because the initialization now captures the principal directions of the RL-relevant subspace, the adapters start in a sensible place instead of at random. A residual matrix is computed by subtracting the scaled initial adapter product from the original weights, and this residual is frozen during training. Since the frozen residual plus the trainable adapter equals the original weight at step zero, the model's output is unchanged at initialization. During training, the optimizer can only move within the rank-r manifold defined by the adapters, while the frozen residual keeps the model anchored near its pre-trained state.

All of this uses dense matrix multiplications rather than irregular sparse operations, which is what lets the parameter savings translate into actual wall-clock and memory savings. Evaluations use GRPO with rank r=16, scaling factor α=32, sparsity ρ=0.2, and adapters applied to all linear layers.

Why This Matters

Impact on research. The paper connects mechanistic analyses of RLVR (that updates are "off the principal" and geometrically constrained) to a concrete PEFT design. It challenges the assumption that initialization strategies validated under SFT transfer to RL post-training, and it supplies a reusable diagnostic toolkit — normalized spectral shift, head/tail subspace alignment, Jaccard overlap of geometric priors, and singular-value spectra — for comparing adaptation methods under RL.

Real-world applications:

• Post-training reasoning models in mathematics and science, where teams need to raise AIME-style or competition-level accuracy without destroying general knowledge.
• Domain specialization in medicine, where the paper's medical RLVR setup shows transfer to MedQA, MedMCQA and PubMedQA without the large capability regressions seen in FullFT.
• Code generation assistants, where coding RLVR on LiveCodeBench, HumanEval and MBPP is shown to keep pace with FullFT at a fraction of the trainable parameters.
• Memory- and compute-constrained training pipelines, where a 99.5% reduction in trainable parameters and 28.5% lower VRAM relative to FullFT makes larger backbones trainable on existing hardware.

Industry relevance. The efficiency numbers matter directly for cost: GeoRA updates 0.04B parameters versus 8.00B for FullFT, runs 185 s/it versus 231 s/it, and uses 68.43% VRAM versus 95.73%. The initialization overhead is one-time, sub-minute at 72B with randomized SVD, and its memory can be released before training. For teams already running GRPO-style RLVR pipelines, GeoRA is a drop-in adapter swap rather than a new training paradigm.

Future Directions

• Extending beyond reasoning domains. The authors state in their limitations that experiments focus on RLVR in reasoning tasks, and that generalizability needs validation across broader model architectures and reinforcement learning scenarios beyond verifiable rewards.

• Reducing the preprocessing cost. Truncated SVD plus dual-masking is flagged as an extra preprocessing step relative to LoRA's random initialization. Randomized SVD already cuts this by roughly two orders of magnitude, but the question of eliminating the step entirely remains open.

• Learning the geometric priors instead of hand-designing them. The spectral and Euclidean masks are built from fixed quantile thresholds at a fixed sparsity ratio. Whether these masks could be learned or adapted per layer, given their low mutual overlap (4.55% intersection, Jaccard 0.128), is not addressed.

• Interaction with KL regularization. The paper reports robustness under aggressive learning rates and low KL divergence, but how GeoRA interacts with KL-regularized RLVR objectives and the trade-offs described in prior work on exploration diversity is not empirically explored here.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning from verifiable rewards, and parameter-efficient fine-tuning. It is most useful for readers already comfortable with LoRA, SVD, and policy-gradient style training who want to understand why SFT-oriented adapter initialization underperforms in RLVR and what a geometry-aligned alternative looks like. Readers new to PEFT will find the design intuitions accessible, but the spectral analysis and mechanism metrics sections require background in linear algebra and model fine-tuning.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models. Unlike supervised fine-tuning (SFT), RLVR exhibits distinct optimization dynamics and is sensitive to the preservation of pre-trained geometric structures. However, existing parameter-efficient methods face key limitations in this regime. Low-rank adaptation methods, such as PiSSA, are primarily designed for Supervised Fine-Tuning (SFT) and do not account for the distinct optimization dynamics and geometric structures of RLVR. Conversely, directly fine-tuning the unstructured sparse parameter subspace favored by RLVR encounters efficiency bottlenecks on modern hardware. To address these challenges, we propose GeoRA (Geometry-Aware Low-Rank Adaptation), a low-rank adaptation method tailored for RLVR. Specifically, GeoRA exploits the anisotropic and compressible structure of RL update subspace, and extracts its principal directions via Singular Value Decomposition (SVD) to initialize low-rank adapters, while freezing residual components as a structural anchor during training. This design preserves the pre-trained structure and enables efficient dense computation. Experiments on Qwen and Llama models from 1.5B to 32B parameters show that GeoRA consistently outperforms strong low-rank baselines across RLVR settings in mathematics, medicine, and coding, while showing stronger generalization and less forgetting on out-of-domain tasks.