Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning

AI summary

Overview

• Research area: Parameter-efficient fine-tuning (PEFT) of large language models, specifically Low-Rank Adaptation (LoRA) and its variants, with an emphasis on empirical benchmarking methodology and loss-landscape analysis.
• Technical level: Intermediate. The paper assumes familiarity with LoRA, learning rates, and basic optimization concepts, and introduces a second-order (Hessian eigenvalue) analysis that is explained from first principles.
• Scope: A systematic re-evaluation of ten LoRA-based fine-tuning methods under unified, extensively tuned hyperparameters, showing that reported advantages over vanilla LoRA largely disappear once learning rates are properly calibrated.

What This Paper Is About

Many recent LoRA variants — new initialization schemes, architecture changes, and optimizer adjustments — claim substantial improvements over vanilla LoRA. The authors argue these claims often rest on training configurations inherited from prior work or tuned only in narrow ranges, rather than on method-specific hyperparameter searches. This paper re-runs nine representative LoRA variants head-to-head against vanilla LoRA with broad searches over learning rate, batch size, rank, and training duration, and explains the remaining differences through Hessian analysis.

Key Contributions

1. A hyperparameter-tuning audit of the prior literature. The authors collect 54 LoRA publications from major AI conferences and journals over the past three years, plus 10 additional high-impact or recently released preprints, for a total of 64 prior studies. They report that only 1 of the 64 papers simultaneously considers three key hyperparameters (learning rate, batch size, rank), fewer than 30% tune the learning rate, and 46 present results under a fixed learning rate.
2. A large-scale unified re-evaluation of nine LoRA variants plus vanilla LoRA across learning rate, batch size, rank, and training duration, spanning mathematical reasoning, commonsense reasoning, code generation, and instruction following on models from 0.6B to 13B parameters.
3. A demonstration that peak performance converges. Once learning rates are tuned per method, all methods reach similar peak performance within 1–2%, with only subtle rank-dependent behaviors; different methods favor distinct optimal learning rate ranges, and improper learning rates "give a false sense of LoRA advancements."
4. A second-order explanation and five practical heuristics. Hessian eigenvalue analysis links each method's optimal learning rate to its sharpness at initialization, and the authors derive five heuristics covering learning rate, batch size, rank, and training duration interactions.

Main Findings

• Convergence of peak performance. In Figure 1, Qwen3-0.6B fine-tuned on mathematical reasoning with rank 128 and batch size 64, across three orders of magnitude of learning rate, shows all ten methods achieving accuracies within a narrow 0.84% range. The top method (GraLoRA) leads the runner-up (LoRA-GA) by only 0.02% and the least effective method (OLoRA) by 0.84%, averaged over three training runs.
• Distinct optimal learning rate ranges. Different methods operate in different learning rate regimes — Figure 1 shows a 10× difference between PiSSA and LoRA. OLoRA and LoRA-GA require learning rates 18.2× lower than LoRA.
• Similarity holds across scales and tasks. Performance gaps across all methods are 0.52% for Gemma-3-1B on math, 0.43% and 1.75% for Llama-2-7B on math and code, and 1.81% for Llama-2-13B on math.
• Learning rate tuning outweighs batch size tuning. In Table 1 (Gemma-3-1B, math, rank 128), fixing PiSSA's learning rate at 2×10⁻⁵ and tuning only batch size yields a suboptimal maximum accuracy of 16.1%; fixing batch size to 16, 64, or 128 and tuning the learning rate reaches around 20.6%. Optimal learning rate scales proportionally with batch size, consistent with the SGD "scaling rule."
• PiSSA stays effective at large learning rates. On Llama-2-7B (Figure 3), at η = 1.1×10⁻³ PiSSA maintains 27.83% and 26.90% accuracy on math and code, while other methods collapse to near-zero on at least one task.
• Rank-dependent fluctuations within narrow margins. On Gemma-3-1B with batch size 64 (Figure 4), maximum performance differences among methods are 1.67% (Math, r=32) and 2.15% (Code, r=4). PiSSA underperforms LoRA by up to 1.67% at low ranks (r ≤ 32), narrows the gap to within 0.11% at r=64, and gains 0.22% and 0.33% at r=128 and 256. MiLoRA shows the opposite trend, outperforming at lower ranks but not sustaining it. Init[AB] peaks at medium ranks with gains of 0.52% (math) and 1.26% (code) at r=128. DoRA gains specifically in low-rank regimes, peaking at 1.1% (math) and 0.95% (code) at r=8.
• Sharpness explains the learning rate differences. Figure 5 shows the distribution of top Hessian eigenvalue ratios relative to LoRA for Query projection matrices across Transformer layers on Qwen3 at rank 128. All initialization variants start in a higher-curvature state than vanilla LoRA: OLoRA and LoRA-GA up to 100× higher (matching their 18.2× lower optimal learning rate), PiSSA approximately 10× higher (matching its 10× lower learning rate), and Init[AB] and MiLoRA approximately 2× higher (matching lower optimal learning rates by factors of 1.8× and 3.2×).
• Acknowledged counterexample. The authors note LoRA-One, which initializes adapters via the SVD of the one-step full gradient, originally reported approximately 2% performance improvement on Llama under comprehensive learning rate sweeps (citing Zhang et al., Table 3).

Methodology in Plain English

The authors first survey how the field actually tunes hyperparameters, counting how many recent LoRA papers search over learning rate, batch size, and rank. They then build one shared experimental framework so every method is trained and evaluated identically: four decoder-only models (Qwen3-0.6B, Gemma-3-1B, Llama-2-7B, Llama-2-13B) and four task families (commonsense reasoning with 15k examples compiled by Hu et al.; mathematical reasoning with 100k subsampled MetaMathQA examples evaluated on GSM8K and MATH; code generation with 104k subsampled CodeFeedback examples evaluated on HumanEval and MBPP; instruction following with 52k Alpaca examples evaluated with IFEval).

Batch sizes are drawn from {16, 32, 64, 128, 256, 512} and ranks from {4, 8, 16, 32, 64, 128, 256}. Learning rates are swept on a logarithmic scale from 10⁻⁶ to 10⁻³ with four values per order of magnitude (1.1247×10*, 2.0000×10*, 3.5566×10*, 6.3246×10*), giving up to 16 grid points. Learning rates are tuned for every model–task combination, while batch size and rank are tuned only for selected combinations to keep computation feasible. The scaling factor is fixed by setting α equal to r, making γ_r = 1, and other settings such as epoch, adapter placement, and scheduler are held constant. Each configuration is run three times and reported as mean ± standard deviation.

To explain why methods prefer different learning rates, the authors compute the loss Hessian with respect to the trainable LoRA parameters on MetaMathQA at the initialization point, estimating the top eigenvalue block-wise with the Lanczos algorithm and Hessian-vector products rather than forming the full Hessian.

Why This Matters

This work challenges a common evaluation pattern in PEFT research: claiming methodological superiority based on a single or narrowly tuned training configuration. It argues that fair comparison requires method-specific tuning, and that once this is done, vanilla LoRA remains a competitive baseline — potentially indicating that weight-based low-rank adaptation strategies are approaching saturation. The five practical heuristics are intended to help practitioners with limited compute avoid exhaustive searches.

Real-world applications:

• Domain adaptation of LLMs such as medicine and finance, where full-parameter fine-tuning is prohibitively expensive in memory and computation, making PEFT method choice consequential.
• Mathematical reasoning assistants, where the paper fine-tunes on MetaMathQA and evaluates on GSM8K and MATH.
• Code generation tools, using CodeFeedback training data with HumanEval and MBPP evaluation.
• Instruction-following and commonsense question-answering assistants, trained on Alpaca and the 15k-example commonsense suite and evaluated with IFEval.

Industry relevance: organizations that fine-tune LLMs under compute budgets must decide which LoRA variant to use and how to allocate tuning effort. The paper's heuristics suggest prioritizing learning rate tuning over batch size tuning, and using Hessian eigenvalue measurements to predict a reasonable learning rate range, which can reduce search cost before committing to full sweeps.

Future Directions

• Scaling beyond 13B parameters. The study covers decoder-only LLMs from 0.6B to 13B, so whether the findings hold for larger foundation models remains unverified.
• Tuning secondary hyperparameters. Learning rate schedulers, warmup steps, and LoRA adapter placements were held fixed; fine-grained tuning of these could yield further gains or distinct convergence behaviors.
• Extending to other architectures and variants. Findings may not transfer to untested architectures such as encoder-only LLMs, Vision Transformers, and Vision-Language Models, or to all existing LoRA variants.
• Analyzing Hessian evolution for non-initialization variants. The current sharpness analysis focuses on initialization-based variants; because architecture modifications and optimization adjustments may evolve distinct Hessians during training, the authors defer that investigation to future work.
• Exploring alternative adaptation mechanisms. The authors suggest hidden representation fine-tuning and approaches that adapt non-linear functions within layers as directions that may unlock new efficiency dimensions beyond weight-based low-rank adaptation.

Target Audience

PEFT and LLM fine-tuning researchers who benchmark LoRA variants; practitioners selecting a LoRA method under limited compute and needing practical hyperparameter guidance; and reviewers or meta-researchers interested in evaluation methodology, statistical fairness, and reproducibility in machine learning. Readers need basic familiarity with LoRA, learning rates, and gradient-based optimization to follow the empirical sections, while the Hessian analysis is presented accessibly through the classical inverse relationship between sharpness and optimal learning rate.

Authors’ abstract

Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model (LLM) fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies, architectural modifications, and optimization adjustments, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate nine representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches over learning rate, batch size, rank, and training duration. Across tasks spanning mathematical reasoning, commonsense reasoning, code generation, and instruction following at diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under a single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.