Normalized Low-Rank Adaptation

AI summary

Overview

Research area: Parameter-efficient finetuning (PEFT) of large language models, specifically the optimization dynamics and initialization of Low-Rank Adaptation (LoRA).

Technical level: Intermediate. The method itself is a one-line change to a forward pass, but Section 3.3 develops a preconditioning argument that assumes comfort with gradient derivations, low-rank factorization, and curvature-based preconditioning.

Scope: The paper proposes Normalized LoRA (NoRA), which constrains the down-projection matrix of LoRA to unit-norm columns along the rank dimension, and evaluates it across LLM pretraining, supervised finetuning, and reinforcement learning with verifiable rewards.

What This Paper Is About

LoRA initializes the up-projection matrix B to zero, which means the gradient with respect to A is exactly zero at the start of training and the entire early optimization is dictated by the randomly initialized down-projection A. The paper asks whether regularizing A — specifically, controlling the scale of each of its columns — is a useful and underexplored design dimension for improving LoRA. The goal is to fix LoRA's imbalanced per-coordinate learning rates without adding trainable parameters, without inference-time cost, and without giving up LoRA's property that the adapter can be merged exactly into the pretrained weights.

Key Contributions

1. Identification of a neglected design axis. The paper isolates the down-projection matrix A as a previously underexplored but important design dimension in LoRA, showing that its magnitude structure shapes early optimization dynamics, training efficiency, and downstream performance.

2. NoRA and NoRA-init. It proposes NoRA, which normalizes each column of A along the LoRA rank dimension in the forward pass (Δy = α B Norm(A) x), and NoRA-init, which applies the same normalization once at initialization and then trains with standard LoRA parameterization. Neither adds trainable parameters or inference-time computation, and both keep the update linear in the input so the adapter remains exactly mergeable.

3. BIMI, a deterministic structured instance. It introduces Block Identity Matrix Initialization (BIMI), which builds A from repeated identity blocks (plus a truncated identity block when the dimension does not divide evenly), satisfying the unit-column-norm requirement by construction. It also shows that the existing MiSS method is a special case of NoRA with a fixed block-identity down-projection.

4. Empirical validation across three training regimes. It reports experiments in pretraining (MLA and MHA architectures), supervised finetuning (Llama-3.2-3B), and RLVR (DeepSeek-R1-Distill-Qwen1.5B), plus ablations over normalization dimension and initialization distribution.

Main Findings

• Early optimization is governed by A alone. At initialization, the gradient with respect to B is α G (A⁽⁰⁾)ᵀ while the gradient with respect to A is exactly zero, so the input-to-latent projection determines the first phase of training.

• LoRA's hidden preconditioner is diagnosable. The induced merged-weight update is ΔW = −η G P with P = α² Aᵀ A ∈ ℝ^{k×k}. The diagonal entries α²‖a_j‖² act as per-coordinate learning rates, and off-diagonal entries (a_iᵀa_j) act as crosstalk induced by the rank bottleneck.

• Random initialization makes learning rates too small and uneven. With zero-mean i.i.d. entries of variance σ² ∝ 1/k, the expected squared column norm is r σ² ∝ r/k ≪ 1, so the adapter receives a small fraction of the full-finetuning rate. Column norms also fluctuate with relative spread of order 1/√r, meaning at small rank each coordinate's learning rate is assigned randomly and unrelated to data or curvature.

• Rank-dimension normalization fixes the diagonal. Setting every ‖a_j‖₂ = 1 at unit scaling (with α = 1, or equivalently α = r under the α/r convention) makes Diag(P) = I deterministically and E[P] = I over random directions, so the adapter's gradient norm can match full finetuning's in expectation, independently of r.

• The normalization dimension matters, the initialization distribution does not. In the ablation (Table 3, GSM8K / Math / Average), column-wise normalization Norm_r consistently improves over no normalization across three initialization distributions: 𝒰(−1/√k, 1/√k) goes 29.27 → 37.16 average, 𝒩(0, 1/r²) goes 31.05 → 36.95, and 𝒰(−1, 1) goes 29.61 → 36.66. Row-wise Norm_k provides little improvement (29.27 → 29.75, 31.05 → 30.54, 29.61 → 29.58). After normalization all three schemes land near 37, showing the benefit is largely independent of the initialization distribution. BIMI reaches 37.07, comparable to Norm_r.

• Row normalization adds noise rather than removing it. The paper reports that Norm_k leaves Diag(P) random of order r/k and brings no empirical gain, while BIMI uses unit basis vectors (the same diagonal, a different crosstalk pattern) and performs similarly to normalized random initializations, pointing to the diagonal of P as the decisive quantity.

• Initialization does most of the work; the constraint adds stability. Because A's gradient vanishes with B, correcting the rates once at initialization captures most of the benefit. Persistent normalization additionally makes the loss invariant to each column's scale (so gradients are tangential and effective steps anneal automatically) and enforces Diag(P) = I for the whole run on a compact set of directions.

• Pretraining with NoRA-init. On MLA with 10B training tokens and 0.5M batch-size tokens at 342.4M parameters, NoRA-init improves LAMBADA perplexity from 61.83 (standard low-rank) to 50.77, and improves the commonsense average from 40.36 to 40.64. Directly normalizing the latent feature B Norm(Ax) achieves the best LAMBADA perplexity (48.73) and average (41.17) but is not mergeable. On MHA at rank r = 64 for q_proj, k_proj, v_proj, and o_proj, standard low-rank parameterization collapses (LAMBADA and WikiText reported as "-" and a commonsense average of 27.81, with ARC-easy at 0.00), whereas NoRA-init reaches a LAMBADA perplexity of 63.45 and an average of 39.69.

• Supervised finetuning gains. On Llama-3.2-3B, NoRA raises the average SFT score from 37.93 with standard LoRA to 43.37, a gain of 5.44 points, with GSM8K at 61.63 and HumanEval at 42.10. NoRA-init alone reaches 42.38, capturing a large portion of the gain. NoRA also outperforms PiSSA (40.26), OFT (40.43), RSLoRA (41.28), and MiSS (42.70) on average. Applying the same normalization to DoRA improves its average from 38.30 to 41.40 for NoRA-init.

• Better retention of pretrained knowledge. NoRA shows an average change of +0.02 across MMLU, AGIEval, and ARC-C, compared with −0.56 for LoRA, −0.70 for MiSS, −0.51 for RSLoRA, −0.28 for PiSSA, and +0.52 for OFT.

• RLVR results and robustness. Under RLVR with verifiable rewards on DeepSeek-R1-Distill-Qwen1.5B, NoRA raises the overall average from 41.0 (base model) to 44.4, versus 42.8 for standard LoRA — improvements of 3.4 and 1.6 points respectively — with clear gains on AMC (72.7 vs 68.0), MATH500 (84.5 vs 81.3), and Minerva (32.0 vs 30.1). Spectral initialization methods degrade sharply in this setting: MiLoRA averages 18.0 and PiSSA averages 0.2.

Methodology in Plain English

The researchers start from an architectural cue rather than a purely algebraic one. Multi-head Latent Attention (MLA) normalizes the latent representation before it is projected back out, and that normalization is associated with more stable and faster training. Importing that operation directly into LoRA would be Norm(Ax), but because the normalization depends on the input, the result is nonlinear and can no longer be merged into the pretrained weight matrix.

The move that defines the paper is to transfer the scale-control effect from the latent feature to the projection itself: instead of normalizing the output A x, normalize the matrix A. Every column of A corresponds to an input coordinate projected into the rank-r latent space, so normalizing those columns to unit L2 norm means every input coordinate is always projected through a unit-norm direction. The result is linear in the input and therefore exactly mergeable. A small constant ε is included in the denominator for numerical stability, and the matrix is viewed with columns a_1, …, a_k ∈ ℝ^r.

The authors then offer two ways to apply this. NoRA normalizes A on every forward pass, adding a constraint throughout training. NoRA-init normalizes A once at initialization (A⁽⁰⁾ = Norm(A_init), B⁽⁰⁾ = 0) and then trains with ordinary LoRA, requiring no modification to the optimizer or training loop. BIMI replaces random initialization with a block-identity construction that satisfies unit column norms by construction and remains fully trainable.

To explain why this helps, the authors derive the effective weight update for LoRA at initialization and show it is full finetuning with an input-side preconditioner P = α² Aᵀ A. They decompose this into diagonal terms (each coordinate's own update, scaled by the squared length of its latent vector) and off-diagonal terms (crosstalk between coordinates introduced by the rank bottleneck). Random initialization makes the diagonal both far below one and randomly dispersed, which they measure empirically as gradient norms far smaller than full finetuning. Normalization sets the diagonal to one exactly.

Evaluation proceeds in three stages. Pretraining uses the FLA repository on the MLA and MHA architectures with SlimPajama at 10B tokens and 0.5M batch-size tokens, evaluated with lm-evaluation-harness. Supervised finetuning uses PEFT-Arena with Llama-3.2-3B on MetaMath and CodeFeedback, evaluated on GSM8K, Math500, HumanEval, and MBPP, with MMLU, AGIEval, and ARC-C used to measure catastrophic forgetting. Reinforcement learning uses the PeRL framework with DeepSeek-R1-Distill-Qwen1.5B on open-r1/DAPO-Math-17k-Processed, evaluated with Avg@32 and Avg@4 metrics on AIME24, AIME25, AMC, HMMT, MATH500, and Minerva. An ablation sweeps the normalization dimension (row versus column) across three initialization distributions.

Why This Matters

Impact on research. The paper reframes LoRA's behavior as a preconditioning problem, giving a concrete analytical reason (the diagonal of P = α² Aᵀ A) why some LoRA variants work and why random initialization is weak. It also connects several previously separate methods: MiSS becomes a fixed block-identity special case of NoRA, and MLA's normalization becomes the architectural inspiration for a linear, mergeable analogue. Because the intervention is a one-line change that adds no parameters and no inference cost, it is directly composable with the many LoRA variants that modify scaling, parameterization, or rank allocation.

Real-world applications.

• Domain adaptation of open-weight LLMs (medical, legal, scientific) where retaining general capability matters as much as acquiring the new task, given NoRA's +0.02 average retention change versus −0.56 for LoRA.
• Code assistants and math tutors built by finetuning mid-size models such as Llama-3.2-3B, where NoRA reports HumanEval 42.10 and GSM8K 61.63.
• Post-training reasoning models via RLVR, where spectral initialization methods collapse (PiSSA averaging 0.2) but NoRA remains stable at 44.4 average.
• Large-scale pretraining or continued pretraining on limited compute, where the MHA collapse of standard low-rank parameterization is avoided.

Industry relevance. The method requires no SVD of pretrained weights, no activation statistics, no gradient estimation, no extra data, and no test-time overhead, and the adapter still merges into the base weights. That combination matters for teams serving many adapters per base model under tight latency budgets, and for training pipelines where an SVD pass over every weight matrix is impractical.

Future Directions

• Scope of the preconditioning account. The theory explains the early phase, while the authors note the neglected term α B ΔA = −η α² B Bᵀ G is second order in B. How the picture changes once B grows is not

Authors’ abstract

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.