Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

AI summary

Overview

Research area: Parameter-efficient fine-tuning (PEFT) for continual learning in large language models, with a specific focus on geometrically principled merging of LoRA adapters.

Technical level: Advanced. The paper assumes familiarity with LoRA, low-rank matrix factorization, Riemannian optimization, the Stiefel manifold, and orthogonal Procrustes problems.

Scope: The paper proposes SOAR (Stiefel Optimization and Aligned Rotation), a two-stage framework that makes the LoRA factorization identifiable enough to be merged stably across tasks, and evaluates it against continual-learning and LoRA-merging baselines on three benchmark suites and three models.

What This Paper Is About

When a model is adapted to a sequence of tasks using LoRA, each task produces an update ΔW = BA. The problem is that this factorization is non-identifiable: for any invertible matrix R, (BR)(R⁻¹A) gives the exact same update, so two adapters trained on two tasks can live in completely different latent coordinate systems. Merging them directly then mixes up directions that are not geometrically comparable, producing interference that looks like task conflict but is actually coordinate mismatch.

The paper's goal is to remove that ambiguity so that a single shared adapter can be consolidated task after task, preserving old knowledge while learning new tasks, and without memory growing linearly with the number of tasks.

Key Contributions

1. SOAR framework for single-adapter continual LoRA learning. Consolidates knowledge at constant memory cost — adapter memory stays at O((m+n)r) instead of growing as O(T(m+n)r) — by explicitly targeting representational misalignment rather than treating interference as task conflict.

2. Stiefel constraint on the input factor A. Constraining A to the row-Stiefel manifold (AAᵀ = I_r) removes the general linear invariance of the factorization. Lemma 3.1 proves that any remaining invertible transformation R between two Stiefel-constrained A matrices must satisfy RRᵀ = I, i.e., R ∈ O(r). This reduces ambiguity from GL(r) to orthogonal rotations only, yielding a stable shared coordinate system across tasks.

3. Alignment-aware merging for the output factor B. The previously merged B is rotated into the current task's coordinate frame using Orthogonal Procrustes alignment (R* = UVᵀ from the SVD of B_newᵀ B_old), followed by rank-wise, forgetting-aware interpolation driven by directional agreement and magnitude drift.

4. Empirical validation. SOAR is evaluated across three continual-learning benchmarks, three models from two model families, and eight task orders, outperforming both sequential/single-adapter baselines and offline LoRA-merging methods.

Main Findings

• SOAR leads on every benchmark with Qwen3-8B. Average accuracy: 89.48 on the Standard CL Benchmark, 70.96 on Large Number of Tasks, and 51.59 on SuperNI. The strongest baseline, SLAO, reaches 85.99, 68.49, and 47.84 respectively.

• Unconstrained sequential fine-tuning is the weakest setting. SeqLoRA scores 78.02, 61.19, and 30.43 average accuracy — worst or near-worst across all three suites — attributed to severe catastrophic forgetting from a single shared adapter.

• Offline merging methods transfer poorly to the continual setting. KnOTS scores 76.80, 59.41, and 27.03, underperforming several continual-learning baselines because it assumes simultaneous access to all task adapters. LoRA-LEGO is more competitive (82.64, 67.42, 41.02) but treats all rank directions equally and has no forgetting-aware consolidation, which limits it most on SuperNI (41.02 vs. SOAR's 51.59).

• Stiefel optimization alone gives only a modest gain; it is a precondition, not the whole solution. Replacing QR-based initialization with Stiefel optimization under a λ-merge rule moves the Standard CL average from 85.99 to 86.21 and Large Number of Tasks from 68.49 to 69.13. Adding SOAR merging on top of QR initialization yields 87.67, 70.02, and 49.29 — but the full combination (Stiefel + SOAR merging) gives 89.48, 70.96, and 51.59.

• Procrustes alignment removes a measurable confound. With a uniform interpolation coefficient, Procrustes alignment alone recovers 0.52 BWT points on task orders O7–O8 (−5.10 → −4.58), indicating a component of interference that is independent of task conflict.

• Uniform merging degrades as task sequences grow. The gap between Procrustes + λ-merge and full SOAR grows with sequence length: 0.35 on O1–O3, 1.60 on O4–O6, and 1.67 on O7–O8.

• Both gate signals are necessary. With identical merge skeleton and Procrustes alignment, a random gate averages 61.41 (−9.21 relative to full SOAR), cosine-only averages 65.26 (−5.36), residual-only averages 68.09 (−2.53), and the full multiplicative gate averages 70.62.

• Gains hold across model families and sizes. SOAR beats SLAO on Llama-3.2-3B (86.48 vs. 82.51; 62.98 vs. 59.92; 47.96 vs. 45.79), on Llama-3.1-8B (89.72 vs. 87.32; 68.90 vs. 66.60; 50.91 vs. 47.84), and on Qwen3-8B. The SuperNI gains are +3.07 for Llama-3.1-8B and +3.75 for Qwen3-8B.

• CorDA is uneven. It is competitive on Standard CL (83.25) and Large Number of Tasks (67.10) but drops sharply on SuperNI (30.33), which the authors attribute to unreliable nullspace selection under diverse instruction-following tasks.

Methodology in Plain English

Step 1 — Force A onto a rigid basis. Instead of letting the low-rank input factor A drift anywhere in matrix space during fine-tuning, the authors keep it on the Stiefel manifold, so its rows form an orthonormal set (AAᵀ = I_r). This is done during training with Riemannian Adam rather than by projecting A afterwards. The authors argue post-hoc projection is insufficient because it breaks the coupling between A and B that was established during joint optimization, causing alignment errors to compound across tasks. The effect of the constraint is to shrink the space of transformations that can silently relabel the latent coordinates from all invertible matrices down to rotations only.

Step 2 — Align B before merging it. Only B is consolidated across tasks; A is simply carried over from the current task's fine-tuning. Before blending the new B with the previously merged B, the older one is rotated into the new one's coordinate frame using the closed-form orthogonal Procrustes solution: take the thin SVD of B_newᵀ B_old = UΣVᵀ and set R* = UVᵀ.

Step 3 — Decide per rank direction how much to keep. Rather than using one global interpolation weight, SOAR computes a retention score for each rank direction. It measures (a) cosine similarity c_i between the aligned old column and the new column, capturing directional agreement, and (b) a normalized residual δ_i, capturing how far the current task has moved that direction relative to its original scale. These combine into a gate γ_i = max(c_i, 0)·exp(−δ_i/τ), which is high only when a direction is both positively aligned and barely altered. The merged factor is B_new + (B̃_old − B_new)Γ, where Γ is the diagonal matrix of gates. τ, which controls sensitivity to magnitude drift, is set to 0.5.

Experimental setup. LoRA rank 32, alpha 64, learning rate 5e-5, targeting the q, k, v, and o modules, on a single H100 80GB GPU. Each configuration is run three times on seeds {42, 43, 44}. The Large Number of Tasks benchmark uses 2500 randomly sampled training samples and 500 test samples per class; SuperNI uses 1000 training instances and 100 for validation/testing per task. Evaluation uses ACC (final accuracy averaged over all tasks) and BWT (average accuracy change per task after learning new tasks).

Why This Matters

The paper reframes a practical failure mode in continual learning — adapters that interfere when merged — as a geometric identifiability problem rather than a data or task-similarity problem. If the diagnosis is right, it means a whole class of merging instabilities can be fixed by constraining the parameterization, not by adding memory or routing machinery. This matters because single-adapter continual learning is the only variant that keeps inference cost constant as task count grows.

Real-world applications:

• Continuously updated assistant models. A deployed assistant that must absorb new instruction-following behaviors over time without losing earlier capabilities, while keeping a single adapter for cheap serving.
• Domain adaptation in regulated enterprises. Incrementally specializing a base model to new document types, customer verticals, or languages where retaining past task-specific adapters is undesirable or costly.
• Edge and on-device deployment. Fixed O((m+n)r) adapter memory is a hard constraint on hardware where storing one adapter per task is infeasible.
• Multi-task composition without joint training. Task sequences that arrive in arbitrary order and at arbitrary times, where re-running a full multi-task fine-tune is impossible.

Industry relevance: The work comes from Sony Research India and targets the deployment economics of PEFT — the authors explicitly frame linear memory growth as the reason per-task adapters are impractical. Beyond the headline accuracy, the ablation showing that Stiefel training (not post-hoc projection) is a precondition for well-conditioned alignment is directly actionable for teams already running single-adapter merge pipelines.

Future Directions

• Characterizing the rank and τ trade-offs more fully. The paper reports that smaller ranks cause modest degradation on Large Number of Tasks and SuperNI, that excessively large ranks overfit, and that r = 32 is the best trade-off, with a separate study on τ. The provided content truncates at this point, so the full D.1 and D.2 results are not reported here.

• Determining when the asymmetry assumption fails. The framework relies on the observation from prior work that A stays relatively stable while B carries most task-specific adaptation, supported by an ablation in Appendix D.6. Whether this holds for architectures or task types where input-side subspaces shift substantially is left open.

• Reducing the cost of constrained optimization. Training on the Stiefel manifold with Riemannian Adam is more involved than standard optimization; whether this overhead can be reduced while preserving the conditioning benefits of R* is not reported.

• Extending identifiability arguments beyond LoRA. The same GL(r)-to-O(r) reduction could in principle apply to other low-rank or factorized PEFT parameterizations, but the paper does not explore this.

Target Audience

• Continual-learning and PEFT researchers who want a geometric framing of adapter merging and a concrete identifiability argument (Lemma 3.1) they can build on.
• Machine learning engineers deploying LoRA at scale, for whom the constant-memory single-adapter result and the practical precondition of Stiefel training during fine-tuning are the actionable takeaways.
• Graduate students and advanced practitioners comfortable with manifold optimization, SVD-based alignment, and low-rank factorization, since the paper does not explain these foundations from scratch.
• Model-merging researchers interested in why offline merging methods such as KnOTS and LoRA-LEGO transfer imperfectly to the sequential, data-free continual setting.

Authors’ abstract

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.