KeepLoRA: Continual Learning with Residual Gradient Adaptation

AI summary

Overview

Research area: Continual learning (CL) for pre-trained vision-language models, specifically parameter-efficient adaptation of large models without forgetting.

Technical level: Advanced. The paper relies on singular value decomposition (SVD), subspace projections of gradients and weights, low-rank adaptation (LoRA), and two formal propositions.

Scope: This paper (arXiv:2601.19659v1 [cs.CV], 27 Jan 2026, CC BY 4.0, code at https://github.com/MaolinLuo/KeepLoRA) proposes KeepLoRA, a LoRA-based continual learning method that confines parameter updates to the residual subspace of pre-trained weights to jointly preserve pre-trained knowledge, retain previous tasks, and acquire new tasks.

What This Paper Is About

When a pre-trained vision-language model is continually trained on a sequence of narrow, specialized datasets, it tends to lose the broad general knowledge that made it useful in the first place. The paper argues that CL must balance three competing goals at once: plasticity (learning new tasks), backward stability (not forgetting earlier learned tasks), and forward stability (not degrading the pre-trained knowledge that guarantees general transferability).

KeepLoRA's goal is to infuse new task knowledge directly into the existing parameter space — without replaying pre-training data, without storing reference data, and without adding extra modules that raise inference cost — by learning only in directions that do not disturb existing knowledge.

Key Contributions

1. An empirical analysis of where knowledge lives in the parameter space. Using SVD on backbone attention weights, the authors find that general knowledge is primarily encoded in the principal subspace (large singular values) and domain-specific knowledge in the residual subspace (small singular values).

2. The KeepLoRA method. A LoRA-based continual learner that initializes the down-projection matrix from a gradient projected onto a subspace orthogonal to both the pre-trained principal subspace and the dominant feature directions of previous tasks, then freezes it and trains only the up-projection matrix.

3. Theoretical validation. Proposition 3.1 shows that LoRA with a frozen down-projection matrix is equivalent to gradient descent constrained to that matrix's span; Proposition 3.2 shows KeepLoRA's initialization solves a constrained optimization problem that enforces orthogonality (stability) while maximizing the projected gradient norm (plasticity).

4. Experiments on two model families. Results on dual-encoder CLIP (MTIL benchmark) and encoder-decoder LLaVA (MLLM-DCL and UCIT benchmarks), reporting Transfer, Average, and Last scores, with a "KeepLoRA+" architecture-extension variant.

Main Findings

• General knowledge is robust to removing low-energy components; domain knowledge is not. In Figures 1(a) and 1(b), reconstructing attention weights from only the top principal singular components leaves general-domain performance highly robust, while performance on most specific-domain datasets degrades sharply as more low-energy components are removed.

• KeepLoRA leads the "architecture-kept, no extra data" group on MTIL. On the classification tasks of MTIL (Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, Cars, Sun397), KeepLoRA reports an Average of 77.5% versus InfLoRA 76.3%, SD-LoRA 72.5%, O-LoRA 73.3%, and ZSCL 75.4%.

• Forward forgetting appears largely avoided. The zero-shot Transfer average is 65.4%, while KeepLoRA reaches 69.0% and the architecture-extension variant KeepLoRA+ reaches 69.9% — i.e., accuracy on unseen tasks improves relative to the untrained baseline rather than degrading. IAP reports 69.2%, MoE-Adapters 68.9%, ZSCL 68.1%.

• Strongest Last scores. KeepLoRA reports a Last average of 86.1%, the highest among methods that keep the architecture and use no extra data, ahead of InfLoRA (85.2%), while KeepLoRA+ reports 88.2% overall, ahead of IAP (85.7%) and MoE-Adapters (85.0%).

• Gains on LLaVA VQA benchmarks. On MLLM-DCL, KeepLoRA reports Transfer 33.71%, Average 54.19%, and Last 64.41%, versus SEFE at 33.49%, 51.86%, and 59.91%. On UCIT, KeepLoRA reports Transfer 28.40%, Average 55.37%, and Last 67.84%, versus SEFE at 27.48%, 54.29%, and 66.48%.

• Interference across tasks is suppressed. In Figure 2's heatmaps of the average L2 norm of the LoRA module's output, standard LoRA, LoRA with a frozen A, and LoRA with a gradient-informed frozen A all show bright off-diagonal cells and bright vertical bars (high interference and high backbone impact). KeepLoRA shows a bright diagonal with dark off-diagonal cells and a consistently dark vertical bar — updates focus on the current task with minimal impact on others.

• Plasticity is preserved. Figure 3 compares KeepLoRA against an unconstrained LoRA, showing that KeepLoRA keeps stability with minimal sacrifice in adaptive capability.

Methodology in Plain English

1. Diagnose the parameter space. The authors decompose each pre-trained attention weight matrix W into U S Vᵀ. Directions with large singular values form the principal subspace W_p, retained to satisfy an energy ratio threshold ϵ_w in (0, 1) via ‖W_p‖²_F ≥ ϵ_w‖W‖²_F. Directions with small singular values form the residual subspace.

2. Build a unified principal subspace. After each task, the model extracts the dominant feature directions of that task — computed from X̂_t = X_t − W_p W_pᵀ X_t − M_{t−1} M_{t−1}ᵀ X_t, with the number of retained directions k set by an energy threshold ϵ_f — and appends them to an accumulated matrix M_t. Since W_p and M_t are both sets of orthogonal direction vectors in the same d_in-dimensional feature space, they are concatenated into one subspace M′_t = [W_p, M_t]. The paper states this unified subspace has a size not exceeding the square of the feature dimension.

3. Initialize LoRA in the residual subspace. For a new task, the first-step gradient G_t is projected as Ĝ_t = G_t − W_p W_pᵀ G_t − M_{t−1} M_{t−1}ᵀ G_t. The first term preserves plasticity; the subtracted terms remove gradient components that would damage pre-trained and previous-task knowledge. SVD of Ĝ_t gives A = U_{:,1:r} and B = S_{1:r} Vᵀ_{:,1:r}. The matrix A is frozen; only B is trained.

4. Keep the initial forward pass identical. Because A·B is non-zero at initialization, the frozen base weight is adjusted to W′ = W − (α/r)AB, and after training the adapter is merged back with W = W′ + (α/r)A_tB_t. Merging means no extra inference cost.

5. Why this works, formally. Proposition 3.1 shows that training only B is equivalent to gradient descent along the projection of G_t onto span(A_t), with ΔW = −c A_t A_tᵀ G_t and c = ηα²/r². Proposition 3.2 shows KeepLoRA's A_t minimizes ‖G_t − A_t A_tᵀ G_t‖²_F subject to W_pᵀ A_t = M_{t−1}ᵀ A_t = 0 — that is, the initialization captures the dominant gradient directions while being exactly orthogonal to the subspaces holding pre-trained and previous-task knowledge.

Why This Matters

Research impact. Most prior LoRA-based continual learning methods (O-LoRA, InfLoRA, SD-LoRA) focus on backward forgetting — losing knowledge of previously learned sequential tasks. This paper argues that forward stability, the preservation of general pre-trained transferability, has not been explicitly analyzed, and shows the two can be handled by a single orthogonality constraint. It also places the approach in the same family as gradient-projection methods such as GPM and CoSo while avoiding replay data and reference datasets.

Real-world applications (potential, as implied by the setting):

• Adapting a general vision-language model to a sequence of clinical or scientific imaging datasets without losing its general-purpose capabilities.
• Continually updating perception models in autonomous driving or robotics across changing environments and sensor domains.
• On-device personalization of a model, since the adapter merges into the base weights and adds no inference overhead — unlike prompt-pool or MoE-adapter designs that increase inference cost and complicate deployment.
• Settings where the original training corpus is proprietary or unavailable, so replay of pre-training data is not an option.

Industry relevance. Deployment-friendly continual learning is valuable because merged LoRA weights preserve existing inference stacks, and because avoiding reference-data curation removes a data-governance and compute burden that the paper notes makes reference-data regularization sensitive to the choice and volume of reference data.

Natural strengths. The method uses only the first-step gradient for initialization and top singular vectors for storage, keeping the persistent state bounded by the model's own feature dimension rather than growing external modules per task.

Future Directions

• Threshold sensitivity. The method introduces ϵ_w and ϵ_f to decide how many principal and task directions to retain. The truncated content does not report the values used or a sensitivity study, so how to set these across different backbones and task orders is an open question.
• Subspace capacity under long task sequences. The accumulated direction matrix M_t grows with each task, and the number of orthogonal vectors is upper-bounded by d_in. How the method behaves when long task sequences approach that bound is not addressed in the material presented.
• Beyond CLIP and LLaVA. The reported experiments cover a dual-encoder model (CLIP) and an encoder-decoder model (LLaVA) on MTIL, MLLM-DCL, and UCIT. Applicability to other architectures and modalities is untested here.
• Interaction with architecture extension. KeepLoRA+ adds a prototype vector for a class name and improves every reported metric, suggesting that combining residual-subspace updates with lightweight external components is a promising but underexplored direction.

Target Audience

Researchers and graduate students working on continual learning, parameter-efficient fine-tuning, and vision-language models, especially those already familiar with LoRA, SVD-based subspace methods, and benchmarks such as MTIL, MLLM-DCL, and UCIT. It is also relevant to practitioners who need to adapt large pre-trained models across successive tasks under constraints on stored data and inference latency. Readers should be comfortable with matrix decompositions and constrained optimization to follow the propositions.

Authors’ abstract

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.