PLAN: Proactive Low-Rank Allocation for Continual Learning

AI summary

Overview

Research area: Continual learning (CL) with parameter-efficient fine-tuning (PEFT), specifically low-rank adaptation (LoRA) applied to large pre-trained vision models.

Technical level: Intermediate — the paper assumes familiarity with LoRA, low-rank adaptation parameters, orthogonality constraints, and standard continual-learning evaluation metrics, though the core ideas are explainable without deep mathematics.

Scope: The paper proposes PLAN (Proactive Low-Rank Allocation), a LoRA-based continual learning framework that selects task-specific parameter subspaces from a fixed orthogonal basis and trains them with a perturbation-based min-max objective, evaluated on CIFAR-100, DomainNet, and ImageNet-R using ViT-B/16 backbones.

What This Paper Is About

Continual learning asks a model to learn a sequence of tasks without forgetting earlier ones, and a common approach is to give each new task its own small LoRA update so the shared backbone stays frozen. The problem is that new updates can still interfere with old ones, so prior methods like O-LoRA and InfLoRA try to avoid interference passively by forcing new updates to be orthogonal to old ones. PLAN instead claims to proactively plan which subspaces each task should use, anticipating future interference during the training of the current task rather than merely reacting to it.

Key Contributions

1. A shift from passive avoidance to proactive planning. PLAN anticipates future parameter conflicts and strategically reserves subspaces for tasks that have not yet been seen, rather than only isolating new updates from old ones.
2. A min-max optimization objective for the LoRA matrix B_t. The objective perturbs the currently unallocated subspace and optimizes the current task's parameters to be robust against worst-case interference from future tasks, with a closed-form expression derived for the perturbation via a first-order Taylor approximation.
3. An orthogonal basis selection mechanism for A_t. Instead of learning subspaces, PLAN picks rows from a predefined standard orthogonal basis set, choosing the basis vectors that consistently experience the smallest perturbations during the previous task's training (tracked by a frequency counter over a sliding window).
4. Empirical state-of-the-art claims across benchmarks. PLAN is reported to outperform L2P, DualPrompt, CODA-Prompt, Inc-LoRA, O-LoRA, and InfLoRA on CIFAR-100, DomainNet, and ImageNet-R under both supervised and self-supervised (iBOT) pre-training.

Main Findings

• ImageNet-R with ViT-B/16 (supervised pre-training). On 5 tasks, PLAN reaches 77.79% Acc (±0.24) versus InfLoRA's 77.09% (±0.33), with AAA of 81.93% (±0.63) versus InfLoRA's 81.96% (±0.28). On 10 tasks, PLAN reaches 75.25% Acc / 80.41% AAA versus InfLoRA's 74.37% / 80.37%. On 20 tasks, PLAN reaches 71.06% Acc / 77.93% AAA versus InfLoRA's 69.83% / 76.83%. AAA on the 5-task setting is marginally below InfLoRA.
• CIFAR-100 and DomainNet (N=5). On CIFAR-100, PLAN records 87.54% Acc / 92.21% AAA, above InfLoRA's 86.50% / 91.23%. On DomainNet, PLAN records 72.12% Acc / 77.52% AAA; its accuracy is the highest reported, while its AAA is below InfLoRA's 78.29%.
• Stability across task sequences. Figure 3 reports that PLAN maintains more stable and higher accuracy across sequential tasks on ImageNet-R and CIFAR-100, while compared methods show pronounced fluctuations and sharp drops when new tasks arrive.
• Ablation confirms both components matter. Removing the A_t selection (random selection from E_t) yields 75.97% / 80.69% (N=5), 72.14% / 78.56% (N=10), and 68.35% / 76.33% (N=20) on ImageNet-R. Removing the perturbation-based B_t optimization yields 76.66% / 80.38%, 74.97% / 79.57%, and 70.65% / 76.42%. Full PLAN yields 77.79% / 81.93%, 75.25% / 80.41%, and 71.06% / 77.93%. Both variants still beat Inc-LoRA (72.36% / 79.60%, 63.69% / 74.54%, 52.12% / 67.73%).
• Basis initialization matters. On CIFAR-100, a random orthogonal basis gives 81.21% Acc, LoRA-GA initialization gives 84.30%, and the standard orthogonal basis gives 87.54%. On ImageNet-20 (ImageNet-R, N=20), the three give 69.42%, 69.40%, and 71.06% Acc respectively. The authors argue LoRA-GA's SVD-based initialization produces vectors that are poorly aligned with features needed for later tasks, harming plasticity.
• The choice of norm p is sensitive. On CIFAR-100, p=1 yields 80.32% Acc, p=2 yields 87.54%, and p→∞ yields 79.94%. Extreme values are reported to destabilize the min-max optimization.
• Storage efficiency. On ImageNet-R (N=20), PLAN uses 0.70 MB of expanded parameters and 0 MB of stored features, the same parameter footprint as InfLoRA (0.70 MB) but with no stored feature space (InfLoRA stores 67.35 MB; O-LoRA stores 13.36 MB). Prompt-based methods store no features but use more parameters (DualPrompt 14.65 MB, CODA-Prompt 5.57 MB, L2P 1.85 MB, Inc-LoRA 1.41 MB).
• Self-supervised backbone results. With iBOT-1k ViT-B/16, all methods perform lower than with supervised pre-training. On CIFAR-100, PLAN reaches 65.93% Acc / 74.46% AAA (InfLoRA 65.28% / 74.11%). On ImageNet-R (N=20), PLAN reaches 78.39% Acc / 86.61% AAA, with CODA-Prompt at 78.78% Acc / 86.63% AAA and InfLoRA at 78.11% / 86.47%.
• Hyperparameter settings. ρ=0.01 performs best on ImageNet-R (N=5) at 77.79% Acc / 81.93% AAA, versus 75.23% / 78.94% at ρ=0.1 and 76.38% / 79.36% at ρ=0.001. The sliding window S=50 is used, with Appendix A.1 reporting that selected basis indices stabilize quickly and S=50 nearly matches a full window of S=100.

Methodology in Plain English

PLAN keeps the pre-trained model frozen and gives each task its own LoRA update of the form ΔW_t = B_t A_t, but it changes how the two matrices are obtained.

For the A_t matrix, the authors do not train it at all. They start from a fixed standard orthogonal basis — vectors that are all zeros except for a single one — and simply pick a set of rows for each task, removing those rows from the pool so later tasks cannot reuse them. The total basis size equals the model's feature dimension, which is large relative to the number of tasks, so the pool rarely runs out.

To choose which rows a new task gets, PLAN uses information gathered while training the previous task. During that training, it repeatedly computes a worst-case perturbation along all the still-unallocated directions and measures how large that perturbation is for each direction. A frequency counter records how often each direction ranks among the smallest perturbations over a sliding window of 50 training steps, and the new task takes the directions with the highest counts. The intuition is that directions that were rarely perturbed are unlikely to disrupt past knowledge.

For the B_t matrix, PLAN solves a min-max problem: the inner step finds the perturbation, along the not-yet-assigned directions, that most increases the loss on the current task; the outer step then adjusts B_t so the model is robust to that perturbation. A first-order Taylor expansion gives a closed-form solution for the worst-case perturbation, and to avoid computing Hessians the authors treat that perturbation as a constant during the gradient update. This makes the current task's parameters robust in advance, and it also produces the sensitivity scores that drive the next task's basis selection.

The reported training setup used Adam (β1=0.9, β2=0.999), batch size 128, 50 epochs on ImageNet-R, 20 on CIFAR-100, and 5 on DomainNet, with LoRA modules inserted into the key and value components of the attention mechanism.

Why This Matters

The paper reframes continual learning with LoRA as a resource allocation problem: instead of asking how to repair interference after it happens, it asks which parts of the parameter space a task should claim so that future tasks cannot damage it. If the reported gains and the zero stored-feature cost hold up, PLAN offers a practical route to adapting large foundation models over long task sequences without a growing memory of gradients or features.

Real-world applications cited or implied by the work:

• Autonomous driving, where perception models must adapt to new environments and conditions without losing earlier competence.
• Robotics, where a deployed policy encounters shifting tasks and surroundings over its lifetime.
• Edge or on-device personalization, since PLAN's storage overhead on ImageNet-R (N=20) is 0.70 MB of expanded parameters and 0 MB of stored features, far below InfLoRA's 67.35 MB of stored features.
• Any long-horizon deployment of a frozen backbone, where fine-tuning must be cheap and forgetting must be bounded.

Industry relevance: The combination of LoRA-compatible parameter counts, no stored gradient subspace, and compatibility with both supervised and self-supervised ViT pre-training makes the approach a plausible drop-in for teams already serving LoRA adapters, who would otherwise need additional storage and selection machinery to keep them from overwriting each other.

Future Directions

1. Relaxing strict orthogonality to allow positive backward transfer. The authors state that PLAN's strict orthogonality prevents forgetting but does not explicitly promote backward transfer, and suggest selectively relaxing orthogonality as a future direction.
2. Extending beyond ViT architectures. The paper notes experiments focused on ViT-based models and that evaluating on ConvNets or other data modalities is a valuable next step.
3. Adaptive basis generation for heterogeneous task sequences. Although the standard orthogonal basis performed best in the reported comparison, the authors flag adaptive basis generation as an open question for highly heterogeneous tasks.
4. Robustness of the min-max balance. The p-ablation shows the framework is sensitive to how perturbation magnitude is scaled (p=2 at 87.54% Acc versus 80.32% for p=1 and 79.94% for p→∞), leaving open how to keep the minimization and maximization components balanced in general.

Target Audience

This paper is most useful to researchers and practitioners working on continual learning, parameter-efficient fine-tuning, or adapter-based deployment of large pre-trained models. It is also relevant to engineers maintaining long-lived model services who need to add tasks incrementally while controlling storage and forgetting, and to readers already familiar with O-LoRA, InfLoRA, and prompt-based CL baselines who want to understand how a proactive subspace-allocation strategy differs from passive orthogonality constraints.

Authors’ abstract

Continual learning (CL) requires models to continuously adapt to new tasks without forgetting past knowledge. In this work, we propose \underline{P}roactive \underline{L}ow-rank \underline{A}llocatio\underline{N} (PLAN), a framework that extends Low-Rank Adaptation (LoRA) to enable efficient and interference-aware fine-tuning of large pre-trained models in CL settings. PLAN proactively manages the allocation of task-specific subspaces by introducing orthogonal basis vectors for each task and optimizing them through a perturbation-based strategy that minimizes conflicts with previously learned parameters. Furthermore, PLAN incorporates a novel selection mechanism that identifies and assigns basis vectors with minimal sensitivity to interference, reducing the risk of degrading past knowledge while maintaining efficient adaptation to new tasks. Empirical results on standard CL benchmarks demonstrate that PLAN consistently outperforms existing methods, establishing a new state-of-the-art for continual learning with foundation models.