Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning

AI summary

Overview

Research area: Machine learning — specifically continual learning (CL) with parameter-efficient fine-tuning (PEFT), focusing on how LoRA updates are merged into a single inference model.

Technical level: Advanced. The method relies on second-order Taylor expansions of task losses, diagonal Fisher/Hessian approximations, and low-rank adapter algebra.

Scope: The paper proposes a LoRA merging framework (M²LE) that replaces a single shared merge coefficient with an analytically derived, element-wise modulation computed from a joint objective over past and current tasks, plus a training-time perturbation scheme that makes merged updates more merge-stable.

What This Paper Is About

Continual learning systems that build on LoRA typically keep one model for inference by folding each new task's low-rank update into the accumulated weights, scaled by a single number. But a single number treats every parameter element the same, even though different elements affect old and new tasks differently. The paper shows this shared-factor assumption is unnecessarily restrictive and replaces it with a per-element merge strength derived from loss information, along with a training procedure that makes the resulting update more robust to later merges.

Key Contributions

1. Hessian Information-guided Adaptive Merging (HIM): A parameter-adaptive merging rule derived from a joint loss objective over previous and current tasks. Under a local quadratic approximation with diagonal Fisher matrices, the optimum takes a closed form that scales each element of the current LoRA update individually — no search over a global coefficient is performed.

2. Task Simulation Perturbation for Merging (TSP): A training strategy that splits each mini-batch into two label-disjoint subsets, uses one subset to induce a local parameter perturbation, and evaluates the loss change on the other. This simulates cross-task interference without stored old data and encourages flat (merge-stable) solutions.

3. Empirical demonstration that a shared scalar is suboptimal: A controlled two-group merge experiment on 10-task ImageNet-R shows the joint-loss optimum lies off the diagonal where both groups share one coefficient.

4. State-of-the-art results across benchmarks and task counts: Consistent gains over PEFT-based CL baselines on ImageNet-R, ImageNet-A, CIFAR100, CUB, and DomainNet, plus an ablation isolating the effect of each component and a comparison against other merge strategies (EMA-style averaging, LoRA-P&M, sequential fine-tuning).

Main Findings

• Two merge strengths beat one: In the preliminary analysis on 10-task ImageNet-R (first merge step, splitting the update by normalized magnitude at a 0.5 threshold), the best two-coefficient point was (0.40, 0.80) with 91.84% average accuracy on tasks 0 and 1, versus 91.04% for the best scalar merge on the diagonal at (0.60, 0.60). The low-loss region in the loss landscape is shifted off-diagonal.

• Main benchmark gains (10 tasks, no feature replay): M²LE reaches $\mathcal{A}{Last}$ / $\mathcal{A}{Avg}$ of 81.34 / 86.53 on ImageNet-R, 62.45 / 71.02 on ImageNet-A, 89.97 / 93.31 on CIFAR100, and 81.40 / 89.23 on CUB, against LoRA-P&M's 79.95 / 85.29, 56.57 / 65.35, 88.45 / 92.89, and 78.29 / 83.39 respectively.

• Reported improvement margins: Relative to LoRA-P&M, $\mathcal{A}_{Last}$ improves by 1.39%–5.88% on the 10-task benchmarks and 0.43%–2.25% under the additional settings (5-task ImageNet-R, 20-task ImageNet-R, 20-task ImageNet-A, 10-task DomainNet).

• Additional task settings: 82.67 / 87.31 on 5-task ImageNet-R, 78.62 / 84.62 on 20-task ImageNet-R, 53.87 / 63.69 on 20-task ImageNet-A, and 84.78 / 89.74 on 10-task DomainNet.

• With classifier alignment (M²LE+CA): 82.68 / 87.05 on 10-task ImageNet-R, 65.96 / 74.02 on 10-task ImageNet-A, 91.86 / 94.62 on CIFAR100, and 90.35 / 93.92 on CUB. It outperforms MACIL in $\mathcal{A}_{Last}$ by 0.80%–1.82% on the 10-task ImageNet-R/ImageNet-A settings and by 1.69%–4.89% on their 20-task settings. The authors note the margins are smaller on CIFAR100 and CUB, attributing this to strong pre-trained representations.

• Ablation (both components matter): On 10-task ImageNet-R, the baseline without HIM or TSP scores 73.05 / 80.40; HIM alone 80.36 / 85.78; TSP alone 74.20 / 80.67; both 81.34 / 86.53. On 10-task ImageNet-A the same ordering holds: 52.56 / 63.36, 61.35 / 70.23, 56.57 / 66.26, 62.45 / 71.02. On 20-task ImageNet-R: 66.14 / 73.72, 78.04 / 84.31, 69.17 / 77.16, 78.62 / 84.62. On 20-task ImageNet-A: 46.06 / 59.43, 52.80 / 62.64, 49.50 / 60.45, 53.87 / 63.69. HIM contributes the larger share; TSP adds a smaller but consistent gain.

• Merge-strategy comparison (10-task, Table 4): Sequential fine-tuning scores 73.05 / 80.40 on ImageNet-R and 52.56 / 63.36 on ImageNet-A; EMA-style averaging 79.14 / 85.08 and 59.64 / 69.52; LoRA-P&M 78.89 / 84.99 and 59.93 / 69.87. The remainder of this table is truncated in the provided content, so the full set of rows is not reported here.

Methodology in Plain English

The setting is rehearsal-free class-incremental learning: tasks arrive one at a time with disjoint label sets, past data cannot be stored, and at test time the model must predict across all classes seen so far without being told which task a sample came from. The backbone is a frozen ViT, and each task trains a LoRA module (a product of two low-rank matrices).

Step 1 — Train the task update with simulated interference (TSP). Each mini-batch is randomly split into two label-disjoint halves. One half's gradient defines a small step away from the current parameters; the loss is then measured on the other half at that perturbed point, and the perturbed and clean losses are blended with a weight β. The halves are then swapped and the process repeats, so within one mini-batch the model is updated twice. The goal is not to fit the current task harder but to land in a flatter region, so that later merges disturb old knowledge less. The authors contrast this with SAM, which perturbs and evaluates on the same task loss and therefore cannot mimic one task's update affecting another.

Step 2 — Decide how much of each element to merge (HIM). Rather than picking a scalar α, they define two quantities: how much a candidate merged update would raise the losses on all previous tasks, and how much it would raise the current task's loss relative to the fully trained update. The trade-off between them is controlled by λ. Because old data is gone, both are approximated by local second-order Taylor expansions around the known parameter points, with first-order terms treated as approximately zero. Setting the derivative to zero yields a linear system; approximating each Hessian by its diagonal empirical Fisher (as in EWC) reduces it to a per-element formula: each element of the current update is multiplied by λF_t / (F_{1:t−1} + λF_t + ε), where F_t is the current task's diagonal Fisher and F_{1:t−1} is the running sum from earlier tasks. Elements that the current task considers important keep more of their update; elements that matter for the accumulated history get suppressed. The running Fisher sum is then updated for the next merge.

Step 3 — Merge and repeat. The modulated update is added to the inference weights, the LoRA module is discarded, and the process repeats for the next task, always maintaining one model at inference time.

Setup: ViT-B/16 pre-trained on ImageNet-21K, trained with SGD and a cosine annealing schedule for 20 epochs per task at batch size 48, learning rate 0.02 on ImageNet-R and 0.01 elsewhere, on a single NVIDIA RTX 3090. λ = 5 for all datasets; TSP's ρ = 0.3 on ImageNet-A and 0.5 elsewhere; β = 0.6 on ImageNet-A and 0.8 elsewhere. Results are averaged over 3 random seeds.

Why This Matters

Impact on research. The paper challenges a default assumption in LoRA-based continual learning — that one scalar merge coefficient is enough — and shows a measurable gap even with a crude two-group split. It also connects merge-strength selection to the flat-minimum literature, arguing that where an update lands matters as much as how it is scaled, and it gives a closed-form, well-grounded alternative to hyper-parameter search over α. Because the Fisher statistics are accumulated cheaply and diagonally, the method stays compatible with the single-model, no-replay constraint that makes LoRA merging attractive in the first place.

Real-world applications:

• On-device personalization: a phone or laptop adapting a vision or language model to a user's evolving data without storing old examples or keeping one adapter per task.
• Continual visual recognition in deployed systems: robotics or camera networks that encounter new object categories over time and cannot revisit earlier footage.
• Enterprise models updated on proprietary data streams: where retaining historical data may be prohibited by policy, and where a single servable model is much cheaper than a growing library of per-task adapters.
• Medical or scientific imaging with sequentially arriving cohorts: where new classes appear over time, data sharing is restricted, and forgetting an earlier category has real cost.

Industry relevance. Single-model inference, no stored data, and one hyper-parameter per dataset translate directly into lower serving cost and simpler deployment than multi-adapter or replay-based designs. The method adds a diagonal Fisher computation and a two-pass mini-batch update — modest overhead relative to the memory savings of not keeping past data or separate LoRA modules.

Future Directions

• Beyond the diagonal approximation. The analytic solution assumes diagonal Fisher matrices and near-zero first-order terms. How much is lost by ignoring cross-parameter curvature, and would block-diagonal or Kronecker approximations pay for themselves, is not established here.
• Cheaper or better stability shaping. TSP operates within a mini-batch using two label-disjoint subsets, which is only a proxy for genuine cross-task interference. Alternative perturbation schemes, or ones drawing on richer signal than a single batch split, could improve the flatness guarantee.
• Hyper-parameter sensitivity. λ, ρ, and β are set per dataset (λ = 5 everywhere, but ρ and β differ between ImageNet-A and the rest). Whether these can be selected automatically, or made task-adaptive, is not addressed.
• Scope of evaluation. All benchmarks are image classification with a ViT-B/16 backbone. Extension to generative models, language tasks, or long task sequences well beyond 20 tasks remains untested in the reported results.
• Interaction with classifier alignment. M²LE+CA gains come partly from a stage borrowed from prior work (SLCA), and margins shrink on CIFAR100 and CUB. Disentangling how much headroom remains in the merging step itself, versus the classifier stage, is an open question.

Target Audience

Researchers and graduate students working on continual learning, parameter-efficient fine-tuning, or model merging who are comfortable with second-order approximations and Fisher information. It is also relevant to practitioners building single-model continual-learning systems who want a concrete, drop-in replacement for scalar LoRA merging — though those readers will need to work through the derivations in the appendices to implement HIM faithfully. Readers new to the area should start with the introduction and the preliminary analysis in Figure 1, which motivate the method without requiring the math.

Authors’ abstract

Continual learning (CL) aims to adapt models to new tasks while preserving previously learned knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL due to its efficiency and scalability. Several LoRA-based merging methods maintain a single inference model by scaling the newly learned LoRA update with one coefficient and merging it into the previous weights after each task. However, they usually control the whole update with a shared scalar factor, although different update elements may affect previous and current tasks in different ways. To address this issue, we propose Merging Strength Modulation via Joint Loss Estimation (M2LE). It consists of Hessian Information-guided Adaptive Merging (HIM) and Task Simulation Perturbation for Merging (TSP). Instead of searching for a global merge coefficient, HIM directly solves for the target merged update under a joint objective over previous and current tasks. Solving this objective gives an analytic solution that naturally takes the form of element-wise modulation over the current task LoRA update. To improve the stability of the learned update under subsequent merging, we introduce TSP which simulates different tasks interference. Experiments on multiple benchmarks show that M2LE consistently outperforms existing PEFT-based continual learning methods. Code is available at https://github.com/bbdeeer/M2LE_NeurIPS2026.