SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

AI summary

Overview

• Research area: Multimodal continual instruction tuning (MCIT) for Multimodal Large Language Models (MLLMs), specifically Mixture-of-Experts (MoE) architectures combined with LoRA adapters.
• Technical level: Advanced. The paper assumes familiarity with MLLM training pipelines, LoRA, MoE routing, covariance/eigendecomposition, and continual-learning forgetting metrics.
• Scope in one sentence: The paper diagnoses two failure modes — router drift and expert drift — in MoE-based continual instruction tuning, proposes three stabilization mechanisms to address them, and introduces a new 10-task benchmark (TriGap) on which the method reports state-of-the-art average accuracy.

What This Paper Is About

Multimodal LLMs are typically instruction-tuned once on large multi-task mixtures, but real deployments require them to learn new vision-language tasks sequentially without forgetting earlier ones. Recent MCIT methods use sparse MoE routing with LoRA experts to encourage task specialization, yet they still degrade badly on early tasks as training proceeds. This paper shows that the degradation comes from two distinct sources — the router reassigning old inputs to new experts over time (router drift), and the experts themselves being overwritten by later tasks (expert drift) — and builds a method named SAME (StAbilized Mixture-of-Experts) to counter both.

Key Contributions

1. A diagnostic decomposition of forgetting in MoE-based MCIT. The authors insert MoE modules into the FFN layers of an MLLM, save router and expert snapshots after each of eight sequential tasks, and show that Task 1 expert-activation distributions progressively shift away from their original snapshot (router drift). They then freeze experts and re-train only the router on Task 1 after later tasks, and show accuracy on the Task 1 test set still fails to recover the Task 1 baseline, with particularly severe drops after Task 2 and Task 5, and with decreasing routing entropy — evidence of expert drift independent of misrouting.
2. Spectral-aware routing to counter router drift. The router's input covariance is accumulated across tasks, decomposed into orthogonal high-energy and low-energy subspaces, and raw router gradients are projected so that updates emphasize task-relevant directions while leaving old-task predictions approximately unchanged. Within the task-relevant subspace, singular values are rescaled using a sliding-window average so that directions with smaller relative singular values are updated less aggressively.
3. Curvature-aware Riemannian scaling to counter expert drift. Expert updates are preconditioned by the inverse of the historical input covariance (computed as a damped pseudo-inverse from the retained low-rank factors), so that directions heavily used by previous tasks are moved less. This is done in a rehearsal-free manner, using only the accumulated covariance rather than stored past data.
4. Adaptive expert activation plus a new benchmark. An expert is temporarily frozen during a task if its activation score — normalized current-task utilization minus normalized historical importance — falls below a threshold, where historical importance is approximated by routing-weighted input energy as a lightweight proxy for the trace of the Average Gradient Outer Product. Frozen experts are reactivated in later tasks and at inference. The paper also introduces TriGap, a 10-task benchmark built from datasets absent from LLaVA's training mixture, with 10k–40k samples per task and over 250k training instances total.

Main Findings

• TriGap benchmark results: SAME reaches an average accuracy of 46.53%, against 44.45% for MoE-LoRA (described as the second-best result), an improvement the paper states as +2.08%. Gains are reported as particularly clear on DocVQA (43.87%) and FloodNet (81.09%).
• CoIN benchmark results: SAME achieves 66.82% average accuracy, above HiDe-LLaVA at 63.95%, which is the second-best among the listed methods. Per-task scores include ScienceQA 78.35%, TextVQA 60.69%, ImageNet 90.21%, GQA 61.70%, VizWiz 54.13%, REC 59.87%, VQAv2 66.04%, and OCR-VQA 63.59%.
• UCIT benchmark results: SAME achieves 67.12% average, above ModalPrompt at 65.52%. Per-task scores include ImageNet-R 83.83%, ArxivQA 91.40%, Vizcap 51.33%, IconQA 65.27%, CLEVER 53.50%, and Flickr30k 57.43%.
• Ablation on CoIN: Starting from the MoELoRA baseline at 50.58% accuracy, adding spectral-aware routing raises it to 61.32%, adding curvature-aware scaling raises it to 65.89%, and adding adaptive expert activation raises it to 66.82%, indicating complementary contributions from all three components.
• Per-task ablation effects: ScienceQA moves 62.02 → 71.44 → 80.29 → 78.35 across baseline, router, expert, and activation variants; TextVQA moves 52.05 → 59.64 → 66.85 → 60.69; ImageNet moves 37.21 → 70.54 → 84.49 → 90.21.
• Routing stability analysis: Tracking the router output distribution on the Task 1 test set after each subsequent task, the unconstrained MoELoRA router drifts steadily, while the version with spectral-aware routing shows a markedly smaller distribution shift, indicating more consistent expert selection over time.
• Re-routing analysis: The paper reports that adding curvature-aware scaling improves re-routing accuracy on Task 1, indicating stronger preservation of early-task expert functionality (this analysis is introduced in Figure 4; the provided text is truncated at that point).
• Diagnostic entropy finding: After re-training the router on Task 1 while freezing experts from each stage, the normalized routing entropy decreases across tasks, indicating a more peaked but increasingly constrained routing decision.

Methodology in Plain English

The researchers first ran a forensic experiment: they trained a multimodal LLM through a sequence of tasks with MoE layers in the feed-forward blocks, kept snapshots after every task, and asked two questions. First, does the router keep sending the same Task 1 images to the same experts over time? It does not. Second, even if you forgive the router and re-teach it Task 1 while holding the experts fixed, do the experts still work? They do not. That established two separate problems.

To fix the router problem, they keep a running estimate of the statistics (covariance) of the inputs the router sees, then split that into directions the data varies a lot along and directions it barely varies along. Router weight updates are restricted mostly to the high-variance directions, which are the ones that matter for the new task, so the low-variance directions that encode earlier behavior change little. Directions with small singular values get damped further using a smoothed local average.

To fix the expert problem, they reuse the same covariance and use it as a measuring stick: an update to an expert is penalized in proportion to how much it would change that expert's output on inputs resembling previously seen data. Practically, this means scaling the gradient by the inverse covariance, computed via a damped pseudo-inverse of the low-rank factors already stored. Nothing from previous tasks needs to be kept as raw data, so the method is rehearsal-free.

To save computation and reduce interference, they also periodically check each expert's average routing weight on the current task against a running measure of how important that expert has been historically (approximated by routing-weighted input energy). Experts that are rarely used now but were important before are temporarily frozen during the current task's training and unfrozen afterwards and at inference.

The evaluation uses the existing CoIN (eight tasks) and UCIT (six tasks) benchmarks plus the new TriGap benchmark. Training uses LLaVA-v1.5-7B as the backbone and CLIP-L/14-336 for visual and textual features, with LoRA modules inserted only into the FFN layers of the language model at rank 8, one epoch per task, warm-up ratio 0.03, learning rate 2e-4 with cosine decay, and batch size 6, on 8 NVIDIA RTX 5090 GPUs. Baselines include MoELoRA, Continual LLaVA, ModalPrompt, SEFE, ProgLoRA, LLaVA-CMoE, CL-MoE, and HiDe-LLaVA.

Why This Matters

Impact on research. The paper reframes forgetting in MoE-based continual learning as two coupled failure modes rather than one, and provides a diagnostic protocol (snapshot routers and experts, then test whether re-fitting the router rescues old-task accuracy) that other researchers can reuse. It also argues that existing MCIT benchmarks are too easy — short sequences, limited domain variation, balanced data scales — and supplies TriGap as a harder, leakage-controlled alternative with 10 tasks, heterogeneous domains, and imbalanced per-task sizes (10k–40k samples).

Real-world applications:

• Continually updating a deployed visual assistant with new document, chart, or infographic understanding skills without retraining from scratch or losing prior skills.
• Scientific and medical image question answering systems that must absorb new specialties (for example, chemistry or pathology) while retaining existing ones.
• Remote sensing and disaster-response analysis tools that need to add new sensor or scene types over time while keeping earlier detection abilities.
• Enterprise vision-language products that onboard new customer-specific visual tasks sequentially under storage or privacy constraints that rule out replaying old data.

Industry relevance. The method is rehearsal-free, parameter-efficient (LoRA rank 8, adapters only in FFN layers), and explicitly motivated by reducing redundant computation via expert freezing — all of which matter for teams that cannot retain customer data or cannot afford full retraining. The reported hardware footprint (8 NVIDIA RTX 5090 GPUs) is modest relative to frontier-scale training, which makes replication more feasible. Code is released at https://github.com/LAMDA-CL/Prism.

Future Directions

• Choosing the freezing aggressiveness. The activation score threshold controls how many experts are frozen, but the provided content does not report its value or a sensitivity analysis; the paper defers detailed hyperparameter configurations and sensitivity analyses to Appendix F, which is outside the truncated text. How best to set that threshold in general remains open.
• Extending beyond the tested scale. All experiments use LLaVA-v1.5-7B with rank-8 LoRA in FFN layers under a single-epoch-per-task schedule. Whether the drift analyses and stabilization mechanisms behave the same way at larger backbones, higher LoRA ranks, or with adapters in attention layers is not established in the reported content.
• Sequence length and task ordering. The diagnostic uses eight tasks and TriGap uses ten. Whether router and expert drift compound differently over much longer horizons, or under adversarial task orderings, is not reported.
• Theoretical guarantees. The paper argues that projecting onto the approximate null space keeps old-task router predictions approximately unchanged (an approximate-zero argument), but the content does not report formal bounds on forgetting or on how the damping constant affects the quality of the inverse covariance approximation.

Target Audience

Researchers and graduate students working on continual learning, parameter-efficient fine-tuning, and multimodal LLMs will get the most from this paper, particularly those interested in MoE routing dynamics. Practitioners who deploy vision-language models in settings where new tasks arrive over time and old data cannot be stored will find the rehearsal-free design and the TriGap benchmark directly relevant. Readers without a background in MoE routing, LoRA, or covariance-based regularization will need to consult the cited background work first, since the method section is written at an advanced level.

Authors’ abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. We also introduce a new benchmark to evaluate MCIT with long task sequence, and extensive experiments demonstrate SAME's SOTA performance. Code is available at https://github.com/LAMDA-CL/Prism.