PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

AI summary

Overview

• Research area: Continual learning and parameter-efficient fine-tuning for multimodal large language models, specifically Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA).
• Technical level: Intermediate — readers should be comfortable with LoRA's low-rank decomposition, mixture-of-experts routing, and the standard catastrophic-forgetting framing of continual learning.
• One-sentence scope: The paper diagnoses why routers and LoRA experts drift apart during sequential instruction tuning and proposes a routing-plus-regularization scheme anchored in each expert's own low-rank pathway to reduce forgetting without adding parameters.

What This Paper Is About

When a multimodal large language model is tuned on a stream of tasks, a common trick is to split the work across several LoRA "experts" and let a router decide which expert handles each input. The problem is that the router and the experts are usually updated together with no shared reference point, so both gradually wander away from the input–expert pairings established early on. The authors call this Misaligned Co-drift, and their goal is to keep routing decisions and expert parameters locked to the same coordinate system so that earlier capabilities survive later training.

Key Contributions

1. A new definition of "pathway activation subspace" (PASs). The authors define each expert's subspace as the span of its LoRA down-projection rows (span(Aᵀ)), arguing this gives a capability-tied coordinate system unlike conventional activation subspaces derived from intermediate feature statistics.

2. PASs-guided Reweighting (PASs-RW). Instead of learning a separate router network, the mixture weights are computed directly from each expert's low-rank activation energy, tying routing to the expert's own functional response.

3. PASs-aware Rank Stabilization (PASs-RS). The same activation signal is accumulated across tasks to estimate which individual rank directions matter most, and stability penalties are applied selectively to those directions.

4. Empirical validation under a fixed parameter budget. On the MLLM-CTBench continual instruction tuning benchmark, the method improves average performance and reduces forgetting relative to traditional continual learning baselines and MoE-LoRA variants, with no increase in model capacity.

Main Findings

• Best overall accuracy and least forgetting on MLLM-CTBench. The method reaches an average performance (AP) of 48.46 and backward transfer (BWT) of −2.15, versus 43.36 and −6.64 for the strongest MoE-LoRA baseline (softmax MoELoRA). That is a 5.1-point AP lead over the second-best method.

• Task-order robustness. Under an alternative task ordering the authors report a 9.46-point AP improvement over the second-best method, suggesting the gains are not an artifact of one specific sequence.

• Both components contribute. Adding PASs-RW alone lifts AP from 43.36 to 46.31 and improves BWT to −4.24; adding PASs-RS on top brings AP to 48.46 and BWT to −2.15. The authors also note the Top-k routing baseline performs notably worse in the continual setting, so it is treated only as a reference point.

• Routing drift tracks with old-task degradation. Tracking the gating distribution on fixed old-task inputs before and after the full task stream, the baseline shows substantially larger drift (measured by Jensen–Shannon divergence) that coincides with lower final accuracy on those tasks; the proposed method shows smaller drift and higher retained accuracy.

• Rank importance is sparse and concentrated. The aggregated importance map over the expert–rank grid is highly non-uniform, with a small subset of rank directions dominating prior-task behavior, which motivates selective rather than uniform stabilization.

• Stabilization reshapes updates away from important directions. Across layers 1, 15, and 29, the median log update magnitude generally decreases as prior-task importance increases, meaning high-importance directions are held steady while low-importance ones retain flexibility. The strength of this coupling varies by layer.

• Hyperparameter trade-offs. Performance peaks at a moderate expert count (E = 6) and degrades slightly at E = 8. Increasing the LoRA-A regularization strength monotonically reduces forgetting but eventually hurts new-task learning, producing a non-monotonic AP curve. Regularizing LoRA-B helps but is more sensitive to its weighting, with moderate settings giving the best AP/BWT balance.

• Small, bounded overhead. With 6 experts, training time rises from roughly 130.7 to 140.9 units and inference time from 127.6 to 132.8 compared with MoELoRA, while AP improves by about 5 points.

Methodology in Plain English

The authors keep a fixed pool of LoRA experts and never grow it. Each expert is a pair of matrices: a down-projection A that maps the input into a small r-dimensional space, and an up-projection B that maps back to the output. The vector A·h is an "activation signature" telling you how strongly that expert responds to a given input along each of its r rank directions.

Two ideas follow from this.

First, replace the router. Rather than training a separate gating network, the authors compute each expert's activation energy (the squared norm of A·h, normalized by rank) and run a softmax over these energies to produce the mixture weights. Routing therefore becomes an implicit byproduct of how the experts themselves respond, so the router cannot drift independently of expert behavior.

Second, protect what matters. For each expert and each rank direction k, the authors record a running importance score: the expected squared response along direction k, weighted by how often that expert was selected, accumulated over all previous tasks. During training on a new task, they add two penalty terms that push the current A rows and B columns toward their previous values, scaled by these importance weights. Directions that mattered a lot historically barely move; directions that mattered little stay free to adapt.

The overall loss is simply the task loss plus these two weighted stabilization terms. The base model is LLaVA-v1.5-7B with LoRA rank 128 on all language-model linear layers, six experts, and CLIP-L/14-336 for visual features.

Why This Matters

Impact on research. The paper reframes MoE-LoRA routing as a subspace-alignment problem rather than a gating-architecture problem. The PASs definition — deriving a meaningful subspace from the parameters of the low-rank adapter rather than from observed activations — is a conceptual move that could be reused for other PEFT forms, and the "co-drift" diagnosis gives a concrete failure mode to measure rather than a vague appeal to forgetting.

Real-world applications:

• Continuously updated multimodal assistants that must absorb new skills or domains each month without regressing on previously shipped capabilities.
• Medical and scientific VQA systems where new specialties or subfields come online over time and earlier diagnostic knowledge cannot be discarded.
• Document and OCR pipelines deployed in enterprise settings where document formats, languages, and templates evolve and older formats must still be handled.
• Domain-specific customer support or analytics models that are fine-tuned repeatedly on new product lines while retaining knowledge of legacy ones.

Industry relevance. The method adds no parameters and no extra inference-time modules, which matters for deployment cost. Its main overhead is a stability penalty during training and maintaining per-rank importance statistics. For teams already running MoE-LoRA adapters, the change is largely a loss-term and routing-weight modification rather than a re-architecture, which lowers the barrier to adoption.

Future Directions

1. Capacity expansion and replay. The authors restrict themselves to fixed-capacity MoE-LoRA with no access to past data. Whether PASs generalizes to growing expert pools or replay buffers — where co-drift dynamics differ — is untested.

2. Beyond LoRA. PASs relies specifically on LoRA's factorization. Extending it to adapters, prefix tuning, or other PEFT families requires redefining what a "capability-aligned coordinate system" means in each case.

3. Better routing signals and calibration. Activation energy is used as a proxy for input–expert compatibility. Its reliability under severe distribution shift, ambiguous instructions, or scarce task data has not been established, and importance estimates can be noisy.

4. Richer evaluation and interpretability. The current metrics are AP and BWT. The authors themselves call for direct measurements of routing mismatch, better interpretability of how expert responsibilities evolve, and evaluation under realistic online distribution dynamics rather than curated task sequences.

Target Audience

Researchers and graduate students working on continual learning, parameter-efficient fine-tuning, or multimodal LLMs — particularly those already familiar with LoRA and MoE routing who want a concrete failure-mode analysis plus a lightweight fix. Practitioners deploying sequentially updated vision-language models in production will also find the fixed-budget, no-extra-inference-cost design relevant, though they should expect to tune the two stabilization coefficients and the expert count for their own task streams.

Authors’ abstract

Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input--expert specialization. We term this as Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting. To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE--LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE--LoRA variants in both accuracy and resistance to forgetting, without increasing model parameters. Our code is publicly available at https://github.com/yueluoshuangtian/PASs-MoE.