Multimodal Continual Learning with MLLMs from Multi-scenario Perspectives

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models (MLLMs) / Continual Learning (CL). Keywords listed by the authors: Machine Learning, ICML.

Technical level: Advanced. The paper assumes familiarity with catastrophic forgetting, LoRA-style parameter-efficient fine-tuning, vision encoders, knowledge distillation, and the distinction between rehearsal-based and rehearsal-free continual learning.

Scope: The paper introduces the MSVQA dataset of four visual scenarios (high-altitude, underwater, low-altitude, indoor), and proposes Unifier, a continual learning framework built on Vision Representation Expansion (VRE) and Vision Consistency Constraint (VCC), evaluated over 5-, 10-, and 20-step cross-scenario task streams.

What This Paper Is About

MLLMs deployed on real devices encounter visual conditions that keep changing — different backgrounds, capture devices, and perspectives — and learning a new scenario often destroys the model's ability to handle previously learned ones (catastrophic forgetting in the visual components, not just the language side). The paper builds a benchmark to expose this problem and proposes a framework that lets one model accumulate knowledge within a scenario and improve across scenarios instead of degrading.

Key Contributions

1. MSVQA dataset and continual learning benchmark. A multimodal visual understanding dataset covering four distinct scenarios and perspectives — high-altitude, underwater, low-altitude, and indoor — with a corresponding CL benchmark to evaluate how on-device MLLM performance changes under complex scenario variation. The dataset is stated to be open-sourced (dataset link given as https://huggingface.co/datasets/Kaij00/MSVQA; a contributions bullet separately notes the link was omitted for double-blind review).
2. The Unifier framework. A CL framework that aligns visual representations across scenarios by learning each scenario with a new branch inside the vision blocks and constraining features across scenarios to obtain consistent vision representations, thereby reducing forgetting caused by scenario shifts.
3. Two named components: VRE and VCC. Vision Representation Expansion isolates per-scenario parameters in the vision encoder and projects their outputs into a unified feature space via a single inference path, without needing a routing/gating network; Vision Consistency Constraint applies a softer distillation-style constraint across branches.
4. Empirical validation. Experiments on MSVQA showing improved last-step VQA scores by 2.70% to 10.62% and last-step F1 scores by 3.40% to 7.69% over the state-of-the-art method QUAD in 20-step cross-scenario continual learning tasks, with the authors stating no additional inference cost.

Main Findings

• Headline result vs. QUAD (20 steps): Unifier improves the last-step VQA scores by 2.70% to 10.62% and last-step F1 scores by 3.40% to 7.69% compared to QUAD.
• Specific 20-step (T=20) gaps: Unifier outperforms the runner-up by 10.62% on last-step VQA in the high-altitude scenario and by 7.69% on last-step F1 in the underwater scenario.
• Specific 5-step (T=5) gaps: Unifier outperforms the runner-up by 6.54% on the last-step VQA score in the high-altitude scenario and by 4.96% on the last-step F1 score in the indoor scenario.
• No performance decline while learning: Most compared approaches show a significant decline when learning a new scenario, while Unifier improves performance in both seen and unseen scenarios at T=10, indicating knowledge accumulation without forgetting across scenarios.
• Better stability/plasticity trade-off: In the two-scenario sequential comparisons (Underwater→Indoor, High altitude→Low altitude, Indoor→Underwater, Low altitude→High altitude), Unifier achieves a higher average score on both seen and novel scenarios.
• Ablation (T=10) — VRE alone helps substantially: Baseline high-altitude VQA average 29.74 / last 1.60, versus w/ VRE 51.03 / 51.82, and w/ VRE+VCC 60.00 / 61.42. High-altitude F1 moves from 30.54 / 15.82 (baseline) to 71.32 / 72.49 (w/ VRE+VCC).
• Looser constraints balance better than strict ones: Rows 3-5 of the ablation show that ℓ2-distance without channel reduction, ℓ2 with channel reduction (penalizing only global changes), and relative entropy (soft logits) perform progressively better — the paper concludes looser constraints better balance stability and plasticity.
• The projector matters: Removing the projector (row 6, "w/o Projector") shows that simple feature addition constrains the model's capacity to learn the new scenario, demonstrating the projection layer is effective.
• Routing-free design advantage: VRE outperforms a simple multi-branch (multi-LoRA) approach, and the unified representation is argued to be more robust for ambiguous scenarios than routing-based methods.
• Inference cost: Adding the VRE module does not impact inference efficiency; the paper reports no additional inference cost and an acceptable increase in parameters (detailed analysis in Appendix Sec. F).
• Order effects: The appendix (Sec. K) states the effect of learning order is eliminated.
• Visualization evidence: Unifier keeps attention shifts confined to a few regions after learning a new scenario, showing only minor missed detections and bounding-box offsets, whereas Finetune's attention shifts to unrelated areas, producing many false alarms and missed detections.
• Not reported in the paper content: The total number of images, questions, or annotations in MSVQA is not reported in the available text.

Methodology in Plain English

Continual learning here means the model sees a stream of tasks, one scenario per step, and must not forget earlier ones. The authors first build the evaluation: MSVQA is assembled from existing source datasets — high-altitude imagery from a large-scale optical remote sensing dataset (Sun et al., 2022), underwater imagery from (Chenping et al., 2023), and low-altitude imagery from (Zhu et al., 2021) — and converted into question-answer pairs. Preprocessing includes separating airport-containing images, sliding-window cropping of high-resolution imagery so slices do not exceed 1500 pixels with a 200-pixel backtracking step (so each target appears completely in at least one slice), adaptive scaling to under 1500×1500 pixels, coordinate recalculation with the crop/scale parameters, and filling a JSON file with a designed VQA template. Each scenario has five question types: Counting, Classification, True or False, Visual Grounding, and Fine-grained Visual Grounding. Counting maxima differ by scenario, e.g., at least 30 airplanes for high altitude and at least 80 creatures for underwater.

For the method: rather than adding one LoRA branch (which still forgets badly) or many branches requiring routing (which itself suffers forgetting without stored data), Unifier puts a Cross-Scenario Representation (CSR) module only inside the vision blocks of the vision encoder. The CSR module runs in parallel with the FFN layer, contains multiple lightweight branches (down-projection to a smaller dimension, activation, up-projection), and a projector that merges the branches' outputs into a unified feature space. Only the branch for the current scenario is trained; other branches stay frozen.

To stop the unified representation from drifting, VCC computes a scenario prototype — the mean of the K branch outputs — and instead of a strict ℓ2-distance, it penalizes global changes using relative entropy (KL divergence) between mean vectors taken along the feature and embedding channels, at a temperature τ. A second KL term aligns the projector's intermediate output between the old and new models. The two terms are averaged over all layers to form the VCC loss.

Experimental setup: Qwen2.5VL-3B is used to support mobile deployment, with the AdamW optimizer at an initial learning rate of 10⁻⁵, batch size 64 across 8 Nvidia H200 GPUs, 20 epochs for the initial task and 10 epochs for each subsequent task. The stream is split into T = 5, 10, and 20 steps, with data from one scenario per step, partitioned at the image level (not annotation level) so images in each step are mutually exclusive. Evaluation uses F1 from object detection combined with the VQA score, reported as average performance Ā = (1/T)Σ A_t and last performance A_T. Comparisons cover rehearsal-free methods (EWC, Tailor), rehearsal-based methods (ER, PODNet, VQACL, QUAD), a lower bound (Finetune), an upper bound (Joint), and Zero-shot.

Why This Matters

Impact on research. The paper argues that existing continual learning work focuses more on language models and neglects visual components, and that no prior benchmark evaluated MLLM performance under complex scenario variation for on-device deployment. MSVQA and the accompanying benchmark give the field a way to measure visual forgetting, and the VRE/VCC design offers a routing-free alternative to multi-branch approaches. Zero-shot results (e.g., 20.55 on high-altitude VQA, 19.30 on low-altitude VQA, 23.74 on underwater VQA) and the Joint upper bound (e.g., 64.97 high-altitude VQA, 87.20 indoor VQA, 74.93 indoor F1) frame the gap that continual methods are trying to close.

Real-world applications (drawn from the scenarios the paper constructs and the deployment framing it uses):

• Drones and aerial platforms operating in high-altitude and low-altitude environments, where targets are small and dense within complex backgrounds.
• Underwater exploration and marine monitoring, where light attenuation, scattering, and camouflage colors make detection hard and objects such as echinus and fish are diminutive.
• Indoor robots and assistants that must handle scene changes as rooms, lighting, and viewpoints vary.
• On-device MLLM assistants that continuously receive data from different devices, locations, and time periods without a curated retraining stage.

Industry relevance. The work targets on-device MLLMs rather than cloud-only models, and explicitly emphasizes that the method incurs no additional inference cost and an acceptable parameter increase — a practical constraint for edge deployment. The base model (Qwen2.5VL-3B) is chosen specifically to facilitate deployment on mobile platforms, and the appendix studies other foundation models and sizes (Qwen2.5VL-3B, Qwen2.5VL-7B, Qwen3VL-4B) to examine generality.

Future Directions

• How far do the results scale? The paper sets T = 5, 10, and 20; behavior at longer streams and with more than the four covered scenarios is a natural open question.
• Storing visual data. The authors note that storing all textual questions is feasible but storing all images or videos is impractical, and that visual data has lower information density and frequent noise. Rehearsal-free visual continual learning remains a constraint to work around.
• Order and model-dependence. The appendix addresses learning order (Sec. K) and compares upper bounds across foundation models and sizes (Sec. G, H), including 10-step experiments with Qwen3VL-4B — these are the axes along which robustness still needs confirmation.
• Beyond images. The motivation discusses real-world data streams of images or videos collected by varying devices; how VRE/VCC extend to video and to ambiguous scenario boundaries is raised by the design but not fully explored in the available content.

Target Audience

Researchers and engineers working on multimodal large language models, continual learning, and on-device/edge deployment, as well as practitioners in aerial remote sensing, underwater vision, and robotics who need models to keep adapting to new visual conditions without retraining from scratch. The paper is most useful to readers already comfortable with PEFT methods and continual learning metrics; newcomers will find the benchmark and scenario descriptions accessible but the VRE/VCC formulation mathematically dense.

Authors’ abstract

Multimodal large language models (MLLMs) deployed on devices must adapt to continuously changing visual scenarios such as variations in background and perspective, to effectively perform complex visual tasks. To investigate catastrophic forgetting under real-world scenario shifts, we construct a multimodal visual understanding dataset (MSVQA), covering four distinct scenarios and perspectives: high-altitude, underwater, low-altitude, and indoor environments. Furthermore, we propose UNIFIER (mUltimodal coNtInual learning with MLLMs From multi-scenarIo pERspectives), a continual learning (CL) framework designed to address visual discrepancies while learning different scenarios. Compared to existing CL methods, UNIFIER enables knowledge accumulation within the same scenario and mutual enhancement across different scenarios via Vision Representation Expansion (VRE) and Vision Consistency Constraint (VCC). Experimental results show that UNIFIER improves the last-step VQA scores by 2.70%~10.62% and the last-step F1 scores by 3.40%~7.69% compared to the state-of-the-art method, QUAD, in 20-step cross-scenario continual learning tasks. MSVQA dataset is available at https://huggingface.co/datasets/Kaij00/MSVQA.