Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models

AI summary

Overview

Research area: Continual learning (CL) for Unified Multimodal Generative Models (UMGMs) — autoregressive transformers that perform both visual understanding and image generation.

Technical level: Intermediate. The paper assumes familiarity with LoRA, mixture-of-experts routing, and continual-learning evaluation protocols, though the core ideas are explained clearly.

Scope: The paper identifies, explains, and mitigates a previously underexplored failure mode — forgetting that occurs across modalities when a unified model is continually fine-tuned on text-output tasks — and proposes a lightweight adapter architecture called Modality-Decoupled Experts (MoDE) that addresses this alongside conventional within-modality forgetting.

What This Paper Is About

Unified Multimodal Generative Models such as Chameleon and Janus-Pro can both answer questions about images and generate images. When such a model is fine-tuned one task at a time (continual instruction tuning), it forgets earlier knowledge. Prior work has studied forgetting within one modality (intra-modal), but the paper shows that tuning on visual question answering tasks also degrades the model's ability to generate images — a phenomenon the authors call inter-modal catastrophic forgetting. The goal is an architecture that preserves both multimodal understanding and pre-trained image generation ability while continually learning new tasks.

Key Contributions

1. Identification and empirical validation of inter-modal catastrophic forgetting in UMGMs under continual instruction tuning, contrasted with the more commonly studied intra-modal forgetting.
2. A theoretical account attributing inter-modal forgetting to modality gradient conflict (Definition 1), with a formal proof (Proposition 1) that text-gradient updates on shared parameters increase visual loss at order O(η), and Proposition 2 showing modality decoupling reduces this interference to O(η²).
3. Modality-Decoupled Experts (MoDE), a lightweight architecture combining a text mixture-of-LoRA-experts (T-MoE) with a single visual LoRA adapter (V-Adapter), plus logit-level knowledge distillation from the frozen pre-trained teacher to preserve image generation.
4. Extensive experiments showing MoDE outperforms baselines such as CL-MoE and Model Tailor when both intra- and inter-modal forgetting must be mitigated simultaneously.

Main Findings

• Naive sequential tuning hurts image generation. Tuning Chameleon with LoRA on a sequence of three VQA datasets degraded both overall image quality and prompt-image alignment. In Figure 2, the prompt "A photo of a car" produced an image of a building after the third VQA task.
• Modality gradient conflict is real. Figure 5 shows a cosine distance distribution between text- and image-modality gradients in modality-coupled MoE LoRA on ScienceQA, with substantial conflict; Figure 6 shows MoDE has perfectly orthogonal gradients between modalities, i.e., no conflict.
• MoDE best preserves image generation. In Table 1, MoDE reaches an FID of 53.74, close to the zero-shot pre-trained value of 52.13, versus 65.87 for CL-MoE, 65.16 for MoE-LoRA, 56.12 for Seq LoRA, 56.08 for DualPrompt, and 55.47 for Model Tailor.
• MoDE gives the best multimodal understanding accuracy. Average accuracy (ACC) of 33.47% for MoDE, versus 33.01% (MoE-LoRA), 32.86% (CL-MoE), 32.62% (Model Tailor), 31.92% (DualPrompt), 28.43% (Seq LoRA), and 22.48% for zero-shot.
• MoDE has the lowest overall drop. Δ = 22.78 for MoDE, compared with 24.31 (MoE-LoRA), 24.46 (CL-MoE), 24.70 (Model Tailor), 25.40 (DualPrompt), and 28.57 (Seq LoRA).
• DualPrompt's low forgetting is misleading. It reports 6.82% average forgetting, but its best ImageNet accuracy is only 24.55% while other methods exceed 70% — little forgetting because little was learned.
• MoE-LoRA handles intra-modal forgetting but worsens inter-modal forgetting, with FID 65.16 against 56.12 for vanilla Seq LoRA.
• Ablation (Table 3) shows both components matter: T-MoE alone preserves generation (FID 51.28, image alignment 0.5317) but yields lower accuracy (33.03%); removing knowledge distillation (MoDE w/o KD) raises FID to 54.61 and lowers accuracy to 33.07%; full MoDE achieves 53.74 FID and 33.47% accuracy with the lowest forgetting (25.99).
• Qualitative examples (Figure 4, Table 4) show MoDE capturing prompt details that baselines miss, e.g., yellow leaves around a barn for "Barn in the fall season with leaves all around," which Model Tailor and CL-MoE fail to render.
• The distillation weight was tuned. The authors set λ = 0.3 based on optimal performance in a sweep (Appendix D).
• Training time, memory, and parameter overhead are stated to be reported in Appendix H, but no numerical values appear in the provided content.

Methodology in Plain English

The model's original weights are frozen. Instead of updating the shared backbone, the authors attach two separate, trainable modules to the transformer's linear (MLP) layers:

• A V-Adapter — a low-rank (LoRA) adapter that handles only image tokens and is responsible for both understanding and generating images.
• A T-MoE — a mixture of LoRA experts plus a router that handles only text tokens; the router selects which experts contribute, which lets the model absorb new understanding tasks without overwriting old ones.

Because text and image tokens flow through disjoint parameter sets, their gradients cannot conflict — the paper proves this bounds visual-loss drift at O(η²) rather than O(η). To further protect generation quality, the V-Adapter is trained with a combined loss: the standard cross-entropy objective plus a logit-level knowledge-distillation loss (Kullback–Leibler divergence between softened teacher and student predictions, scaled by β²) computed on a small subset of images from LAION-5B. The final loss is L_CE + λ·L_KD with λ = 0.3. Evaluation used two backbones, Chameleon and Janus-Pro, five instruction-tuning datasets (ScienceQA, TextVQA, ImageNet, GQA, VizWiz) and the CustomConcept101 image-generation dataset (101 concepts in 16 categories).

Why This Matters

Impact on research. The paper reframes continual learning for multimodal models: when a single backbone produces more than one output modality, forgetting is not a single-axis problem. The gradient-conflict formalization gives a principled reason why prior MoE-based CL methods — which share experts across modalities — can actively worsen generation quality even while protecting understanding. It also supplies a diagnostic lens for evaluating whether a CL method is genuinely retaining knowledge or merely failing to learn.

Real-world applications (implied by the setting):

• Incrementally adding new visual question-answering or classification capabilities to a deployed multimodal assistant without retraining from scratch or losing image generation.
• Personalized image generation where new concepts or styles are learned over time (the paper uses CustomConcept101, a concept-centric benchmark) while preserving the base model's general fidelity.
• Domain-shift adaptation, where models must absorb new task data as it arrives (e.g., new product categories, new document types) under compute and storage limits that rule out full retraining.
• On-device or resource-constrained deployment, since MoDE trains only lightweight adapters while the backbone stays frozen.

Industry relevance. Freezing the backbone and training small adapters is directly compatible with adapter-serving infrastructure, and the distillation anchor uses a small reference image subset rather than the full pre-training corpus — important when proprietary or licensed data cannot be reused.

Future Directions

• Beyond the visual direction studied here. The paper states that complementary results on understanding forgetting induced by tuning on image-generation tasks appear in Appendix F, which the provided content does not include — extending the analysis in that direction remains an open area.
• Scaling the expert count and routing. T-MoE's number of experts n and its routing behavior are only partially explored; how routing scales with many more tasks and whether expert growth can be made automatic are open questions.
• Broader backbone coverage. Main results use Chameleon; Janus-Pro results are reported in Appendix G. Testing on a wider class of autoregressive UMGMs with different vision tokenizers (VQGAN, CLIP, SigLIP) and decoders (VQ-VAE, diffusion-based generators) would test generality.
• Efficiency characterization. The paper points to Appendix H for training time, memory, and parameter overhead, describing a "favorable trade-off" but reporting no figures in the provided content — quantifying adapter overhead against accuracy gains is a natural next step.

Target Audience

Researchers and engineers working on continual learning, multimodal large language models, and parameter-efficient fine-tuning (LoRA, mixture-of-experts), as well as practitioners building unified understanding-plus-generation systems who need models that can be updated repeatedly in deployment. Readers without a background in gradient-based optimization or adapter methods will find the methodology sections demanding, though the problem statement and empirical results are accessible.

Authors’ abstract

Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git