Continual Adaptation Across Modalities: Learning and Forgetting in Unified Multimodal Models

AI summary

Overview

Research area: Continual learning for unified multimodal models (UMMs) — models that both understand images/text and generate images within a single framework.

Technical level: Advanced.

Scope: A systematic empirical study of how architectural coupling and adaptation method (LoRA-based DreamBooth personalization vs. prompt tuning vs. supervised fine-tuning) determine intra-modal learning and cross-modal interference in four representative UMMs.

What This Paper Is About

Unified multimodal models such as Chameleon, JanusPro, BAGEL and OmniDiffusion can answer questions about images and also generate images, but in deployment they must keep learning new tasks and concepts without losing old abilities. Prior work showed that updating a UMM on one modality can damage another, but only examined a single fully shared autoregressive architecture and treated generation as a static capability to be preserved. This paper builds a controlled continual-learning testbed spanning both understanding tasks and personalized image generation across four architectures that sit at different points on the parameter-sharing spectrum, in order to map how learning and forgetting actually behave.

Key Contributions

1. A continual learning framework for UMMs that covers both sides of the model: sequential fine-tuning on seven vision-language understanding tasks (six VQA plus one captioning) and sequential personalized image generation over 19 visual concepts, measuring both intra-modal retention and bidirectional cross-modal interference.
2. A controlled comparison across four architectures ordered by parameter coupling — Chameleon (fully coupled, autoregressive), OmniDiffusion (coupled backbone, masked discrete diffusion over MAGVIT-v2 tokens), JanusPro (decoupled visual encoders — SigLIP for understanding, VQ tokenizer for generation — over a shared autoregressive transformer), and BAGEL (Mixture-of-Transformer-Experts with separate FFN experts and shared self-attention) — with LoRA hyperparameters grid-searched per model so differences reflect architecture rather than optimization.
3. Empirical evidence that identically placed, similarly sized LoRA updates (39.9M trainable parameters for Chameleon, 43.1M for OmniDiffusion, 37.4M for JanusPro, all in the shared backbone; 12.6M for BAGEL, confined to generation-specific attention projections) produce very different interference levels, ruling out adapter placement or capacity as the cause.
4. A characterization of what gets forgotten: post-training gains are the most fragile, and purely post-training capabilities (personalized generation) collapse while pretraining capabilities (generic generation, general understanding) remain comparatively stable — plus a method-dependence result showing prompt-based personalization largely avoids this collapse.

Main Findings

• Parameter isolation reduces cross-modal interference (generation → understanding). Continual DreamBooth personalization degrades understanding by 3.6 percentage points (pp) on average in fully coupled Chameleon, 2.4 pp in OmniDiffusion, almost nothing in JanusPro, and not at all in BAGEL, whose Mixture-of-Transformer-Experts keep understanding and generation updates separate over shared attention. Per-task records show BAGEL's deltas are exactly zero, while JanusPro's personalization mildly improves most benchmarks (Fin +0.9, Med +2.0, RS +1.4 pp mean per stage) with Autonomous Driving as the exception (−2.4 pp).

• The same ordering holds in reverse (understanding → generation). Understanding training degrades Chameleon's personalized generations most (Q-Align Δ −16.5 pp) and leaves BAGEL's at Δ ≈ 0, with the other two models in between. Generic generation is far less affected: Q-Align Δ −5.8 pp for Chameleon and near zero for JanusPro and BAGEL.

• Generation is markedly more fragile than understanding within the same model. After the same seven sequential understanding tasks, forgetting (BWT) ranges from −4.7 pp for BAGEL to −10.4 pp for Chameleon (−7.2 pp for OmniDiffusion, −10.1 pp for JanusPro). On the generation side, sequential DreamBooth over 19 concepts destroys earlier concepts: Q-Align falls from 0.46 to 0.06 for Chameleon (87%), 0.81 to 0.09 for BAGEL (89%), 0.58 to 0.06 for JanusPro (90%), and 0.33 to 0.05 for OmniDiffusion (84%) — a drop of 84% to 90% in every model.

• What post-training adds is the first to be forgotten. Within the understanding sequence, forgetting concentrates on the tasks that training helped most relative to zero-shot: for Chameleon the three tasks with the largest gains (Medical, Remote Sensing, Autonomous Driving, each improved by roughly 45 pp) also lose the most (11.5 to 18.3 pp), while tasks with smaller gains are nearly retained. JanusPro follows the same trend almost monotonically, from 0.1 pp of forgetting for a 1.6 pp gain to 17.3 pp of forgetting for a 45.8 pp gain.

• Different metrics tell different stories about generation collapse. The near-total collapse is specific to semantic-consistency scores: Q-Quality falls from 0.68 to 0.06 for Chameleon, 0.50 to 0.07 for OmniDiffusion and 0.67 to 0.07 for JanusPro, but only 0.86 to 0.49 (−43%) for BAGEL, whose generation expert still produces visually plausible images. CLIP-I erodes more slowly (0.80 to 0.51 for Chameleon, 0.59 to 0.36 for OmniDiffusion, 0.74 to 0.23 for JanusPro, 0.71 to 0.23 for BAGEL), and CLIP-T stays nearly flat (e.g., 0.21 to 0.18 for Chameleon).

• Why generation collapses faster: data density, not just sequence length. Although the generation sequence has more items than the understanding sequence (19 concepts vs. 7 tasks), each concept is compressed into dense, identity-specific weights from only 6 to 10 reference images, so later updates overwrite earlier ones; understanding tasks are spread over large training sets and a shared feature space that supports retention.

• General understanding can be disrupted then recover, but not always. Chameleon's zero-shot MMBench drops from 37.63 to 20.70 after the captioning task and ends at 63.36, its peak. JanusPro's MMBench falls to 17.53 after vizwiz and recovers to within 0.6% of its peak, while its MMMU declines from 24.92 to 15.72. Chameleon's OCR diagonal (7.54) ends below its already weak zero-shot OCR (10.40); JanusPro's OCR improves from 49.37 to 52.40 and BAGEL's dips from 85.58 to 81.10.

• Adaptation method changes the forgetting profile. DreamBooth personalization written into parameters after understanding training degrades Chameleon's Q-Align from 0.41 to 0.24 and Q-Quality from 0.39 to 0.24, OmniDiffusion by 0.100 and 0.096, JanusPro by 0.045 and 0.051, and BAGEL by within 0.003. Prompt-based personalization (YoPrompt, a 16-token soft prompt) is essentially unaffected for Chameleon, OmniDiffusion and JanusPro (Chameleon slightly positive, +0.001 to +0.017), with BAGEL the exception (largest CLIP-I change Δ −0.038).

• Prompt tuning can be competitive with parameter updates. Per-concept, YoPrompt improves on DreamBooth for BAGEL (Q-Align 0.805 to 0.845) and especially OmniDiffusion (0.341 to 0.412), while DreamBooth stays ahead for JanusPro (0.577 vs. 0.521) and Chameleon (0.409 vs. 0.386). BAGEL's Q-Quality is 0.856 (DreamBooth) and 0.875 (YoPrompt), versus 0.37 to 0.40 for Chameleon and OmniDiffusion.

• Decoupling is a double-edged sword. Full decoupling nearly removes cross-modal interference, but it also severs the cross-modal synergy that motivates unified models in the first place. BAGEL is architecturally immune to understanding-side updates yet is still destroyed by sequential DreamBooth, because the first concept's identity lives in the same Generation Expert that later concept updates overwrite.

Methodology in Plain English

The authors take four publicly available UMMs chosen to span the design space from "everything shares one set of weights" to "generation and understanding have mostly separate weights," and run two continual-learning sequences on each.

For the understanding side, they fine-tune each model one task at a time on seven vision-language benchmarks (Finance, Science, Medical, Remote Sensing, Autonomous Driving, and OCR as VQA tasks, plus vizwiz captioning), always in the same order. After each new task they re-test all previous tasks, producing a lower-triangular score matrix in which the diagonal is how well the model learned the task it just saw and the entries below it are how well it remembers. MMBench and MMMU are used as zero-shot probes of general understanding rather than training tasks, and a backward-transfer (BWT) row summarizes average forgetting.

For the generation side, they teach each model 19 different visual concepts one after another, each from only 6 to 10 reference images, following the DreamBooth recipe — all concepts written into a single LoRA adapter so later concepts can overwrite earlier ones. They also run a second, parameter-free scheme (YoPrompt) that learns a 16-token soft prompt per concept while the model stays frozen. Generation quality is scored with CLIP-I (identity similarity), CLIP-T (text alignment), and Q-Align / Q-Quality (an LLM-based scorer over five quality categories combined into a 0–1 score with weights 1.0, 0.75, 0.5, 0.25 and 0.0). Captioning is scored with SPICE.

To measure cross-modal interference, they combine the two sequences: personalize first and watch understanding, then train understanding first and watch both generic and personalized generation. All adaptation uses LoRA with learning rate, rank and alpha chosen by grid search per model, so that architectural differences are not confounded with tuning. They also report where the LoRA update actually lands in each model, showing that Chameleon, OmniDiffusion and JanusPro all place their entire update in the shared backbone while BAGEL's 12.6M parameters sit only in generation-specific projections — evidence that the observed interference gradient tracks architecture.

Why This Matters

The paper gives the first systematic map of what happens when a unified multimodal model is asked to keep learning after deployment, and it identifies parameter coupling as the lever that controls cross-modal interference. It also reframes forgetting: the abilities most at risk are not the model's foundational skills but precisely the ones added by post-training, which is exactly what continual learning is supposed to deliver. For anyone building self-improving multimodal systems, that means the standard assumption "fine-tuning adds capability safely" does not hold on the generation side.

Real-world applications:

• Personalization services where users add their own subjects, products or characters to an image generator over time, and earlier users' concepts must not silently degrade.
• Continually updated domain assistants in medical, remote sensing, autonomous driving or OCR pipelines, where new specialties are added without eroding older ones.
• Foundation-model maintenance in production, where vendors must decide whether to isolate modality-specific parameters or keep a shared backbone.
• Choosing an adaptation method: parameter-free prompt tuning versus parameter-modifying LoRA, given the very different forgetting profiles the paper measures.

Industry relevance: The finding that post-training capabilities collapse by 84% to 90% while pretraining capabilities stay stable is directly actionable for anyone shipping iterative model updates; it suggests that personalization knowledge should be stored outside the shared weights, or that architecture (expert routing) should be chosen with continual adaptation in mind rather than only with benchmark accuracy in mind.

Future Directions

• Closing the gap between isolation and synergy: full decoupling removes interference but also removes the cross-modal transfer that justifies unification, and the paper explicitly frames this as an open trade-off rather than a solved problem.
• Making generation-side continual learning survivable — since even BAGEL, protected from understanding-side updates by construction, is destroyed by sequential DreamBooth, the paper's results imply a need for retention mechanisms on the generation expert itself.
• Explaining the BAGEL/YoPrompt anomaly, where prompt-based personalization shows small but consistently negative deltas (largest CLIP-I Δ −0.038); the authors offer only a tentative routing-based explanation and state their data do not allow definitive attribution.
• Extending the study beyond the four architectures and two personalization schemes examined, and beyond LoRA-based adaptation, to test whether the coupling-versus-interference gradient generalizes as a design rule for continually evolving UMMs.

Target Audience

Researchers and engineers working on continual learning, multimodal foundation models, or diffusion/autoregressive image generation who need to understand the practical trade-offs of architecture and adaptation strategy. It is also valuable for practitioners deciding how to add personalization or new domain skills to a deployed UMM without regressing existing capabilities. Readers should be comfortable with LoRA, DreamBooth-style personalization, and standard continual-learning evaluation (BWT, task matrices).

Authors’ abstract

Unified multimodal models (UMMs), which can process and generate diverse modalities within a single framework, have recently emerged as a powerful paradigm. However, when these models undergo continual learning on multiple tasks across different modalities, their learning and forgetting dynamics remain poorly understood. In this paper, we systematically investigate how architectural design and adaptation method shape intra-modal learning and cross-modal interference between the generation and understanding sides. Specifically, we report three findings. First, the severity of cross-modal interference decreases with the degree of modality-specific parameter isolation. Second, a within-model asymmetry: the generation side of a UMM is markedly more fragile than its understanding side. Third, what post-training adds is the first to be forgotten: purely post-training capabilities such as personalized generation collapse by 84\% to 90\%, while pretraining capabilities remain stable. Together, these results clarify how parameter isolation and post-training fragility shape UMMs' learning and forgetting dynamics, offering a starting point for developing continually evolving UMMs.