Is Forgetting Unified? Exploring Concept Unlearning in Unified Multimodal Models

AI summary

Overview

Research area: Machine unlearning and concept erasure in unified multimodal models (UMMs) — models that perform both text-to-image generation and image-to-text visual understanding with a shared parameter set.

Technical level: Intermediate. The paper is readable with a general grasp of vision-language models and fine-tuning, but the method section involves closed-form weight editing and low-rank adaptation.

Scope: The paper asks whether unlearning a concept in one pathway of a unified multimodal model carries over to the other, introduces a 30-concept diagnostic benchmark (UUBench) to measure this, and proposes a coordinated editing method (UniForget) that suppresses a target concept in both generation and understanding while limiting collateral damage.

What This Paper Is About

Unified multimodal models can both draw a concept and recognize it, yet prior unlearning work only ever targeted one pathway at a time — either suppressing image generation of a concept or erasing factual knowledge from a model's text answers. The authors ask whether existing single-pathway methods are enough to make a model both stop generating a concept and stop identifying it, and they find that they are not. Their goal is to define, measure, and then close this "cross-pathway" forgetting gap without destroying image quality, answer informativeness, or knowledge of similar neighboring concepts.

Key Contributions

1. Cross-pathway unlearning formulation. The authors give what they describe as the first systematic study of joint concept unlearning across image generation and visual understanding in a single UMM, and explicitly separate pathway-specific forgetting, joint forgetting, and utility-preserving joint forgetting.

2. UUBench, a diagnostic benchmark. A controlled benchmark of 30 target concepts (10 each of characters, landmarks, and objects), each paired with two same-category neighbors, for 60 neighbors and 90 entities in total, backed by 720 reference images. It pairs text-to-image prompts with identity and non-identity visual questions about identical entities, uses strictly separated seen and held-out splits, and defines ten metrics including two joint ones (Unified Forget, Unified Strict Forget).

3. UniForget, a coordinated unlearning method. A two-stage approach that first performs closed-form weight editing on the generation pathway and then trains LoRA adapters on the understanding pathway, merging the two in parameter space with a calibrated fraction ρ.

4. Empirical analysis of trade-offs. Component ablations, a composition-strength sweep, multi-concept scaling up to 30 concepts per checkpoint, and robustness tests to prompt attacks and image perturbations, showing how joint forgetting trades off against image quality and answer utility.

Main Findings

• Single-pathway methods do not transfer. TIME reaches 100.0% Target Forget (TF) on BAGEL-7B-MoT and 96.7% on Janus-Pro-7B, yet only 24.0% and 30.7% Direct Forget (DF) respectively. UCE reaches 83.3% TF on BAGEL and 80.0% on Janus, with only 4.0% and 15.3% DF.

• Understanding-side baselines barely forget even in their own pathway. GD and KL produce 0.0% TF on both backbones. On Janus-Pro-7B, GD reaches 22.7% DF and KL 14.0%; on BAGEL-7B-MoT, GD reaches 0.7% DF and KL 0.0% DF.

• Naive composition is weak. TIME + GD, combining the strongest baseline from each side, achieves only 31.3% Unified Forget on BAGEL-7B-MoT and 41.3% on Janus-Pro-7B.

• UniForget substantially improves joint forgetting. It reaches 80.0% Unified Forget on BAGEL-7B-MoT and 92.7% on Janus-Pro-7B, with 100.0% TF on both backbones. On the stricter joint metric it improves over TIME + GD by 18.9 and 27.0 percentage points, landing at 47.6% and 58.0% Unified Strict Forget.

• Forgetting and preservation pull in opposite directions, differently per architecture. Janus-Pro-7B performs better on both joint metrics (92.7% Forget, 58.0% Strict Forget), while BAGEL-7B-MoT preserves neighbor generation better, with 84.6% Strict Neighbor Retention (SR) versus 73.2% for Janus.

• General capabilities are mostly stable on understanding but degrade on generation. UniForget moves MMBench from 88.4% to 87.8% and MME-P from 83.3% to 82.1% on BAGEL-7B-MoT; on Janus-Pro-7B, from 79.1% to 77.2% and 76.8% to 76.1%. GenEval drops from 72.7% to 63.4% (BAGEL) and 78.9% to 70.1% (Janus), and DPG-Bench from 83.4% to 79.9% and 83.1% to 74.2%. BAGEL's MME-C falls from 85.5% to 77.7%, while Janus is nearly unchanged (35.0% to 34.6%).

• Full prompt construction preserves neighbors. Compared with a Neighbors-only variant, the full construction holds 100.0% TF on both models while raising NR from 90.4% to 98.1% on BAGEL-7B-MoT and from 85.7% to 92.9% on Janus-Pro-7B.

• Each understanding-side loss component buys utility. On BAGEL, adding Masked KL to the configuration with Pairwise preference raises DF from 77.2% to 80.0% and TU from 63.8% to 72.2%. On Janus, adding Anti-collapse keeps DF at 92.7% while raising TU from 67.1% to 75.6%.

• Pushing the understanding update harder trades utility for forgetting. Raising ρ from the selected 0.4 to 0.6 on BAGEL improves DF from 80.0% to 93.3% but drops TU from 72.2% to 55.6%. On Janus, going from 1.0 to 1.2 moves DF only from 92.7% to 93.3% while TU falls from 75.6% to 55.0%.

• The two pathways scale differently with concept count. Across N ∈ {1, 5, 10, 20, 30} concepts per checkpoint, generation-side editing holds 100.0% TF, but at N = 30 neighbor retention retains 81.5% (BAGEL) and 41.8% (Janus) of its single-concept value, with GenEval at 39.2% and 32.7%. Understanding-side DF degrades from 80.0% to 51.2% (BAGEL) and from 92.7% to 37.1% (Janus), while MMBench stays nearly stable.

• Robustness is strong for generation, mixed for understanding under prompt attacks. Generation holds 100.0% TF under instruction override, target-name repetition, and paraphrasing, with SF of at least 86.7% and NR of at least 92.9%. Under Gaussian blur, center cropping, and JPEG compression, target-name suppression ranges from 84.6% to 86.7% on BAGEL and 95.1% to 95.7% on Janus. Prompt attacks are model-specific: name exposure yields 61.7% suppression on BAGEL versus 97.5% on Janus, while forced-short prompts yield 100.0% and 77.3% respectively.

Methodology in Plain English

The authors start by defining what "forgetting" should mean for a concept like the Eiffel Tower: the model should stop drawing it, and when shown a photo of it, should decline to name it but still describe it correctly as, say, an iron lattice tower in Paris — rather than refusing, hallucinating, or calling it something else. They then build UUBench around that definition, with each target paired to two visually or functionally similar concepts (Eiffel Tower with Tokyo Tower and Blackpool Tower) so that over-aggressive editing is caught.

For the method, they treat the two pathways separately and then reconcile them. On the generation side they use a closed-form ridge-regression edit of selected projection matrices: the model is pushed to produce the same output for "Eiffel Tower" prompts as for a generic category phrase, while a preservation term keeps outputs for generic, non-target, and neighbor prompts unchanged, and a third term keeps the edited weights close to the originals. On the understanding side they freeze the backbone and train LoRA adapters with a composite loss that redirects identity questions to the generic category, supervises detailed name-free descriptions, ranks informative name-free answers above name-leaking ones, protects neighbor recognition, and adds masked KL and an anti-collapse margin. Finally, the learned LoRA update is folded back into the generation-edited weights, scaled by a fraction ρ (0 gives generation-only editing; 1 gives the full merge), which becomes the knob for trading forgetting against utility.

Evaluation uses two architecturally different 7B models — BAGEL-7B-MoT with modality-specific Transformer parameters, and Janus-Pro-7B with a shared Transformer backbone and decoupled visual encodings — compared against UCE, TIME, GD, KL, and a TIME + GD composition.

Why This Matters

Impact on research. The paper reframes unlearning in multimodal models as an outcome that must be verified across modalities rather than assumed from shared parameters. UUBench provides paired generation and understanding tasks plus metrics that jointly require success in both, which existing multimodal unlearning benchmarks — largely text-response based — do not offer. The negative result that generation editors and understanding editors do not compose well is itself useful: it sets a baseline that future unified methods must beat.

Real-world applications:

• Removing copyrighted characters or branded landmarks from generative image tools while keeping the tool's ability to describe those images accurately.
• Enforcing privacy or right-to-be-forgotten requests for individuals or properties that appear in both generated images and image-question answering.
• Safety and policy compliance, where a deployed assistant must neither depict nor confidently identify a disallowed subject.
• Model auditing and compliance reporting, where a single benchmark score can attest that a concept is suppressed on both output surfaces.

Industry relevance. Companies shipping unified models as products face requests to remove a concept once, not twice, and to prove it stayed removed. The paper's utility-preservation metrics — refusal-free generic descriptions, neighbor retention, GenEval and DPG-Bench scores — map directly onto the product risks that make unlearning teams reluctant to deploy aggressive edits.

Future Directions

• Scaling beyond a handful of concepts. Joint forgetting degrades sharply at N = 30 concepts per checkpoint, especially for understanding (DF down to 51.2% on BAGEL and 37.1% on Janus) and neighbor retention on Janus (41.8% of its single-concept value). Methods that hold up when many concepts must be removed at once remain open.

• Closing model-specific robustness gaps. Under prompt attacks the two models diverge widely (name exposure: 61.7% suppression on BAGEL versus 97.5% on Janus; forced-short prompts: 100.0% versus 77.3%), and the paper does not explain why.

• Reconciling the forgetting–utility trade-off. The ρ sweep shows forgetting can be pushed to 93.3% DF only by dropping target utility to roughly 55%, so finding operating points that improve both rather than trading them is unresolved.

• Generalization and stronger guarantees. The authors list as limitations that only two 7B UMMs and 30 target concepts were tested, that image quality and visual utility are judged by automated metrics and LLM judges rather than humans, and that evaluation captures behavioral forgetting — certified deletion from model weights is explicitly out of scope.

Target Audience

Researchers and engineers working on machine unlearning, concept erasure, safety, or model editing for multimodal and generative models; benchmark designers who need paired evaluation across output modalities; and product or compliance teams at organizations deploying unified image-generation-and-understanding models who need to reason about what "removing a concept" actually requires.

Authors’ abstract

Unified multimodal models (UMMs) integrate image generation and visual understanding, but are existing single-pathway unlearning methods sufficient to suppress target concepts across both? In this work, we study cross-pathway concept unlearning. Our goal is to prevent a model from both generating images of a target concept and identifying it in input images, while preserving other capabilities. To this end, we introduce UUBench, a diagnostic benchmark covering 30 target concepts and 60 semantically related neighboring concepts. Experiments on representative UMMs show that the evaluated single-pathway methods achieve limited joint forgetting: generation-side editors strongly suppress target generation but yield limited forgetting in visual understanding, while understanding-side baselines struggle to achieve substantial forgetting even in their own pathway. Their direct composition also yields limited joint forgetting. We therefore propose UniForget, achieving joint forgetting rates of 80.0% on BAGEL-7B-MoT and 92.7% on Janus-Pro-7B, outperforming representative baselines. Further analysis reveals trade-offs between joint forgetting and capability preservation.