Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning

AI summary

Overview

Research area: Machine unlearning for multimodal large language models (MLLMs), specifically privacy protection and the "right to be forgotten."

Technical level: Advanced. The paper combines multimodal model fine-tuning, parameter-update analysis (weight-space masking), and a purpose-built benchmark with custom evaluation metrics.

Scope: The paper defines a stricter goal for MLLM unlearning called benign memory forgetting, introduces the S-MLLMUn Bench to measure it, and proposes the Sculpted Memory Forgetting Adapter (SMFA) to achieve it.

What This Paper Is About

Multimodal large language models can accidentally memorize privacy-sensitive information from their training data — including information tied to specific images — and can reveal it when prompted. Existing unlearning methods can erase this content, but they usually cause serious collateral damage: the model loses general image understanding abilities it should keep. This paper argues that unlearning must remove targeted private knowledge precisely, leaving everything else intact, and builds both a benchmark and a method to pursue that standard.

Key Contributions

1. A stricter problem definition. The authors formalize benign memory forgetting as the requirement that an unlearned model simultaneously satisfy three conditions: targeted erasure on the forget set, memory retention on the retain set, and benign preservation of general image understanding.

2. S-MLLMUn Bench. The first benchmark, per the authors, designed to jointly and quantitatively assess both knowledge erasure and preservation of image understanding in MLLMs, using a dual structure that pairs sensitive memory data with fundamental understanding queries for each image.

3. Sculpted Memory Forgetting Adapter (SMFA). A new unlearning framework that derives a Memory Forgetting Adapter (MFA) via refusal-label fine-tuning, then sculpts it with a retaining anchor-guided masking mechanism to suppress over-generalized forgetting.

4. Evidence that existing methods fail the standard. Extensive experiments on S-MLLMUn Bench show representative unlearning methods cannot balance forgetting and retention, while SMFA serves as an effective baseline.

Main Findings

• Existing methods degrade foundational abilities. On a preliminary study using 1,000 synthetic image-question-answer pairs, IDK Tuning (an LLM unlearning method) and MANU (an MLLM-specific method) both achieved forgetting at the cost of the model's general visual understanding performance.

• Catastrophic forgetting in gradient- and merging-based baselines. GA Difference, KL Minimization, and TIES-Merging produced severely low scores on the retain set. GA Difference and KL Minimization rely on gradient ascent on the forget set, which disrupts the model's weight distribution; TIES-Merging struggles to merge the highly conflicting and imbalanced forgetting and retaining objectives.

• Model Tailor, MANU, and IDK Tuning forget but do not balance. They performed effective forgetting on target data yet showed poor Overall Memory scores, indicating they cannot unlearn selectively.

• SMFA achieves the best trade-off. On LLaVA-OneVision with a 5% forget ratio, SMFA reached Overall Memory scores of 0.199 (ROUGE-L) and 2.30 (Fact Score), compared with 0.121 and 1.99 for IDK Tuning and 0.006 and 0.07 for MANU. On Qwen2.5-VL with a 5% forget ratio, SMFA reached 0.147 and 1.92, versus 0.109 and 1.46 for IDK Tuning and 0.008 and 0.06 for MANU.

• Image understanding is better preserved. Most baselines showed a noticeable, comprehensive decline in image understanding across both the forget and retain sets, whereas SMFA preserved this ability far more effectively.

• Output quality is maintained. GA Difference and KL Minimization collapsed into corrupted or meaningless text with severely low Meaningful Scores. SMFA consistently achieved either the best or second-best Meaningful Score (for example, 7.98 on LLaVA-OneVision at a 5% forget ratio and 7.88 on Qwen2.5-VL at 5%).

• Both masking criteria are necessary. Ablation on LLaVA-OneVision at a 5% forget ratio shows the unsculpted MFA over-generalizes: image understanding on the retain set fell from 0.694 (Original) to 0.664, and image memory on the retain set fell from 0.705 to 0.486. Using only directional conflict or only relative magnitude masking preserved retention better (retain image memory 0.682 and 0.681 respectively) but left forgetting too weak. Combining both criteria gave the balanced result (forget image memory 0.460, retain image memory 0.622).

• The forgetting strength is tunable and stable. As the hyperparameter k increases, forgetting improves (ROUGE-L on the forget set drops) while retain-set performance stays largely stable, with only a decline in image memory when k becomes excessively large.

• Ophthalmic images are more vulnerable than facial images. Understanding scores on facial images remained relatively stable across methods, but ophthalmic medical images degraded much more easily. On the forget set, MANU and IDK Tuning showed sharp declines in ophthalmic understanding, with IDK Tuning comparatively more stable on the retain set; SMFA remained robust.

• Partial robustness to prompt attacks. Under prefix injection, refusal suppression, distractor, and style injection attacks, some attacks partially weakened SMFA's forgetting effectiveness, but the paper reports SMFA still achieved better forgetting performance. (The provided text is truncated mid-sentence at this point.)

Methodology in Plain English

The authors first reframe the unlearning problem. A model fine-tuned on a multimodal dataset is split into a forget set (sensitive image-question-answer pairs to erase), a retain set (unrelated knowledge to keep), and a new understanding set (general image questions that do not depend on memorized identities). Because full access to the retain set is unrealistic, the unlearning method may only use the forget set and a few-shot subset of the retain set whose size matches the forget set.

Their method, SMFA, works in two stages. First, they fine-tune the model's linear layers with LoRA on the forget set, replacing the real answers with refusal labels such as "I don't know" (following IDK Tuning), while mixing in a few-shot subset of the retain set to avoid degenerate, uniform refusals. The resulting parameter change — the difference between fine-tuned and original weights — is stored separately and named the Memory Forgetting Adapter, or MFA.

The problem is that the MFA over-generalizes: the refusal behavior spills over into queries that should still be answered. So in the second stage they build a retaining anchor by fine-tuning on the few-shot retain subset, producing a parameter update that represents desirable retention. They then compare the forgetting update against this anchor using two criteria. Directional conflict flags entries where the forgetting update points in the opposite direction from the retain update. Relative magnitude flags entries where the forgetting update is large relative to the (normalized) anchor update. Only entries satisfying both criteria — both harmful and dominant — are masked out. The sculpted update is added back to the original model to produce the final unlearned model.

The benchmark itself is built from 1,000 synthetic profiles of fictitious personal information. Facial images are randomly sampled from the thispersondoesnotexist dataset (based on StyleGAN), textual attributes are produced using Qwen-VL-Plus, and each record is augmented with an ophthalmic medical image and description randomly sampled from DeepEyeNet. The forget set comes in three splits — 5%, 10%, and 15% of the full dataset. Evaluation queries are paraphrased or rephrased by Qwen-VL-Plus so that methods cannot overfit to template cues. Metrics include ROUGE-L, Fact Score (0 to 10, judged by Qwen-Plus), the newly proposed Meaningful Score (0 to 10, also judged by Qwen-Plus, penalizing gibberish outputs), plus three aggregate metrics: Overall Image Understanding, Overall Memory (retain memory minus residual forget memory), and Overall Meaningful.

Experiments use LLaVA-OneVision-7B and Qwen2.5-VL-7B, each first fine-tuned on the benchmark's fine-tuning dataset to produce an "original model" that has genuinely memorized the target knowledge. Baselines are GA Difference, KL Minimization, IDK Tuning, MANU, Model Tailor, and TIES-Merging, with LoRA, all linear layers fine-tuned, and k set to 5.

Why This Matters

Impact on research. The paper shifts the evaluation standard for MLLM unlearning from "did it forget?" to "did it forget only what it should?" By arguing that retaining general image understanding is a first-class requirement rather than an afterthought, it exposes a weakness shared by current methods and provides a benchmark and metrics for measuring the trade-off. The Meaningful Score also addresses a real loophole: methods can inflate forgetting rates by emitting nonsense, which this metric penalizes.

Real-world applications:

• Consumer privacy and the right to be forgotten. Individuals who appear in training data — for example, social media images — could request removal of identity-linked knowledge without the model losing its ability to describe people in general.
• Regulatory compliance. Organizations subject to data protection rules need demonstrable, targeted removal of personal data from deployed models without a full costly retrain.
• Medical imaging systems. The benchmark deliberately includes ophthalmic medical images from DeepEyeNet, and the findings show this modality is especially fragile during unlearning — relevant for clinical AI that must purge specific patient data while retaining diagnostic ability.
• Model maintenance and customization. Treating forgetting as a modular, removable adapter update allows practitioners to apply or reverse unlearning effects on a base model rather than retraining from scratch.

Industry relevance. Fine-tuning-based deployment pipelines routinely face takedown requests and data-licensing constraints. A method that surgically edits weights while preserving benchmarked capabilities is more practical than retraining, and a benchmark that quantifies collateral damage gives engineering teams a way to certify that a deployed unlearning operation did not quietly break the product.

Future Directions

• Scaling beyond few-shot anchors. The retaining anchor is built from only a few examples and leans on the model's generalization ability. How robust this remains across domains, model sizes, and much larger forget sets is not established.

• Extending to more modalities and data types. The benchmark covers facial images and ophthalmic medical images with text attributes. Video, audio, and other structured privacy data are not addressed.

• Stronger adversarial evaluation. The prompt attack experiments are reported as only partially weakening SMFA's forgetting, and the provided text is truncated mid-sentence. A fuller characterization of attack resistance, including adaptive attacks designed against masking-based unlearning, remains open.

• Principled selection of the masking hyperparameter k. The parameter analysis shows a stability region, but the paper does not report an automatic or task-agnostic rule for choosing k, which would be needed for routine deployment.

Target Audience

Researchers and graduate students working on machine unlearning, privacy-preserving machine learning, and multimodal foundation models; practitioners responsible for data removal and compliance in deployed vision-language systems; and benchmark designers interested in evaluation protocols that measure capability trade-offs rather than forgetting alone. Readers should be comfortable with transformer fine-tuning and parameter-level analysis.

Authors’ abstract

Multimodal large language models (MLLMs) can inadvertently memorize privacy-sensitive information during training. While existing unlearning methods can remove such content, they often severely degrade the model's foundational capabilities, such as general image understanding. This critical shortfall motivates our investigation into benign memory forgetting, the precise removal of targeted, privacy-sensitive knowledge while rigorously preserving unrelated capabilities. To pioneer and evaluate progress toward this objective, we introduce S-MLLMUn Bench, the first benchmark designed to jointly and quantitatively assess an unlearning method's efficacy in knowledge erasure and the preservation of image understanding. Furthermore, we propose the Sculpted Memory Forgetting Adapter (SMFA), a new framework that enables benign memory forgetting. SMFA confines forgetting to designated memory regions, maintaining overall model performance. By initially fine-tuning the model to replace sensitive outputs with refusals, SMFA generates a memory forgetting adapter, followed by a retaining anchor-guided masking mechanism that safeguards unrelated knowledge. Extensive experiments on S-MLLMUn Bench demonstrate that existing methods fail to achieve benign forgetting, whereas our proposed SMFA serves as an effective baseline, successfully achieving targeted knowledge erasure without compromising the model's foundational visual capabilities. Code and data are available at https://github.com/zeng-zhen/S-MLLMUn.