AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models

AI summary

Overview

• Research area: Machine unlearning for Multimodal Large Language Models (MLLMs), specifically removing fine-grained visual concepts (e.g., a specific person's identity) from models trained on image-text data.
• Technical level: Advanced. Familiarity with MLLMs, vision-language alignment, adversarial training, LoRA adapters, and Gumbel-Softmax sampling is required to follow the methodology.
• Scope: This paper proposes AUVIC, an adversarial framework for precise visual-concept unlearning in MLLMs, and introduces VCUBench, the first benchmark for evaluating such unlearning in single-person and multi-person settings.

What This Paper Is About

Multimodal LLMs are trained on massive internet-scraped image-text datasets that often contain private or copyrighted content, and regulations like GDPR's "right to be forgotten" require the ability to remove specific data without retraining from scratch. Existing unlearning methods, originally designed for text, either damage overall model performance or over-generalize—erasing not just the target concept but also visually or semantically similar ones, which is especially harmful in group photos where forgetting one person can corrupt recognition of others nearby. The paper's goal is to enable precise, per-concept forgetting in MLLMs while preserving performance on related non-target concepts.

Key Contributions

1. AUVIC framework: A novel adversarial unlearning method that combines a trainable perturbation generator, multimodal (text and image) perturbations, and a dynamic anchor preservation mechanism to erase a target visual concept with high specificity.
2. Dynamic anchor preservation: A Gumbel-Softmax-based mechanism that automatically selects the most semantically similar concepts to the target and explicitly protects them during unlearning, directly mitigating collateral forgetting.
3. VCUBench benchmark: The first benchmark dedicated to evaluating visual concept unlearning in MLLMs, containing 15,000+ image–question–answer triples covering both single-person and multi-person (group photo) scenarios, with dedicated non-target control data.
4. State-of-the-art empirical results: AUVIC achieves the best trade-off between forgetting the target and retaining non-target concepts across six target identities and five metrics, with minimal loss of language fluency.

Main Findings

• Naive unlearning causes severe collateral damage: Applying Gradient Ascent to forget "Donald Trump" dropped recall for "Joe Biden" from high levels to 11% and for "Boris Johnson" to 23%, while dissimilar concepts like "Hello Kitty" were largely unaffected. Collateral forgetting is asymmetric and feature-dependent.
• AUVIC dominates baselines on targeted forgetting: On the Trump example, AUVIC achieved 93.64% Target Forgetting Accuracy (TFA), 83.17% Non-Target Retain Accuracy (NTRA), and 88.10% Group Retain–Forget F1—versus GA's 84.48% TFA but only 30.17% NTRA.
• Strong results across six target concepts: Averaged over six identities, AUVIC reached 96.99% TFA, 75.34% NTRA, 84.94% GRF-F1, and 96.82% Efficacy, outperforming GA, Preference Optimization (PO), and GA+KL on every headline metric.
• Language fluency is preserved: AUVIC achieved the lowest masked perplexity (8.34 average) among compared methods, indicating that vision-side fine-tuning did not degrade caption generation.
• Both components matter: Ablation showed that removing adversarial perturbations dropped GRF-F1 to 70.98, removing Gumbel sampling dropped it to 72.37, and removing both collapsed it to 38.55—confirming each mechanism is essential.
• Precision in group scenes: In qualitative examples, AUVIC erased Trump's identity (referring to him only as "a man with blonde hair") while still correctly naming Joe Biden in the same image, whereas GA forgot both.

Methodology in Plain English

The researchers treat unlearning as a two-player game. A small generator network (three linear layers) takes CLIP image features and produces a tiny, imperceptible pixel-level perturbation (bounded by ε = 8/255) designed to make the target concept as active as possible inside the model. The MLLM (acting as discriminator) is then trained to do three things at once under those adversarial conditions:

1. Suppress the target: push the model's confidence about the target concept toward "not this person" (the forgetting loss).
2. Preserve related concepts: keep correct predictions on a set of semantically similar identities selected dynamically by comparing embedding similarity to the target, with Gumbel-Softmax noise adding controlled randomness (the preservation loss).
3. Stay consistent: penalize KL divergence between the model's outputs on clean versus perturbed images, so behavior outside the target region doesn't drift (the consistency loss).

To make training efficient, they freeze most of LLaVA-1.5 and only insert LoRA adapters (rank 32) into the CLIP vision tower. On the text side, they vary prompts (greetings, rephrasing, distractors) and prepend the target name so the model's attention is anchored. The generator and the model are updated alternately. Because existing benchmarks didn't test multi-person scenarios, they built VCUBench using verified celebrity photos—both solo and group shots—paired with VQA-style questions probing presence, recognition, spatial relations, and context.

Why This Matters

• Research impact: This is the first work to systematically address fine-grained visual concept unlearning in MLLMs, showing that text-based unlearning methods transfer poorly to multimodal settings and that adversarial perturbation is a powerful tool for localizing forgetting in visual representation space. VCUBench provides a reproducible evaluation standard for a previously unmeasured problem.
• Real-world applications:
  • GDPR compliance: Companies can honor individual erasure requests applied to multimodal model weights without retraining from scratch.
  • Copyright and likeness removal: Artists, public figures, or rights holders can request removal of their visual identity from a deployed model.
  • Safety and misinformation: Suppress recognition of harmful visual concepts or prevent models from identifying protected individuals in sensitive contexts (e.g., witnesses, minors).
  • Content moderation and privacy in group imagery: Enable selective anonymization of one person in a photograph while preserving recognition of others.
• Industry relevance: As multimodal models proliferate in consumer products and enterprise deployments, the ability to surgically remove data-derived knowledge—without breaking model utility—becomes a regulatory and reputational necessity. The parameter-efficient LoRA-based approach makes this practical for real deployment on commodity hardware (the authors trained on 3× RTX 4090s).

Future Directions

• Scaling to non-person visual concepts: Extending AUVIC beyond public-figure identities to objects, logos, styles, or scenes, where "similarity" is harder to define than in face recognition.
• Robustness against adversarial recovery: Evaluating whether forgotten concepts can be re-extracted through prompt engineering, fine-tuning attacks, or membership inference—a standard concern in unlearning research.
• Broader modality coverage: Applying the adversarial unlearning paradigm to audio, video, or larger generative MLLMs where perturbation design must be rethought.
• Theoretical analysis of forgetting isolation: Formalizing why adversarial perturbations localize erasure in visual feature space and deriving bounds on collateral forgetting to guide anchor-set selection.
• Benchmark extension: Expanding VCUBench with more identities, demographic diversity, and adversarial group compositions to test generalization of unlearning methods.

Target Audience

This paper is most valuable to machine learning researchers working on model unlearning, privacy, and multimodal systems, particularly those focused on vision-language models. It also serves AI safety and compliance practitioners who need concrete techniques for GDPR-style data removal, and engineers deploying MLLMs in production who require parameter-efficient, selective forgetting methods. Readers without prior exposure to adversarial training or LoRA fine-tuning will find the methodology section demanding, but the motivation analysis and benchmark contributions are accessible to a broader AI audience.

Authors’ abstract

Multimodal Large Language Models (MLLMs) achieve impressive performance once optimized on massive datasets. Such datasets often contain sensitive or copyrighted content, raising significant data privacy concerns. Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning. This technique allows for the removal of target data without resource-consuming retraining. However, while well-studied for text, visual concept unlearning in MLLMs remains underexplored. A primary challenge is precisely removing a target visual concept without disrupting model performance on related entities. To address this, we introduce AUVIC, a novel visual concept unlearning framework for MLLMs. AUVIC applies adversarial perturbations to enable precise forgetting. This approach effectively isolates the target concept while avoiding unintended effects on similar entities. To evaluate our method, we construct VCUBench. It is the first benchmark designed to assess visual concept unlearning in group contexts. Experimental results demonstrate that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.