AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning

AI summary

Overview

Research area: Machine unlearning for multimodal large language models (MLLMs), specifically privacy-driven removal of identity-specific knowledge (names, occupations, affiliations tied to faces), with an emphasis on the computer-vision side of the pipeline.

Technical level: Intermediate to Advanced. The paper's motivation and analysis are accessible, but the method relies on Fisher information, KKT conditions, and Jacobian-based derivations.

Scope: The paper proposes and evaluates AIM, a two-stage, vision-encoder-only unlearning method that suppresses identity knowledge in fine-tuned MLLMs without access to retain images, original training questions, or ground-truth answers at deletion time.

What This Paper Is About

MLLMs fine-tuned on image-text data memorize identity-specific facts about real people, so an individual who requests deletion of their data needs the model to stop answering questions about them. Most existing MLLM unlearning methods quietly assume they still have retain images, auxiliary data, or ground-truth answers available at deletion time — assumptions that often do not hold in practice. AIM targets this stricter setting: only the forget identity's images and constructed probing questions are available, and general visual perception plus non-deleted identities must still be preserved.

Key Contributions

1. A representation-level analysis of identity versus perception knowledge. The authors contrast last-layer LLM hidden states for identity questions and visual-perception questions on the same fine-tuned images, on LLaVA-1.5-7B and Qwen3-VL-8B fine-tuned on MLLMU-Bench, showing the two question types occupy distinct regions (PCA) and follow different grouping rules (t-SNE): identity questions cluster by person ID, perception questions cluster by question type.

2. AIM, a two-stage unlearning method. Stage 1 learns a universal visual prompt that anchors an identity-forgetting (IDK-inducing) target for the forget identity; Stage 2 updates only the vision encoder so unprompted forget-image features match that anchored target, while the language model is held fixed.

3. A retain-data-free preservation mechanism via cached Fisher statistics. The retain-preservation signal comes from an identity-balanced Fisher statistic cached once at the end of fine-tuning, decomposed as the full Fisher minus the forget Fisher at deletion time, so no retain images, original questions, or ground-truth answers are needed.

4. Evaluation under a strict data scope on two benchmarks and two models, including a continual unlearning extension where the cached full Fisher is reused across sequential deletion requests.

Main Findings

• Identity and perception knowledge are separable but organized differently. PCA of last-layer hidden states shows identity questions and visual-perception questions occupy distinct regions; t-SNE shows identity questions cluster by image identity regardless of question text, while perception questions cluster by question type regardless of which person appears. The authors state (Appendix C.1) that this separation is a consequence of SFT rather than a property of the pretrained model.

• A pilot visual prompt transfers selectively. A visual prompt trained without epsilon constraints on 4 identity questions to elicit IDK responses transferred to 11 held-out identity questions while leaving perception responses largely intact. Reported with GPT-4o-mini judgments: LLaVA-1.5-7B reached 100.0 IDK-Rate on held-out identity questions with 10.0 IDK-Rate and 63.3 Semantic Preservation on visual perception; Qwen3-VL-8B reached 85.0 IDK-Rate with 0.0 IDK-Rate and 85.2 SP.

• AIM competes with retain-utilizing baselines without retain data. On MLLMU-Bench with LLaVA-1.5-7B at the 5% split, AIM's retain and celebrity metrics stay close to the vanilla model (Cls_r 38.1 vs. 39.0, ROUGE_r 52.3 vs. 55.4, Cls_c 51.3 vs. 52.4, ROUGE_c 26.9 vs. 24.4), matching or exceeding retain-utilizing baselines such as MMUnlearner and MANU on LLaVA.

• Forgetting generalizes beyond memorized training images. On ReMem (LLaVA), the test-set keyword exact match EM_t drops from vanilla's 92.9/94.0 (5%/10%) to 45.2/35.1, which the authors interpret as forgetting generalizing to identities rather than to exact training images.

• Forget-only baselines collapse where AIM stays stable. GA and NPO collapse as iterations accumulate: by the 10% split they either reach near 0 on every MLLMU-Bench column (LLaVA) or drive retain metrics to single digits (NPO on Qwen3-VL). AIM maintains a balanced forget–retain trade-off across forget ratios.

• Remaining gap on ReMem's Exposure metric is attributed to the Fisher constraint. The paper states that capping per-step update magnitude to preserve retain features limits how deeply the forget signal propagates into the internal token distribution.

• Visual perception is preserved after unlearning. On MLLMU-Bench at 5%, AIM scores 57.8 and 68.9 against the pretrained LLaVA and 87.5 and 82.7 against the vanilla model on ROUGE-L and GPT semantic preservation, exceeding the best retain-utilizing method shown. The "Appr." (coherent, on-topic, non-refusal) fraction is 97.3 for AIM, 96.4 for vanilla, 88.4 for MMUnlearner, 99.1 for NPO, and 0.0 for GA.

• AIM extends to continual unlearning. On ReMem, for the 10%→15% transition AIM reports EM_f 50.0, EM_t 56.3, ROUGE 85.2, and EM_r 71.8, while GA and NPO collapse entirely (0.0 for most metrics) at the second step.

• Each Stage 1 loss term is necessary. Ablating the norm term causes retain to collapse along with forget; ablating the alignment term leaves metrics near vanilla with no effective update; ablating the CE term makes T ≈ 0 a trivial solution. The target-feature ablation shows a random-vector target fails to forget from the start, while IDK and Blank targets both work "because only a coherent, aligned target is required."

• The method is stable under image-side perturbations. On the 5% forget split of MLLMU-Bench, forget ROUGE-L changes by at most 4.7 points relative to the clean condition, and retain ROUGE-L decreases by at most 1.8 points under Gaussian noise, JPEG compression, horizontal flipping, and Gaussian blur.

Methodology in Plain English

The authors first check whether identity knowledge and general visual perception are stored in the same place inside a fine-tuned MLLM. They feed (image, question) pairs into LLaVA-1.5-7B and Qwen3-VL-8B fine-tuned on MLLMU-Bench and take the last-layer hidden state right before the model starts answering, then visualize those states. Two things show up: the two question types sit in different regions, and they group differently — identity questions group by who is in the picture, perception questions group by what is being asked.

That motivates the core idea: intervene on the visual pathway, leave the language model alone. In Stage 1, AIM learns a single visual prompt — a small perturbation added to every forget image — trained with three terms: one that pushes the model toward "I don't know"-style refusals, one that keeps the perturbation small, and one that makes the perturbation shift all forget images in a consistent direction. That consistent direction matters because the Stage 2 gradient would otherwise average out to roughly zero.

In Stage 2, the prompt is frozen and the vision encoder is updated so that the original (unperturbed) forget images produce the same features the perturbed images produced. The update direction is preconditioned by a Fisher matrix computed at the end of the original fine-tuning, which acts as a budget on how much features of non-deleted identities may drift. Because the full Fisher is cached once and the forget Fisher can be computed from forget images alone, deletion never requires retain images, the original fine-tuning questions, or ground-truth answers. Each identity is weighted equally when aggregating Fisher statistics, so identities with many images do not dominate those with few.

Experiments use MLLMU-Bench and ReMem at 5% and 10% forget splits (with 15% in the appendix) on LLaVA-1.5-7B and Qwen3-VL-8B-Instruct, compared against retain-utilizing baselines (KL_Min, GA_Diff, MMUnlearner, MANU) and forget-only baselines (GA, NPO). AIM is trained with 10 epochs of Stage 1 visual prompt learning, 100 iterations of Fisher approximation, and 50 epochs of vision-encoder updates, on a single NVIDIA A100 80GB GPU using one fixed random seed. Forget-only baselines are reported at 1 epoch because the authors observed they collapse rapidly under their standard schedule.

Why This Matters

The paper addresses a realistic constraint that most prior MLLM unlearning work sidesteps: when someone asks a deployed system to forget them, the operator typically has that person's images but not the original training corpus, its questions, its answers, or a clean set of retain examples. AIM shows that useful identity forgetting is still possible under that constraint, and it does so without degrading the model's ability to see and describe the same images.

Real-world applications:

• Right-to-be-forgotten compliance. A deployed multimodal assistant or photo-tagging service receives a deletion request and must suppress that person's identity knowledge using only their images, without retraining from scratch on the full dataset.
• Continual deletion in production. Because the cached full Fisher is independent of which identities are forgotten, sequential requests can be handled one at a time; the paper reports that AIM survives a second deletion step where GA and NPO collapse entirely.
• Preserving service quality during deletion. Visual question answering, captioning, and general image understanding on the affected images are largely retained, as measured against both the pretrained and vanilla model, avoiding the degenerate outputs observed from forget-only baselines.
• Privacy auditing of fine-tuned MLLMs. The representation analysis offers a diagnostic for whether identity knowledge is separable from perception knowledge in a given model, and (per the authors) whether that separation arose from supervised fine-tuning.

Industry relevance: Any organization fine-tuning or serving multimodal models on user or scraped data faces deletion obligations, and the strict setting AIM targets matches what most production systems actually have on hand. The method's small surface — vision encoder only, cached Fisher, no retain data — lowers the operational cost of honoring deletion requests, while its documented limitation (text-only identity knowledge is untouched) defines a clear boundary for deployment.

Future Directions

1. Cross-modal deletion. Because AIM updates only the vision encoder and keeps the language model fixed, it does not remove identity knowledge in the text-only modality. The authors propose combining AIM with a complementary LLM-unlearning method for broader cross-modal deletion.

2. Removing the precomputed-Fisher requirement. The full Fisher must be precomputed during fine-tuning, when retain images are implicitly accessible, so the framework cannot be applied directly to models lacking such a cache. The authors list methods that need neither retain data nor a precomputed cache as an important direction.

3. Longer schedules and larger steps. The retain Fisher is treated as fixed at its initial value, an approximation the authors say may break down under longer schedules or larger steps; AIM also requires small learning rates, which may contribute to the gap on ReMem's Exposure metric.

4. Broader attack evaluation. The robustness test covers image-side perturbations only. The authors state that prompt-side, cross-modal, and adaptive attacks remain untested and call for evaluating AIM under more diverse attack settings while remaining robust at inference time.

Target Audience

Researchers and practitioners working on machine unlearning, privacy-preserving machine learning, and multimodal model deployment will get the most from this paper — particularly those who need unlearning methods that do not assume access to retain data. It is also relevant to engineers implementing deletion workflows for production vision-language systems, and to readers interested in representation-level analysis of how fine-tuning organizes identity versus perceptual knowledge. Readers without a background in Fisher information or unlearning objectives will find the analysis section accessible but the Stage 2 derivation (deferred to the appendix) demanding.

Authors’ abstract

Multimodal large language models (MLLMs) can memorize identity-specific facts about people in their fine-tuning data, creating privacy risks when a person requests deletion. Existing MLLM unlearning methods often assume access to retain images or ground-truth answers during deletion, which is unrealistic in many practical scenarios. We study identity unlearning when retain images are unavailable at deletion time. Our analysis shows that identity and visual-perception questions occupy distinct regions in fine-tuned hidden states and are organized differently: identity questions cluster by person, whereas perception questions cluster by question type. This suggests that identity knowledge can be suppressed without erasing general visual perception. Building on this observation, we propose AIM, a two-stage method that anchors an identity-forgetting target with a universal visual prompt and then matches the vision encoder to that target under a Fisher-based constraint. Extensive experiments show that AIM achieves competitive identity forgetting while preserving non-deleted identities, prior knowledge, and visual perception on the same images.