PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models

AI summary

Overview

Research area: Machine unlearning for multimodal large language models (MLLMs), specifically person-centric knowledge removal and benchmark design.

Technical level: Intermediate. The paper builds on established unlearning methods (gradient ascent, preference optimization, knowledge editing) and assumes familiarity with MLLM evaluation, but the benchmark's motivation and findings are accessible without deep mathematical background.

Scope: The paper introduces PRMU, a benchmark of 1,080 public figures with 50,649 textual probes and 41,303 visual probes for evaluating corpus-free, person-centric knowledge unlearning in MLLMs, plus a lightweight baseline method called Similarity-Gated Projection Editing (SGPE).

What This Paper Is About

Multimodal large language models memorize substantial person-related knowledge during pre-training, and deleting a specific person's knowledge after the fact is difficult without retraining. Existing unlearning methods and benchmarks generally assume access to the original forget and retain corpora, or they inject target knowledge through fine-tuning before measuring its removal — neither matches realistic deletion requests, which typically provide only a target identity and no training data. PRMU addresses this by evaluating whether models can remove naturally acquired person-related knowledge, given only the original model plus a target name and reference image, while preserving related knowledge and general multimodal ability.

Key Contributions

1. The PRMU benchmark. A person-centric multimodal unlearning benchmark containing 1,080 public figures, 50,649 textual probes, and 41,303 visual probes. Targets are sampled from the Pantheon public-figure ranking across 20 occupational categories and 74 nationalities, stratified into Head, Torso, and Long-tail HPI tiers, and filtered with a Native Knowledge Score (NKS) procedure using an unfine-tuned Qwen2.5-VL-32B as reference model so that retained probes reflect knowledge a strong MLLM already holds.

2. A corpus-free unlearning protocol and evaluation framework. Methods receive only the original model and a target specification of the target name plus a reference image. PRMU supplies a target-specific proxy corpus generated by the original MLLM (biographies, atomic facts, and question-answer pairs) so existing methods can be adapted without the original forget or retain corpora, and it evaluates forgetting, locality preservation, utility retention, and adversarial robustness using FB, QA, MCQ, and AA probes under both text-only and vision-language inputs.

3. Neighbor-based locality assessment. For each target, PRMU builds a Neighbor Set of the top-5 semantically related non-target individuals using five signals: occupation match, nationality match, normalized birth-year distance, normalized HPI (popularity) distance, and CLIP image-embedding cosine similarity. This enables fine-grained measurement of collateral damage to related people rather than only general knowledge retention.

4. SGPE, a lightweight corpus-free unlearning baseline. Similarity-Gated Projection Editing combines self-induced knowledge displacement from proxy knowledge, bidirectional protected editing in an orthogonal subspace, and proxy-contrastive multimodal locality gating that activates the edit only for target-relevant inputs — without requiring forget corpora, retain data, or neighbor examples.

Main Findings

• Existing baselines trade forgetting against locality. GA, RT, NPO, and DPO can be adapted to the corpus-free setting via the proxy corpus, but their Forget Set improvements are accompanied by severe drops on the Neighbor and Other Sets. SGPE reduces parameter disturbance and gives a better trade-off between target suppression and locality preservation.

• Vision-language probes reveal residual knowledge that text-only probes miss. Across methods, vision-language probes consistently show higher residual scores than text-only probes after unlearning, indicating that multimodal associations contribute to target knowledge retention and that text-only evaluation underestimates removal difficulty.

• Neighbor knowledge suffers more interference than unrelated knowledge. The Neighbor Set experiences significantly larger performance drops than the Other Set after unlearning, meaning unlearning one target can unintentionally affect semantically related individuals.

• Scaling to multiple targets increases forgetting difficulty and collateral interference. Under nested target subsets, batch-target unlearning introduces cross-target interference among deletion objectives, degrading the forgetting-locality trade-off.

• Batch-target results across three models. On LLaVA-1.5-7B, SGPE reduced Forget Set QA (text) from 0.630 to 0.332 and FB (text) from 0.629 to 0.273 while Neighbor Set QA (text) moved from 0.601 to 0.505 and Other Set QA (text) from 0.619 to 0.529. On Qwen2.5-VL-7B, SGPE reduced Forget Set QA (text) from 0.606 to 0.119, with Neighbor Set QA (text) at 0.459 versus 0.568 before unlearning. On Qwen3-VL-8B, SGPE reduced Forget Set QA (text) from 0.777 to 0.399, with Neighbor QA (text) at 0.578 versus 0.752 before.

• General utility is largely preserved by SGPE. In the batch-target setting, SGPE's MMBench score was 0.691 versus 0.692 before unlearning on LLaVA-1.5-7B, 0.876 versus 0.877 on Qwen2.5-VL-7B, and 0.900 versus 0.900 on Qwen3-VL-8B.

• Proxy corpus beats a Wikipedia-based corpus. On Qwen3-VL-8B with GA-based unlearning, the proxy setting achieved stronger forgetting (Forget QA text 0.593, QA vision 0.503, AA text 0.726, AA vision 0.690) than the Wikipedia-based corpus (0.624, 0.542, 0.739, 0.707) while retaining comparable Neighbor Set scores, suggesting target-specific proxy supervision is better suited to corpus-free unlearning.

• Deletion scale matters. With nested subsets from the same 150-target pool, GA at N=150 lowered Forget QA (text) from 0.777 to 0.593 but also dropped Neighbor QA (text) from 0.752 to 0.585; at N=50 the Forget QA (text) moved from 0.666 to 0.597 and Neighbor QA (text) from 0.665 to 0.602.

• Locality degradation is uneven across relationship types. The radar analysis over five person-level relationships shows unlearning effects are not uniformly distributed, so semantically or visually related individuals experience different levels of collateral change.

• Proxy knowledge forms are complementary. On GA-based unlearning, the complete proxy corpus achieved the best forgetting-locality trade-off (Forget QA text 0.593, Neighbor QA text 0.585). QA-only supervision gave stronger target suppression (Forget QA text 0.575) but weaker neighbor preservation (Neighbor QA text 0.571), while biography-only was least effective (Forget QA text 0.649, Neighbor QA text 0.629).

• Ablation confirms each SGPE component. Removing the displacement anchor degraded forgetting (QA text 0.445 versus 0.399 for full SGPE). Removing protected projection led to larger Neighbor Set degradation (Neighbor QA vision 0.473 versus 0.346). Removing the locality gate gave slightly stronger forgetting but increased collateral changes (Neighbor AA vision 0.493 versus 0.576).

Methodology in Plain English

The researchers first built a target pool from the Pantheon public-figure popularity ranking, filtering candidates for information availability, sampling across 20 occupations, three popularity tiers, many nationalities, and both genders. For each figure, they assembled a factual profile from Wikidata and Wikipedia covering nationality, occupation, birthplace, education, works, and awards. GPT-5.4 then generated diverse probes from these profiles: fill-in-the-blank, open-ended QA, multiple-choice derived from those, and adversarial attack probes spanning 14 attack styles such as paraphrasing, role playing, cross-lingual prompting, and biographical clue chaining. Visual probes were created from text probes that survived filtering, using target images collected from Wikipedia/Commons and manually verified.

To ensure the benchmark tests knowledge a model genuinely acquired rather than obscure trivia, they applied Native Knowledge Score filtering: an unfine-tuned Qwen2.5-VL-32B acted as a fixed reference model, and only probes it answered correctly were kept. This was done first on text probes, then extended to visual probes with visual-side filtering. Targets with too few valid probes were dropped, leaving 1,080 targets.

Because corpus-free unlearning means no original forget or retain data, they generated a proxy corpus per target by prompting the original MLLM for biographies, atomic facts, and QA pairs, and verified it against the source profiles. For locality evaluation, they computed a composite neighbor score from occupation match, nationality match, birth-year distance, popularity distance, and CLIP image similarity, selecting the top-5 neighbors per target.

Evaluation uses three sets — Forget, Neighbor, and Other — across MCQ, FB, QA, and AA probes under text-only and vision-language inputs. ROUGE-L recall is used for FB, QA, and AA probes, accuracy for MCQs, and MMBench for general multimodal utility. They ran experiments on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B against GA, RT, NPO, and DPO, plus their own SGPE baseline, with single-target evaluation on 100 targets and batch-target evaluation on 150 targets, plus batch sizes of 50 and 100 drawn as nested subsets.

SGPE itself works in three stages. It derives alternative, semantically close responses from the proxy corpus to act as displacement anchors that steer the model away from target knowledge without adding external facts. It then writes the edit into a protected parameter subspace, computed by projecting editing keys away from a principal-activation basis built from generic calibration prompts, so unrelated knowledge is less disturbed. Finally, a similarity gate compares input representations against target and background prototype banks and activates the edit only when target-related textual or visual evidence is detected.

Why This Matters

Impact on research. PRMU shifts multimodal unlearning evaluation away from artificially injected knowledge and toward naturally acquired knowledge with no access to original corpora, which is closer to what real deletion requests look like. The Neighbor Set adds a locality dimension that prior person-centric benchmarks such as PPU-Bench do not evaluate, and the finding that vision-language probes retain more knowledge than text-only probes suggests text-only evaluation can overstate unlearning success. The release of SGPE as a corpus-free baseline gives the community a starting point that avoids the most common failure mode reported here.

Real-world applications:

• Responding to individual deletion or erasure requests for public figures whose attributes and biographical facts are memorized by deployed multimodal models.
• Reducing unintended disclosure of identity-related attributes and biographical information surfaced through image inputs, not just text prompts.
• Auditing model controllability before deployment, by checking whether removed knowledge can be reactivated through paraphrasing, role playing, cross-lingual prompting, or multimodal context.
• Regulatory and compliance workflows that need evidence that a specific entity's knowledge has been suppressed while unrelated entities and general capability remain intact.

Industry relevance. Organizations deploying MLLMs face deletion requests and privacy obligations but rarely retain the original pre-training corpora, so methods requiring forget and retain sets are impractical. PRMU's corpus-free protocol and proxy-corpus design map directly onto that constraint, and the batch-target results flag a scalability problem for teams that must process many simultaneous deletions rather than one at a time.

Future Directions

• Closing the multimodal gap. Vision-language probes consistently retain more target knowledge than text-only probes, so unlearning objectives and evaluation that explicitly target cross-modal associations remain an open problem.
• Reducing collateral damage to related individuals. Neighbor knowledge degrades more than unrelated knowledge, and degradation is uneven across relationship types; better locality mechanisms are needed beyond the input-conditioned gating SGPE uses.
• Scaling to large simultaneous deletion sets. Batch sizes of 50, 100, and 150 show growing cross-target interference, raising the question of how methods should behave at far larger deletion scales.
• Broadening the reference model and target pool. PRMU uses a single fixed reference model (Qwen2.5-VL-32B) to keep evaluation targets consistent; whether NKS filtering with a different or multiple reference models changes which knowledge counts as "native" is left open.

Target Audience

Researchers and engineers working on machine unlearning, privacy-preserving model deployment, and multimodal model evaluation, particularly those who must remove knowledge without access to pre-training data. The paper also suits practitioners responsible for deletion requests, compliance auditing, and model controllability in production MLLM systems, as well as benchmark designers interested in locality-aware and adversarial evaluation protocols.

Authors’ abstract

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in storing and recalling rich person-related knowledge, raising increasing concerns about reliable knowledge removal. However, existing machine unlearning approaches for MLLMs typically assume access to original forget and retain corpora, which are often unavailable in realistic deletion scenarios. To address this limitation, we introduce PRMU, a benchmark for evaluating corpus-free multimodal unlearning under realistic person-centric deletion requests. PRMU focuses on naturally acquired person-related knowledge and evaluates whether models can remove target knowledge while preserving related knowledge through diverse textual and visual probes, including adversarial evaluation and fine-grained locality analysis. To facilitate research in this setting, we further introduce Similarity-Gated Projection Editing (SGPE), a lightweight corpus-free unlearning baseline with knowledge displacement, protected parameter-space editing, and locality-aware multimodal control. Extensive experiments on representative MLLMs reveal that existing unlearning methods often suffer from unfavorable forgetting-locality trade-offs, with significant locality degradation under aggressive forgetting settings, and remain vulnerable to multimodal knowledge reactivation. Meanwhile, SGPE provides a competitive trade-off between target forgetting, locality preservation, and general multimodal utility. We hope PRMU can facilitate future research toward realistic and scalable multimodal machine unlearning. Code and dataset will be released at https://github.com/2231122/PRMU.