Approximate Domain Unlearning for Vision-Language Models

AI summary

Overview

Research area: Machine unlearning for pre-trained Vision-Language Models (VLMs), specifically a new task called Approximate Domain Unlearning (ADU) that sits at the intersection of approximate unlearning, domain adaptation/generalization, and parameter-efficient VLM tuning.

Technical level: Advanced. The paper assumes familiarity with CLIP-style contrastive vision-language pretraining, prompt tuning (deep prompting, vision prompts), Transformer attention mechanics, and Maximum Mean Discrepancy in a Reproducing Kernel Hilbert Space.

Scope: The paper defines ADU, proposes a two-component method (Domain Disentangling Loss plus Instance-wise Prompt Generator) to solve it, and evaluates it against CLIP fine-tuning and VLM unlearning baselines on four multi-domain benchmarks.

What This Paper Is About

Pre-trained VLMs generalize so broadly that they recognize objects across many visual styles at once — a car is recognized as a car whether it appears in a photograph, a painting, a clipart, or a sketch. Existing approximate unlearning work only targets classes (making the model forget an object category entirely), but many practical systems need finer control: an autonomous driving system must still recognize real cars while ignoring illustrated cars on roadside advertisements. This paper introduces Approximate Domain Unlearning (ADU), which requires reducing recognition accuracy for images from specified domains while preserving accuracy for the others, and proposes a method that achieves this by separating domain distributions in the feature space rather than simply penalizing the target domains.

Key Contributions

1. A new problem setting — Approximate Domain Unlearning (ADU). The paper extends approximate unlearning from the class level to the domain level, formally defining a set of domains to memorize (D_memorize) and a set to forget (D_forget = D \ D_memorize), with the goal of preserving accuracy on the former and reducing it on the latter.

2. Domain Disentangling Loss (DDL). A loss that explicitly separates domain distributions in the latent feature space by combining a domain-classification cross-entropy term with a maximized Maximum Mean Discrepancy term — inverting the conventional use of MMD, which is normally minimized for domain invariance.

3. Instance-wise Prompt Generator (InstaPG). A module embedded in an intermediate Transformer block of the image encoder that generates instance-specific vision prompts via cross-attention, using the learnable vision prompt as query and image patch features as keys and values, so the model adapts to per-image variation in how strongly a domain manifests.

4. Extensive empirical validation. Experiments on four multi-domain benchmarks showing substantial gains over strong baselines built from state-of-the-art CLIP fine-tuning methods and a state-of-the-art VLM class unlearning method.

Main Findings

• Naive unlearning fails at the domain level. A "Baseline" that uses the standard class-unlearning losses (L_memorize and L_forget) reaches H = 52.59 on Office-Home and 62.07 on Mini DomainNet for |D_forget| = 1, but its forgetting metric For lags far behind what the proposed method achieves, because domain distributions are entangled in the VLM latent space.

• Large gains over state-of-the-art CLIP tuning methods. Compared with LP++ and CLIPFit, the proposed method outperforms them by more than 20% in H on Office-Home and Mini DomainNet, and by 5.71% on ImageNet, regardless of the number of domains to be forgotten.

• Forgetting accuracy above 60% where baselines stay below 40%. On Office-Home and Mini DomainNet, the For metric of CLIPFit and LP++ is below 40%, while the proposed method achieves over 60%. For example, at |D_forget| = 1 on Office-Home, the proposed method reaches For = 64.34 versus 18.55 for LP++, 29.40 for CLIPFit, 19.74 for BBF, and 39.88 for the Baseline.

• Class unlearning is not sufficient for domain unlearning. Against Black-Box Forgetting (BBF), the current state-of-the-art class unlearning method for VLMs applied to ADU, the proposed method achieves over 30% higher For on all datasets.

• Beat the naive Baseline in forgetting by over 20%. On Office-Home and Mini DomainNet, the method surpasses the Baseline by over 20% in For, showing the limits of naive domain-penalizing strategies.

• Both components contribute, and together they are best. In ablation on Office-Home at |D_forget| = 3, H rises from 59.47 (no DDL, no InstaPG) to 63.12 (DDL only) and 70.60 (InstaPG only), reaching 75.89 when both are combined. The same ordering holds on Mini DomainNet (68.82 to 77.60 to 74.17 to 81.78).

• CE and MMD are complementary inside DDL. On Office-Home at |D_forget| = 3, using MMD alone gives H = 66.76 and CE alone gives H = 71.53, while using both gives H = 75.89.

• Domain separability is directly measurable. Domain classification accuracy on Office-Home when Art is forgotten rises from 25.80% before unlearning to 79.43% with the proposed method; removing DDL and InstaPG drops it to 31.06%.

• t-SNE confirms disentanglement. Zero-shot CLIP features are heavily entangled across domains, whereas the proposed method separates domains in the feature space, enabling per-domain control of memorization and forgetting.

• Attention is redirected away from forgotten-domain objects. With Real as the forgotten domain on Mini DomainNet, Zero-shot CLIP attention concentrates on objects; after unlearning, attention on objects disappears or weakens for Real data while being maintained or strengthened for Painting, Clipart, and Sketch.

• Fine-grained control survives visually similar domains. In Mini DomainNet, forgetting Sketch reduces its accuracy from 72.54% to 20.64%, while visually similar clipart and painting are affected within 1%.

• Some domains are harder to forget. On Office-Home, forgetting Real only decreases its accuracy from 81.29% to 55.48%; the paper attributes this to CLIP being heavily pre-trained on image-text pairs dominated by real-world photos.

• Robustness to hyperparameters. Performance saturates around γ = 10 and peaks around λ = 10, remaining stable across a wide range of values beyond those thresholds.

• More training shots help the proposed method but not the Baseline. The proposed method improves consistently with more shots per domain, while the Baseline struggles to benefit, especially on Mini DomainNet, suggesting overfitting.

• Additional results on DomainNet. With 177K test samples, six domains, and 345 classes, the proposed method reaches H = 66.81, 67.81, and 68.83 for |D_forget| = 1, 2, and 3 respectively, versus 38.45, 39.36, and 38.79 for the Baseline and a constant 36.85 for Zero-shot CLIP.

Methodology in Plain English

The researchers start from a CLIP model with ViT-B/16 as the image encoder and the text prompt "a photo of a [class]", then learn vision prompts in the style of deep prompting: eight learnable context tokens optimized within the first nine transformer layers. Training runs 50 epochs with SGD at a learning rate of 0.0025, using eight labeled samples per domain (both class and domain labels), and the text encoder is left untouched except for the class prompt.

The obvious route — keep the standard cross-entropy loss on data from domains to preserve and maximize entropy on data from domains to forget — is applied first as the Baseline, and it does not work well because VLMs entangle different domains in their feature space. The fix has two parts. First, the Domain Disentangling Loss adds a small domain classifier head and trains it with cross-entropy so domain labels become predictable from features; on top of that, it maximizes the Maximum Mean Discrepancy between domain distributions measured in a Reproducing Kernel Hilbert Space, actively pushing domains apart instead of pulling them together. Second, the Instance-wise Prompt Generator sits inside an intermediate Transformer block and produces extra prompts for that specific image through cross-attention, with the learnable prompt as query and image patches as keys and values.

The total objective is simply L_memorize + L_forget + L_domain, with weighting hyperparameters γ = 30 and λ = 10 for the cross-entropy and MMD terms. Evaluation uses three metrics: Mem (accuracy on memorized-domain classes), For (error rate on forgotten-domain classes), and H, their harmonic mean — all reported as averages over three runs with different random seeds, and averaged over all possible domain forget/retain combinations for |D_forget| in {1, 2, 3}. ImageNet has only two domains, so only |D_forget| = 1 is tested there.

Why This Matters

Impact on research: This is described as the first investigation of domain-level approximate unlearning, opening a direction that conventional class unlearning cannot address. It also inverts a long-standing convention in domain adaptation by maximizing rather than minimizing MMD, reframing domain separability as a goal rather than an obstacle. The paper frames ADU as a new challenge for the community, aimed at more fine-grained and practical unlearning in VLMs.

Real-world applications:

• Autonomous driving: recognizing real cars and pedestrians for distance control and collision avoidance, while avoiding misrecognition of illustrated cars in roadside advertisements that could trigger unsafe behavior.
• Content moderation or filtering systems: selectively suppressing recognition of stylized or synthetic imagery while keeping real-world recognition intact.
• Edge and embedded deployment: discarding irrelevant knowledge reduces excessive computational resource consumption.
• Privacy and information leakage control: selectively removing domain-specific knowledge from a deployed model limits what the model can expose.

Industry relevance: Because the method operates by tuning learnable prompts rather than retraining the whole model, and the authors state that exact unlearning typically requires full retraining (computationally prohibitive for deep networks), the approach fits the practical demand for efficient, selective forgetting in proprietary or large-scale models. The DomainNet results covering 345 classes across six domains indicate the method is intended to scale beyond toy settings. Code is released at a project page linked in the paper.

Future Directions

• Removing the domain-label assumption. The method assumes domain labels are available for all training samples, which the authors acknowledge may not hold in real scenarios. They show in Appendix C.3 that a simple pseudo-labeling domain-estimation technique preserves substantially better performance than no estimation even when a large fraction of domain labels is absent, and suggest combining ADU with more advanced domain estimation methods.

• Handling hard-to-forget domains. The paper reports that forgetting the Real domain on Office-Home only reduces accuracy from 81.29% to 55.48%, likely because CLIP's pre-training is dominated by real-world photos. Closing this gap for strongly encoded domains remains open.

• Extending ADU to black-box or proprietary VLMs. BBF was designed for black-box unlearning; the paper positions ADU as a new task and does not report a black-box variant of its own method.

• Testing under more realistic data regimes and larger domain sets. The paper notes that the number of available samples per domain can vary substantially in practice, motivating the sensitivity analysis, while DomainNet results are relegated to an appendix.

Target Audience

Researchers and practitioners working on machine unlearning, vision-language models, and parameter-efficient tuning who need selective control over what kind of data a model recognizes rather than which classes it recognizes. It is also relevant to engineers building deployed perception systems (such as autonomous driving or content filtering) that must avoid reacting to stylized or illustrated inputs, and to readers interested in domain disentanglement as an alternative to domain-invariant representation learning. The paper is not beginner-friendly: it assumes working knowledge of CLIP, prompt tuning, attention, and kernel-based distribution distances.

Authors’ abstract

Pre-trained Vision-Language Models (VLMs) exhibit strong generalization capabilities, enabling them to recognize a wide range of objects across diverse domains without additional training. However, they often retain irrelevant information beyond the requirements of specific downstream tasks, raising concerns about computational efficiency and potential information leakage. This has motivated growing interest in approximate unlearning, which aims to selectively remove unnecessary knowledge while preserving overall model performance. Existing approaches to approximate unlearning have primarily focused on class unlearning, where a VLM is retrained to fail to recognize specified object classes while maintaining accuracy for others. However, merely forgetting object classes is often insufficient in practical applications. For instance, an autonomous driving system should accurately recognize real cars while avoiding misrecognition of illustrated cars depicted in roadside advertisements as real cars, which could be hazardous. In this paper, we introduce Approximate Domain Unlearning (ADU), a novel problem setting that requires reducing recognition accuracy for images from specified domains (e.g., illustration) while preserving accuracy for other domains (e.g., real). ADU presents new technical challenges: due to the strong domain generalization capability of pre-trained VLMs, domain distributions are highly entangled in the feature space, making naive approaches based on penalizing target domains ineffective. To tackle this limitation, we propose a novel approach that explicitly disentangles domain distributions and adaptively captures instance-specific domain information. Extensive experiments show that our approach outperforms baselines built upon VLM tuning techniques, paving the way for practical and fine-grained unlearning in VLMs. Code: https://kodaikawamura.github.io/Domain_Unlearning/.