Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models

AI summary

Overview

• Research area: Computer vision and generative AI, specifically privacy protection against unauthorized personalization of text-to-image diffusion models.
• Technical level: Advanced. The paper combines a formal convergence analysis of loss functions with a new training objective derived from preference optimization and a nested dual-path training algorithm.
• Scope: The paper proposes an Anti-Personalized Diffusion Model (APDM) that edits a pre-trained diffusion model's parameters so it can no longer be personalized to a specific subject, while remaining useful for other subjects and general image generation.

What This Paper Is About

Personalization techniques such as DreamBooth and Custom Diffusion let a user fine-tune a diffusion model on a handful of photos to generate new images of a specific person or object, which creates a privacy risk when that capability is misused. Existing defenses work at the image level: they add imperceptible adversarial noise (via Projected Gradient Descent) to a subject's photos so that later personalization attempts fail. The authors argue this is impractical and fragile — it requires poisoning an entire personal photo collection, it breaks down if even one clean image is available, it can be undone by simple image transformations, it puts the burden on non-expert users, and it conflicts with privacy regulations such as the GDPR that place the obligation on service providers. The goal is therefore to move protection from the data to the model itself.

Key Contributions

1. APDM framework: The first approach the authors describe that protects against unauthorized personalization by directly updating the model parameters of a pre-trained diffusion model, rather than perturbing images, which removes the data dependency and impractical assumptions of prior work.
2. Theoretical proof of failure for naive approaches: The authors prove that naively transferring image-perturbation losses (a term maximizing the personalization loss plus a prior-preservation loss, written as L_adv) to the model parameters fails to converge, because the two gradient terms would need to point in the same direction while the derived inequalities force contradictory requirements (Eq. 11 and Eq. 12).
3. Direct Protective Optimization (DPO): A new loss function, inspired by Direct Preference Optimization and the Bradley-Terry model, that pairs positive (generic, synthetic) images with negative (subject) images so the model is explicitly told what to encourage and what to suppress, while preserving generation quality.
4. Learning to Protect (L2P): A dual-path optimization strategy that alternates between a personalization path (simulating future personalized model states over N_per steps) and a protection path (accumulating protection gradients along that simulated trajectory), so protection anticipates how personalization would evolve.

Main Findings

• Protection performance: APDM achieved a DINO score of 0.1375 for "person" and 0.0959 for "dog" (average 0.1167), and BRISQUE of 40.25 and 60.74 (average 50.50). DreamBooth with only clean images scored 0.6994 / 0.6056 (average 0.6525) DINO and 11.27 / 22.33 (average 16.80) BRISQUE.
• Data-poisoning baselines degrade with clean images: For example, SimAC at 0 clean images scored an average DINO of 0.4411, rising to 0.5181 with 1 clean image and 0.6181 with N−1 clean images, while Anti-DreamBooth went from 0.4680 to 0.5393 to 0.6127 respectively. APDM kept its protection despite only using clean images.
• Preservation of generation quality: On the COCO 2014 validation set, Stable Diffusion scored FID 25.98, CLIP 0.2878, TIFA 78.76, and GenEval 0.4303; APDM scored FID 28.85, CLIP 0.2853, TIFA 75.91, and GenEval 0.4017.
• Other subjects remain personalizable: For "cat", "sneaker", and "glasses", APDM-protected models reached a DINO average of 0.6334 versus 0.5991 for DreamBooth, with BRISQUE averages of 24.41 versus 22.49.
• Image pairing matters: Without pairing positive and negative images, DINO was 0.2770 (person) and 0.3487 (dog) with BRISQUE 27.32 and 29.87; with pairing, DINO dropped to 0.1375 and 0.0959 and BRISQUE rose to 40.25 and 60.74.
• L2P matters: Without L2P, DINO was 0.4454 (person) and 0.3689 (dog), with BRISQUE 24.70 and 30.62; with L2P the same scores improved to 0.1375 / 0.0959 and 40.25 / 60.74.
• Lower beta gives stronger protection: beta = 1 gave DINO 0.1375 / 0.0959 and BRISQUE 40.25 / 60.74; beta = 10 gave 0.5392 / 0.3885 and 13.58 / 15.14; beta = 100 gave 0.5962 / 0.4755 and 12.21 / 14.10.
• More personalization steps in L2P help: N_per = 5 gave DINO 0.3371 / 0.1923 and BRISQUE 37.89 / 39.48; N_per = 10 gave 0.2096 / 0.1342 and 38.14 / 47.15; N_per = 20 gave 0.1375 / 0.0959 and 40.25 / 60.74.
• Robustness to perturbed inputs and unseen images: APDM reached 0.1702 DINO and 40.20 BRISQUE when the inputs were perturbed by Anti-DreamBooth, and maintained DINO scores of 0.1616, 0.1994, and 0.1873 when 4, 8, and 12 unseen images per subject were introduced.

Methodology in Plain English

The researchers start from a pre-trained text-to-image diffusion model and, instead of touching any images, they fine-tune the model's own weights toward a "safeguarded" state using a small set of images of the subject to be protected.

First, they check whether the obvious approach works — taking the adversarial loss used by image-poisoning methods and applying it to the weights. They analyze the gradients of its two terms (the personalization loss and the prior-preservation loss) and derive a theorem showing those terms impose contradictory requirements, so the optimization cannot converge to a point that satisfies both.

To fix this, they borrow the idea behind Direct Preference Optimization: rather than pushing the model away from something, they teach it to prefer one image over another. For every subject image, they generate a paired generic image with a generic prompt, label the generic one positive and the subject one negative, and use a Bradley-Terry-style objective with a sigmoid term (scaled by beta) so the model is nudged to produce generic content and suppress subject-specific content. They add the standard prior-preservation loss to this DPO loss to keep overall generation quality.

Second, because personalization itself is an iterative fine-tuning process, protecting against a single static snapshot is not enough. Their L2P algorithm nests two loops: an inner loop simulates what would happen if an attacker ran personalization for N_per steps starting from the current protected model, and at each simulated state it records the protection gradient; an outer loop sums those gradients and applies one protection update, repeating this for N_protect rounds. This makes protection trajectory-aware, anticipating how personalization would unfold rather than reacting to a single step.

Why This Matters

• Impact on research: It reframes anti-personalization as a model-editing problem rather than a data-poisoning problem, gives a formal argument for why the naive transfer of image-level adversarial losses fails at the parameter level, and introduces a preference-optimization-style objective plus a trajectory-simulation training scheme that other privacy and safety researchers can build on.
• Real-world applications:
  • Service providers running image-generation platforms could disable generation of a specific individual's likeness on request, without requiring that person to poison every photo they have ever posted.
  • Compliance with privacy regulations such as the GDPR, which place anti-personalization obligations on service providers rather than on end users.
  • Combating non-consensual synthetic media and fake news depicting real individuals.
  • Protecting commercial or proprietary objects (the paper tests subjects such as "sneaker", "glasses", "clock", "cat", and "dog") from unauthorized reproduction.
• Industry relevance: The method runs on a single NVIDIA RTX A6000 GPU and took about 9 GPU hours to protect a model, and it preserves image quality and personalization of untouched subjects, which are practical requirements for deployment in a production service.

Future Directions

• The paper reports that scaling tested subjects and transformations is only partially covered, with additional transform (flipping, blurring) and subject results placed in Appendix B — broader coverage of attacks and subjects remains open.
• Extending the evaluation of varied text prompts, different unique identifiers, and the user study (Appendices B and D) into the main benchmarking regime.
• Whether the approach generalizes beyond the tested Stable Diffusion 1.5 and 2.1 backbones and personalization methods (DreamBooth, with Custom Diffusion results in Appendix C) is not established in the reported content.
• The trade-off between protection strength and preservation is visible in the results (FID rises from 25.98 to 28.85, TIFA falls from 78.76 to 75.91), so reducing the cost to general generation quality is a natural next question.

Target Audience

This paper is most useful to researchers and engineers working on diffusion model safety, privacy-preserving machine learning, and adversarial defenses, as well as to platform engineers responsible for content-policy and anti-impersonation systems. It also suits readers with a background in optimization or generative modeling who want to understand why naive adversarial transfers to model weights fail. The mathematical derivation of Proposition 1 and Theorem 1 means the theoretical sections assume familiarity with gradient-based optimization and Taylor approximations.

Authors’ abstract

Recent advances in diffusion models have enabled high-quality synthesis of specific subjects, such as identities or objects. This capability, while unlocking new possibilities in content creation, also introduces significant privacy risks, as personalization techniques can be misused by malicious users to generate unauthorized content. Although several studies have attempted to counter this by generating adversarially perturbed samples designed to disrupt personalization, they rely on unrealistic assumptions and become ineffective in the presence of even a few clean images or under simple image transformations. To address these challenges, we shift the protection target from the images to the diffusion model itself to hinder the personalization of specific subjects, through our novel framework called Anti-Personalized Diffusion Models (APDM). We first provide a theoretical analysis demonstrating that a naive approach of existing loss functions to diffusion models is inherently incapable of ensuring convergence for robust anti-personalization. Motivated by this finding, we introduce Direct Protective Optimization (DPO), a novel loss function that effectively disrupts subject personalization in the target model without compromising generative quality. Moreover, we propose a new dual-path optimization strategy, coined Learning to Protect (L2P). By alternating between personalization and protection paths, L2P simulates future personalization trajectories and adaptively reinforces protection at each step. Experimental results demonstrate that our framework outperforms existing methods, achieving state-of-the-art performance in preventing unauthorized personalization. The code is available at https://github.com/KU-VGI/APDM.