Targeted Data Protection for Diffusion Model by Matching Training Trajectory

AI summary

Overview

Research area: AI safety and privacy — specifically protecting images from unauthorized use in text-to-image diffusion model personalization (DreamBooth-style fine-tuning).

Technical level: Advanced. The paper assumes familiarity with denoising diffusion probabilistic models, latent diffusion, LoRA fine-tuning, DreamBooth, adversarial attacks (FGSM/PGD), and dataset distillation / trajectory matching (MTT).

Scope: The paper proposes TAFAP, a trajectory-matching adversarial perturbation method that redirects the entire fine-tuning trajectory of a protected image toward a user-specified target concept, and evaluates it versus prior targeted and untargeted protection baselines on face datasets.

What This Paper Is About

When someone fine-tunes a diffusion model on a person's photos, they can generate unauthorized images of that person. Existing defenses mostly just degrade output quality, which offers no control and no proof that protection worked. This paper aims to move from passive degradation to "targeted data protection," where protected images steer the model into learning a completely different, user-chosen identity or visual pattern instead.

Key Contributions

1. TAFAP (Trajectory Alignment via Fine-tuning with Adversarial Perturbations): the first method to realize Targeted Data Protection by aligning the entire fine-tuning trajectory rather than individual model snapshots.
2. Trajectory matching for protection rather than data synthesis: the principle from Matching Training Trajectory (MTT) dataset distillation (Cazenavette et al. 2022) is repurposed to craft imperceptible adversarial perturbations for pre-existing images, instead of generating a synthetic dataset.
3. First successful targeted transformation in diffusion models: simultaneous control over both identity and low-level visual patterns, backed by quantitative and qualitative comparisons against T-ASPL, Mist, and ACE.
4. Robustness and transfer evidence: protection survives common image preprocessing (blur, JPEG, rescaling) and shows partial cross-model transfer to different model configurations.

Main Findings

• Prior TDP attempts fail due to snapshot matching: Mist computes perturbations only from the initial pre-trained model, and T-ASPL recomputes perturbations at each fine-tuning step but only from the immediate model state, so protective influence is diluted as training progresses. The paper argues that considering multiple snapshots is not equivalent to considering the entire trajectory.
• Identity redirection succeeds where prior work did not: using Identity Score Matching (ISM) with ArcFace, TAFAP reaches ISM 0.202 against the protected identity and 0.393 against the target identity, compared with No defense (0.536 / 0.042), T-ASPL (0.226 / 0.147), Mist (0.368 / 0.108), and ACE (0.405 / 0.177). ISM was evaluated on randomly sampled protect-target pairs.
• Image quality is preserved: TAFAP scores BRISQUE 11.51, FDFR 0.000, and SER-FIQ 0.80, versus No defense (1.40 / 0.000 / 0.78), T-ASPL (23.69 / 0.134 / 0.48), Mist (26.70 / 0.014 / 0.65), and ACE (32.74 / 0.019 / 0.67). The paper cautions that high image quality alone does not imply successful protection under TDP without identity redirection.
• Noise is semantic, not an afterimage: Mist, T-ASPL, and ACE use losses that minimize latent-space distance between target and protected data, which produces visible overlapping or afterimage artifacts in image space; TAFAP's trajectory-alignment objective does not.
• Chaotic pattern redirection works: when a Mist pattern image is used as the target, the method redirects generation toward that chaotic pattern at noise budgets ε of 0.05 and 0.10.
• Protection degrades gracefully under preprocessing: without manipulation, ISM is 0.202 (protected) / 0.393 (target). Under Gaussian blur, scores range from 0.207 / 0.353 (kernel = 3) to 0.496 / 0.157 (kernel = 9); under JPEG compression, from 0.297 / 0.246 (quality = 70) to 0.374 / 0.105 (quality = 10); under resizing, 0.496 / 0.228 (down 0.5x then up to 1x) and 0.389 / 0.304 (up 2x then to 1x), all against the No defense baseline of 0.536 / 0.042.
• Progressive identity change: as adversarial noise is updated, identity changes gradually, altering overall content rather than just textures, suggesting the degree of identity transfer could be controlled by the number of noise update iterations.
• Cross-model effects observed: protection optimized with SD1.4 + LoRA still causes noticeable deviation from the protected identity when fine-tuning without LoRA on SD1.4 and on SD2.1, though the paper describes target resemblance as varying in degree.

Methodology in Plain English

The researchers start by first fine-tuning a model on the target data (the identity or pattern they want the output to become) and saving the model weights at each training iteration. This saved sequence is called the "expert trajectory."

They then take the protected image, add a small adversarial noise pattern to it, and simulate fine-tuning the model on that noisy image. After a few update steps, they compare the resulting weights against the saved target weights at the matching point in the expert trajectory. The normalized L2 distance between those two weight sets (Eq. 5) is the loss.

They backpropagate through this comparison to update the noise, but they use only the sign of the gradient (a sign-based rule inspired by FGSM and PGD), not its magnitude, and they clip the noise to a fixed budget ε so the image stays visually close to the original. This process repeats N times, and the saved target weights are randomly selected each round so the noise learns to follow the whole path rather than one point.

Implementation details: the perturbation budget ε is 0.05 (12.75/255 in pixel space [0, 255]); noise is optimized over N = 2,000 iterations; PGD uses 6 steps with step size α = 0.001; three expert trajectories (target LoRA models) are used with snapshots captured every iteration; the network update step count is k = 8; trajectory matching spans iterations 1 to 600 of expert training; and the loss is scaled by a constant λ = 100 purely to prevent gradient underflow in the unrolled bi-level optimization, which the paper states does not change the outcome because only the gradient sign is used.

Experimental setup: CelebA HQ and VGGFace2 datasets, Stable Diffusion 1.4 base, DreamBooth with LoRA (rank 4), 12 images each for protected and target data at 256x256, some identities reduced to 11 images after manual curation, DreamBooth trained for 1,000 iterations, all runs on a single GeForce RTX 3090 GPU 24GB, random seed 0. LoRA checkpoints require about 3MB of storage (6MB with optimizer states). LoRA learning rate is 1e-4 with no scheduler or warmup, batch size 1, no gradient accumulation; the text encoder is frozen and LoRA is applied only to the U-Net. For baselines, only the targeted attack loss components of Mist and ACE were used, with their chaotic targets replaced by identity images to fit the TDP evaluation, under identical target images, PGD configuration, noise budget, and personalization settings.

Evaluation uses ISM with the ArcFace recognizer, FDFR with RetinaFace, BRISQUE (0-100, lower is better), and SER-FIQ (0-1, higher is better). The paper notes that TDP evaluation requires the opposite interpretation from UDP: TDP wants low FDFR to keep faces detectable, and dual ISM evaluation where lower similarity to the protected identity and higher similarity to the target both indicate success.

Why This Matters

Data protection for diffusion models has been stuck in a destructive mode: make the output ugly and hope that discourages misuse. This paper argues that destruction alone fails three requirements the authors identify — identity concealing, verifiability, and traceability — because untargeted methods may leave residual identity traces, provide no proof protection is active, and offer no way to attribute misuse after the fact. TAFAP's trajectory-level control targets all three at once: the model learns a different concept (concealing), the redirection is measurable with standard recognition metrics (verifiable), and the redirection is directed at a chosen signature (traceable).

Real-world applications:

• Individual privacy for public figures: people whose images are widely scraped could release protected versions that steer unauthorized fine-tuning toward a different, chosen identity.
• Deepfake and identity-theft deterrence: preventing the identity-theft and non-consensual content risks the paper cites as motivation for protection mechanisms.
• Copyright and intellectual property protection: artists or rights holders could embed a targeted redirection so unauthorized model training produces attributable, off-brand output.
• Provenance and misuse tracing: the distinctive transformation embedded in outputs gives a signal for identifying when protected data was used without authorization.

Industry relevance: the method is designed for low-resource deployment. It targets LoRA-based personalization (rank 4), requires only about 3MB per LoRA checkpoint, and runs on a single 24GB consumer GPU, while the paper notes that cross-model experiments suggest resilience even when the attacker's model differs from the one used during protection.

Future Directions

• Controlling the degree of transformation: the paper observes a progressive identity change as noise updates accumulate and suggests controlling the degree of identity transfer through the number of noise update iterations; this remains a suggestion rather than a demonstrated control knob.
• Interpreting identity blending: the paper notes that observed identity blending in some cases "suggests intriguing possibilities for controllable interpolation between protection levels" — treating partial failure as an open research direction.
• Stronger cross-model transfer: protection was optimized with SD1.4 + LoRA; the paper reports initial evidence on SD1.4 without LoRA and SD2.1 and frames broader transferability across model versions and architectures as an open direction worth further exploration.
• Extending beyond faces and patterns: the paper positions identity transformation as a proof of concept for the TDP paradigm and anticipates the work serving as a cornerstone for more sophisticated and controllable data protection technologies, but does not report experiments on other concept types or larger image resolutions.

Target Audience

This paper is most useful to researchers and practitioners working on adversarial machine learning, dataset distillation, and generative model safety — particularly those studying defenses against unauthorized diffusion-model fine-tuning. It also suits privacy engineers and rights-management teams evaluating whether targeted redirection is a viable alternative to quality-degradation defenses, and readers familiar with diffusion personalization pipelines who want to understand why snapshot-based perturbation methods fail and how trajectory matching addresses that gap.

Authors’ abstract

Recent advancements in diffusion models have made fine-tuning text-to-image models for personalization increasingly accessible, but have also raised significant concerns regarding unauthorized data usage and privacy infringement. Current protection methods are limited to passively degrading image quality, failing to achieve stable control. While Targeted Data Protection (TDP) offers a promising paradigm for active redirection toward user-specified target concepts, existing TDP attempts suffer from poor controllability due to snapshot-matching approaches that fail to account for complete learning dynamics. We introduce TAFAP (Trajectory Alignment via Fine-tuning with Adversarial Perturbations), the first method to successfully achieve effective TDP by controlling the entire training trajectory. Unlike snapshot-based methods whose protective influence is easily diluted as training progresses, TAFAP employs trajectory-matching inspired by dataset distillation to enforce persistent, verifiable transformations throughout fine-tuning. We validate our method through extensive experiments, demonstrating the first successful targeted transformation in diffusion models with simultaneous control over both identity and visual patterns. TAFAP significantly outperforms existing TDP attempts, achieving robust redirection toward target concepts while maintaining high image quality. This work enables verifiable safeguards and provides a new framework for controlling and tracing alterations in diffusion model outputs.