Diffusion-Driven Progressive Target Manipulation for Source-Free Domain Adaptation

AI summary

Overview

Research area: Computer vision, specifically source-free domain adaptation (SFDA) — training a model to work on a new target domain when the original labeled source data is no longer available.

Technical level: Advanced. The method combines latent diffusion models, DDIM inversion, and Fourier-based frequency decomposition, so familiarity with diffusion models and domain adaptation is assumed.

Scope: The paper proposes a generation-based SFDA framework called Diffusion-Driven Progressive Target Manipulation (DPTM) that builds and iteratively refines a "pseudo-target" domain with a latent diffusion model, and evaluates it on four standard domain adaptation benchmarks.

What This Paper Is About

Source-free domain adaptation methods are limited by the gap between the source domain (used to train the model) and the target domain (where the model is applied). Existing approaches either trust noisy pseudo-labels generated by the source model, or generate a "pseudo-source" domain that reintroduces and can even enlarge the domain gap. This paper's goal is to break that bottleneck by generating a pseudo-target domain instead — taking unlabeled target images, splitting them by prediction confidence, and using a diffusion model to semantically "move" the unreliable images toward newly assigned labels while keeping them inside the target distribution.

Key Contributions

1. The DPTM framework: A generation-based SFDA method that progressively constructs and refines a pseudo-target domain using unlabeled target data as references with a latent diffusion model, rather than generating a pseudo-source domain.

2. A three-part manipulation strategy for unreliable samples: Target-guided Initialization (builds the diffusion sampling start point from the target image's low-frequency content and random-noise high-frequency content), Semantic Feature Injection (injects the assigned label's semantics along the sampling trajectory via DDIM inversion), and Domain-specific Feature Preservation (re-injects target-domain low-frequency features at each denoising step to keep generated samples in the target distribution).

3. A Progressive Refinement Mechanism: An iterative procedure that re-partitions the target data and refines the pseudo-target domain over R iterations, so the trust set grows, the manipulated non-trust set shrinks, and the residual domain discrepancy decreases.

4. State-of-the-art results across four benchmarks of different scales (Office-31, Office-Home, VisDA, DomainNet-126), with gains reported up to 18.6% in scenarios with large source-target gaps.

Main Findings

• Office-31 (ResNet-50, small-scale): DPTM reaches 95.8% average accuracy, versus 93.7% for the best generation-based method (DM-SFDA), 90.3% for ISFDA, 89.9% for ProDe, and 80.7% for the source-only baseline. The paper reports a 9.3% improvement on D→A and 8.2% on W→A over the existing SOTA method, and an average accuracy gain of 5% over the best non-generation methods in all tasks except D→W.

• Office-Home (ResNet-50, medium-scale): DPTM reaches 91.2% average accuracy. The paper states an average accuracy gain of 11.7% over DM-SFDA (79.5%) and 10.1% over ProDe (81.1%). On the challenging Ar→Cl, Pr→Cl, and Rw→Cl tasks it outperforms ProDe by 22.7%, 21.0%, and 21.6% respectively.

• VisDA (ResNet-101, large-scale): DPTM reaches 97.6% average accuracy, which the paper reports as an 8.5% average accuracy gain over ISFDA (88.4%) and 8.2% over ProDe (88.7%).

• DomainNet-126 (ResNet-50, large-scale): DPTM reaches 85.2% average accuracy. The paper reports 17.6% higher average accuracy than the generation-based CPGA (67.6%) and a 3.7% margin over ProDe (81.5%). The introduction separately highlights a gain of 24.4% over the existing generation-based method and 6.3% over SOTA for the C→P task.

• Large domain gaps are where the method helps most: The abstract states DPTM can enhance performance by up to 18.6% in scenarios with large source-target gaps; the introduction attributes an 18.6% gain over SOTA to the Rw→Cl task on Office-Home, while the results section reports 21.6% for that same task. The paper motivates this focus with a prior observation that the same source model yields over 90% accuracy for Ar→Cl but about 60% for Ar→Pr on Office-Home.

• Threshold choice matters: With R = 10 refinement iterations, an entropy threshold E of 0.001 gives 80.7% average accuracy on Office-Home, E = 0.005 gives 86.7%, and E = 0.01 gives 91.2% — the value used in the main experiments.

• Diffusion backbone is roughly interchangeable: With E = 0.001 and R = 3, Stable Diffusion XL and Stable Diffusion v1.5 both average 75.6% on Office-Home. SDXL natively produces 1024×1024 images and SD15 produces 512×512; the authors choose SD15 because SDXL's computational cost is higher.

• Qualitative manipulation quality: A figure comparing manipulated samples with and without each of the three components shows the full method produces the best semantic alignment with the assigned labels — using the example labels Alarm Clock, Curtains, Computer, and Bottle — and the best preservation of target-domain characteristics.

Methodology in Plain English

The pipeline starts with a model trained on labeled source data. That model is applied to the unlabeled target images, and each prediction is scored by entropy (a measure of uncertainty).

1. Split the data. Images with low-entropy predictions go into a "trust set" and are trained on directly with their pseudo-labels. High-entropy images go into a "non-trust set."

2. Assign fresh labels. Each non-trust image is given a new class label in a round-robin fashion, cycling through all classes so no class gets over-represented and any mismatch from dropping leftover samples is avoided.

3. Manipulate the image with diffusion. The non-trust image is transformed so it plausibly belongs to its newly assigned class while still looking like it came from the target domain. This is done in three stages:

   • Target-guided Initialization: Take the target image's low-frequency content (which carries domain style) via Fast Fourier Transform, take high-frequency content from semantically neutral Gaussian noise (so no wrong semantic leaks in), combine them, encode the result, and add noise to get the diffusion start point.
   • Semantic Feature Injection: During denoising, at each step, reverse-engineer the latent with DDIM inversion driven by the assigned class label, then keep only the high-frequency part of that result — the part carrying new semantic information — and discard low-frequency artifacts.
   • Domain-specific Feature Preservation: Perturb the original clean target latent with noise matched to the current timestep, extract its low-frequency component, and merge it with the injected high-frequency semantics so the final sample stays inside the target distribution.

4. Retrain and repeat. The manipulated samples plus the trust set form a pseudo-target domain, and the source model is fine-tuned on it. Then the whole process repeats for R iterations: the model re-splits the data, the trust set grows, the manipulated set shrinks, and the pseudo-target domain gets closer to the real target domain.

Implementation settings reported: Stable Diffusion v1.5, 512×512 images, 20 denoising steps, γ₁ = 5.5, γ₂ = 0, E = 0.01, R = 10. ResNet-50 is used for Office-31, Office-Home, and DomainNet-126; ResNet-101 for VisDA. Large-scale datasets train for 20K iterations with batch size 128 and learning rate 3e-3; Office-31 and Office-Home train for 15K iterations with batch size 32 and learning rate 1e-3. Weight decay is 5e-4 throughout. Comparisons cover 21 existing methods across a source-only baseline, six generation-based methods (CPGA, ASOGE, ISFDA, PS, DATUM, DM-SFDA), and fifteen non-generation methods including SHOT, NRC, GKD, HCL, AaD, AdaCon, CoWA, SCLM, ELR, PLUE, CRS, CPD, TPDS, DIFO, and ProDe.

Why This Matters

Impact on research: The paper challenges the dominant generation-based SFDA paradigm, which converts SFDA into conventional UDA by synthesizing a pseudo-source domain. By showing that generating a pseudo-target domain instead yields large gains — particularly on the hardest adaptation pairs where prior methods collapse — it reframes where the domain-gap bottleneck actually lives and gives a concrete recipe (frequency-selective latent manipulation) for addressing it.

Real-world applications:

• Deploying vision models in a new operating environment (different cameras, lighting, or sensors) when the original training data cannot be shared or retained.
• Medical imaging, where patient privacy and institutional rules often prevent moving source data, but unlabeled scans from a new scanner or hospital are available.
• Industrial and manufacturing inspection, where a model trained on one production line's imagery must transfer to a new line or plant.
• Autonomous driving and robotics, where a perception model trained in one city or weather condition must adapt to another using only unlabeled footage.

Industry relevance: The source-free constraint is the practical one — regulations, contracts, and privacy policies frequently prohibit retaining or transferring source data. A method that needs only a frozen pre-trained model plus unlabeled target data, and that works across small (Office-31), medium (Office-Home), and large (VisDA, DomainNet-126) benchmarks, maps directly onto how deployed models are actually maintained and updated.

Future Directions

• Sensitivity to the entropy threshold and iteration count. The paper notes that "setting E and R to other values may obtain superior performance," leaving systematic tuning of the trust/non-trust boundary and refinement depth as open work.
• Cost of stronger diffusion backbones. SDXL and SD15 perform comparably in the ablation, but SDXL is rejected for its higher computational cost. Making the manipulation pipeline practical with stronger, higher-resolution generators is an unresolved engineering question.
• Quantifying the refinement trajectory. The ablation on the Progressive Refinement Mechanism is truncated in the available text and its detailed performance trajectory is deferred to the supplementary material, so the precise dynamics of how trust-set size and manipulated-set size evolve across R iterations remain to be examined in the main paper.
• Extension beyond classification benchmarks. All four evaluations are object-recognition datasets with ResNet backbones; whether the pseudo-target generation approach transfers to detection, segmentation, or other task families is not addressed.

Target Audience

Researchers and graduate students working on domain adaptation, test-time adaptation, or diffusion-model applications in vision; practitioners who must adapt deployed perception models without access to source data; and readers interested in how frequency-domain decomposition can be used to separate "what an image shows" from "what domain it comes from." A working knowledge of diffusion models and pseudo-labeling is needed to follow the method section, but the problem framing and results tables are accessible to anyone familiar with standard image classification benchmarks.

Authors’ abstract

Source-free domain adaptation (SFDA) is a challenging task that tackles domain shifts using only a pre-trained source model and unlabeled target data. Existing SFDA methods are restricted by the fundamental limitation of source-target domain discrepancy. Non-generation SFDA methods suffer from unreliable pseudo-labels in challenging scenarios with large domain discrepancies, while generation-based SFDA methods are evidently degraded due to enlarged domain discrepancies in creating pseudo-source data. To address this limitation, we propose a novel generation-based framework named Diffusion-Driven Progressive Target Manipulation (DPTM) that leverages unlabeled target data as references to reliably generate and progressively refine a pseudo-target domain for SFDA. Specifically, we divide the target samples into a trust set and a non-trust set based on the reliability of pseudo-labels to sufficiently and reliably exploit their information. For samples from the non-trust set, we develop a manipulation strategy to semantically transform them into the newly assigned categories, while simultaneously maintaining them in the target distribution via a latent diffusion model. Furthermore, we design a progressive refinement mechanism that progressively reduces the domain discrepancy between the pseudo-target domain and the real target domain via iterative refinement. Experimental results demonstrate that DPTM outperforms existing methods by a large margin and achieves state-of-the-art performance on four prevailing SFDA benchmark datasets with different scales. Remarkably, DPTM can significantly enhance the performance by up to 18.6% in scenarios with large source-target gaps.