Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise

AI summary

Overview

• Research area: Computer vision — diffusion-based image generation, specifically image-based virtual try-on (VTON).
• Technical level: Intermediate. The paper is conceptually argued rather than equation-heavy in its core message, but it assumes familiarity with latent diffusion models, classifier-free guidance, UNet backbones, and evaluation metrics such as FID, KID, SSIM, and LPIPS.
• Scope: A diagnosis of why full fine-tuning fails in single-network, spatial-concatenation VTON models, plus a training and inference recipe (DeCo-VTON) that fixes it without changing the architecture.

What This Paper Is About

Diffusion-based virtual try-on has two main designs: dual-UNet models that add a second network just for the garment (strong results, doubled cost), and single-network models that simply paste the garment and person latents together ("spatial concatenation", cheaper but weaker). A puzzling pattern has appeared across this second family: both CatVTON (an 860M UNet) and Voost (an 11.9B DiT) report that full fine-tuning does not help, and both fall back to training only the attention layers. The paper asks why full fine-tuning fails under spatial concatenation and whether the failure can be fixed. The authors answer by showing that garment conditioning must be kept separate from the denoising process, deriving three design principles from that insight, and packaging them as a recipe — not an architecture change — that unlocks effective full fine-tuning.

Key Contributions

1. Analysis. To the authors' knowledge, the first visualization study of dual-UNet reference network behavior. By decoding each reference UNet's noise prediction at different timesteps, the paper compares IDM-VTON (frozen weights), OOTDiffusion (fine-tuned but fixed at t=0), and Leffa (fine-tuned with the same timestep t as the denoising UNet), and concludes that successful garment conditioning requires decoupling the conditioning pathway from the denoising process.
2. Diagnosis and principles. The paper identifies three functional conflicts caused by spatial concatenation — guidance leakage, gradient competition, and train-test discrepancy — and derives one design principle for each: Garment-Free Guidance, Decoupled Loss, and Clean Latent Anchoring.
3. Results. Implementing these principles as a pure recipe on an unchanged SD1.5 inpainting UNet produces DeCo-VTON (860M parameters), reported as the first method to unlock effective full fine-tuning under spatial concatenation, achieving single-network state of the art and matching the dual-UNet state of the art at half the parameters.
4. Cross-architecture argument. The paper points out that CatVTON (860M UNet) and Voost (11.9B DiT) reach the same conclusion about full fine-tuning from very different architectures and scales, which the authors read as evidence that the problem lies in spatial concatenation itself, not in any particular backbone.

Main Findings

• Decoupling is the unifying insight. Dual-UNet models succeed because their reference UNet runs in a separate forward pass on clean garment input and carries no denoising loss. Spatial concatenation removes that structural separation by placing the garment inside the denoising target.
• Reference UNet timestep behavior explains dual-UNet quality gaps. IDM-VTON's frozen weights expect noisy input matching timestep t, creating a distributional mismatch with the clean garment latent; features are unreliable at high t, and at low t the output has low variance. OOTDiffusion's fixed t=0 produces strong but timestep-invariant features. Leffa's fine-tuned, timestep-aligned reference UNet produces stage-adaptive features that move from coarse structure at high t to fine texture at low t.
• Conflict 1 — guidance leakage. Under spatial concatenation, CatVTON's unconditional branch removes the clean garment latent but keeps the noisy one, so garment information leaks in; increasing the guidance scale ω suppresses rather than enhances garment detail.
• Conflict 2 — gradient competition. The training loss covers the garment region too, so the network is pulled simultaneously toward pixel-level garment reconstruction and toward transferring garment information onto the person. Dual-UNet reference networks carry no reconstruction loss.
• Conflict 3 — train-test discrepancy. At training, forward diffusion gives the garment region a correctly scheduled noise level; at inference, the garment latent is updated by the model's own predictions, so the noise level drifts away from what the network expects and error accumulates over steps.
• Full fine-tuning failure is confirmed and then overcome. In the progressive ablation on VITON-HD, CatVTON full fine-tuning (paired FID 5.250, KID 0.402) improves only marginally over attention-only training (5.425 / 0.411). Adding Garment-Free Guidance gives the largest single jump (4.538 / 0.074), Decoupled Loss adds a further step (4.517 / 0.068), and Clean Latent Anchoring completes the model (4.438 / 0.010).
• Quantitative results on VITON-HD (paired FID / KID / SSIM / LPIPS; unpaired FID / KID). DeCo-VTON: 4.438 / 0.010 / 0.880 / 0.047; 8.266 / 0.517. Leffa: 4.540 / 0.050 / 0.899 / 0.048; 8.520 / 0.320. CatVTON: 5.425 / 0.411 / 0.870 / 0.057; 9.015 / 1.091. DeCo-VTON takes the best paired FID, KID, and LPIPS and the best unpaired FID, trailing Leffa only in SSIM (0.880 vs 0.899) and unpaired KID (0.517 vs 0.320).
• Quantitative results on DressCode (same metric order). DeCo-VTON: 2.175 / 0.062 / 0.914 / 0.031; 4.310 / 0.628. Leffa: 2.060 / 0.070 / 0.924 / 0.031; 4.480 / 0.620. The paper notes DeCo-VTON matches Leffa in LPIPS and is lower in KID, with the best unpaired FID; Leffa remains lower on paired FID (2.060 vs 2.175) and on SSIM (0.924 vs 0.914).
• Efficiency. DeCo-VTON reports 859.54M parameters, 973.99 GFLOPs, 1.3s latency, and 2.26 GB VRAM at 512×384, identical to CatVTON and the same as its backbone. Leffa is listed at 1802.72M parameters, 1012.03 GFLOPs, 2.7s, and 3.91 GB. The paper states this is 2.1× faster inference and 42% lower VRAM than Leffa. IDM-VTON and OOTDiffusion are listed at 7003.26M / 2679.45 GFLOPs / 6.6s / 14.62 GB and 2229.73M / 1225.16 GFLOPs / 1.5s / 5.93 GB respectively.
• Guidance-scale robustness. At ω=1.0, CatVTON already produces blurred garments (Conflicts 2 and 3), and higher ω adds artifacts and over-sharpening (Conflict 1). DeCo-VTON preserves readable logos at ω=1.0 and stays visually stable across scales; the paper reports CatVTON's FID degrading sharply beyond ω=2.5 while DeCo-VTON remains stable. The default scale in the experiments is ω=2.5.
• User study. A randomized blind study with 25 participants and 30 samples (both p < 0.001) prefers DeCo-VTON over CatVTON at approximately 3.5:1 and over Leffa at approximately 2:1.
• Higher resolution. The paper states that at 1024×768 (Appendix B) DeCo-VTON attains the best paired FID on both benchmarks; the specific values are in an appendix not included in the provided content.
• Reported cross-architecture numbers. The paper cites Voost (11.9B DiT) as reporting full fine-tuning at FID 6.351 versus attention-only training at FID 5.269, with less than a quarter of the trainable parameters.

Methodology in Plain English

The authors start from a puzzle rather than a new architecture. They take the reference network of three dual-UNet systems, pass the garment through it at several timesteps, and decode the predicted noise back into image space to see what information each network supplies at each stage. From the pattern that emerges — reliable, stage-adaptive garment features when the reference network is fine-tuned and given the same timestep as the denoising network — they infer a general rule: the garment should act only as a condition, never as something the model is asked to reconstruct or denoise.

They then show that spatial concatenation breaks this rule in three concrete ways and fix each one while leaving the network untouched:

1. Garment-Free Guidance. Instead of CatVTON's unconditional branch, which still carries the noisy garment latent, they build a strictly garment-free unconditional input by zeroing out all garment-related latents, and use that in the classifier-free guidance formula. During training they drop the garment condition with probability 10% so the model learns the garment-free prediction.
2. Decoupled Loss. Because the person and garment latents are stacked along the height dimension, the UNet output splits naturally into a person part and a garment part. They compute the loss only on the person region, applying DREAM-style target and input rectification (with λ = 10.0) exclusively to that region and leaving the garment latent untouched.
3. Clean Latent Anchoring. Since the clean garment latent is available as input, they re-derive the garment's noisy latent at each step directly by forward diffusion using one fixed noise sample, instead of letting the model's own predictions update it. This removes error accumulation and keeps the garment noise level aligned with timestep t. It is an inference-only change; training already uses correctly scheduled noise.

Everything is evaluated on the same backbone (SD1.5 inpainting UNet, 860M), the same input formulation, and the same optimizer and learning rate as CatVTON, with full fine-tuning enabled. Training is at 512×384 with AdamW, batch size 128, learning rate 1×10⁻⁵, weight decay 1×10⁻², gradient clipping norm 1.0, gradient accumulation 2, BF16 mixed precision, a linear noise scheduler, ε-prediction, and 16K steps on VITON-HD and 32K steps on DressCode on 2 NVIDIA H200 GPUs (approximately 10 and 20 hours). Evaluation uses VITON-HD (11,647 training / 2,032 testing upper-body pairs) and DressCode (48,392 training / 5,400 testing full-body pairs across tops, bottoms, and dresses), with FID, KID, SSIM, and LPIPS for paired evaluation and FID and KID for unpaired evaluation.

Why This Matters

The paper reframes a widely observed but unexplained failure — full fine-tuning not working under spatial concatenation — as a conditionable design flaw rather than a dead end. Its practical appeal is that it requires no new architecture and no extra parameters: the same 860M backbone simply trains better once the conditioning is decoupled. That makes the recipe cheap to adopt for anyone already using a CatVTON-style pipeline.

Real-world applications:

• Fashion e-commerce product pages, where shoppers upload a photo and see a garment on themselves, a use case the paper links to improved customer experience and reduced return rates.
• Retail visualization at scale, since the model's 2.26 GB VRAM and 1.3s latency on an H200 make single-GPU or modest-server deployment plausible relative to dual-UNet or 7B-parameter alternatives.
• Catalog and marketing content generation, where garment logos, text, and patterns must survive faithfully — an area the qualitative comparison highlights.
• Reference-based inpainting and outpainting, which the paper explicitly identifies as other tasks where a known signal is embedded inside the denoising target and may benefit from the same decoupling principles.

Industry relevance: the headline trade-off — matching dual-UNet quality at half the parameters, 2.1× faster inference, and 42% lower VRAM — targets the cost side of generative try-on services, where latency and memory directly determine serving economics.

Future Directions

• Extending beyond UNet and classifier-free guidance. Only Garment-Free Guidance is tied to the CFG mechanism; Decoupled Loss and Clean Latent Anchoring address gradient competition and the train-test discrepancy, which the authors argue come from spatial concatenation itself and should transfer to DiT and flow-matching backbones with minor adaptation. Adapting GFG to non-CFG architectures would need a different way to keep garment information out of the baseline prediction.
• Testing the principles on flow-matching systems such as Voost, which the paper says show the same full fine-tuning failure, to confirm whether the diagnosis is truly architecture-agnostic.
• Applying the framework to other spatial-concatenation tasks such as reference-based inpainting and outpainting, where a known signal is likewise embedded in the denoising target.
• Determining where the gains saturate, since the paper's higher-resolution results are reported in an appendix and the recipe has so far been validated on a single 860M UNet backbone at 512×384 and 1024×768.

Target Audience

Researchers and engineers working on diffusion-based image generation, particularly those building or deploying virtual try-on systems. It is most useful to readers already familiar with latent diffusion, classifier-free guidance, and standard generative metrics, and to practitioners who want a training-and-inference recipe they can apply to an existing spatial-concatenation pipeline without redesigning the network. Readers interested in conditioning mechanisms more broadly — inpainting, editing, and multimodal conditioning — will also find the decoupling argument relevant.

Authors’ abstract

Virtual Try-On (VTON) synthesizes realistic images of a person wearing a target garment, with broad applications in e-commerce and fashion. Diffusion-based dual-UNet methods achieve strong results but double the parameters by dedicating a separate network to garment conditioning. Spatial concatenation offers a simpler single-network alternative, yet both UNet- and DiT-based instantiations report that full fine-tuning is ineffective, and the community has settled for attention-only training. We ask: why does full fine-tuning fail, and can this be resolved? Through what is, to our knowledge, the first visualization study of dual-UNet reference network behavior, we identify a unifying insight: garment conditioning must be decoupled from the denoising process. Spatial concatenation violates this by embedding the garment within the denoising target, causing three conflicts: guidance leakage, gradient competition, and train-test discrepancy. We derive three design principles to restore this decoupling and implement them as a pure recipe atop a standard architecture with no modification. The resulting model, DeCo-VTON (860M params), achieves single-network state of the art, matching the dual-UNet state of the art at half the cost while being preferred in human evaluation.