The devil is in the details: Enhancing Video Virtual Try-On via Keyframe-Driven Details Injection

AI summary

Overview

Research area: Computer Vision — video virtual try-on (VVT), diffusion transformers (DiTs), and dataset construction. Technical level: Advanced (requires familiarity with diffusion models, DiT backbones, flow matching, and LoRA finetuning). Scope: This paper introduces KeyTailor, a DiT-based video virtual try-on framework that injects keyframe-derived garment and background details into a standard DiT backbone, together with ViT-HD, a 15,070-sample high-definition try-on dataset.

What This Paper Is About

Video virtual try-on replaces the clothing a person wears in a video with a target garment image while keeping motion, background, and garment appearance consistent across frames. Existing DiT-based methods miss fine-grained garment dynamics (wrinkles, backside textures, motion-induced changes), blur or distort backgrounds, and add costly interaction modules to the DiT backbone; public datasets are also small and low-resolution. The paper's goal is to improve garment fidelity and background integrity without modifying the DiT architecture, and to supply a larger, higher-quality dataset for training and evaluation.

Key Contributions

1. KeyTailor framework. A DiT-based VVT framework that uses a keyframe-driven details injection strategy to improve garment fidelity and background integrity, without introducing additional interaction layers into the DiT backbone.
2. Instruction-guided keyframe sampling plus two lightweight injection modules. An instruction-guided keyframe sampling (IKS) approach selects informative keyframes, and two keyframe-driven modules — the garment dynamic details enhancement (GDDE) module and the collaborative background details optimization (CBDO) module — inject details into the DiTs.
3. ViT-HD dataset. A new large-scale, high-definition dataset collected from multiple e-commerce platforms containing 15,070 video samples at a resolution of 810 × 1080, covering various garment styles.
4. Broad empirical validation. Experiments on ViT-HD, two widely used VVT datasets (VVT and ViViD), and two image-based virtual try-on datasets (VITON-HD and DressCode) show KeyTailor outperforming state-of-the-art baselines in garment fidelity and background consistency in both dynamic and static scenarios.

Main Findings

• Better results on ViT-HD and VVT. On ViT-HD, KeyTailor reaches VFID_I^p 7.5267, VFID_R^p 0.1628, SSIM 0.9066, LPIPS 0.0397, VFID_I^u 13.6628, and VFID_R^u 0.3519, versus MagicTryOn at 14.0587, 0.2461, 0.8622, 0.0828, 19.2253, and 0.5587. On VVT, KeyTailor records 1.226, 0.0059, 0.968, and 0.016, compared with MagicTryOn at 1.991, 0.0084, 0.958, and 0.024.
• Best results on ViViD. On ViViD, KeyTailor scores VFID_I^p 9.5285, VFID_R^p 0.2761, SSIM 0.8985, LPIPS 0.0585, VFID_I^u 14.5381, VFID_R^u 0.3869; the version additionally fine-tuned on its own dataset (KeyTailor§) improves to 8.2164, 0.1854, 0.9023, 0.0522, 14.1521, 0.3106. DreamVVT§ shows no comparable gain when trained on its own dataset (11.0180, 0.2549, 0.8737, 0.0619, 16.9468, 0.4285).
• Strong static-scenario generalization. On VITON-HD, KeyTailor reports FID_p 5.293, KID_p 0.720, SSIM 0.9201, LPIPS 0.0566, FID_u 8.528, KID_u 0.788. On DressCode it reports FID_p 2.746, KID_p 0.517, SSIM 0.9621, LPIPS 0.0347, FID_u 5.147, KID_u 1.012.
• Low parameter overhead. Relative to its backbone, KeyTailor adds only a 2.10% increase in parameters, versus 15.11% for MagicTryOn and 157.10% for ViViD. It trains on 0.2057B parameters (versus 16.446B for MagicTryOn) while maintaining computational demands comparable to Wan 2.1 (194607.28 G FLOPs and 12.58 s/it versus 193899.00 G and 12.38 s/it).
• Ablation confirms each module matters. Removing the distillation component D causes the largest degradation (VFID_I^p 22.5241, VFID_R^p 0.5869, SSIM 0.7658, LPIPS 0.2106, VFID_I^u 25.3632, VFID_R^u 0.7901) versus the full model's 7.5267, 0.1628, 0.9066, 0.0397, 13.6628, 0.3519. Removing IKS, S_r(f), Q_key, L_key^bg, the fusion step, CBDO, or GDDE, and setting F_key = 1, all degrade performance.
• User preference. In a user study with 80 completed feedback forms, each questionnaire containing 15 randomly selected generated videos, the majority of participants preferred KeyTailor over state-of-the-art methods across visual quality, semantic consistency, and overall quality, with particularly clear advantages over CatV2TON and ViViD.
• Dataset scale and quality. ViT-HD provides 15,070 samples at 810 × 1080, exceeding VVT (791 samples at 192 × 256) and ViViD (9,700 samples at 632 × 824) in scale, and is marked as having multi-class garment diversity, no start-frame overexposure, and subject integrity.

Methodology in Plain English

The key idea is that a few well-chosen "keyframes" in a video already contain the garment's changing appearance (front, back, wrinkles from motion) and the background's stable structure — so they can be used as extra guidance instead of redesigning the DiT.

• Selecting keyframes. An instruction-guided keyframe sampling module uses a large visual-language model (e.g., QWen) to parse a view–action instruction into target views and actions, generates standardized multi-anchor pose frames, and scores every input frame by combining a motion-difference score with a garment-area ratio score via S_f(f) = 1 − S_m(f) + λ · S_r(f). A dual-selection strategy with two thresholds constrains both score difference and temporal interval to avoid redundancy.
• Enriching garment details. The garment dynamic details enhancement module crops the first agnostic frame, applies a pre-trained single-image try-on model with LoRA layers, encodes the result into a latent L_g with a VAE encoder, then distills garment variation features from the keyframes into it using a lightweight component D implemented as two 1 × 1 convolution layers followed by a LayerNorm.
• Repairing background details. The collaborative background details optimization module has two branches: a coarse global background encoding branch using a mask guider E_BG (four 3D convolutional layers with channel dimensions 32, 96, 192, and 256, with a zero-initialized linear layer), and a fine-grained branch that encodes keyframe background regions and blends the frame with the highest background completeness score via L̄_bg = α · L_bg + (1 − α) L_key^max, with α defaulting to 0.3.
• Injecting into the DiT. A three-step fusion strategy combines pose and mask latents into tokens, fuses them with the garment latents, concatenates the result with patchified noise, and adds the background latents via an "addto" operation. The garment latents also replace the original text tokens in cross-attention. No DiT component is architecturally modified; LoRA finetunes self-attention and cross-attention.
• Training setup. The base model is Wan2.1-I2V-14B-720P; each training video sample has 81 frames, batch size is 1, training runs 14,500 iterations with the AdamW optimizer at a fixed 1e-4 learning rate, and video inference uses 25 steps. FiTDiT serves as the image-based try-on model. Training combines ViT-HD (13,070 paired training samples, 2,000 test samples after partitioning) with ViViD (7,759 paired video training samples).

Why This Matters

Impact on research. The paper shows that detail quality in VVT can be improved by better use of existing frames rather than by adding heavyweight interaction modules to the DiT backbone, and it supplies both a strong baseline and a larger high-definition dataset for the field.

Real-world applications:

• E-commerce online garment fitting, letting shoppers see a specific garment on a model in motion.
• Short-video and social platforms, where virtual try-on can be offered as an engaging creative experience.
• Garment marketing and showcase video production from a single high-resolution garment image plus a model video.
• Person-to-video garment transfer, which the paper demonstrates as a more challenging scenario.

Industry relevance. The efficiency comparison targets practical deployment concerns: KeyTailor adds only 2.10% more parameters than its backbone, trains on far fewer parameters than MagicTryOn (0.2057B versus 16.446B), and reduces inference time relative to MagicTryOn (281.65 s versus 345.27 s) under the stated 64-frame ViViD evaluation setting.

Future Directions

• Scaling and diversifying the data further. ViT-HD already contains 15,070 samples at 810 × 1080, but the paper leaves open whether still larger or more varied catalogues further improve generalization.
• Extending keyframe selection. The ablation shows IKS and multi-keyframe usage matter; how the instruction-guided sampling behaves under harder view or action descriptions is an open question.
• Broader scenario coverage. The paper reports gains in dynamic and static settings plus person-to-video transfer; robustness in more complex or in-the-wild conditions remains an area for further study.
• Closing the remaining quality gap. The ablation reveals measurable degradation when any module is removed, and residual errors such as unnatural color shifts, distorted backgrounds, hallucinated objects, and skin tone changes are still observed in variants, indicating room for improvement in the full model's failure cases.

Target Audience

Researchers and engineers working on generative video models, virtual try-on, and diffusion transformers, as well as practitioners building e-commerce or short-video fashion applications. Readers need prior exposure to diffusion models, DiT architectures, and parameter-efficient finetuning to follow the methodology, though the dataset contribution and comparative results are accessible to a broader computer vision audience.

Authors’ abstract

Although diffusion transformer (DiT)-based video virtual try-on (VVT) has made significant progress in synthesizing realistic videos, existing methods still struggle to capture fine-grained garment dynamics and preserve background integrity across video frames. They also incur high computational costs due to additional interaction modules introduced into DiTs, while the limited scale and quality of existing public datasets also restrict model generalization and effective training. To address these challenges, we propose a novel framework, KeyTailor, along with a large-scale, high-definition dataset, ViT-HD. The core idea of KeyTailor is a keyframe-driven details injection strategy, motivated by the fact that keyframes inherently contain both foreground dynamics and background consistency. Specifically, KeyTailor adopts an instruction-guided keyframe sampling strategy to filter informative frames from the input video. Subsequently,two tailored keyframe-driven modules, the garment details enhancement module and the collaborative background optimization module, are employed to distill garment dynamics into garment-related latents and to optimize the integrity of background latents, both guided by keyframes.These enriched details are then injected into standard DiT blocks together with pose, mask, and noise latents, enabling efficient and realistic try-on video synthesis. This design ensures consistency without explicitly modifying the DiT architecture, while simultaneously avoiding additional complexity. In addition, our dataset ViT-HD comprises 15, 070 high-quality video samples at a resolution of 810*1080, covering diverse garments. Extensive experiments demonstrate that KeyTailor outperforms state-of-the-art baselines in terms of garment fidelity and background integrity across both dynamic and static scenarios.