BooM-VVT: Boosting Mask-Free Video Virtual Try-On with Image-Level Pseudo Data

AI summary

Overview

• Research area: Computer vision, specifically generative video synthesis for virtual try-on, using diffusion/DiT models.
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, LoRA fine-tuning, positional encodings (3D-RoPE), and standard video-generation evaluation metrics (FVD, SSIM, LPIPS).
• Scope: The paper proposes a mask-free keyframe-driven pipeline for video virtual try-on, supported by a new multi-view dataset (OmniView) and a multi-stage training scheme that leans on image-level pseudo data.

What This Paper Is About

Video virtual try-on (VVT) aims to generate a realistic video of a person wearing a target garment. Existing keyframe-driven methods still depend on explicit masks to localize the try-on region, which breaks down under large body motion and severe occlusion, and they also suffer from coarse keyframe selection and a shortage of multi-view try-on data. BooM-VVT addresses all three problems by removing the mask requirement, choosing keyframes based on garment-relevant body regions, and releasing a new 6,110-sample multi-view dataset called OmniView.

Key Contributions

1. BooM-VVT, a mask-free VVT framework built on the keyframe-driven paradigm, using a multi-stage training strategy that learns try-on region localization from image-level pseudo data instead of costly video-level pseudo data.
2. Garment-Sensitive Keyframe Sampling (GSKS), which selects keyframes by scoring garment-relevant limbs and body regions rather than global pose change, plus Frame-Shared 3D-RoPE, which assigns matching positional encodings to keyframe try-on tokens and their corresponding video-frame tokens to strengthen garment-detail transfer.
3. OmniView, a large-scale multi-view try-on dataset of 6,110 samples (versus 1,009 in MVG), with at least two viewpoints per person, 88% of samples containing back-view images, front- and back-view flat garment images, an added outerwear category, and synthesized pseudo data for cross-category and layered try-on.

Main Findings

• ViViD-S results: On 180 ViViD-S samples, BooM-VVT (mask-free) reaches FVD_u 101.92, ABC 4.61, GC 3.95, OVQ 4.21, FVD_p 79.12, SSIM 0.891, and LPIPS 0.061, improving over MagicTryOn (110.66, 4.15, 2.38, 3.12, 86.31, 0.869, 0.068), CatV2TON, and ViViD.
• WildVVT results: On the new 100-sample in-the-wild benchmark, BooM-VVT achieves FVD 219.76, ABC 4.33, GC 3.72, OVQ 3.94, compared with ViViD (301.80, 2.13, 1.55, 1.57), CatV2TON (350.72, 1.39, 1.28, 1.23), and MagicTryOn (250.55, 2.12, 1.84, 1.65).
• Cost comparison: BooM-VVT uses 57.9G GPU memory and 281s inference time for a 65-frame, 624×832 video on a single NVIDIA A800, versus 342s and 63.5G for MagicTryOn.
• Mask-free beats masked variant: BooM-VVT (Stage 1, which uses masks) scores 241.72 FVD / 4.33 ABC / 3.07 GC / 3.26 OVQ on WildVVT, while the full mask-free model reaches 219.76 / 4.33 / 3.72 / 3.94.
• Ablation on GSKS: Removing GSKS raises FVD to 227.88 and lowers GC to 3.42 (full model: 219.76 and 3.72).
• Ablation on Frame-Shared 3D-RoPE: Removing it gives FVD 224.15, GC 3.27, OVQ 3.81; it scores a slightly higher ABC (4.34 vs 4.33) but loses garment transfer quality, and the paper reports it reconstructs incorrect shoulder straps and misses white chest stripes.
• Ablation on training stages: Dropping Stage 1 causes the largest degradation (FVD 242.61, GC 3.26, OVQ 3.66); dropping Stage 2 gives FVD 237.89, GC 3.37, OVQ 3.75.
• Data efficiency of Stage 2: The full model with 500 video-level pseudo samples performs comparably to the no-Stage-2 variant trained with 2,500 samples. For a fair-cost comparison, the no-Stage-2 variant was trained with 400 additional video-level pseudo samples to match the construction cost of the 30K image-level pseudo samples used in Stage 2.
• Qualitative behavior: On ViViD-S, mask-based baselines remain constrained by the original garment structure when source and target garment shapes differ substantially; on WildVVT, ViViD and MagicTryOn fail to preserve garment structure under turning motions and severe occlusion.

Methodology in Plain English

The pipeline has two parts. First, a keyframe try-on stage picks a small set of frames from the input video and generates try-on images for them. Second, a video generation model uses those try-on images as appearance conditions to synthesize the full try-on video.

For keyframe selection, the method uses DWPose to detect limbs and SAM to estimate the garment-relevant body region. It scores each frame by combining limb visibility and region visibility with a weight λ = 0.2, picks the highest-scoring frame, then iteratively adds frames that show the largest viewpoint difference from those already chosen, measuring difference as average cosine distance between skeletal joint direction vectors. Only limbs relevant to the target garment category are considered.

The selected keyframes are horizontally concatenated into one composite image, passed through a pretrained MM-DiT-based image try-on model in a single forward pass, then split back into individual frames; this is what gives cross-frame consistency. The model is fine-tuned on OmniView through rank-64 LoRA adapters in the self-attention layers, and it accepts either single-view or multi-view garment references.

The video generation model is built on Wan-Animate. Video and pose are encoded into tokens by a Video VAE, pose tokens are added to video tokens, and keyframe try-on tokens are concatenated along the sequence dimension before entering the DiT backbone. Frame-Shared 3D-RoPE gives each keyframe try-on token the same 3D positional encoding as its matching video-frame token, shortening their relative distance so attention between them is stronger.

Training happens in three stages. Stage 1 trains an Appearance LoRA (rank-128, 10K steps) on 20K unpaired human-centric videos, using garment masks to strip clothing and asking the model to reconstruct the original video. Stage 2 freezes the Appearance LoRA and trains a rank-64 Location LoRA for 20K steps on 30K multi-view image pairs (frames sampled from Stage 1 videos plus OmniView), where one image is edited with Qwen-Image-Edit-2511 to create a pseudo target, forcing the model to learn mask-free localization. Stage 3.1 uses the Stage 1 model to synthesize pseudo videos from 2,500 selected videos; Stage 3.2 jointly fine-tunes both LoRAs for 5K steps on those pseudo videos. All training used AdamW on 4 NVIDIA RTX Pro 6000 GPUs, learning rate 1×10⁻⁴, batch size 1. Inference samples two keyframes per video, uses 8 steps with a Qwen-Image-Edit-Lightning acceleration LoRA, and 20 video-generation steps with guidance scale 1.

Evaluation uses FVD, SSIM, and LPIPS, plus Gemini 3.1 Pro as a judge scoring 0–5 on Appearance and Background Consistency (ABC), Garment Consistency (GC), and Overall Video Quality (OVQ). OmniView data was annotated with Qwen3-VL-32B and pseudo data synthesized with Qwen-Image-Edit-2511, with manual verification.

Why This Matters

• Research impact: The work shows that mask-free try-on, previously demonstrated mainly for images, can be extended to temporally coherent video by learning localization from cheap image-level pseudo data rather than expensive video-level pseudo data. It also contributes a dataset and two targeted architectural ideas (garment-aware keyframe sampling and shared positional encodings) that are transferable to other keyframe-conditioned video generation tasks.
• E-commerce: Shoppers could see garments rendered on their own body across a video rather than a single still image, including back views and turning motions.
• Digital content creation: Creators could restyle or layer outfits in existing footage without manual rotoscoping.
• Virtual fitting and retail catalogs: Multi-view garment references plus the added outerwear category and layered try-on support more realistic product visualization.
• Robustness in unconstrained video: The mask-free design targets user-generated footage where masks are unreliable due to occlusion and large motion, and the reported inference cost (57.9G memory, 281s) places it in a practical range for production experimentation.

Future Directions

• Reducing dependence on off-the-shelf visual priors: The limitations section notes that GSKS still relies on DWPose and SAM, so extremely severe occlusion or inaccurate detection can lead to suboptimal keyframe selection.
• Robustness to lighting: The paper states that extreme illumination changes may affect mask-free localization, leaving this unaddressed.
• Lower inference cost: The authors list reducing inference cost as an important direction, given the reported 281s for a 65-frame video at 624×832.
• Filling the multi-view data gap further: OmniView is larger than MVG, but the paper frames the scarcity of multi-view try-on data as an ongoing constraint on handling challenging viewpoints and diverse try-on tasks.
• Direct comparison with other keyframe-driven methods: The paper reports that public implementations of existing keyframe-driven VVT methods were unavailable, so it used its own Stage-1 variant as the keyframe-driven baseline instead.

Target Audience

Researchers and engineers working on generative video, diffusion transformers, and virtual try-on; dataset builders interested in multi-view garment and person data; and applied teams in e-commerce or content creation evaluating mask-free, keyframe-driven video pipelines. Readers without a background in diffusion models or video generation architectures will find the methodology section dense, while the dataset and benchmark contributions are accessible to a broader audience.

Authors’ abstract

Video virtual try-on (VVT) aims to generate realistic videos of a person wearing a target garment. Recent methods leverage a keyframe-driven video generation paradigm to improve in-the-wild performance, yet they still rely on masks to localize try-on regions, making them vulnerable to large motions and severe occlusions. Although mask-free image-based try-on methods have shown promising results by leveraging large-scale pseudo data, extending this paradigm to videos remains difficult, as constructing video-level pseudo data is prohibitively expensive. Furthermore, coarse keyframe sampling and the scarcity of multi-view try-on data limit existing keyframe-driven methods in maintaining garment consistency and handling diverse try-on tasks. To address these challenges, we propose BooM-VVT, a mask-free VVT framework built upon the keyframe-driven paradigm. To achieve mask-free VVT, we introduce a multi-stage training strategy that leverages image-level pseudo data for mask-free localization learning, substantially reducing the need for costly video-level pseudo data. To improve garment consistency, we propose Garment-Sensitive Keyframe Sampling, which selects keyframes based on garment-relevant body regions to better capture garment appearance. We further introduce Frame-Shared 3D-RoPE to establish spatiotemporal correspondences between keyframes and target video frames for accurate garment-detail transfer. Finally, we construct OmniView, a large-scale multi-view try-on dataset to support reliable try-on video generation under complex camera viewpoints and diverse try-on tasks. Extensive experiments demonstrate that BooM-VVT achieves superior temporal consistency and garment fidelity over existing methods. Project page: https://boomvvt.github.io/boomvvt.