ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

AI summary

Overview

• Research area: Computer vision, specifically image-based virtual try-on, extended to 3D multi-view footwear (virtual footwear try-on, VFTON).
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), rotary position embeddings, attention mechanisms, and virtual try-on pipelines.
• Scope: Introduces MVShoes, a 7305-triplet multi-view shoe try-on dataset, and ShoeFit, a dual-image-stream DiT framework built on FLUX.1-Lite that adds Multi-View Conditioning with 3D RoPE and a LayeredRefAttention module.

What This Paper Is About

The paper defines virtual footwear try-on as a task that is distinct from garment try-on: given product shoe images and a human foot image, generate a realistic image of the person wearing those shoes. The authors argue that existing garment-focused try-on methods fail here for three reasons: no public footwear try-on dataset exists, the product image viewpoint rarely matches the target foot pose, and reflective shoe materials pick up color contamination from backgrounds. Their goal is to build both the missing dataset and a generation architecture that handles those failures.

Key Contributions

1. Formal definition of the VFTON problem, identifying three specific challenges: Data Scarcity, Viewpoint Misalignment, and Background-induced Color Distortion, all described as first explicitly proposed in this paper.
2. MVShoes dataset, described as the first multi-view footwear try-on dataset, containing 7305 annotated high-resolution image triplets. Each triplet has two product shoe images from different views plus a corresponding try-on result.
3. A dual-image-stream DiT architecture (ShoeFit) built on FLUX.1-Lite that replaces the text stream with a replicated conditioning image stream, reported to save approximately 40% of parameters, including those for T5 and CLIP models.
4. Two mechanism-level modules: Multi-View Conditioning (Reference Concatenation plus 3D Rotary Position Embedding) to address viewpoint misalignment, and LayeredRefAttention to decouple shoe appearance from background lighting.

Main Findings

• Dataset scale and construction: MVShoes contains 7305 annotated high-resolution image triplets, drawn from approximately 25,000 initially collected raw shoe product image sets, each with 2 to 5 images. Processing used Qwen-VL / Qwen2.5-VL-7B for product-versus-model classification, CLIP with a similarity threshold of 0.9 for deduplication, SAM for shoe segmentation, DINO feature matching with an inner-similarity cutoff of 0.8 to reject mismatched pairs, manual filtering, and DWPose plus Grounding DINO for foot poses and shoe-agnostic masks.

• Dataset composition: Four model scenarios are curated: top-down foot close-up, horizontal foot close-up, half-body model, and full-body model. Horizontal foot close-ups constitute the largest proportion. The authors note that reflections introduce background color contamination particularly in boots, leather shoes, and high heels.

• Quantitative comparison (single view): ShoeFit-1V reaches SSIM 0.772, LPIPS 0.159, FID 21.07, KID 0.944, DISTS 0.1059. Best baseline scores on the same setting were CatVTON-Flux-Lite at SSIM 0.745, LPIPS 0.185, FID 22.71, KID 3.327, DISTS 0.1179.

• Quantitative comparison (two views): ShoeFit-2V reaches SSIM 0.780, LPIPS 0.149, FID 20.18, KID 0.485, DISTS 0.0941, versus CatVTON-Flux-Lite-2V at SSIM 0.756, LPIPS 0.171, FID 22.64, KID 3.318, DISTS 0.1040.

• Baseline behavior: FLUX.1-Fill-Redux and FLUX.1-Fill produced the worst FID and DISTS scores in the comparison. OOTDiffusion and CatVTON showed complementary performance across unpaired and paired evaluations. OOTDiffusion tended to show extreme color variations; FLUX.1-Fill-Redux and FLUX.1-Fill were noted as possibly limited by only LoRA (r=64) being trainable.

• Ablation on the dual-image-stream design: The Vanilla Model (text stream plus image stream) scored SSIM 0.745, LPIPS 0.185, FID 22.71, KID 3.327, DISTS 0.1179; adding the dual-image-stream ("+ D-i-s") improved these to SSIM 0.761, LPIPS 0.174, FID 21.57, KID 1.493, DISTS 0.1038.

• Ablation on LayeredRefAttention: Adding LayeredRefAttention ("+ D-i-s + LRA") gave SSIM 0.772, LPIPS 0.159, FID 21.07, KID 0.944, DISTS 0.1059. A naive alternative, "+ D-i-s + Standard Attn (w/ mask)", scored worse on every metric: SSIM 0.756, LPIPS 0.181, FID 21.70, KID 1.698, DISTS 0.1056. The authors attribute the naive masking failure to discarding illumination cues embedded in the product image.

• Ablation on Multi-View Conditioning and 3D RoPE: The full model ("+ D-i-s + LRA + MC") scored SSIM 0.780, LPIPS 0.149, FID 20.18, KID 0.485, DISTS 0.0941. Removing 3D RoPE ("w/o 3D RoPE") scored SSIM 0.778, LPIPS 0.152, FID 20.18, KID 0.695, DISTS 0.0971, so the RoPE change gave gains with no additional computational overhead reported.

• Efficiency (NVIDIA H20, batch size 1, 25 steps, torch.bfloat16): Cat-VTON with image plus text streams took 22.27 seconds per image and 35.5 GB; ShoeFit's dual-image-stream variant took 17.44 seconds and 22.0 GB; ShoeFit plus LayeredRefAttention took 21.15 seconds and 28.5 GB.

• Training and inference configuration: Trained at 768 x 768 resolution, with 6305 pairs for training and 1000 pairs for testing, for 7 days on 8 80GB-A100 GPUs with DeepSpeed ZeRO-2, batch size 4, AdamW optimizer, constant learning rate of 3e-5. All DiT parameters are trainable. Inference runs on a single A10 GPU for 25 steps.

• Commercial framing: The introduction cites Nike's annual footwear sales reaching 33,427 million dollars.

• Stated limitations: The model occasionally fails on small logos and intricate text, which the authors note are critical for e-commerce sellers. It also fails on severely occluded rear-shoe regions, where keypoint and shoe detection modules misclassify the area as background so only the front foot is masked for inpainting. The paper also discusses a "shoe expansion" phenomenon, attributed to stiffness in shoe tongues causing outward bulging in real-world data, which the model replicates by fitting the training distribution.

Methodology in Plain English

The team first built the dataset because none existed. They gathered roughly 25,000 raw product image sets from the internet, then used vision-language and vision models to separate product photos from photos of people wearing shoes, remove duplicates, and verify that the product images matched the worn shoes by comparing segmented features. Human reviewers filtered remaining errors. Foot keypoint and mask annotations were then extracted automatically, and the pose estimator was fine-tuned on lower-body and close-up leg poses because standard pose tools struggle with feet. A leg-pose-aware mask strategy combines leg pose with an expanded bounding box from shoe detection, allowing cross-category try-on such as swapping boots or flip-flops for sneakers.

For the model, they started from FLUX, which normally has a text stream and an image (denoising) stream. Because try-on results are driven by the shoe images rather than text prompts, they removed the text stream and instead duplicated part of the denoising stream's architecture and weights to create a separate conditioning image stream. This separates preserving shoe detail from generating the final image. Multiple shoe views are concatenated along the spatial dimension and fed into that conditioning stream, and a three-dimensional rotary position embedding adds a per-view index channel alongside height and width so the model can tell which view a token came from.

For background contamination, the LayeredRefAttention module uses a Squeeze-and-Excitation block with global average pooling to compute per-view channel weights, then two linear layers produce scale and shift parameters that modulate the foreground shoe features. The conditioning stream still sees unmasked latents, preserving illumination information, but the denoising stream's attention is restricted to modulated foreground shoe features and the target human image. In effect, the background informs how the shoe should be modulated without its colors leaking into the generated shoe.

Why This Matters

The paper identifies footwear try-on as a largely unexplored subtask of virtual try-on with distinct failure modes that garment methods do not handle, and supplies both a benchmark dataset and a baseline architecture for it. Because no open-source footwear try-on dataset previously existed — the authors attribute this to companies keeping such data confidential due to commercial value — MVShoes gives subsequent researchers something to train and compare against.

Real-world applications:

• E-commerce product pages that render a specific shoe on a shopper's uploaded foot photo, using the multiple product views retailers already publish.
• Reducing returns and fit uncertainty by letting buyers visualize shoes on themselves before purchase.
• Cross-category visualization, for example showing a person currently wearing boots or flip-flops in a different shoe style, which the leg-pose-aware mask design explicitly supports.
• Automated catalog and advertising content generation for footwear brands, where faithful material and color reproduction matters.

Industry relevance is direct: the dataset sources span full-body, half-body, and close-up e-commerce scenarios, the authors include Alibaba Group affiliations, and the paper frames small logos and brand text as a high-value remaining problem for sellers.

Future Directions

• Detailed glyph conditioning: The authors plan a supplementary conditioning method that pre-detects and enhances injection of small logos and intricate text, which the model currently renders poorly.
• Explicit 3D geometric and material priors: Introducing these is proposed to achieve more robust multi-view representation and refined visual fidelity.
• Robustness under severe occlusion: The failure case where a heavily occluded rear shoe is misclassified as background is framed as an inherent challenge for any explicit masking strategy, and handling it is left to future work.
• Scaling and generalization: The paper positions itself as establishing a strong baseline, leaving open how the architecture behaves across broader categories, poses, and data conditions beyond what MVShoes covers. The appendix notes some dataset distribution statistics (Tables 4 and 5) cover shoe category counts and scene distributions, though the full tables are not included in the provided content.

Target Audience

Researchers and engineers working on virtual try-on, conditional image generation, and diffusion transformer architectures, particularly those interested in multi-reference conditioning and attention-level control. It is also relevant to practitioners in fashion e-commerce and retail technology evaluating try-on systems, and to dataset builders interested in the multi-stage automatic-plus-manual curation pipeline. Readers without a background in diffusion models, DiT blocks, or RoPE will find the method sections difficult without supplementary reading.

Authors’ abstract

Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse footwear and model images while maintaining 3D consistency and texture authenticity. Unlike conventional garment-focused VTON methods, VFTON presents unique challenges due to (1) Data Scarcity, which arises from the difficulty of perfectly matching product shoes with models wearing the identical ones, (2) Viewpoint Misalignment, where the target foot pose and source shoe views are always misaligned, leading to incomplete texture information and detail distortion, and (3) Background-induced Color Distortion, where complex material of footwear interacts with environmental lighting, causing unintended color contamination. To address these challenges, we introduce MVShoes, a multi-view shoe try-on dataset consisting of 7305 well-annotated image triplets, covering diverse footwear categories and challenging try-on scenarios. Furthermore, we propose a dual-stream DiT architecture, ShoeFit, designed to mitigate viewpoint misalignment through Multi-View Conditioning with 3D Rotary Position Embedding, and alleviate background-induced distortion using the LayeredRefAttention which leverages background features to modulate footwear latents. The proposed framework effectively decouples shoe appearance from environmental interferences while preserving high-quality texture detail through decoupled denoising and conditioning branches. Extensive quantitative and qualitative experiments demonstrate that our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis.