FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On

AI summary

Overview

Research area: Computer vision, specifically virtual try-on (VTO) and synthetic-to-real (sim2real) data generation for generative models.

Technical level: Advanced. The paper assumes familiarity with diffusion models, normal maps, LoRA fine-tuning, cloth physics simulation, and standard generative evaluation metrics.

Scope: The paper introduces FIT, a 1.13M-sample dataset with precise body and garment measurements for teaching virtual try-on models to render realistic garment fit (not just appearance), plus a baseline model (Fit-VTO) trained on it.

What This Paper Is About

Existing virtual try-on systems are good at pasting a garment's texture onto a person, but they ignore a basic question shoppers ask: "How will this actually fit me?" A size 3XL shirt on an XS person should look oversized and draping; current models render it as perfectly tailored anyway. The core obstacle is data: no large-scale dataset pairs real try-on images with precise numeric measurements for both the person and the garment, because retail catalog photos almost never contain ill-fitting examples. FIT solves this by generating the training data synthetically through 3D physics simulation rather than scraping the web.

Key Contributions

1. The FIT dataset — the first large-scale benchmark explicitly built for fit-aware try-on: 1,137,282 training and 1,000 test triplets, each with a target try-on image, a paired person reference image (same person and pose, different garment), a layflat garment image, and precise person/garment measurements in centimeters. It spans 168 body shapes (XS–3XL), 528 poses, and 158,483 unique garment designs.

2. A scalable synthetic data generation pipeline — garments are procedurally drafted with GarmentCode (giving exact ground-truth measurements) and draped onto differently-sized 3D bodies via physics simulation, then transformed into photorealistic images by a geometry-preserving re-texturing model.

3. A solution to the paired-data bottleneck — because the pipeline simulates the same 3D subject and pose wearing two different garments, it produces genuine ground-truth pairs, enabling supervised (rather than self-supervised reconstruction or hallucinated pseudo-triplet) VTO training.

4. Fit-VTO, a baseline fit-aware try-on model that replaces Flux.1-dev's text conditioning with a custom numeric measurement encoder, achieving state-of-the-art fit accuracy as measured by a new size-aware IoU metric.

Main Findings

• Current VTO methods fail at fit. Any2AnyTryon, Nano Banana Pro, COTTON, and IDM-VTON all produce plausible textures but render garments as well-fitted regardless of the actual size relationship between person and garment.

• FIT is the only dataset meeting all four criteria. Compared against SV-VTO, SIZER, VITON-HD, Size4Men, LAION-Garment, SewFactory, and GCD, FIT is the only one that is simultaneously realistic, includes ill-fit cases, provides measurements, and supplies true triplets — at roughly 19x the scale of the next-largest dataset with measurements (SewFactory, 1M) and over 1,000x the scale of SIZER.

• Synthetic paired-person generation beats all baseline strategies. For identity preservation in non-garment regions, the proposed method achieves a Masked L1 distance of 1.61, versus 2.29 (VTO-based), 3.91 (inpainting-based), and 4.45 (VLM-based). VLM approaches notably drift in limb pose and chest volume.

• Fit-VTO leads on the FIT benchmark. It reaches SSIM 0.914, FID 10.381, LPIPS 0.050, KID 0.144, and size-aware IoU 0.955 — the best in every column, including beating the size-conditioned COTTON on IoU.

• Trade-off between fit accuracy and general realism. On real-world VITON-HD, IDM-VTON edges ahead (SSIM 0.849 vs. 0.817) partly because it trains directly on that dataset; fine-tuning Fit-VTO on VITON-HD closes the gap (SSIM 0.833, FID 9.320).

• Both data sources are necessary. Ablations show that training without FIT data destroys size accuracy (IoU 0.844), training on FIT only overfits and degrades badly on real images (VITON-HD SSIM 0.732), and swapping the measurement encoder for the original T5/CLIP text encoders yields decent real-world scores but poor IoU (0.932 vs. 0.955). The full combination is the only configuration that performs well on both.

• Size control is independently steerable, up to a point. Holding the person fixed while varying garment size produces realistic tight-to-loose variation, but the model struggles to distinguish "tight" from "very tight" (both fit to skin in simulation) and is sensitive to measurement correlations — increasing width tends to slightly increase length and sleeve length too.

• Fabric and detail realism come from prompting, not geometry. A library of 72 fabric types (leather, cotton, silk, etc.) is injected into text prompts, and the re-texturing model inpaints details like pockets, buttons, and seams that the coarse 3D mesh lacks.

Methodology in Plain English

The authors sidestep the data problem entirely by manufacturing the data. For each sample, they:

1. Generate a garment as a 2D sewing pattern using GarmentCode, a parametric programming language for clothing design, which gives exact centimeter measurements for length, bust, sleeve length, waist, and out-seam.

2. Cross-drape it onto the wrong-sized body. A pattern drafted for one body size is simulated (via physics) onto a body of a different size, producing realistic tight and loose fits with wrinkles and stretch. Two technical fixes were needed: realigning the initial box-mesh panels to the new body to prevent simulation failures, and draping top and bottom garments in separate steps so shirts can appear "tucked out."

3. Convert the 3D render into a photograph. Rather than training on cartoonish renders, they fine-tune Flux.1-dev to regenerate realistic textures from surface normal maps — a representation that captures shape and lighting but no color, so geometry is strictly preserved. Text prompts describe the person and garment. Faces, hair, and shoes are separately inpainted with Nano Banana Pro and stitched into the normal map, since the 3D meshes are bald and barefoot.

4. Create a paired reference image. The same synthetic body in the same pose is draped with a different garment. An "identity map" — the original try-on image with both garment regions masked out — conditions a second generation step, forcing the model to preserve the person's face, skin, and background while dressing them differently.

5. Generate the layflat garment image by running Nano Banana Pro as a "virtual try-off" model on the result.

The resulting triplets train Fit-VTO, which is Flux.1-dev with LoRA adapters plus a custom measurement encoder. Numbers (seven measurements, Fourier-encoded into a 7×16 tensor, projected to the model's 3072-dimensional hidden space) are fed in through cross-attention in place of the usual T5 text embeddings. Real-world fashion images (330,559 of them, with measurements set to null) are mixed in during training to prevent overfitting to synthetic aesthetics.

Why This Matters

Research impact: This is a proof of concept that physics-grounded synthetic supervision can teach generative models to respect precise physical quantities. It attacks a structural limitation of 2D pseudo-triplet generation — that training on a teacher model's hallucinated outputs propagates the teacher's artifacts — by producing geometrically exact ground truth instead. It also supplies the first benchmark with a size-aware metric (masked-garment IoU), giving the field something concrete to measure progress against.

Real-world applications:

• E-commerce sizing confidence — showing shoppers how a specific size will actually look on their body, reducing returns driven by fit mismatch.
• Size recommendation systems — a visual complement to numeric size charts, letting users preview a range of sizes before buying.
• Fashion design prototyping — visualizing how a pattern drafted at one size drapes across the full size run without physically manufacturing each variant.
• Virtual styling and social commerce — enabling "oversized" versus "fitted" looks to be previewed faithfully, which matters for trend-driven retail.

Industry relevance: Apparel return rates driven by poor fit are a major cost center for online retail, and the demand for VTO has grown alongside online shopping. A dataset and model that quantify fit — rather than merely simulating appearance — addresses the exact failure mode that limits commercial VTO adoption.

Future Directions

• Extend beyond upper-body tops to pants, dresses, and full-body garments, and eventually to multi-layered or structurally complex apparel that GarmentCode cannot currently express.
• Improve tightness fidelity — the simulation renders "tight" and "very tight" nearly identically, so neither the dataset nor the model captures degrees of snugness well.
• Decouple correlated measurements so that changing garment width does not inadvertently change length or sleeve length, enabling finer-grained user control.
• Scale pose and camera diversity beyond the standardized front-facing views and casual poses used here, to handle in-the-wild inputs.
• Question worth pursuing: how far synthetic-to-real transfer can be pushed before the bounded structural diversity of the garment engine becomes the limiting factor.

Target Audience

Researchers and engineers working on virtual try-on, sim2real data generation, or conditional diffusion models; practitioners in fashion e-commerce and sizing technology evaluating VTO architectures; and graduate students looking for a case study in how synthetic data pipelines can be designed to supply supervision that the real world simply does not provide. Readers should be comfortable with diffusion model conditioning mechanisms and standard generative image metrics to get full value from the experiments section.

Authors’ abstract

Given a person and a garment image, virtual try-on (VTO) aims to synthesize a realistic image of the person wearing the garment, while preserving their original pose and identity. Although recent VTO methods excel at visualizing garment appearance, they largely overlook a crucial aspect of the try-on experience: the accuracy of garment fit -- for example, depicting how an extra-large shirt looks on an extra-small person. A key obstacle is the absence of datasets that provide precise garment and body size information, particularly for "ill-fit" cases, where garments are significantly too large or too small. Consequently, current VTO methods default to generating well-fitted results regardless of the garment or person size. In this paper, we take the first steps towards solving this open problem. We introduce FIT (Fit-Inclusive Try-on), a large-scale VTO dataset comprising over 1.13M try-on image triplets accompanied by precise body and garment measurements. We overcome the challenges of data collection via a scalable synthetic strategy: (1) We programmatically generate 3D garments using GarmentCode and drape them via physics simulation to capture realistic garment fit. (2) We employ a novel re-texturing framework to transform synthetic renderings into photorealistic images while strictly preserving geometry. (3) We introduce person identity preservation into our re-texturing model to generate paired person images (same person, different garments) for supervised training. Finally, we leverage our FIT dataset to train a baseline fit-aware virtual try-on model. Our data and results set the new state-of-the-art for fit-aware virtual try-on, as well as offer a robust benchmark for future research. We will make all data and code publicly available on our project page: https://johannakarras.github.io/FIT.