OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

AI summary

Overview

Research area: Computer vision — generative virtual try-on (VTON) and model evaluation methodology.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, self-supervised vision encoders (DINOv3), vision-language models, and segmentation foundation models (SAM3), plus standard generative-image metrics (FID, LPIPS, SSIM).

Scope in one sentence: The paper releases a ~100K-pair, up-to-1536×1536 open benchmark for virtual try-on and a hybrid VLM-plus-representation evaluation protocol that tracks human preference far more closely than conventional pixel metrics.

What This Paper Is About

Virtual try-on systems can now generate convincing images of a person wearing a new garment, but the field has no reliable way to measure how good those images actually are. Existing datasets are small, low-resolution, and shot in clean studio conditions, while the standard metrics (FID, SSIM, LPIPS) measure global statistics or low-level pixel patches and are blind to the things that matter commercially — a distorted logo, a wrong fabric weave, a changed body shape. The paper addresses both halves of this problem: a large, diverse, high-resolution dataset, and an evaluation protocol whose scores agree with what human raters actually prefer.

Key Contributions

1. OpenVTON-Bench dataset. Approximately 100,000 (exactly 99,925) high-resolution image pairs ranging from 1024×1024 up to 1536×1536, drawn from e-commerce and social-media sources, filtered for quality, and balanced across 20 fine-grained garment categories. Each sample is a triplet (reference garment, masked person, ground-truth person) with dense semantic captions totaling over 3 million words.

2. Hybrid multi-modal evaluation protocol. A two-track system that pairs a VLM-as-a-Judge module (Qwen-VL-Plus scoring five semantic dimensions) with a Multi-Scale Representation Metric built on DINOv3 features and SAM3 segmentation, so that boundary-alignment errors can be separated from internal texture artifacts.

3. Five interpretable quality axes. VTON quality is decomposed into background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism, replacing single opaque scalar scores with a diagnostic profile.

4. Diagnostic benchmarking of nine state-of-the-art systems. The paper benchmarks both open-source diffusion models and commercial-grade systems, revealing a consistent "texture-realism gap" and demonstrating that the proposed metric ranks models in agreement with human judgment (Kendall's τ = 0.833) far better than SSIM (0.611).

Main Findings

• The proposed metric is the best predictor of human preference. The aggregated Representation score achieves the highest Kendall's τ (0.833) and Spearman ρ (0.933) against human ratings, compared to 0.611 for both PSNR and SSIM, 0.667 for LPIPS, and 0.722 for FID and the VLM score alone. High Kendall's τ matters because it means the metric correctly predicts which of two models a human would prefer when compared head-to-head.

• A "texture-realism gap" separates photorealism from garment fidelity. General-purpose diffusion models such as FLUX.1-Kontext-dev score well on background (4.428) and overall realism (4.137) but drop sharply on texture (3.574), indicating that large-scale pre-training produces attractive images without preserving specific garment patterns zero-shot.

• Commercial systems are more balanced. YingHui achieved the top average scores in both VLM (4.372) and human (4.608) evaluation, and retained high fidelity even under the deepest mask erosion (S_rep(3) = 0.823), suggesting it learned genuine internal texture representations rather than relying on edge completion.

• Erosion reveals which models depend on boundaries. As masks erode inward, all methods lose similarity, but OOTD decays fastest (0.797 → 0.669), while YingHui, Qwen-Editor, and Nanobanana hold up. This isolates the models that merely align contours from those that reproduce fabric.

• Pixel metrics can be actively misleading. Qwen-Editor wins on PSNR (26.343) and SSIM (0.905) but tends to smooth textures to minimize pixel error, whereas YingHui has the best FID (7.372), indicating better distribution-level realism.

• Generation backbones matter enormously. FLUX.2-dev dramatically outperforms FLUX.1-Kontext-dev at every erosion level (mean representation similarity 0.841 vs. 0.754), showing that a stronger generative backbone is central to preserving high-frequency detail.

• Global scores hide local failures. Nanobanana and Qwen-Editor reach global consistency of 0.936, matching YingHui, yet fall behind on local garment similarity — a model can produce a pleasing whole image while failing to preserve the merchandise.

• VLM judges track human ratings closely. Across most dimensions the VLM scores mirror human scores, supporting their use as a scalable substitute for costly manual annotation.

Methodology in Plain English

Building the dataset. The team gathered over 3 million raw image pairs from two sources: a refined subset of the IMAGDressing-v1 social-media dataset and large-scale e-commerce crawls. They kept only images where both dimensions were at least 1024 pixels and the longer side no more than 1536 pixels, leaving roughly 300,000 candidates. Human annotators (over 1,000 of them) verified that each standalone garment actually matched the garment in the person image, discarding mismatches and severe occlusions. Faces were generatively anonymized for privacy, and the release is research-only with a takedown mechanism.

Balancing the categories. Fashion data naturally skews toward simple items like plain white t-shirts. To fix this, the team ran every garment image through DINOv3, a self-supervised vision encoder chosen over CLIP because it captures object-level structure rather than just text alignment, then performed hierarchical clustering into 20 fine-grained garment classes and sampled evenly from each. The result is 99,925 balanced samples — the "100K" dataset.

Generating annotations and triplets. Gemini 2.0 Flash produced dense captions using a two-tier prompt: first classify the garment as upper- or lower-body, then extract structure, texture, and design details appropriate to that category. GroundingDINO and SAM3 located garment regions, and a black occlusion layer was applied to create the masked person image, forming the (garment, masked person, ground truth) triplets.

Evaluating results. Each generated try-on is scored three ways. First, a VLM receives the garment, the ground truth, and the generated image along with a task prompt, and outputs five scores from 1 to 5 covering the five quality axes. Second, DINOv3 embeddings compare the generated and ground-truth images globally (cosine similarity of whole-image features) and locally: SAM3 segments the garment, and a series of progressively eroded masks narrows the comparison from the garment outline inward to the fabric interior, producing a curve of similarity scores that separates boundary errors from texture errors. Third, conventional PSNR, SSIM, LPIPS, and FID are reported for compatibility.

Validating the metrics. Seventy-six participants produced over 90,000 valid ratings on five-point Likert scales along the same five dimensions, with each image group reviewed at least twice and averaged. Correlations between these human scores and each objective metric establish which metric actually reflects human preference.

Why This Matters

Impact on research. The paper reframes virtual try-on evaluation from a single scalar to an interpretable, multi-axis diagnosis. By releasing a large, high-resolution, category-balanced dataset with paired supervision and in-the-wild diversity, it removes the "studio-centric bias" that let models look good on VITON-HD and DressCode while failing on real customer photos. The Kendall's τ result gives the community a metric that can be trusted for model selection.

Real-world applications:

• E-commerce and fashion retail: reliable pre-deployment screening of try-on features, where a distorted logo or shifted pattern is a brand-damaging defect, not a rounding error.
• Shopping apps and virtual fitting rooms: higher-fidelity previews reduce returns driven by "not what I expected" mismatches.
• Model selection and procurement: businesses comparing vendors or open-source models need a benchmark whose rankings match what their customers perceive.
• Advertising and content creation: accurate garment reproduction matters for campaign imagery that must match catalog photography.

Industry relevance. The benchmark is explicitly designed for "commercial-grade" assessment, and the results expose a real divide: open-source diffusion models produce attractive images but hallucinate fine garment details, while commercial systems with proprietary try-on data perform more evenly. That finding is actionable for anyone deciding whether to build or buy, and it identifies texture preservation as the specific capability where open models still lag.

Future Directions

• Reducing inherited bias from foundation models. The pipeline depends on Gemini and DINOv3 for captioning and filtering, so any semantic biases or hallucinations in those upstream models propagate into the benchmark. Refining or diversifying the automated annotation stage is an open task.

• Expanding topological and occlusion diversity. Complex multi-layer outfits and extreme, acrobatic poses remain underrepresented compared to standard studio-like poses, leaving a coverage gap the authors flag for future iterations.

• Testing whether training on the benchmark closes the texture gap. The paper includes fine-tuning results in its appendix, but whether open models can match commercial systems on deep-erosion texture fidelity after training on OpenVTON-Bench is a natural follow-up question.

• Generalizing the hybrid protocol beyond try-on. The combination of VLM semantic judging and multi-scale masked representation scoring could transfer to other garment- or object-centric generation tasks where boundary accuracy and internal detail are confounded.

Target Audience

Researchers and engineers working on generative image synthesis, particularly virtual try-on, image editing, and person-centric generation. It is also aimed at practitioners who must evaluate or procure VTON systems for retail and e-commerce, and at benchmark builders interested in hybrid evaluation protocols that combine vision-language models with self-supervised representation metrics. Readers should be comfortable with diffusion model terminology, similarity metrics, and standard evaluation statistics to get the most from the experimental tables.

Authors’ abstract

Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck. Traditional metrics struggle to quantify fine-grained texture details and semantic consistency, while existing datasets fail to meet commercial standards in scale and diversity. We present OpenVTON-Bench, a large-scale benchmark comprising approximately 100K high-resolution image pairs (up to $1536 \times 1536$). The dataset is constructed using DINOv3-based hierarchical clustering for semantically balanced sampling and Gemini-powered dense captioning, ensuring a uniform distribution across 20 fine-grained garment categories. To support reliable evaluation, we propose a multi-modal protocol that measures VTON quality along five interpretable dimensions: background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism. The protocol integrates VLM-based semantic reasoning with a novel Multi-Scale Representation Metric based on SAM3 segmentation and morphological erosion, enabling the separation of boundary alignment errors from internal texture artifacts. Experimental results show strong agreement with human judgments (Kendall's $τ$ of 0.833 vs. 0.611 for SSIM), establishing a robust benchmark for VTON evaluation.