UniFit: Towards Universal Virtual Try-on with MLLM-Guided Semantic Alignment

AI summary

Overview

Research area: Computer Vision — image-based virtual try-on (VTON), multimodal generative modeling, diffusion/flow-matching transformers.

Technical level: Advanced. The work assumes familiarity with diffusion models, Diffusion Transformers (DiT), flow matching, cross-attention mechanisms, and Multimodal Large Language Models (MLLMs).

Scope: The paper presents UniFit, a single instruction-guided framework that unifies six different virtual try-on tasks under one model by aligning text instructions with reference images through a multimodal LLM and a progressive self-synthesis training pipeline.

What This Paper Is About

Existing virtual try-on systems are mostly built for one narrow task, such as placing a single shirt on a person, and the few multi-task frameworks that exist rely on abstract text prompts that do not ground well in concrete visual details like fabric texture or logo shape. This creates a semantic gap between what the text says and what the image shows, and public datasets simply do not contain enough examples of harder tasks like dressing someone in a full outfit or transferring clothes between two models. UniFit attacks both problems at once, aiming to build one universal try-on model that can follow natural-language instructions across a wide range of try-on scenarios.

Key Contributions

1. A universal VTON framework (UniFit) that handles six tasks in one model: single-garment try-on, model-free try-on, garment reconstruction, multi-garment try-on, model-to-model try-on, and multi-view try-on — a superset no prior method covers.

2. An MLLM-Guided Semantic Alignment Module (MGSA) that feeds instructions and reference images jointly into a multimodal LLM (Qwen2-VL-2B) and uses learnable query tokens to distill the long multimodal sequence into a compact, task-aware semantic representation. A cosine-similarity semantic alignment loss ties these queries to features of the ground-truth target image.

3. A spatial attention focusing loss that supervises the DiT's cross-attention maps against ground-truth spatial masks, pushing the model to concentrate on task-relevant regions (e.g., the region where the garment should appear) instead of spreading attention thinly across the image.

4. A two-stage progressive training strategy with a self-synthesis pipeline, in which a base model trained on foundational tasks generates its own pseudo-paired training data for advanced tasks, filtered for quality, allowing UniFit to learn multi-garment and model-to-model try-on despite scarce real data.

Main Findings

• Best-in-class garment reconstruction: On VITON-HD, UniFit achieves an FID of 12.58, LPIPS of 0.281, and DISTS of 0.202, beating TryOffDiff and Any2AnyTryon on perceptual and distributional metrics, though Any2AnyTryon retains a slight edge in SSIM.

• Strong single-garment try-on: UniFit reaches FID 8.799 and KID 0.702 on VITON-HD, outperforming CatVTON, FitDiT, and Any2AnyTryon on those generation-quality metrics, with a competitive LPIPS of 0.065.

• Superior garment consistency in model-free try-on: CLIP-I of 0.914 and MP-LPIPS of 0.078 surpass IMAGDressing-v1 and Any2AnyTryon, indicating the synthesized person wears the input garment far more faithfully.

• Competitive multi-view performance: On the MVG dataset UniFit improves SSIM (0.935) and FID (35.62) over MV-TON, while MV-TON is slightly better on LPIPS and KID — a mixed but favorable result.

• Competitive multi-garment and model-to-model results: Against closed-source tools OutfitAnyone and Kolors-VTON, UniFit better preserves garment shape, texture, and silhouette in multi-garment combinations, and handles model-to-model transfer without any explicit garment masks.

• Every component matters: Ablations on the base model show performance drops when MGSA is replaced by a T5 text encoder (SSIM 0.851 vs. 0.887), when the alignment loss is removed (0.863), and when the focusing loss is removed (0.872), confirming both losses contribute.

• Efficiency gains: UniFit totals about 4.8B parameters and runs single-garment try-on in ~14s using 16GB VRAM on an A800, versus Any2AnyTryon's 17B parameters, 36GB, and 56s for a similarly broad task range.

Methodology in Plain English

UniFit combines two pretrained systems: a multimodal language model (Qwen2-VL) that "reads" both the text instruction and the reference images together, and a diffusion transformer (based on Stable Diffusion 3.5 Medium) that actually paints the output image.

The language model is not used to generate text. Instead, the researchers attach a fixed set of 486 learnable "query" tokens to the end of its input. As the language model processes everything, these queries act like sponges, soaking up the task-relevant information from the instruction and images into a compact summary. A separate loss then forces each query token to resemble a corresponding patch of the real target image, so the summary is not just abstract — it is visually grounded in what the output should look like. This compact summary, plus low-level visual features from a VAE encoder, plus a noisy version of the target latent, all get concatenated and fed into the diffusion transformer, which iteratively denoises them into the final image.

Two extra mechanisms sharpen the results. A pose block copies the first transformer block and injects pose information so the person's posture is respected. The attention-focusing loss watches the cross-attention maps — essentially "where the model is looking" — and penalizes them whenever attention drifts away from the region it should care about, using segmentation masks as ground truth. For garment reconstruction it supervises a reference-centric map (where to extract details); for try-on it supervises an output-centric map (where to paste them).

The training happens in two rounds. Round one teaches the basics on public datasets (VITON-HD, DressCode). Round two uses that trained model as its own data factory: it reconstructs individual garments from full-body photos to create multi-garment training pairs, and generates new models wearing given garments to create model-to-model pairs. Every synthetic sample passes a two-stage filter — a perceptual similarity check (DreamSim) followed by a semantic consistency check by Qwen2.5-VL — before being added to the joint fine-tuning set. Training uses a combined objective of flow-matching loss, semantic alignment loss, and the attention-focusing loss.

Why This Matters

Research impact: The paper shows that a multimodal LLM can serve as a semantic bridge between language instructions and visual references in a generative pipeline, an idea that generalizes beyond try-on to any instruction-conditioned image generation task. The self-synthesis loop also offers a template for bootstrapping hard tasks when labeled data is essentially nonexistent.

Real-world applications:

• E-commerce product pages where shoppers upload a photo and preview any outfit in combinations of tops and bottoms without needing separate garment cutouts.
• Virtual fashion catalogs and lookbooks, where a single garment photo can generate try-on images across many different models and poses.
• Fashion design and merchandising tools that let stylists type an instruction and instantly visualize a garment transferred onto a target model or avatar.
• Content creation for social media and games, where users can restyle photos of themselves by following a short text prompt rather than manual editing.

Industry relevance: By running at roughly 4.8B parameters and 16GB VRAM, UniFit is deployable on far more modest hardware than 17B-parameter competitors, which matters for cloud cost and for whether such systems can be brought closer to real-time retail experiences. The mask-free, instruction-driven design also reduces the preprocessing burden that typically hampers production pipelines.

Future Directions

• In-the-wild robustness: The authors note performance may degrade under extreme lighting or severe occlusion because training data is mostly in-shop studio imagery; extending to casual, uncontrolled photos is a clear next step.

• Layered and text-editable try-on: UniFit cannot currently output layered garments or allow fine-grained text edits to specific clothing attributes after generation — both are natural extensions.

• Scaling task coverage and data synthesis quality: The self-synthesis pipeline is filtered by fixed thresholds and a single judge model; improving the generation and verification of pseudo-labels could unlock even harder tasks like full-body layering or accessory transfer.

• Backbone generalization: The appendix reports experiments swapping the DiT backbone for Flux.1 Fill, hinting at a broader question of how much of the MGSA approach transfers to other generative architectures.

Target Audience

This paper is best suited for computer vision and generative AI researchers working on virtual try-on, instruction-conditioned image synthesis, or diffusion transformer architectures. It is also valuable for applied engineers in e-commerce and fashion tech evaluating whether a universal, instruction-driven try-on model is practical to deploy, and for graduate students looking for a well-structured example of combining MLLMs with generative backbones under data-scarce conditions. Readers without a background in diffusion models or attention mechanisms will find the technical sections demanding.

Authors’ abstract

Image-based virtual try-on (VTON) aims to synthesize photorealistic images of a person wearing specified garments. Despite significant progress, building a universal VTON framework that can flexibly handle diverse and complex tasks remains a major challenge. Recent methods explore multi-task VTON frameworks guided by textual instructions, yet they still face two key limitations: (1) semantic gap between text instructions and reference images, and (2) data scarcity in complex scenarios. To address these challenges, we propose UniFit, a universal VTON framework driven by a Multimodal Large Language Model (MLLM). Specifically, we introduce an MLLM-Guided Semantic Alignment Module (MGSA), which integrates multimodal inputs using an MLLM and a set of learnable queries. By imposing a semantic alignment loss, MGSA captures cross-modal semantic relationships and provides coherent and explicit semantic guidance for the generative process, thereby reducing the semantic gap. Moreover, by devising a two-stage progressive training strategy with a self-synthesis pipeline, UniFit is able to learn complex tasks from limited data. Extensive experiments show that UniFit not only supports a wide range of VTON tasks, including multi-garment and model-to-model try-on, but also achieves state-of-the-art performance. The source code and pretrained models are available at https://github.com/zwplus/UniFit.