RefTon: Reference person shot assist virtual Try-on

AI summary

Overview

Research area: Computer vision, specifically image-based virtual try-on (generative image editing with diffusion/flow-matching models).

Technical level: Advanced. The paper builds on the Flux-Kontext diffusion transformer, flow matching, Rotary Position Embeddings (RoPE), and LoRA fine-tuning, and assumes familiarity with latent diffusion and DiT architectures.

Scope: RefTon is a person-to-person virtual try-on framework that generates a photo of a target person wearing a target garment directly from a source image and garment image, with optional extra "reference" images of the same garment worn by a different person, and without requiring masks, pose estimates, or segmentation models at inference.

What This Paper Is About

Virtual try-on systems today typically require a stack of auxiliary inputs — parsing masks, DensePose, warped cloth masks, or text prompts — and each conditional input has to be produced by a separate external model, whose quality directly affects the final result. A second problem is that a flat garment image often cannot convey what the garment actually looks like when worn: the paper's motivating example is a garment that could be either green translucent fabric or light green opaque fabric, which only becomes clear from an image of a person wearing it. RefTon addresses both problems by training a single Flux-Kontext-based model that needs only a source person image and the target garment, and that can additionally accept reference images of the garment worn by other people to sharpen texture and detail fidelity.

Key Contributions

1. Reference images in the try-on pipeline. The authors incorporate additional reference images (the target garment worn on a different individual) into virtual try-on. They report this delivers state-of-the-art performance in preserving fine garment design details.
2. A reference data generation framework. Using Qwen2.5-VL and Flux-Kontext, they synthesize reference images for existing benchmarks, building the VFR dataset on top of VITON-HD, DressCode, ViViD (and also drawing on FashionTryOn and IGPairs), as a new training and evaluation resource.
3. An adapted Flux-Kontext model. They modify Flux-Kontext's Rescaled Position Indexing so the first position-index channel carries discrete condition labels instead of a binary flag (the other two channels store rescaled spatial coordinates), supporting flexible multi-condition and multi-resolution inputs in one DiT. Combined with a two-stage training strategy, one model supports both mask-based and mask-free, person-to-person try-on, and varying numbers and types of reference images.
4. Strong generalization. They report state-of-the-art results across mask-based and mask-free settings, plus cross-dataset and in-the-wild transfer.

Main Findings

• Reference images help consistently. Adding references ("+R") improves all four reported metrics on both benchmarks. On VITON-HD, RefTon goes from LPIPS 0.057 / SSIM 0.873 / FID_p 5.45 / KID_p 0.82 to LPIPS 0.049 / SSIM 0.879 / FID_p 4.69 / KID_p 0.68; unpaired FID_u moves from 8.58 to 8.43 and KID_u from 1.06 to 0.91. On DressCode, RefTon goes from LPIPS 0.037 / SSIM 0.912 / FID_p 3.48 / KID_p 1.20 to LPIPS 0.031 / SSIM 0.918 / FID_p 2.94 / KID_p 0.95, with unpaired FID_u 5.31 to 5.07 and KID_u 1.36 to 1.15.
• Mask-free performance is close to mask-based. RefTon/R+M/F on VITON-HD reports LPIPS 0.053, SSIM 0.872, FID_p 5.11, KID_p 0.82, FID_u 8.32, KID_u 0.78; on DressCode, LPIPS 0.035, SSIM 0.906, FID_p 3.34, KID_p 1.15, FID_u 5.02, KID_u 1.28. The authors describe the drop from mask-based to mask-free as "mild metric fluctuations."
• Outperforms baselines on public benchmarks. In Table 1, RefTon and RefTon+R are reported against CAT-DM, IDM-VTON, OOTDiffusion, CatVTON, CatVT2ON, OmniVTON, PromptDresser, Any2AnyTryon, TryOffDiff, and mask-free CatVTON. The paper states its method performs consistently better than prior baselines in try-on fidelity and pose alignment.
• Strong generalization to StreetTryOn without domain training. RefTON reaches FID 28.991 (shop-to-street), 8.870 (person-to-person), 25.429 (person-to-street), and 16.452 (street-to-street), versus OmniVTON's 33.919 / 8.983 / 33.450 / 23.470 and the StreetTryOn baseline's 34.054 / 12.185 / 34.191 / 33.039. The model was neither trained on the StreetTryOn domain nor designed for the absence of garment images during training.
• Mixed-dataset training generalizes across benchmarks. Training on the mixed set ("Mixed-Virtual-Ref") with 103,936 image pairs, RefTon beats OOTDiffusion on most metrics despite not being trained on DressCode or VITON-HD individually. Paired VITON-HD: RefTon 0.851 SSIM / 6.23 FID_p / 0.80 KID_p / 0.072 LPIPS versus OOTDiffusion 0.839 / 11.22 / 2.72 / 0.123; with references, 0.859 / 5.13 / 0.62 / 0.060. Paired DressCode: RefTon 0.896 / 3.70 / 1.13 / 0.045 versus OOTDiffusion 0.915 / 11.96 / 1.21 / 0.061; with references 0.903 / 3.14 / 0.97 / 0.038.
• Rescaled position index outperforms the original. In the ablation at 0.5× and 0.25× condition scales, the rescaled position index beats the original Flux-Kontext index on FID and KID in both masked and mask-free settings. At 0.5× scale: FID 5.09 vs 5.29, KID 0.79 vs 0.84, FID_MF 4.71 vs 4.75, KID_MF 0.69 vs 0.72. At 0.25× scale: FID 6.01 vs 6.08, KID 0.77 vs 0.93, KID_MF 0.71 vs 0.76 (FID_MF is 5.37 with the rescaled index versus 5.35 without).
• Qualitative advantage on difficult materials. The authors report better rendering of hollow and semi-transparent fabrics, lace perforation and transparency, printed letters and logos, and reflective materials such as leather and metallic fabrics, where baselines produce solid textures, spurious dotted artifacts, over-smoothing, or distortion.
• Mask quality affects mask-dependent models. The ablation shows overly aggressive masks remove items such as handbags and conservative masks retain unwanted regions such as legs, leading to incorrect garment geometry; the mask-free RefTon is reported to be stable across these conditions.

Methodology in Plain English

The team starts from Flux-Kontext, a strong image-editing transformer, and adapts it rather than building a pipeline from scratch.

Training happens in two stages. In stage 1, they train a mask-based try-on model that takes an agnostic person image, clothing image, DensePose map, and warp mask and produces a person wearing a selected garment. At inference, they feed it a random unpaired garment from the same clothing category (a dress for a dress, not a shirt) to synthesize a person image wearing something they are not actually wearing. This creates the unpaired data that public benchmarks do not provide.

In stage 2, they train the person-to-person model on triplets of garment, unpaired person image, and target garment, so the model learns to dress a person directly. During training the model sees either the original agnostic image or the stage-1 synthesized person image, sampled with equal 50% probability. A reference image of the garment worn by a different person is supplied with 25% probability. The encoder and decoder of Flux-Kontext are frozen; only the transformer blocks are fine-tuned with LoRA (rank 64, alpha = 128, AdamW).

To handle multiple different condition images at once, they change how positions are encoded. Flux-Kontext generates a three-channel position index whose first channel is a binary flag; RefTon turns that channel into discrete labels identifying which condition an image is (person, garment, reference, and so on), while the second and third channels hold spatial coordinates rescaled by the resolution ratio between the target image and each condition image. This keeps spatial alignment when conditions arrive at different resolutions.

For references, they build a generation pipeline rather than collecting data. Qwen2.5-VL describes the person's appearance in the target image and produces an opposite description. The opposite description plus action and non-target garment descriptions becomes the positive prompt; the original appearance description becomes the negative prompt. Flux-Kontext then edits the target image into a reference showing a visibly different person wearing the same garment, with different non-target clothing and poses. The design has three stated requirements: preserve the target garment faithfully, make the new person clearly different (to prevent the model from taking a shortcut and copying the reference), and vary the non-target garments.

Data quality is handled by comparing CLIP features to drop duplicated samples and using Qwen2.5-VL to flag distorted, unclear, or back-facing images.

Evaluation covers paired settings (SSIM, LPIPS, FID, KID against ground truth) and unpaired settings (FID, KID for distributional similarity). All generated images were resized to 512 × 384 for the main quantitative comparison. Single-dataset models were trained for 20,000 steps on VITON-HD and 48,000 steps on DressCode with batch size 128 on 8 NVIDIA H100 GPUs.

Why This Matters

Impact on research. Most virtual try-on work has grown more complex over time, adding warping modules, parsing networks, and condition-specific branches. RefTon argues in the opposite direction: one adapted editing backbone, no external condition extractors at inference, and an optional new input modality (reference images) that prior benchmarks cannot support. The paper also contributes the VFR data-generation pipeline as a resource, which shifts part of the problem from model architecture to data construction.

Real-world applications:

• Online fashion retail, where a shopper wants to see a garment on their own photo rather than on a stock model.
• Product pages that show garments on multiple models, matching the behavior the paper describes of users preferring model shots over flat garment images.
• Preserving translucency, lace, prints, logos, and leather sheen in e-commerce imagery, where the paper reports its clearest qualitative advantage.
• In-the-wild or cross-domain try-on, where StreetTryOn results and mask-free operation matter because clean masks and pose annotations are not available.

Industry relevance. The framework is designed to work without pose estimators, human parsing models, or segmentation models, and without structural guidance, which removes a chain of external dependencies that each add failure modes and latency in production. The reported single-model support for both mask-based and mask-free try-on also simplifies deployment. Code is released at https://github.com/360CVGroup/RefTon, and the work is supported by 360 AI Research.

Future Directions

• Cost of reference generation. References are synthesized with Flux-Kontext rather than collected; the paper does not report how well synthetic references match real model shots, and it does not report inference latency or generation cost for either training or deployment.
• Reference selection. The model accepts varying numbers and types of reference images, but the paper does not report how many references are optimal or how to pick good ones, nor does it study failure modes when a reference is a poor match.
• Shortcut avoidance. The authors explicitly design reference diversity to stop the model from copying the reference directly. How much diversity is enough, and whether stronger diversity helps further, is left open.
• Scaling and dataset naming. The mixed training set is described as both VFR and Mixed-Virtual-Ref across different sections, and the exact per-source composition of the 103,936 image pairs is not broken down in the main text (details are deferred to Appendix A, which is not included in the provided content).

Target Audience

Researchers and engineers working on generative image editing, diffusion transformers, and virtual try-on; applied ML teams building fashion e-commerce or avatar/personalization features; and readers already comfortable with latent diffusion, flow matching, LoRA, and DiT position encoding who want to see how an editing backbone can be repurposed for multi-condition, multi-resolution try-on with a new reference-image modality.

Authors’ abstract

We introduce RefTon, a flux-based person-to-person virtual try-on framework that enhances garment realism through unpaired visual references. Unlike conventional approaches that rely on complex auxiliary inputs such as body parsing and warped mask or require finely designed extract branches to process various input conditions, RefTon streamlines the process by directly generating try-on results from a source image and a target garment, without the need for structural guidance or auxiliary components to handle diverse inputs. Moreover, inspired by human clothing selection behavior, RefTon leverages additional reference images (the target garment worn on different individuals) to provide powerful guidance for refining texture alignment and maintaining the garment details. To enable this capability, we built a dataset containing unpaired reference images for training. Extensive experiments on public benchmarks demonstrate that RefTon achieves competitive or superior performance compared to state-of-the-art methods, while maintaining a simple and efficient person-to-person design.