OmniFabric: Coherent UV Space Texture Synthesis for 3D Garment Reconstruction

AI summary

Overview

Research area: Computer vision and computer graphics, specifically 3D garment reconstruction and neural texture synthesis, with connections to digital fashion production pipelines.

Technical level: Intermediate to Advanced. The paper assumes familiarity with diffusion models, Diffusion Transformers (DiT), UV mapping, LoRA fine-tuning, and garment sewing-pattern representations, though the two-stage pipeline structure is described coherently enough for a motivated non-specialist to follow.

Scope: OmniFabric is a two-stage framework that converts a single in-the-wild clothing photograph into a simulation-ready garment sewing pattern carrying a globally coherent, artifact-free texture map.

What This Paper Is About

Reconstructing production-ready 3D garments from a single image is hard not because of geometry alone, but because textures synthesized by existing 3D texturing methods bake environmental lighting, cast shadows, and geometry-induced wrinkles directly into the texture map, and lose global structural coherence across garment panels. OmniFabric addresses this by synthesizing texture maps directly in the flat, canonical 2D sewing pattern (UV) space, where folded geometry and perspective distortion do not exist, and where the result is directly usable by cloth simulation engines.

Key Contributions

1. OmniFabric, a framework for synthesizing high-fidelity, globally coherent textures from a single in-the-wild image for 3D garment reconstruction.
2. An automated data engine for creating realistic, diverse textured sewing patterns at scale, providing training data for learning complex garment textures.
3. A canonical 3D mesh as a spatial anchor to map image pixels to sewing pattern UV space, enabling globally aligned texture synthesis (rather than the local tileable patches used by FabricDiffusion).
4. A coarse-to-fine texture generation pipeline that initializes globally aligned textures and then rectifies distortion and baked-in artifacts with a DiT-based UV space normalization network.

Main Findings

• Quantitative superiority on the synthetic dataset: In Table 1, OmniFabric achieves LPIPS 0.092, SSIM 0.868, MS-SSIM 0.905, DISTS 0.121 and CLIP-s 0.963. The baselines score: FabricDiffusion 0.273 / 0.645 / 0.707 / 0.259 / 0.906; Paint3D 0.311 / 0.657 / 0.704 / 0.266 / 0.890; Hunyuan3D-2.0 0.223 / 0.712 / 0.717 / 0.220 / 0.924. Each baseline receives the identical frontal rendering and rest-posed mesh for fair comparison.

• User preference on real-world images: A user study with 13 participants evaluating results from 10 real-world image inputs shows OmniFabric attaining the best overall rank of 1.09 (lower is better) versus Hunyuan3D-2.0 at 2.10, FabricDiffusion at 3.40 and Paint3D at 3.41, and the best scores on fidelity (4.44), back-view plausibility (4.35) and global coherence (4.41) on a 1-to-5 scale.

• Reproducibility without closed-source models: A fully open-source variant, using FLUX.2 as the image generation backbone with Ministral 3 for prompt up-sampling, scores LPIPS 0.114, SSIM 0.847, DISTS 0.149 and CLIP-s 0.955, close to the default setting of 0.092 / 0.868 / 0.121 / 0.963.

• Texture normalization absorbs accumulated error: Replacing intermediate outputs with ground-truth counterparts, the cumulative pipeline (predicted reposed image plus predicted multi-views) scores LPIPS 0.144, SSIM 0.804, DISTS 0.166, CLIP-s 0.938, but adding the sewing pattern normalization step improves this to LPIPS 0.092, SSIM 0.868, DISTS 0.121, CLIP-s 0.963.

• All three conditioning signals matter: Removing the coarse texture map causes a severe drop (LPIPS 0.407, SSIM 0.601, DISTS 0.306, CLIP-s 0.887), while removing the frontal view rendering gives LPIPS 0.097 / SSIM 0.850 / DISTS 0.129 / CLIP-s 0.962 and removing the position map gives 0.098 / 0.848 / 0.130 / 0.961. Without the position map, elements such as a lemon were generated in the wrong location.

• Distortion augmentation is necessary: Training on TPS-distorted textures lets the model restore normalized structured patterns (for example a checker pattern on sleeves) and clean text and logos.

• Baselines fail on high-frequency and asymmetric content: Both Hunyuan3D and Paint3D fail to preserve high-frequency details, and FabricDiffusion, while producing normalized tileable patterns, cannot produce coherent global appearance.

• Failure modes exist: OmniFabric may synthesize mismatched textures when the generated multi-views are inaccurate, and it cannot correct geometry predicted incorrectly by ChatGarment, although it still normalizes textures for whatever geometry it is given.

Methodology in Plain English

The pipeline has two stages.

Stage 1 creates a coarse but globally aligned texture. Given a reference photo, an image generation model (Nano Banana Pro) re-poses the garment into a canonical A-pose whose silhouette matches the frontal render of the garment mesh simulated from predicted sewing patterns. That frontal render then gets a 360-degree spinning video generated by a large video model (Veo 3), from which four orthogonal views (front, back, left, right) are sampled. These views are projected onto the canonical mesh and mapped to UV coordinates on the sewing pattern, producing a coarse texture map that covers the whole garment. This map preserves global design but contains baked-in shadows, physical wrinkles, unpainted regions and projection distortion.

Stage 2 cleans that coarse map. A Diffusion Transformer, fine-tuned with LoRA on synthetic data, takes a multi-modal conditioning set — the coarse texture map, the textured frontal rendering, and a 3D position map created by projecting mesh vertex coordinates onto the 2D panel layout — and predicts a normalized texture. All conditions and the noisy latent are treated as a unified token sequence with joint self-attention; patches at the same spatial coordinates across the coarse map, position map and target share identical positional encodings, while the frontal rendering keeps its own coordinate system. The position map teaches the model spatial connectivity between separate UV panels so textures stay seamless across fragmented islands.

Training data is synthesized automatically. From GarmentCodeData, 3,000 unique garment samples are drawn. An LLM writes descriptions of texture layout, semantic elements and color palettes based on DeepFashion references; a vision-language model then synthesizes diverse textures, which are overlaid on sampled sewing patterns to yield 30,000 textured sewing patterns (roughly 10 texture variations per garment style). A random Thin Plate Spline warp is applied to the texture, with control-point perturbation scaled by a magnitude parameter of 25 and the texture resolution, so the model has to learn undistortion. The distorted pattern is simulated onto an A-pose SMPL body using NVIDIA Warp with per-sample random seeds, and random subsets of white point lights introduce varied illumination. The rest-posed mesh, position map, coarse texture map and frontal rendering form each training tuple against the original undistorted texture as ground truth. A held-out split uses a test/train ratio of 0.1.

Practical details: training runs on a single NVIDIA A6000 GPU with batch size 8, with the VAE and primary DiT backbone frozen; maps are resized and padded to 1024×1024 for training, and an image super-resolution model upscales outputs at inference. A single inference takes roughly 5 minutes depending on Gemini server load. The reported failure rate of the Gemini models (producing unrelated textures on the frontal rendering or multi-views) is lower than 2%, estimated from randomly sampled generations. Sewing pattern prediction uses ChatGarment, and simulated results are produced in CLO using its default Fabric_Matte preset for roughness, metallic, reflection intensity and normal map.

Why This Matters

Impact on research. The paper reframes garment texturing from a 3D painting problem into a UV-space normalization problem. It argues that the right domain for fabric texture synthesis is the sewing pattern rather than the draped mesh, because the flat panel layout removes the geometric distortion and self-occlusion that make projection-based texturing fragile. It also shows that a general-purpose diffusion transformer, fine-tuned with LoRA on purely synthetic data, can serve as a texture normalizer that recovers clean base color from a noisy, artifact-laden initialization.

Real-world applications:

• Gaming and interactive entertainment, where garments must be simulated and relit dynamically rather than appearing as static baked appearance.
• E-commerce and virtual try-on, where a shopper's photograph of a garment must become a relightable, animatable asset.
• Cinematic visual effects, where the same cloth asset is rendered under many lighting conditions and camera angles.
• Digital fashion design and manufacturing workflows, since the output lives in the same sewing pattern representation used to construct real garments.

Industry relevance. The output is directly importable into cloth simulation engines such as CLO, with the normalized RGB map used as albedo and remaining material properties supplied externally. That puts the method on the same footing as garment production tooling rather than leaving it as a visualization-only artifact.

Future Directions

• Joint prediction of complete PBR material maps — roughness, metallic and normal maps alongside albedo — which the authors identify as an immediate and promising extension for greater photorealism under diverse lighting.
• Reducing dependence on multi-view generation accuracy, since mismatched or inaccurate generated multi-views can produce mismatched textures.
• Geometry correction, because OmniFabric cannot refine a mismatched mesh predicted by ChatGarment; it only normalizes textures for the geometry it receives.
• Higher-resolution training and broader pattern-prediction compatibility, since the LoRA-based fine-tuning allows higher training resolution without GPU memory issues, and the supplementary material reportedly covers adaptability to sewing pattern prediction methods other than ChatGarment.

Target Audience

Researchers and graduate students in 3D vision and graphics working on texture synthesis, neural rendering, or garment digitization; practitioners building generative pipelines for cloth assets; and engineers in digital fashion, e-commerce or VFX who need simulation-ready garments with relightable, artifact-free textures. Readers without background in diffusion models, UV parameterization or sewing pattern representations will find the method sections demanding, but the problem framing and experimental comparisons remain accessible.

Authors’ abstract

Automated generation of production-ready 3D garment assets from a single image is a central challenge in digital content creation. While recent generative models have significantly advanced 3D geometry reconstruction, synthesizing high-quality textures remains a bottleneck. Existing methods often bake environmental illumination and shadows directly into the texture map, or they fail to maintain global structural coherence, making the resulting assets unusable for physical simulation and relighting. In this work, we introduce OmniFabric, a novel approach that synthesizes globally coherent texture maps directly within the 2D sewing pattern space. Given a single reference image, our pipeline utilizes an estimated 3D mesh and generative priors of powerful Vision-Language Models (VLM) to establish a complete but coarse texture initialization across the unwrapped sewing patterns. We then leverage a specialized diffusion transformer, trained via an automated synthetic data engine and conditioned on 3D positional features, to refine this initialization directly in the canonical UV domain. This effectively removes distortion and baked-in artifacts to extract a clean and normalized texture map that preserves the original garment design. Extensive experiments demonstrate that OmniFabric significantly outperforms state-of-the-art baselines, yielding photorealistic 3D garments with high-quality textures.