ReWeaver: Towards Simulation-Ready and Topology-Accurate Garment Reconstruction

AI summary

Overview

Research area: Computer Vision, specifically 3D garment reconstruction and structured sewing-pattern recovery from multi-view images.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, multi-view encoders, hyper-networks, Hungarian matching, and Chamfer Distance losses.

Scope: A single framework, ReWeaver, that reconstructs both 3D garment geometry and corresponding 2D sewing patterns (panels, seams, connectivities) from as few as four multi-view RGB images, together with a new large-scale training dataset, GCD-TS.

What This Paper Is About

Existing garment reconstruction methods typically output unstructured representations — point clouds, meshes, distance fields, or 3D Gaussian Splats — that can approximate a garment's visible surface but do not encode how a garment is actually constructed (its panels, seams, and sewing structure). Because physics-based simulators and industry garment design workflows both require that sewing structure, these unstructured outputs are hard to simulate, edit, or retarget. ReWeaver's goal is to recover a structured, topology-accurate garment — simultaneously in 3D space and in 2D sewing-pattern space — from only a sparse set of ordinary multi-view photographs, with the reconstruction aligned to the input images.

Key Contributions

1. A unified 2D–3D garment reconstruction framework. ReWeaver is presented as the first framework to jointly reconstruct structured 3D garments and 2D sewing patterns while maintaining explicit 2D–3D correspondence. It predicts 3D curves, 3D patches, patch–curve connectivity, and the corresponding 2D edges.

2. The GCD-TS dataset. An extension of the large-scale GCD dataset that adds garment and body texture assets and structured 3D point-cloud annotations with explicit correspondences to 2D panels. GCD-TS contains over 100,000 synthetic samples (roughly 100,000 textured multi-view samples) covering a wide range of complex geometries and topologies.

3. Strong topological generalization with geometric fidelity. The paper reports that ReWeaver outperforms prior methods in pattern reconstruction, achieving strong topological generalization while maintaining high geometric fidelity to the input images.

4. Refinement procedures for clean topology and closed boundaries. A topology refinement pipeline (reliability-based filtering, duplicate curve merging, sub-curve removal, 2D loop pruning) and a 2D geometry refinement stage (bad-edge replacement and joint-midpoint alignment) that convert raw, redundant predictions into closed, triangulable panel loops.

Main Findings

• Topology accuracy is the standout result. On 2D panel quality (Table 1), ReWeaver reports Panel Count Accuracy (Acc_p) of 0.9210, Edge Count Accuracy (Acc_e) of 0.7175, and Overall Accuracy (Acc_o) of 0.6608. Baselines report: Sewformer 0.3761 / 0.4802 / 0.1806; ChatGarment 0.5557 / 0.8012 / 0.4452; AIpparel 0.4561 / 0.6774 / 0.3090. The paper states ReWeaver outperforms on five out of six metrics; ChatGarment reports the highest Edge Count Accuracy in the table.

• 2D geometric precision improves. ReWeaver reports 2D Edge Chamfer Distance (CD_e) of 0.0391 and 2D Panel IoU of 0.8221, versus Sewformer (0.1161 / 0.5844), ChatGarment (0.0906 / 0.6533), and AIpparel (0.0648 / 0.7084). Panel IoU improves from top baseline 0.7084 to 0.8221.

• Refinement steps matter, mostly for edges and panels. With refinement versus without (Table 2): Acc_p 0.9210 vs 0.9101; Acc_e 0.7175 vs 0.5361; Acc_o 0.6608 vs 0.4880; CD_e 0.0391 vs 0.0416; IoU 0.8221 vs 0.7775. Patch and curve Chamfer distances are essentially unchanged or slightly higher with refinement (CD_p^base 0.0232 vs 0.0225; CD_p^adapt 0.0185 vs 0.0188; CD_c 0.0266 vs 0.0255), which the paper attributes to refinement having negligible influence on patch and curve geometry.

• Each topology refinement rule contributes. In the cumulative ablation (Table 3): threshold-only gives Acc_e 0.5361, CD_e 0.0416, IoU 0.7775; adding 2D loop pruning gives 0.6045 / 0.0444 / 0.7584; adding sub-curve removal gives 0.6269 / 0.0418 / 0.7940; adding duplicate merging (full refinement) gives 0.7175 / 0.0391 / 0.8221.

• Adaptive sampling is used for patch point clouds at inference. The model is trained with a fixed 20×20 sampling density, but because the implicit patch mapping is smooth, arbitrary densities can be sampled at inference. The authors pre-sample a 20×20 grid and adaptively retain points based on spatial variance to produce near-uniform density across patches of different sizes; they note Li et al. [25] also predicts 3D point clouds but with fixed resolution. Table 2 reports CD_p^base = 0.0232 and CD_p^adapt = 0.0185 with refinement.

• Texture realism in the dataset is a deliberate design choice. GCD's original textures contain strong seam cues and textureless bodies, which the authors say are unrealistic and can lead to overfitting. GCD-TS replaces them with diverse tileable garment textures and nearly 50 body textures from BEDLAM. The authors note this makes the reconstruction task harder than on GCD's default textures.

• Curves use a fixed 50-point budget. Each curve and its corresponding edge are represented by 50 uniformly sampled points; the paper reports this "is sufficient to capture all curve lengths without noticeable visual artifacts." Patches are represented by a 20×20 point set.

Methodology in Plain English

The system takes multi-view RGB photographs (as few as four) and produces a structured garment description in two linked halves: a 3D half and a 2D half.

Encoding the images. Each image is split into patches and turned into tokens using a DINOv2 backbone. Following VGGT, the model alternates between intra-frame attention (refining each view on its own) and inter-frame attention (sharing information across views), progressively merging local and global geometric cues. The outputs of the final inter- and intra-frame layers are concatenated into one token sequence.

Predicting 3D structure. Rather than relying on a fixed garment template, the model uses a set of learnable "patch queries" and "curve queries" — like object queries in 3D detection. These go through a bi-path transformer (inspired by ComplexGen) where each group first does self-attention within itself, then cross-attention to pull information from the image tokens and from the other group. Three heads then predict: (a) whether each query is valid, (b) the geometry of each curve and patch, and (c) which patches attach to which curves (connectivity, computed as a sigmoid of a dot product between projected patch and curve tokens).

Geometry as functions, not fixed point lists. Each patch and curve is represented as a small neural network (MLP) that maps canonical coordinates — a 1D parameter for curves, a 2D coordinate for patches — into 3D space. These MLPs are generated by hyper-networks conditioned on the query tokens. Because geometry is a continuous function, it can be sampled at any density without losing smoothness.

Flattening to 2D. Using the predicted connectivity, the model groups each valid patch token with its connected curve tokens and runs intra-group attention (self-attention among curve tokens, then cross-attention with the patch token). Another hyper-network then decodes each resulting edge token into a 2D curve in normalized [0, 1]² coordinates. A lightweight MLP predicts a scalar scale factor per patch so the panel can be expressed at realistic physical size.

Cleaning up the output. Predicted topology is refined by filtering with thresholds (patches at ε_p = 0.7; curves and adjacencies at ε_c = 0.5 and ε_adj = 0.5), merging near-duplicate curves (bidirectional Chamfer distance below 0.03), removing short "sub-curves" contained inside others (Chamfer distance below 0.04), and pruning 2D edges that do not improve how well the panel forms a closed loop. A separate geometry refinement replaces badly connected edges (normalized gap above τ_gap = 3.0) with straight interpolated segments and then snaps all edges to joint midpoints using a per-edge 2D similarity transform, closing the loop for triangulation.

Training. Predictions are matched to ground truth with Hungarian matching. Geometry is supervised with Chamfer Distance on sampled point sets, validity and connectivity with binary cross-entropy, and the panel scale factor with an ℓ2 loss. The model was trained on an 8:1:1 train/validation/test split of GCD-TS, initialized with the lightweight 21 MB DINOv2 weights, with 200 patch queries and 70 curve queries (about twice the maximum counts in training data), four 518×518 input images per sample, and 768-dimensional tokens.

Why This Matters

Impact on research. The paper argues that recovering structured sewing patterns rather than unstructured surfaces is what makes garment reconstructions usable downstream. By producing a representation where each panel and edge is explicitly linked to 3D points, ReWeaver aims to bridge two communities that usually work separately: 3D reconstruction and garment pattern design. The GCD-TS dataset (over 100,000 synthetic samples with multi-view RGB images, 3D geometries, textured human body meshes, and annotated sewing patterns) is offered as a training resource for that direction. The authors position the work as addressing a "sim-to-real gap" between reconstructed assets and physical simulation.

Real-world applications named in the paper:

• Digital avatars and digital humans
• Virtual try-on
• Robotic manipulation (requiring 3D structural perception)
• Gaming and garment asset creation, including garment editing and retargeting

Industry relevance. The paper repeatedly emphasizes that industry-standard garment design workflows rely on 2D sewing patterns as the primary medium for design and manufacturing, and that unstructured representations are misaligned with those workflows. Producing panels that can be triangulated into meshes usable for simulation is presented as the key to making reconstructed garments practical rather than merely visually plausible. Authors are affiliated with Zhejiang University, Shanghai Innovation Institute, Westlake University, Xidian University, Fudan University, and Adobe.

Future Directions

The paper does not include an explicit future-work section; the following are open questions raised by its content.

• Transfer from synthetic training to real photographs. The training data is entirely synthetic (over 100,000 samples from a GCD-based simulation pipeline rendered from four viewpoints with randomized camera perturbations). Whether the same topology accuracy holds on in-the-wild photographs without a controlled capture setup — something the paper claims for its output — is not evaluated in the reported content.

• Downstream simulation validation at scale. The paper motivates simulation-readiness and shows simulated meshes in a qualitative comparison against AIpparel, but the reported metrics are topology and geometry metrics (Acc_p, Acc_e, Acc_o, CD_e, IoU, CD_p, CD_c). Quantitative simulation quality, e.g. physical plausibility or drape accuracy, is not reported.

• Robustness to real-world garment diversity and capture conditions. The dataset covers a wide range of complex geometries and topologies, but the paper does not report results on real scanned garments, non-tileable printed patterns, or extreme garment types absent from the GCD sampling distribution.

• Extending to dynamic garments and video. All reported inputs are four static RGB images. Reconstruction of garments in motion — where sewing structure must remain consistent across frames — is not addressed, and the related work section notes that optimization-based methods such as Gaussian Garments require a carefully chosen high-visibility template frame, a limitation the paper does not claim to solve.

Target Audience

This paper is most valuable to researchers and engineers working on 3D garment reconstruction, digital humans, and virtual try-on — particularly those who need simulation-ready assets rather than visual surfaces. It is also relevant to graphics practitioners building garment design and pattern-generation tools, robotics researchers who need structured 3D garment representations for manipulation, and machine-learning researchers interested in multi-view 3D reconstruction with structured, part-based queries. Readers should be comfortable with transformer attention mechanisms, implicit neural representations, and 3D reconstruction evaluation metrics; the writing itself explains the terminology (patches vs. panels, curves vs. edges) clearly enough for newcomers to the garment-reconstruction subfield.

Authors’ abstract

High-quality 3D garment reconstruction plays a crucial role in mitigating the sim-to-real gap in applications such as digital avatars, virtual try-on and robotic manipulation. However, existing garment reconstruction methods typically rely on unstructured representations, such as 3D Gaussian Splats, struggling to provide accurate reconstructions of garment topology and sewing structures. As a result, the reconstructed outputs are often unsuitable for high-fidelity physical simulation. We propose ReWeaver, a novel framework for topology-accurate 3D garment and sewing pattern reconstruction from sparse multi-view RGB images. Given as few as four input views, ReWeaver predicts seams and panels as well as their connectivities in both the 2D UV space and the 3D space. The predicted seams and panels align precisely with the multi-view images, yielding structured 2D--3D garment representations suitable for 3D perception, high-fidelity physical simulation, and robotic manipulation. To enable effective training, we construct a large-scale dataset GCD-TS, comprising multi-view RGB images, 3D garment geometries, textured human body meshes and annotated sewing patterns. The dataset contains over 100,000 synthetic samples covering a wide range of complex geometries and topologies. Extensive experiments show that ReWeaver consistently outperforms existing methods in terms of topology accuracy, geometry alignment and seam-panel consistency.