SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction

AI summary

Overview

Research area: Computer vision — single-image 3D clothed human reconstruction using diffusion/generative priors (2D multiview generation and native 3D generation).

Technical level: Advanced. The paper assumes familiarity with latent diffusion, flow matching, transformer attention, sparse voxel representations, and 3D Gaussian Splatting, though the central idea (letting two generative models correct each other) is conceptually simple.

Scope: The paper proposes a unified framework that couples a 2D multiview image generative model with a 3D native generative model through a synchronization attention mechanism, then decodes the result into a detailed textured human mesh.

What This Paper Is About

Reconstructing a photorealistic, clothed 3D human from a single RGB photo is hard because one view hides most of the body and provides no direct 3D information. Existing methods lean on an estimated SMPL body mesh to guide a 2D multiview generative model, but SMPL estimates are imprecise — especially for occluded or unusual poses — and SMPL only models a naked body, not loose clothing.

SyncHuman's goal is to avoid that fragile dependence: instead of using SMPL as the 3D guide, it uses a 3D native generative model to produce a coarse but structurally sound 3D shape, and lets that shape and the 2D multiview images refine each other in a shared 3D space.

Key Contributions

1. First combination of a 2D multiview generative model and a native 3D generative model in a single framework for single-view human reconstruction, exploiting their complementary strengths (2D detail vs. 3D structural consistency).

2. A 2D-3D synchronization attention mechanism with two directions — 2D-to-3D and 3D-to-2D attention — that explicitly aligns 3D voxel features with multiview image features, plus a joint flow-matching training objective for both branches.

3. A Multiview Guided Decoder (MVGD) that injects DINOv2 features from the generated multiview color and normal maps into the 3D structured latents, carving fine geometric detail and improving texture fidelity beyond what the base decoder produces.

4. A human reconstruction pipeline that does not rely on any human body prior (no SMPL conditioning) and is reported to outperform prior methods in geometric accuracy and visual fidelity, including large-scale 3D generative models trained on datasets "hundreds of times larger."

Main Findings

• State-of-the-art geometry on all three test sets. On X-Humans, SyncHuman reports Chamfer Distance 0.8353, P2S 0.7593, and NC 0.8872, versus PSHuman (1.4377 / 1.1385 / 0.8393) and TRELLIS (2.0043 / 1.5053 / 0.7718). On CAPE-NFP it reports 0.9127 / 0.8113 / 0.8483 and on CAPE-FP 0.6409 / 0.5962 / 0.8958, the best values in each column of Table 1.

• Best appearance metrics on X-Humans. SyncHuman reports PSNR 21.8385, SSIM 0.8741, and LPIPS 0.0786, compared with PSHuman (20.8405 / 0.8523 / 0.0980), SIFU (20.6747 / 0.8455 / 0.1104), GTA (20.0084 / 0.8502 / 0.1129), Human3Diff (19.7181 / 0.8065 / 0.1334), and TRELLIS (17.0786 / 0.7238 / 0.1529). Appearance metrics are reported only for the X-Humans subset in Table 1; the other two datasets show results only for geometry.

• Synchronization attention is the main driver of the gain. In the X-Humans ablation (Table 2), fine-tuned TRELLIS improves from PSNR 17.079 to 20.344 and Chamfer from 2.004 to 1.135; PSHuman reaches 20.840 / 1.438; the full model reaches 21.838 / 0.835. This shows each individual branch improves over its own baseline, and the synchronized version improves further.

• The multiview guided decoder adds measurable detail. With identical structured latents (Table 3), the original decoder gives PSNR 21.083 / LPIPS 0.092 / Chamfer 0.895, the fine-tuned original decoder gives 21.362 / 0.090 / 0.887, and MVGD gives 21.838 / 0.0786 / 0.835.

• Generated 3D structure is more robust than SMPL estimation. Figure 8 compares SMPL estimates from 4D-Humans with the structure generated by SyncHuman on the same inputs; the SMPL estimates show errors such as self-intersection, while the generated structure aligns better with the input.

• Qualitative advantages on challenging poses. Figures 4 and 5 show comparisons against ECON, Human3Diff, SIFU, and PSHuman, where the authors report their method reconstructs complete body structure and richer detail, and avoids the blurred colors and artifacts those baselines show in unseen views and occluded regions.

• Lighting sensitivity is an acknowledged limitation. Because the training data was rendered with a uniform light source, reconstructed textures can show artifacts under extreme lighting, such as localized overexposure or shadows (Figure 9).

• Training scale. The 2D-3D cross-space generative model was trained on 8 NVIDIA H800 GPUs (batch size 32, 50,000 iterations); the multiview branch alone used batch size 32 for 30,000 iterations; the Multiview Guided Decoder was trained on a single NVIDIA H800 (batch size 4, 14,000 iterations, learning rate 1e-4). The multiview model was fine-tuned from SD 2.1 using only approximately 5,000 human scans — a much smaller dataset than the 3D baselines used.

Methodology in Plain English

The pipeline has two stages.

Stage 1 — Generate 2D views and a 3D shape together. Given a single human photo, the model runs two branches at once: a 2D branch that produces color and normal maps for four orthogonal viewpoints (front, back, left, right), and a 3D branch (following the Trellis design) that produces a sparse occupied voxel grid describing the coarse 3D shape. The novel part is that the two branches talk to each other. For every 3D voxel, the model projects the voxel onto the four view planes, pulls the corresponding 2D features, and lets the voxel attend to them (2D-to-3D attention). In the other direction, every 2D pixel feature looks along its ray into 3D space and attends to a column of voxel features (3D-to-2D attention). Both attention outputs use zero-initialized output layers and are added back as residual refinements. The effect: the 3D shape supplies structural guidance that keeps the four generated views mutually consistent, and the detailed 2D views push the 3D shape toward fidelity. Both branches are trained jointly with a flow-matching objective, with the multiview model retargeted from Stable Diffusion 2.1 to the same flow-matching formulation used by Trellis.

Stage 2 — Decode into a detailed textured mesh. The voxel grid is turned into a set of structured latents by a pretrained DiT-based generative model. The authors found that decoding those latents directly loses detail in the face and clothing wrinkles, so they add a feature injection step: DINOv2 features are extracted from the generated multiview color and normal maps (four views × two map types = eight feature vectors per location, taken from DINOv2 layers 4, 11, 17, and 23), concatenated with each latent, passed through an MLP, and added back to the latent. This injection module is inserted after every self-attention in the decoder and is applied to both the mesh decoder and the 3D Gaussian Splatting decoder. Images rendered from the 3DGS representation are then baked onto the mesh to produce the final textured human mesh.

Why This Matters

Impact on research: The paper argues that the field's reliance on SMPL estimation as a conditioning signal is a bottleneck, and demonstrates an alternative — using a native 3D generative model as the structural guide. It also shows that a comparatively small human dataset (~5,000 scans) can, through careful synchronization with a pretrained 3D generative model, produce results the authors report as higher in fidelity and texture quality than a large-scale 3D generative model trained on far more data. If that holds, it suggests a route to strong human-specific generation without training giant models from scratch.

Real-world applications:

• Film and video game asset creation, where full-body characters must be built from limited reference photography.
• AR/VR avatars and immersive experiences.
• Virtual try-on and fashion, where garment geometry and wrinkles matter beyond the naked body shape.
• General image-to-3D content pipelines that need robust handling of unusual or self-occluded poses.

Industry relevance: Studios and platforms that need scalable, pose-robust 3D human capture from a single photo stand to benefit, particularly where users cannot supply controlled multi-camera captures and where SMPL-fitting failures currently produce visible artifacts.

Future Directions

• Scaling the multiview generator. The authors note their multiview model is fine-tuned from SD 2.1 using only ~5,000 human scans, and that scaling up with video generative models or large-scale multiview human datasets is a promising direction.
• Handling non-uniform lighting. Since the training data was rendered under uniform illumination, textures degrade under extreme lighting; training on more varied lighting is an open problem the authors raise.
• Extending beyond the four-view setup. The synchronization attention is defined over front, back, left, and right views; whether more views or continuous view sampling improves structural alignment is not explored in the reported content.
• Generalizing the synchronization idea beyond humans. Because the method avoids human-specific priors, the framework could plausibly be adapted to other single-view 3D generation categories, though the paper does not report such experiments.

Target Audience

Researchers and engineers working on 3D generative models, single-image human reconstruction, and image-to-3D pipelines; practitioners in visual effects, gaming, AR/VR, and virtual try-on who need pose-robust clothed human reconstruction; and graduate students studying how 2D and 3D generative priors can be coupled within one training objective. Readers should be comfortable with diffusion/flow-matching models, transformer attention, and 3D representations such as voxels and Gaussian Splatting.

Authors’ abstract

Photorealistic 3D full-body human reconstruction from a single image is a critical yet challenging task for applications in films and video games due to inherent ambiguities and severe self-occlusions. While recent approaches leverage SMPL estimation and SMPL-conditioned image generative models to hallucinate novel views, they suffer from inaccurate 3D priors estimated from SMPL meshes and have difficulty in handling difficult human poses and reconstructing fine details. In this paper, we propose SyncHuman, a novel framework that combines 2D multiview generative model and 3D native generative model for the first time, enabling high-quality clothed human mesh reconstruction from single-view images even under challenging human poses. Multiview generative model excels at capturing fine 2D details but struggles with structural consistency, whereas 3D native generative model generates coarse yet structurally consistent 3D shapes. By integrating the complementary strengths of these two approaches, we develop a more effective generation framework. Specifically, we first jointly fine-tune the multiview generative model and the 3D native generative model with proposed pixel-aligned 2D-3D synchronization attention to produce geometrically aligned 3D shapes and 2D multiview images. To further improve details, we introduce a feature injection mechanism that lifts fine details from 2D multiview images onto the aligned 3D shapes, enabling accurate and high-fidelity reconstruction. Extensive experiments demonstrate that SyncHuman achieves robust and photo-realistic 3D human reconstruction, even for images with challenging poses. Our method outperforms baseline methods in geometric accuracy and visual fidelity, demonstrating a promising direction for future 3D generation models.