Capture, Canonicalize, Splat: Zero-Shot 3D Gaussian Avatars from Unstructured Phone Images

AI summary

Overview

Research area: Computer vision and graphics — 3D human avatar creation, Gaussian splatting, and large reconstruction models.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian splatting, transformer-based reconstruction models, diffusion/generative view synthesis, and photometric losses such as LPIPS.

Scope: The paper describes a zero-shot, three-stage pipeline that turns a few unstructured smartphone photos into a static, quarter-body, identity-preserving 3D Gaussian splatting avatar trained on a new dataset of high-fidelity dome-captured avatars.

What This Paper Is About

Creating a faithful 3D "digital twin" of a specific person usually requires either calibrated multi-camera rigs or a single image that leaves most of the body unseen. Single-view methods hallucinate geometry for occluded regions such as the back of the head, which damages identity, while models trained on synthetic data (for example RenderPeople) miss high-frequency person-specific detail like skin wrinkles and fine hair, producing an over-smoothed, generic look. This paper's goal is to generate hyperrealistic, identity-preserving 3D avatars from a small set of unstructured phone photos by first canonicalizing those photos into consistent views and then lifting them into Gaussian splats using a model trained on real, high-fidelity scans.

Key Contributions

1. A complete zero-shot pipeline that generates hyperrealistic, identity-preserving static 3D avatars from a few unstructured phone images.
2. A generative canonicalization module that normalizes unstructured multi-view inputs into a 3D-consistent representation with fixed camera parameters, robustly preserving identity.
3. A novel training methodology using a large-scale dataset of high-fidelity Gaussian splatting avatars (3.2K avatars, 5M renders) derived from dome captures of real people, to learn and reproduce fine, person-specific details.
4. A multi-view-to-3D lifting stage: a transformer-based Large Reconstruction Model inspired by GS-LRM that directly predicts a 3D Gaussian splatting representation from the canonicalized views.

Main Findings

• Training data matters more than anything else in this comparison: On the authors' internal test set, a model trained on the high-fidelity Human Avatar Dataset with multi-view input reaches 33.5 PSNR, versus 27.5 for RenderPeople with multi-view input, 27.2 for the Human Avatar Dataset with single input, and 25.3 for RenderPeople with single input (Table 1).
• Multiple input views are critical for identity: The paper reports that reconstruction from a single view often fails to preserve identity, especially for unseen areas, whereas the multi-view approach produces a more faithful and consistent result (Figure 4). A model conditioned on a single front view struggles to maintain identity in the side and back views.
• Synthetic data produces identity shift and over-smoothing: Given a real input photo, a model trained solely on synthetic data such as RenderPeople exhibits an identity shift and an overly smooth, stylized appearance (Figure 3).
• The reconstruction model is highly sensitive to input 3D consistency: Because of this, the view normalization provided by the first stage is described as essential for high-quality, artifact-free reconstructions.
• The canonicalization module performs three functions: view normalization (averaging out lighting and pose variation), 3D consistency enforcement (outputs must be valid projections of a single underlying 3D object), and novel view synthesis (synthesizing views with no direct input, such as 45° views).
• Loss design details are reported: The training objective is a weighted sum of an L1 photometric loss, an LPIPS perceptual loss, an alpha loss supervising the rendered alpha mask against the ground-truth foreground mask (critical for removing floating artifacts and clean silhouettes), and a scale regularization loss that prevents degenerate, needle-like Gaussians. The individual loss weights are not reported in the provided content.

Methodology in Plain English

The pipeline has three named stages: Capture, Canonicalize, Splat.

Capture. The authors first built a new ground-truth dataset rather than relying on synthetic assets. They started from calibrated multi-view dome captures of thousands of real individuals and applied the universal avatar fitting pipeline of prior work to optimize a high-fidelity 3D Gaussian splatting avatar per subject. These avatars retain skin microgeometry, pores, fine wrinkles, and complex hair structure. From 3.2K such avatars they rendered 5M training renders, including canonical multi-view sets and simulated unstructured phone captures generated by perturbing camera position, orientation, and focal length, plus avatars rendered in different body poses.

Canonicalize. At inference, the user supplies a few unstructured phone photos, typically N = 4 (front, back, left, right). The generative canonicalization module aggregates identity information across these views and synthesizes M 3D-consistent canonicalized views with known, fixed camera parameters. This converts ambiguous, uncalibrated input into a structured multi-view format the reconstruction model can consume, and fills in views that were never captured.

Splat. A transformer-based Large Reconstruction Model, inspired by GS-LRM, takes the M canonical images and their camera parameters and predicts the properties of K 3D Gaussians — position, covariance (as scale and rotation), color, and opacity. It is trained end-to-end on the high-fidelity avatar dataset with the weighted combination of L1, LPIPS, alpha, and scale-regularization losses. Rendering the resulting Gaussians yields the final avatar.

Why This Matters

The work attacks the practical bottleneck in digital-human creation: professional captures (camera domes, light stages) are expensive and impractical for casual users, while cheap single-image methods trade away identity. By showing that a generative canonicalization step plus training on real high-fidelity scans closes much of that gap, the paper suggests that consumer-grade inputs can support production-grade realism. It also makes a pointed argument about data: the ablation isolates training data quality and view count as the dominant factors, which pushes back on the assumption that synthetic datasets such as RenderPeople or Objaverse are sufficient for photorealistic humans.

Real-world applications implied by the paper's framing:

• Virtual reality and telepresence, where a user's own likeness appears in shared spaces.
• Entertainment, including games and film, where per-person avatars must be produced at scale.
• Digital fashion, where an individual's avatar is dressed and rendered.
• General "digital twin" creation from smartphone photos for social or communication products.

Industry relevance: the work comes from Meta, and the dataset scale (3.2K avatars, 5M renders, thousands of capture subjects) reflects the infrastructure of a large platform company. The combination of a consumer-capture front end with a Gaussian-splatting back end is directly relevant to any product that needs personalized avatars generated from phones rather than studios. The paper does not report inference speed, model size, or deployment constraints, so cost and latency implications are not established in the provided content.

Future Directions

• Extending beyond static, quarter-body avatars to full-body and animatable or drivable representations — the paper explicitly scopes its output to static quarter-body avatars, and pose variation appears only as a data-augmentation strategy rather than a runtime capability.
• Reducing dependence on the two-stage design: the reconstruction model's sensitivity to input 3D consistency means canonicalization is currently mandatory, so a question is whether that dependence can be relaxed.
• Better handling of truly difficult real-world captures — the simulated "unstructured" inputs are generated by perturbing camera position, orientation, and focal length from dome data, so performance on genuinely uncontrolled phone photos (varied lighting, motion blur, occlusion) is not reported.
• Broader quantitative evaluation: the reported ablation numbers are on an internal test set with PSNR only; the paper content provided does not report comparisons against other methods on public benchmarks, nor perceptual or identity-specific metrics.

Target Audience

Researchers and practitioners in 3D reconstruction, generative modeling, and digital humans who want to understand how generative view canonicalization can be combined with Gaussian-splatting reconstruction models. It is also useful for product and technical teams evaluating whether phone-photo-based avatar capture is viable, and for anyone interested in the argument that real high-fidelity scan data — not synthetic assets — is what unlocks photorealism in avatar generation. The paper is written at an advanced level and assumes prior knowledge of Gaussian splatting and large reconstruction models.

Authors’ abstract

We present a novel, zero-shot pipeline for creating hyperrealistic, identity-preserving 3D avatars from a few unstructured phone images. Existing methods face several challenges: single-view approaches suffer from geometric inconsistencies and hallucinations, degrading identity preservation, while models trained on synthetic data fail to capture high-frequency details like skin wrinkles and fine hair, limiting realism. Our method introduces two key contributions: (1) a generative canonicalization module that processes multiple unstructured views into a standardized, consistent representation, and (2) a transformer-based model trained on a new, large-scale dataset of high-fidelity Gaussian splatting avatars derived from dome captures of real people. This "Capture, Canonicalize, Splat" pipeline produces static quarter-body avatars with compelling realism and robust identity preservation from unstructured photos.