Skullptor: High Fidelity 3D Head Reconstruction in Seconds with Multi-View Normal Prediction

AI summary

Overview

Research area: Computer vision, specifically 3D facial capture — multi-view surface normal estimation and inverse-rendering mesh optimization for production-quality 3D head reconstruction.

Technical level: Intermediate to Advanced. The paper assumes familiarity with vision transformers, cross-attention, inverse rendering, photogrammetry, and differentiable mesh optimization.

Scope: The paper presents a two-stage hybrid pipeline that predicts geometrically consistent multi-view surface normals from sparse images and uses them as geometric priors to optimize a detailed 3D head mesh in seconds.

What This Paper Is About

Producing detailed 3D head geometry from photographs is central to visual effects, gaming, and virtual communication, but the existing options sit at opposite extremes: photogrammetry reaches exceptional detail only with dense camera arrays and heavy computation, while single-image foundation models are fast but miss fine geometric detail. The authors ask whether a learned multi-view normal predictor can supply the missing geometric constraints that let an optimization-based reconstruction reach photogrammetry-quality fidelity from very few cameras. Their goal is to deliver high-fidelity, complete 3D heads with fine features such as wrinkles and skin folds, at a fraction of the capture and compute cost.

Key Contributions

1. A multi-view normal prediction model. The authors adapt the monocular facial foundation model DAViD with a lightweight view-aware cross-attention mechanism inserted between each transformer block and fully connected layer, enforcing geometric consistency across sparse (fewer than 10) input viewpoints.

2. A normal-guided inverse rendering framework. The predicted normals are used as strong geometric priors inside a differentiable-rendering mesh optimization with adaptive remeshing, allowing recovery of high-frequency surface detail.

3. Demonstrated superiority over dense multi-view and single-image baselines. The complete pipeline outperforms Gaussian-splatting methods (2DGS, SuGaR) and single-image approaches while matching photogrammetry fidelity with fewer cameras and less processing time.

4. A code and model release to facilitate future research.

Main Findings

• Photogrammetry-level quality from 10 cameras in 30 seconds. The method reconstructs a highly detailed, complete 3D head from 10 cameras in 30 seconds, and scales down to high-fidelity frontal facial reconstruction from as few as three cameras.

• Normal estimation is competitive on angular error and clearly better on high-frequency detail. On Multiface, Skullptor reaches 9.13 average angular error versus DAViD's 9.16 and Sapiens 0.3B's 10.3; on NPHM, 7.29 versus DAViD's 7.86 and Sapiens 2B's 6.86. The advantage is sharpest on the new Normal Gradient Error metric: 0.234 on Multiface (DAViD 0.250) and 0.166 on NPHM (DAViD 0.190). Inference time for a batch of 10 images is 1.5 s, versus 1.1 s for DAViD and 41.3 s for Sapiens 2B on Multiface.

• Mesh reconstruction beats 2DGS and SuGaR by wide margins and is far faster. On Multiface, Skullptor records 2.43 mm depth error and 0.67 min runtime versus 5.73 mm / 50 min for 2DGS and 5.54 mm / 42 min for SuGaR. On NPHM, 2.33 mm and 0.72 min versus 6.37 mm / 47 min (2DGS) and 3.23 mm / 50 min (SuGaR). The 10-view configuration still achieves 2.99 mm (Multiface) and 2.36 mm (NPHM).

• Comparable to Meshroom photogrammetry with fewer views and an order of magnitude less time. On NPHM, Skullptor reaches 2.33 mm depth error in 0.72 min using 10 views, against Meshroom's 2.54 mm in 9.5 min using 23 views.

• A documented bias on the Multiface benchmark. Both the Multiface ground truth (40 views) and the Meshroom baseline (26 views) are photogrammetry-based. The authors point to a "broken nose" artifact in the Multiface photogrammetry that penalizes their otherwise correct reconstruction: their mesh yields 4.46 mm depth error in that case versus Meshroom's 0.201 mm, over 20× higher, despite comparable visual quality. On NPHM, which uses active scanning rather than photogrammetry, this bias is absent.

• Robustness to very sparse capture. Testing at 3, 6, 10, 16, and 23 views on NPHM, the method holds up at 3 cameras while Meshroom degrades rapidly below 16 views and fails almost completely at 3.

• Cross-attention and multi-view training both matter. Fine-tuning DAViD on the authors' data with monocular input alone (DAViD_ft_mono) does not help and can hurt (NPHM angular error rises to 8.43 from DAViD's 7.80). Multi-view training without cross-attention (DAViD_ft_multi) improves it to 7.13. The full Skullptor model with cross-attention is best at 7.05.

• The normal estimator, not just the optimizer, determines final mesh quality. Substituting monocular predictors into the full pipeline degrades geometry: on NPHM, Sapiens 2B yields 6.76 mm depth error despite performing well on the standalone normal comparison, which the authors attribute to overly smooth normals. Skullptor's own estimator gives 2.39 mm.

• Generalization beyond controlled capture. The DAViD backbone contributes priors from 300K diverse images, and the authors report generalization across four capture setups (TripleGangers, Multiface, NPHM, and their own) plus an in-the-wild result from 12 phone-captured images.

Methodology in Plain English

The pipeline has two stages.

Stage one — predict normals across views. Starting from DAViD, a monocular model that predicts per-pixel surface normals from a single face image, the authors add cross-attention layers so that each image can consult every other image when producing its normal map. The encoded features for each image are 577 tokens of dimension 1024; each view's query attends to keys and values built from all views. Camera poses are injected as positional embeddings: the rotation matrix becomes a unit quaternion, giving a 7-dimensional pose vector (quaternion plus translation) that is projected to the feature dimension and added to every token. Training minimizes a cosine similarity loss between predicted and ground truth normals.

Training data comes from high-quality Triplegangers 3D head scans: 50 subjects, each with 20 static expressions captured from 55 lightstage cameras, with 5 subjects held out for validation and visualization. Rather than training on the fixed lightstage positions, the authors render textured images and camera-space ground truth normal maps from randomly sampled virtual cameras (48 virtual cameras at 512×512 resolution, rendered with nvdiffrast). This augmentation exposes the model to arbitrary camera configurations and multiplies training diversity. The split is 45 subjects / 900 expressions for training and 5 subjects / 100 expressions for validation.

Stage two — fit the mesh. The predicted normals arrive in local camera coordinates, so the authors first calibrate coordinate systems. They detect 2D facial landmarks, triangulate them into 3D by least-squares ray intersection, and use Procrustes analysis with singular value decomposition to find the similarity transform aligning those landmarks to a fixed-topology template normalized to unit radius. Applying the inverse transform to the camera extrinsics and the corresponding rotation to the normals places everything in a canonical space; foreground masking from DAViD removes background.

The mesh itself is initialized to a unit sphere and optimized so that its rendered normals match the predicted normals under differentiable rendering. The loss combines a cosine-similarity normal term with a Laplacian smoothness regularizer. A per-pixel weight matrix, computed as a normalized exponential of the dot product between predicted normal and viewing direction, prioritizes frontal-facing regions over grazing angles. After each optimization step, adaptive remeshing performs edge splits, collapses, and flips to adjust resolution where geometry is complex and to prevent collapsed faces and self-intersections. Using an aligned template head mesh as initialization instead of a sphere made no meaningful difference in time or quality.

Evaluation. Normal prediction is measured with mean angular difference, percentages of angles below 10°, 20°, and 30°, and a new Normal Gradient Error that applies a Sobel filter to predicted and ground truth normals and takes the L1 distance — a proxy for preservation of wrinkles, folds, and fine curvature. Mesh quality is measured with L1 depth error plus the same angular and gradient metrics, rasterized from 12 novel camera views at 512×512 resolution, with a Facer facial mask restricting evaluation to the face region. NPHM evaluation uses 23 virtual camera viewpoints over the first 20 subjects (400 expression scans) since the dataset has no captured images; Multiface uses real captures subsampled at 1 of every 30 frames (347 frames). Mesh comparison uses a random subset of 50 expressions/frames per dataset, 100 reconstructions total. Metrics exclude 6 Multiface and 12 NPHM subjects from the initial 50 faces per dataset where 2DGS or SuGaR failed to produce a reconstruction.

Why This Matters

Impact on research. The paper reframes the trade-off between learned priors and geometric optimization as complementary rather than competing. It shows that a foundation model can be extended to multi-view processing cheaply, and that the resulting normals are strong enough priors to make per-scene inverse rendering viable in sparse-view regimes where it previously needed dense coverage. The Normal Gradient Error metric also offers a way to catch overly smooth reconstructions that angular error alone would rate highly.

Real-world applications:

• Visual effects and game production, where detailed 360-degree head geometry is currently acquired with dense camera arrays and manual cleanup.
• Virtual communication and telepresence, where faster capture with fewer cameras lowers the barrier to creating detailed avatars.
• 4D facial performance capture, where the authors note that multi-view storage of terabytes of data is prohibitive and reduced view counts directly reduce that burden.
• Mobile or consumer capture, supported by the in-the-wild result on 12 phone-captured images.

Industry relevance. The authors are affiliated with Ubisoft La Forge alongside the University of Toronto, ÉTS Montréal, and the Vector Institute, and the framing throughout is explicitly production-oriented, targeting the photogrammetry pipelines that studios rely on for high-end VFX and gaming.

Future Directions

• Joint normal and albedo prediction for complete appearance capture, rather than geometry alone.
• Material and lighting estimation to enable relighting of reconstructed heads.
• Robustness to uncontrolled capture conditions. The method is designed for controlled lighting and synchronized cameras; strong view-dependent reflections, noisy images, and facial props can cause inaccurate normal predictions that propagate into the final geometry.
• Resolving the Multiface evaluation bias, since the photogrammetry-based ground truth creates correlated bias against non-photogrammetry methods — as shown by the "broken nose" case where visual quality and the depth metric disagree.

Target Audience

Researchers and practitioners in 3D face reconstruction, inverse rendering, and multi-view stereo; VFX and game-tools engineers evaluating alternatives to dense photogrammetry rigs; and graduate students working on multi-view transformers or learned geometric priors who want a concrete example of combining feed-forward prediction with per-scene optimization.

Authors’ abstract

Reconstructing high-fidelity 3D head geometry from images is critical for a wide range of applications, yet existing methods face fundamental limitations. Traditional photogrammetry achieves exceptional detail but requires extensive camera arrays (25-200+ views), substantial computation, and manual cleanup in challenging areas like facial hair. Recent alternatives present a fundamental trade-off: foundation models enable efficient single-image reconstruction but lack fine geometric detail, while optimization-based methods achieve higher fidelity but require dense views and expensive computation. We bridge this gap with a hybrid approach that combines the strengths of both paradigms. Our method introduces a multi-view surface normal prediction model that extends monocular foundation models with cross-view attention to produce geometrically consistent normals in a feed-forward pass. We then leverage these predictions as strong geometric priors within an inverse rendering optimization framework to recover high-frequency surface details. Our approach outperforms state-of-the-art single-image and multi-view methods, achieving high-fidelity reconstruction on par with dense-view photogrammetry while reducing camera requirements and computational cost. The code and model will be released.