Geometry Meets Light: Leveraging Geometric Priors for Universal Photometric Stereo under Limited Multi-Illumination Cues

AI summary

Overview

Research area: Computer vision — universal photometric stereo (surface normal estimation from multi-illumination images) combined with pretrained feed-forward 3D reconstruction models.

Technical level: Advanced. The paper assumes familiarity with photometric stereo formulations, Transformer architectures (attention variants, DPT heads), and large-scale 3D reconstruction models such as VGGT and MoGe.

Scope: The paper proposes GeoUniPS, a universal photometric stereo network that fuses synthetic multi-illumination supervision with frozen geometric priors from a pretrained 3D reconstruction model, plus PS-Perp, a synthetic perspective-projection training dataset of 60,297 scenes.

What This Paper Is About

Universal photometric stereo (PS) recovers per-pixel surface normals from images taken under arbitrary, uncalibrated lighting, but it depends on having sufficiently diverse illumination across the images. When lighting variation is weak, biased, shadowed, or self-occluded — common in real-world scenes — existing methods degrade badly, because multi-illumination cues are their only source of information. The paper's goal is to compensate for unreliable lighting cues by injecting high-level geometric knowledge learned from real-world data via a pretrained 3D reconstruction model, and to remove the unrealistic orthographic camera assumption baked into prior PS training datasets.

Key Contributions

1. GeoUniPS architecture: A universal photometric stereo network that integrates synthetic supervision with high-level geometric priors from a large-scale 3D reconstruction model (VGGT), treating such models as visual-geometry foundation models that encode geometric knowledge of real scenes even from a single image.
2. Light-Geometry Dual-Branch Encoder: A two-branch encoder where one branch (Encoder_IL) extracts lighting-aware features from multi-illumination cues using light-axis attention, and the other (Encoder_Geo) extracts lighting-invariant geometric features from a frozen pretrained VGGT aggregator followed by a learnable DPT-head projector. The two 128-dimensional feature maps are concatenated.
3. PS-Perp dataset: Described as the first large-scale synthetic training dataset for universal photometric stereo rendered with a perspective projection model. It contains 60,297 multi-object scenes rendered in Blender's Cycles renderer, with focal lengths sampled from 20–1000mm; 44,220 scenes use focal lengths below 70mm to emphasize strong perspective distortion, and each scene yields 10 16-bit images at 512×512 under randomized directional, point, and environment lighting.
4. Dual-scale pixel-sampling decoder: A decoder design combining a low-scale normal estimation path and a high-scale refinement path that embeds input RGB values, with the paper reporting this as the best trade-off between geometric detail and efficiency among the variants explored.

Main Findings

• State-of-the-art on DiLiGenT (orthographic, all 96 images): GeoUniPS reaches an average mean angular error of 4.65°, compared with SDM-UniPS at 5.80°, Uni MS-PS at 5.38°, and LINO-UniPS at 4.75°.
• State-of-the-art on LUCES (perspective, all 52 images): GeoUniPS averages 9.42°, versus SDM-UniPS at 12.80°, Uni MS-PS at 11.29°, and LINO-UniPS at 9.46°. The paper notes LINO-UniPS uses wavelet-based refinement while GeoUniPS does not.
• Geometric priors help most when illumination cues are scarce: On DiLiGenT with K=1 input image, GeoUniPS achieves 12.86° average; on LUCES with K=1 it achieves 14.33°. The authors state that incorporating geometric priors significantly improves performance when multi-illumination cues are limited.
• More images do not always help: GeoUniPS averages 4.65° with all images, 4.77° with K=16, and 5.91° with K=4 on DiLiGenT. The authors attribute this to training with 3–6 input views, so evaluating on larger sets may cause a performance drop from distribution shift.
• Perspective training data matters: In Table 1, an SDM-UniPS model trained only on PS-Mix (orthographic) yields 22.18° MAE at 15mm focal length, versus 7.18° for PS-Perp and 6.98° for the PS-Perp + PS-Mix hybrid. PS-Perp alone degrades on orthographic images (8.95°), while the hybrid stays consistent (5.62°).
• Both encoder branches are needed: In the encoder ablation, Encoder_IL only reaches 19.03° on DiLiGenT at K=1 and 19.82° on LUCES at K=1, but improves to 4.96° and 10.36° at K=16. The Dual-Branch Encoder reaches 12.84° and 14.82° at K=1 and 5.19° and 9.82° at K=16, performing robustly across both extremes.
• VGGT outperforms MoGe as a geometry backbone: Encoder_Geo with VGGT scores 13.07° (DiLiGenT) and 14.70° (LUCES) at K=1, versus 13.63° and 15.50° with MoGe. The authors note both show similar trends, confirming geometric priors benefit PS regardless of the pretrained model.
• Qualitative gains on complex scenes: On the Multi-illumination Dataset, the method captures plausible normals on floors, walls, and regions with complex depth variation or mirror/transparent surfaces where methods relying solely on multi-illumination cues fail. The high-frequency components of the predicted normal maps are reported as comparable to other methods, indicating fine surface detail is still captured from multi-illumination cues.
• Stress test under minimal lighting: Using four images captured under weak, non-uniform lighting from a selfie ring light with ambient illumination and no object masks, and comparing against 3D scans from an EinScan-SE scanner, the paper reports that SDM-UniPS and LINO-UniPS fail almost completely while GeoUniPS performs significantly better, though not perfectly.

Methodology in Plain English

The authors start from the standard two-stage universal photometric stereo blueprint: an encoder that turns each input image into feature maps, and a decoder that predicts surface normals at randomly sampled pixel locations. They deliberately keep this framework simple so that any improvement can be attributed to geometric priors rather than confounding architectural changes.

Their central idea is to split the encoder in two. One branch does what conventional universal PS does — it looks across the multiple images taken under different lighting and extracts shading-variation features, using a Transformer that attends across images at the same spatial position (light-axis attention) so it can compare how a point changes as lighting changes. The second branch runs the image through the frozen aggregator of VGGT, a pretrained 3D reconstruction model, and projects its intermediate tokens into a feature map with a learnable DPT head. Because VGGT was trained on massive in-the-wild datasets, it carries knowledge of how real scenes are shaped — for instance, that building façades tend to be piecewise planar — even though it never sees the lighting variation. The two feature maps are simply concatenated and passed to the decoder.

For the decoder, they sample 2,048 pixel locations during training and 10,000 during inference, estimating coarse normals first and then refining them by embedding the raw RGB values at those locations. Training uses a sum of mean squared error losses over predicted and ground-truth normals at both scales.

To handle perspective cameras, they render a new synthetic dataset, PS-Perp, with focal lengths spanning 20mm to 1000mm, and train on it together with the orthographic PS-Mix dataset so that the model sees both projection types. They train from scratch on 4 NVIDIA H100 GPUs for 6 days, using 3–6 randomly sampled input images per training batch. Inference takes approximately 13 seconds for 16 images at 512×512 on a single H100, excluding I/O.

Why This Matters

Impact on research: The paper reframes feed-forward 3D reconstruction models as visual-geometry foundation models whose internal knowledge can regularize an entirely different low-level task. It also challenges the long-standing orthographic projection assumption in photometric stereo training data, and provides a dataset (PS-Perp) and an ablation showing that geometric priors and illumination cues are complementary rather than redundant.

Real-world applications:

• Handheld or casually-lit capture, where a user moves a flashlight or phone light around an object and only some regions receive varied illumination.
• Indoor scene understanding, augmented reality, and relighting, where full rooms have complex geometry, mirrors, and transparent surfaces and no controlled lighting.
• Cultural heritage and object digitization, where recreating precise surface detail from limited light control is a practical constraint.
• Robotics and inspection, where geometry must be estimated under weak or uneven ambient lighting without calibrated light sources.

Industry relevance: The work is partly supported by DENSO IT Laboratory, and the code is publicly released at https://github.com/marcotam2002/geounips. The problem it targets — reliable surface normals without controlled lighting — is directly relevant to automotive perception, 3D content creation pipelines, and any product where depth or normal estimation must work in ordinary environments rather than a darkroom.

Future Directions

• Exploring alternative backbones for the geometry branch, such as diffusion-based geometry estimators or other large-scale pretrained models like segmentation models, in place of VGGT.
• Designing better fusion strategies than simple channel-wise concatenation, since the authors note the performance boost under sufficient lighting cues is limited and likely attributable to this simple concatenation.
• Quantitatively validating on more real-world scenes with ground truth, since the Multi-illumination Dataset provides no normal maps and the 3D-scan comparison is described as an intentionally difficult analytical stress test rather than a realistic scenario.
• Extending perspective-projection training coverage beyond PS-Perp, given that perspective cameras with long focal lengths cannot fully replicate orthographic views and the hybrid training was needed to cover both projection types.

Target Audience

Researchers and graduate students working on photometric stereo, inverse rendering, and surface normal estimation; practitioners applying pretrained 3D reconstruction or monocular geometry foundation models to low-level vision tasks; and engineers building 3D capture, relighting, or inspection systems who need normals from uncontrolled, weakly-lit real scenes. Readers should be comfortable with Transformer architectures and photometric stereo fundamentals to follow the encoder and decoder design details.

Authors’ abstract

Universal Photometric Stereo is a promising approach for recovering surface normals without strict lighting assumptions. However, it struggles when multi-illumination cues are unreliable, such as under biased lighting or in shadows or self-occluded regions of complex in-the-wild scenes. We propose GeoUniPS, a universal photometric stereo network that integrates synthetic supervision with high-level geometric priors from large-scale 3D reconstruction models pretrained on massive in-the-wild data. Our key insight is that these 3D reconstruction models serve as visual-geometry foundation models, inherently encoding rich geometric knowledge of real scenes. To leverage this, we design a Light-Geometry Dual-Branch Encoder that extracts both multi-illumination cues and geometric priors from the frozen 3D reconstruction model. We also address the limitations of the conventional orthographic projection assumption by introducing the PS-Perp dataset with realistic perspective projection to enable learning of spatially varying view directions. Extensive experiments demonstrate that GeoUniPS delivers state-of-the-arts performance across multiple datasets, both quantitatively and qualitatively, especially in the complex in-the-wild scenes.