Monocular Normal Estimation via Shading Sequence Estimation

AI summary

Overview

Research area: Computer vision — monocular 3D geometry estimation (normal map prediction from a single image), inverse rendering, and generative video diffusion models.

Technical level: Advanced. The paper assumes familiarity with surface normal maps, photometric stereo and shading models, latent diffusion / video diffusion architectures, and angular-error evaluation protocols.

Scope: The paper reformulates object-based monocular normal estimation as a shading sequence estimation problem, solved with an image-to-video diffusion model that is trained on a new synthetic dataset called MultiShade and whose output is converted to normals analytically.

What This Paper Is About

Given one RGB photo of an object lit by an arbitrary light, the task is to recover a normal map — a per-pixel surface orientation field that describes the object's 3D shape. Existing deep models predict normal maps directly from color, and while the results often look plausible, the surfaces reconstructed from them frequently fail to match the true 3D geometry, a failure the authors call "3D misalignment." The paper's goal is to remove that failure by changing the training target from the normal map itself to a shading sequence — the set of brightness maps the object would produce under a fixed path of canonical parallel lights — which is more sensitive to geometric variation and can be converted back into a normal map exactly.

Key Contributions

1. A new paradigm. The paper reformulates monocular normal estimation as shading sequence estimation, arguing that supervising a model with a representation that varies strongly with geometry (rather than the compact, subtly colored normal map) is what reduces 3D misalignment.
2. RoSE. Under this paradigm the authors build RoSE (Reformulating normal estimation as Shading sequence Estimation), which uses an image-to-video generative model to predict a shading sequence of an object under predefined canonical parallel lights and then derives the normal map analytically with an ordinary least-squares solver.
3. The MultiShade dataset. A synthetic dataset built from roughly 90K pre-filtered 3D models, rendered under parallel, point, and environment lights with material augmentation from the MatSynth dataset, providing about 3 million image-normal pairs plus precomputed shading sequences.
4. Light-path analysis. The paper shows via Lemma 1 that at least six non-coplanar lights (l_min = m × n = 6, with m = 3 lights per point and n = 2 hemispheres) are needed so that every surface point is positively shaded by at least three lights, and it uses a ring-light setup at 45 degrees latitude.

Main Findings

• Shading sequences are more geometry-sensitive than normal maps. The paper validates this with average total variation (TV), the mean magnitude of the first-order gradient of each representation, where a higher TV indicates stronger sensitivity to spatial geometric variation.
• State-of-the-art on DiLiGenT. RoSE reaches 16.36° mean angular error on the parallel-light DiLiGenT benchmark, compared with 17.27° for the previous state of the art, NiRNE.
• State-of-the-art on LUCES. RoSE reaches 14.48° on the near-light LUCES benchmark, compared with 17.44° for Lotus-G.
• Not uniformly best on every object. RoSE does not rank in the top two on certain objects, such as Goblet in DiLiGenT and House in LUCES; the authors attribute this to inherent model variance and to the training set used.
• Best predicted shading sequences on LUCES. RoSE records PSNR 20.74, SSIM 0.7744, and LPIPS 0.2583, against 19.19 / 0.7589 / 0.2724 for the next-best method, Lotus-G.
• Strong on synthetic MultiShade. On the MultiShade test set — 100 unseen Objaverse objects, seven viewpoints each, 2800 test samples — RoSE outperforms all baselines across the reported metrics, with particularly strong performance under tight error thresholds of 3° to 7.5°.
• Competitive detail alignment. On LUCES, RoSE's sharp normal error (SNE) is comparable to NiRNE, a method trained on a dataset nearly 10× larger and with more diverse, complex 3D models.
• Nine lights is the sweet spot. The best performance came with 9 light sources, giving a 0.74° improvement on LUCES over 6 lights, while 12 lights degraded performance by 1.31° under the same settings.
• Negative clamping helps. Clamping negative shading values and rescaling to [-1, 1] via S ↦ S × 2 − 1 makes the shading sequence more sensitive to geometry and improves results over a variant without clamping.
• Material augmentation helps. Training without material augmentation (using only original object materials) yields slightly worse LUCES performance than training with it.
• The dataset transfers to other methods. Retraining Lotus-G on MultiShade ("Lotus-G+M") improves it consistently, and RoSE still beats that variant when both use the same data.
• Dataset changes are not uniformly positive. On the House object, retraining Lotus-G with the authors' dataset hurt performance (35.32° for Lotus-G versus 38.90° for Lotus-G+M).
• Backbone choice matters little. A variant built on Stable Video Diffusion XL instead of SV3D reaches 14.58° versus 14.48° for the main model.
• Grayscale input helps. Replacing the grayscale input with RGB drops performance by 0.79° on LUCES.
• Simpler light paths win. A spiral light path in which elevation decreases from 60° to 30° while rotating 360° around the z-axis produces an MAE of 17.60°, worse than the ring-light setup.
• Inference cost. RoSE takes 10.57 seconds per image, versus 0.31 s for NiRNE, 0.83 s for DSINE, 0.59–0.61 s for Lotus-D/G, 1.52 s for StableNormal, 93.73 s for Neural LightRig, and 101.11 s for GeoWizard. The shading-to-normal OLS step alone adds only 0.045 seconds per object.

Methodology in Plain English

1. Reframe the target. Instead of asking a network to output a normal map, the authors ask it to output a shading sequence: a stack of grayscale images of the same object as if it were lit one at a time by a set of fixed, non-coplanar parallel lights arranged on a latitude ring at 45° in the upper hemisphere. Each shading value is the clamped dot product of the surface normal with a light direction, so shading depends on geometry and light but not on material color.
2. Make it a video problem. Because a shading sequence is an ordered stack of frames, it can be treated as a video. RoSE therefore uses a video diffusion U-Net (built on pretrained SV3D weights) to generate the sequence from a single grayscale input image.
3. Condition the generator. The input is turned into grayscale and replicated to three channels, then used two ways: a CLIP encoder supplies a global semantic embedding injected by cross-attention, and a VAE encoder supplies a spatial latent that is concatenated with the noisy latent at each denoising step. The CLIP and VAE encoders are frozen.
4. Train with a standard diffusion objective. The model learns to predict the noise in the latent space, expressed as a z₀-reparameterization loss on the one-step denoised estimate, given the noisy latent and conditioning.
5. Recover normals analytically. Once the shading sequence exists, the normal map is obtained by solving an ordinary least-squares problem, N = (LᵀL)⁻¹LᵀSˢ, which is exact when the light matrix L is full rank. Because clamping truncates the shading values, the solver uses only shadings greater than zero as valid equations.
6. Train on richer data. MultiShade renders 90,546 filtered Objaverse objects from six viewpoints under one parallel light, one point light, or two HDR environment maps drawn from 780 real-world environments, at 576 × 576, with material augmentation applied with probability 0.5 (0.25 metallic, 0.25 non-metallic materials from MatSynth's 5,657 PBR materials), yielding roughly 3 million image-normal pairs and an extra 42,732 material-varied objects.
7. Fine-tune efficiently. Starting from 1.5B-parameter SV3D weights, only the first convolutional layer and the self-attention and cross-attention parameters are trained — 200M trainable parameters — for 80,000 steps with AdamW at a learning rate of 1 × 10⁻⁵, batch size 16, float16 precision, gradient clipping at norm 1.0, and a 9-frame, 576 × 576 output, taking roughly one day on eight NVIDIA H100 80GB GPUs.

Why This Matters

Impact on research. The paper attacks a failure mode that angular-error metrics alone do not capture: a normal map can score well while the surface reconstructed from it is wrong. By targeting geometry-sensitive supervision rather than appearance, it opens a direction — choosing training targets by their sensitivity to the quantity of interest — that applies beyond normals. It also connects video generative priors to inverse rendering, showing that large-scale video models carry usable lighting priors.

Real-world applications:

• Relighting and rendering. Accurate normals are needed to re-light captured objects and to composite them convincingly into new scenes.
• 3D reconstruction and digital content creation. Normals feed surface reconstruction pipelines used to turn photos into usable assets.
• Augmented reality. Placing virtual objects into real images requires consistent surface orientation so that lighting and shadows line up.
• Robotics. Geometry perception from a single camera supports manipulation and navigation where multi-light rigs or depth sensors are unavailable.

Industry relevance. The method is trained on a purpose-built synthetic dataset and generalizes to real benchmarks (DiLiGenT, LUCES) and to web images, which matters for content pipelines, e-commerce, gaming, and VFX, where casual single-photo capture is the practical constraint. However, its 10.57-second inference time per image is far slower than lightweight baselines like NiRNE (0.31 s), and the authors themselves flag the video-diffusion overhead as a limitation for real-time use.

Future Directions

• Reduce inference cost. The authors identify the computational overhead of video diffusion models as a barrier to real-time application, and the reported 10.57 s per image against sub-second baselines makes this the most immediate open problem.
• Handle extreme lighting. The method degrades when large regions of an object are insufficiently illuminated, producing poor shading quality and unreliable normals in those areas.
• Extend to transparent and semi-transparent objects. RoSE does not produce high-quality normal maps for these materials, and the authors call support for them an important direction.
• Move from objects to scenes. Evaluation is object-centric, focused on robustness to varying lights and reflectance; scene-centric monocular normal estimation remains untested.
• Resolve object-specific failures. The authors note that dataset variations changed accuracy on specific objects, with retraining Lotus-G on MultiShade hurting performance on House, leaving open the question of which data properties help or hurt particular geometries.

Target Audience

This paper suits researchers and graduate students working on inverse rendering, photometric stereo, intrinsic image decomposition, and monocular 3D reconstruction, as well as practitioners building relighting or asset-capture pipelines. It is most valuable to readers already comfortable with diffusion model internals and shading-based geometry, since the central argument rests on comparing training-target representations and on an OLS formulation of shading-to-normal inversion. Readers looking for a lightweight, deployable normal estimator should note the reported inference time and the authors' own stated limitations.

Authors’ abstract

Monocular normal estimation aims to estimate the normal map from a single RGB image of an object under arbitrary lights. Existing methods rely on deep models to directly predict normal maps. However, they often suffer from 3D misalignment: while the estimated normal maps may appear to have a correct appearance, the reconstructed surfaces often fail to align with the geometric details. We argue that this misalignment stems from the current paradigm: the model struggles to distinguish and reconstruct varying geometry represented in normal maps, as the differences in underlying geometry are reflected only through relatively subtle color variations. To address this issue, we propose a new paradigm that reformulates normal estimation as shading sequence estimation, where shading sequences are more sensitive to various geometric information. Building on this paradigm, we present RoSE, a method that leverages image-to-video generative models to predict shading sequences. The predicted shading sequences are then converted into normal maps by solving a simple ordinary least-squares problem. To enhance robustness and better handle complex objects, RoSE is trained on a synthetic dataset, MultiShade, with diverse shapes, materials, and light conditions. Experiments demonstrate that RoSE achieves state-of-the-art performance on real-world benchmark datasets for object-based monocular normal estimation.