Intrinsic Image Fusion for Multi-View 3D Material Reconstruction

AI summary

Overview

Research area: Computer vision / computer graphics — inverse rendering, intrinsic image decomposition, and physically based rendering (PBR) material reconstruction for room-scale 3D scenes.

Technical level: Advanced. The paper assumes familiarity with path tracing, Monte-Carlo rendering noise, BRDF models (Cook-Torrance), diffusion-based material estimators, and distribution matching / KL divergence.

Scope: A probabilistic inverse-rendering framework that distills inconsistent single-view material predictions into consistent 3D PBR textures by fitting an explicit low-dimensional parametric distribution to them.

What This Paper Is About

Recovering physically based materials (albedo, roughness, metallic, plus emission and camera response) from multi-view images is heavily underconstrained, and existing methods rely on analysis-by-synthesis with path tracing, which is expensive and noisy — that noise propagates into the optimization and produces baked-in lighting and biased specular parameters. The goal is to reconstruct sharp, clean, physically grounded PBR textures for entire room-scale 3D scenes by injecting strong single-view generative priors as constraints rather than letting noisy rendering gradients do all the work. The core tension the paper attacks: a diffusion-based estimator such as RGBX produces detailed but inconsistent predictions within a view and across views, so naive aggregation into 3D yields seams and blurred-out detail.

Key Contributions

1. Explicit parametric distribution over plausible materials. The authors model the solution space of a single-view material estimator with a per-image-per-object Laplacian distribution, built from a linear (affine) basis applied to each prediction plus an ambiguity-invariant base pattern and a median-deviation scale estimate. This drastically reduces the number of free parameters that path-traced optimization must handle, limiting the impact of rendering noise.
2. Consistent distribution matching for cross-view aggregation. Instead of averaging single-view predictions, they aggregate them into a 3D-consistent parametric distributional texture using KL-divergence-based distribution matching with soft per-view prediction selection and confidence-based soft multi-view inlier sets, which lets the method use the most consistent predictions rather than blending them.
3. Hybrid optimizer: learned prior plus inverse path tracing. After aggregation, they run factorized inverse path tracing (FIPT-style, with LDR handling following IRIS) to optimize only the remaining per-object affine parameters, plus an emission map and camera response function, yielding a fully re-renderable and relightable decomposition.
4. Demonstrated gains in material disentanglement. Quantitative and qualitative improvements over NeILF++, FIPT, and IRIS on synthetic and real ScanNet++ scenes.

Main Findings

• Quantitative superiority on synthetic scenes (Table 1). Averaged over all views of four synthetic scenes: IIF (Ours) achieves albedo PSNR 20.72, SSIM 0.846, LPIPS 0.201, albedo L2 0.028, roughness L2 0.007, emission L2 0.384. Baselines: NeILF++ 13.18 / 0.733 / 0.375 / 0.103 / 0.047 / N/A; FIPT 10.63 / 0.661 / 0.403 / 0.110 / 0.006 / 2.208; IRIS 15.86 / 0.735 / 0.307 / 0.056 / 0.040 / 2.046. The paper describes the metallic results as "competitive" on the sparse metallic maps rather than dominant.
• Naive aggregation fails in two distinct ways (Figure 2). RGBX predictions are inconsistent within a single view and across views; naive use for 3D texturing produces visible continuity artifacts and blurred-out details.
• Parametric modeling drives most of the gain (Table 2). On the synthetic scenes, RGBX scores albedo PSNR 13.11 / SSIM 0.787 / LPIPS 0.228; per-object mean aggregation 13.21 / 0.641 / 0.563; per-texel mean 13.43 / 0.753 / 0.42; the parametric model alone jumps to 29.53 / 0.909 / 0.176; the full method with distribution matching reaches 30.79 / 0.931 / 0.160. Per-object and per-texel aggregation improve consistency and PSNR but fail to preserve fine patterns (SSIM, LPIPS).
• More 2D predictions monotonically help (Table 3), with no oversmoothing. Albedo PSNR rises from 29.62 with 1 prediction, to 29.77 (2), 30.37 (4), 30.72 (8), and 30.79 (16); SSIM from 0.908 to 0.931; LPIPS falls from 0.177 to 0.160. Reported roughness L2 (×1000) goes from 0.817 to 0.786 and metallic L2 (×1000) from 0.134 to 0.134.
• Model complexity matters (Figure 7). A simpler per-image parametric model (no object segmentation required) yields stable optimization but underfits, producing an average color tone across objects; the per-image-per-object model captures detailed textures.
• Real-world robustness. On ScanNet++ scenes with laser-scan geometry containing holes and unreconstructed reflective surfaces such as windows, prior methods show projected contour artifacts (for example, chair borders visible on walls), while the more constrained objective maintains clean predictions.
• Temperature annealing is a real trade-off (Figure 9). Lower temperature values drive quick convergence to hard assignment but risk local optima; too high a value keeps soft assignment and causes oversmoothing. Assignments converge from entropy 0.5 (naive averaging) to 0 (mode selection) at τ_anneal = 0.85.
• Geometry dependence (Figure 10, Figure 11). Estimated albedo is comparable with laser scan and photometric reconstruction geometry (on ScanNet++ scene 7e09430da7), but geometric errors — missing thin structures, transparent objects, floating artifacts — propagate into the reconstruction. Consistently wrong predictions can also leak through; the book in Figure 2 is shown white and grey due to incorrect predictions.
• Parameter reduction is the mechanism. The base texture uses a neural hashgrid with 32 levels, 2 features per level, and 2^19 hashmap size (28M parameters, optimized only during aggregation). Inverse path tracing optimizes only per-object affine transformations, O × 3 × 4, totaling 1092 parameters for the kitchen scene.

Methodology in Plain English

The pipeline takes posed multi-view images plus reconstructed geometry as input. First, a monocular probabilistic material estimator (RGBX) generates K = 16 candidate decompositions — albedo, roughness, metallic — for every view. Because these candidates disagree, the authors do not average them. Instead, for each object in each prediction they fit a small affine transformation (a linear rescaling and offset) that absorbs the energy/scale ambiguity between lighting and reflectance; the transformed predictions become "base" textures.

They then build a per-image-per-object Laplacian distribution from these bases: learnable assignment logits produce a softmax-weighted mixture over the K predictions to define the distribution's location, and the median deviation of predictions from that mixture defines its scale. On the 3D side, an InstantNGP-based BRDF network queried at randomly sampled 3D points predicts material means and per-channel uncertainty scales, defining a second Laplacian distribution. Distribution matching minimizes the KL divergence between the 2D reference distributions and the 3D predicted distributions, with an additional label loss (an L2 error turned into soft targets) to regularize the assignment logits and an identity regularizer on the affine transformations. This phase runs for 10 epochs and takes about 5 minutes.

Finally, inverse path tracing grounds the result physically. Rendering uses the Cook-Torrance microfacet model, with diffuse term k_d = (1−m)·a and specular k_s = 0.04·(1−m) + a·m. Optimization alternates: first lighting (per-triangle uniform emission, initialized by thresholding aggregated radiance at t = 0.99, then pruned below 5 percent of maximum intensity), then caching of light transport into diffuse and specular shading maps, then fitting the per-object affine parameters together with CRF parameters for LDR inputs. This follows FIPT using Mitsuba 3 and converges in about 55 minutes on a single A6000 GPU. For real ScanNet++ scenes, an environment map of 16 × 32 resolution is added to account for missing geometry and outside emission, with emissive pixels filtered at t = 0.85.

Why This Matters

The work shows that the dominant source of error in inverse rendering — Monte-Carlo rendering noise backpropagating into materials — can be suppressed not by better rendering but by shrinking the optimization to a low-dimensional manifold anchored on a learned prior. That reframes how single-view generative priors and analysis-by-synthesis should be combined: not as a regularizer on top of full-texture optimization, but as the space in which optimization happens.

Real-world applications:

• Relighting and virtual production. Clean albedo/roughness/metallic with an emission map and CRF allows re-rendering scenes under new lighting; the paper demonstrates inserting emissive spheres over smooth trajectories.
• Material editing and content creation. Artists can modify extracted PBR textures without the baked-in illumination that plagues earlier decompositions.
• Virtual object insertion. Physically plausible geometry-plus-materials support compositing new objects with correct shading and specular response.
• Scene digitization for AR/VR and digital twins. Room-scale indoor reconstruction from captured images, demonstrated on ScanNet++ scenes.

Industry relevance is direct for gaming, film and VFX, architectural visualization, e-commerce and real-estate virtual tours, and any pipeline converting captured real spaces into renderable assets. The required inputs (posed images, reconstructed geometry, instance segmentation) match what existing photogrammetry and scanning pipelines already produce.

Future Directions

• Joint geometry and material optimization. The method currently relies on fixed geometry and inherits mesh artifacts; the authors note jointly optimizing geometry is possible and an interesting direction.
• A more compact formulation. Sampling multiple material predictions per view is computationally costly; incorporating the pretrained prior directly into the optimization process could yield a more compact approach.
• Using prediction uncertainty. The method relies on the quality of pretrained material estimators; incorporating prediction uncertainties could allow ignoring incorrect predictions.
• Robustness to reconstruction failures. Thin structures, transparent objects, and consistently biased predictions still leak into the result, indicating a need for handling these cases explicitly.

Target Audience

Researchers and graduate students in inverse rendering, intrinsic image decomposition, and neural rendering; graphics engineers building relighting or asset-capture pipelines; and practitioners working with diffusion-based material estimators who need their predictions made multi-view consistent and physically grounded. Readers should already be comfortable with PBR BRDFs, path tracing, and variational/optimization formulations.

Authors’ abstract

We introduce Intrinsic Image Fusion, a method that reconstructs high-quality physically based materials from multi-view images. Material reconstruction is highly underconstrained and typically relies on analysis-by-synthesis, which requires expensive and noisy path tracing. To better constrain the optimization, we incorporate single-view priors into the reconstruction process. We leverage a diffusion-based material estimator that produces multiple, but often inconsistent, candidate decompositions per view. To reduce the inconsistency, we fit an explicit low-dimensional parametric function to the predictions. We then propose a robust optimization framework using soft per-view prediction selection together with confidence-based soft multi-view inlier set to fuse the most consistent predictions of the most confident views into a consistent parametric material space. Finally, we use inverse path tracing to optimize for the low-dimensional parameters. Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes, producing sharp and clean reconstructions suitable for high-quality relighting.