VisibilitySplat: Ray-Traced Occlusion for Relighting with 3D Gaussian Splatting

AI summary

Overview

• Research area: Computer vision, specifically inverse rendering and relightable 3D scene reconstruction with 3D Gaussian Splatting (3DGS).
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, physically based rendering (PBR), microfacet BRDFs, environment-map lighting, and BVH-accelerated ray tracing.
• Scope (one sentence): The paper adds a sparse, BVH-accelerated visibility ray tracer to a standard rasterized 3DGS inverse-rendering pipeline so that transmittance-weighted occlusion from sampled light directions can constrain material and illumination decomposition, evaluated on the TensoIR synthetic dataset.

What This Paper Is About

Inverse rendering tries to recover a scene's geometry, surface materials, and lighting from ordinary photographs — essentially reversing how light produced the image. 3D Gaussian Splatting is fast and practical for rendering such scenes, but its screen-space alpha compositing gives weak or view-dependent information about shadows and mutual occlusion, which makes it hard to separate what is material from what is lighting. The authors keep the normal 3DGS rasterizer for training and visualization and bolt on a separate ray-traced "visibility oracle" that reports how much light actually reaches each Gaussian from sampled environment directions, testing whether that explicit signal improves relighting and material-illumination disentanglement.

Key Contributions

1. A sparse visibility oracle for 3DGS that estimates how much light reaches each Gaussian from sampled environment directions, implemented with a point-based ray tracer accelerated by a bounding volume hierarchy (BVH).
2. A two-stage, 3DGS-based inverse-rendering pipeline that pretrains Gaussians with per-splat normals in stage 1, then freezes geometry and optimizes BRDF materials jointly with an HDR environment map in stage 2.
3. A reference baseline on the TensoIR synthetic benchmark comparing against NeRF-based (NeRFactor, InvRender, TensoIR), tensor/neural methods (NVDiffrec), and 3DGS-based (GS-IR, IRGS) inverse-rendering methods for novel-view synthesis, albedo, normals, and relighting.
4. Ablations isolating the effect of explicit visibility and of the opacity-mask loss, including a PBR-only variant with neither component.

Main Findings

• Competitive but not best-in-class: On the TensoIR synthetic dataset, VisibilitySplat reaches relighting PSNR 30.538 with SSIM 0.941 and LPIPS 0.077, improving over NeRFactor (23.383), InvRender (23.973), NVDiffrec (19.880), GS-IR (24.374), and TensoIR (28.581). The paper states that IRGS achieves the best overall results on this benchmark, with relighting PSNR 30.646.
• Albedo and NVS results: Albedo PSNR is 30.305 (SSIM 0.934, LPIPS 0.101), slightly above GS-IR's 30.286 but below IRGS's 33.435; NVS PSNR is 34.807 (SSIM 0.967, LPIPS 0.053), below GS-IR (35.333), TensoIR (35.088), and IRGS (35.530).
• Geometry is the weak point: Normal estimation mean angular error (MAE) is 5.111, higher (worse) than IRGS (3.975), TensoIR (4.100), and GS-IR (4.948). The authors attribute their main error source to imperfect surface reconstruction rather than instability of the occlusion ray tracer, noting that inaccurate normals cause visibly curved shadow boundaries and local relighting artifacts.
• Visibility helps relighting, not NVS appearance: In the Lego ablation, removing visibility lowers relighting PSNR from 30.594 to 29.091 and drops albedo PSNR from 30.321 to 30.313, while the PBR-only baseline (no visibility, no opacity-mask loss) actually achieves the highest NVS PSNR (35.697) and lowest LPIPS (0.046) of the variants — which the authors attribute to albedo absorbing illumination information and over-fitting the training lighting, hurting decomposition.
• Opacity-mask loss matters for relighting: Removing it lowers relighting PSNR from 30.594 to 29.939 and albedo PSNR from 30.321 to 29.834, while albedo SSIM and LPIPS stay nearly unchanged, suggesting coarse albedo structure is preserved but radiometric consistency degrades. The paper interprets this via better-defined occluding geometry for the visibility rays.
• Ablation on a single model: The ablation numbers are reported on the Lego model only, not aggregated across the dataset.
• Runtime is not real-time: At the measured 800×800 evaluation resolution on an NVIDIA RTX A6000 GPU, final PBR relighting takes about 0.83 s/image; forward relighting spends about 55% of time on visibility queries and 44% on BRDF/environment-map integration. A full second-stage optimization iteration takes about 1.25 s/it.
• Visibility query cost: With a 16×32 environment map there are 512 candidate directions; after hemisphere culling and empty-space removal, the OptiX visibility pass takes approximately 436 ms at the evaluation resolution, about 127 MRays/s.
• Training budget: Stage 1 pretraining runs 30,000 iterations (about 15 minutes) and stage 2 runs 400 iterations (about 10 minutes) on an RTX A6000. Longer second-stage training degrades relighting quality because albedo begins to absorb illumination information.
• Convergence and rendering details: Rays are not depth-sorted for the visibility pass, because transmittance accumulates multiplicatively and multiplication commutes; rays early-terminate when transmittance falls to a threshold matched to the rasterizer's early-out. Depth is estimated as expected depth along each ray, which the authors found gave the best reconstruction quality and most stable downstream supervision versus inverse-depth variants.

Methodology in Plain English

The pipeline runs in two stages. In stage 1, a standard 3D Gaussian Splatting model is trained from multi-view images, extended with a learnable unit normal per Gaussian and an opacity-mask loss that pulls splats toward the object silhouette so the geometry is well defined. Depth is the expected depth of the Gaussian mixture along each ray, and pseudo-normals derived from image-space gradients of that depth supervise the normals, with an added total variation smoothing term borrowed from GS-IR.

In stage 2, the geometry (positions, covariance, opacity, normals) is frozen and the material parameters (albedo, roughness) plus a compact HDR equirectangular environment map are optimized under an L1 photometric loss. Shading uses a reduced Disney-style principled microfacet BRDF with the metallic term removed and a fixed dielectric base reflectance of 0.04, which reduces the number of learnable parameters in an under-constrained problem.

The distinctive piece is the visibility oracle. Rather than replacing the renderer, the authors run a point-based ray tracer: each Gaussian is enclosed in an anisotropically stretched icosahedron whose triangles go into a BVH, with the proxy scaled by the splat's opacity so nearly transparent splats get tighter bounds ("adaptive clamping"). Rays start at surface points taken from the expected-depth map. Internally, the tracer reuses the unmodified rasterizer by placing a one-pixel virtual camera at the ray origin with its optical axis along the ray direction, so blending stays consistent with the tile-based renderer. This gives transmittance-weighted visibility for sampled lighting directions, which plugs into the rendering equation as the V term.

Why This Matters

• Research impact: The work deliberately studies a direct, explicit visibility formulation rather than a learned or specialized approximation, providing a reference baseline for how much transmittance-weighted visibility contributes to 3DGS-based inverse rendering. It also demonstrates that a ray tracer can be bolted onto an existing 3DGS pipeline as a drop-in companion — pretrained point clouds and the existing rasterizer are reused — rather than requiring training under a different renderer as 3DGRT and IRGS-style approaches do.
• Real-world applications:
  • Relighting captured objects or products (e-commerce, virtual photography) by swapping in a new environment map after reconstruction.
  • Editing material properties such as albedo and roughness in scanned assets while keeping plausible shadows.
  • Content creation pipelines that need interactive previews via the standard rasterizer plus offline high-quality PBR renders.
  • Authoring of synthetic training data and scene assets where decomposed illumination and materials must be separated.
• Industry relevance: The approach preserves the practical 3DGS ecosystem (fast training, interactive visualization, existing tooling), which matters for production pipelines that already rely on Gaussian splats. The reported 0.83 s/image relighting time at 800×800 is not real-time, so current deployment value is in offline or near-interactive workflows; the authors state that real-time deployment would require optimizing both the visibility pass and the PBR integration.

Future Directions

• Indirect lighting: The illumination model accounts for direct lighting only; interreflection, subsurface scattering, and participating media are not modeled. Adding multi-bounce transport is an explicit suggestion.
• Tighter splatting-visibility coupling: The authors propose differentiable or cached variants of the visibility oracle to better integrate it with the rasterization loop.
• Geometric robustness: Higher normal MAE than competing methods points to improved surface reconstruction and normal estimation as a clear next step, since shadow boundary artifacts trace back to geometry.
• Broader and real-world evaluation: Extending to multi-object and full-scene settings, cluttered or dynamic scenes, imperfect segmentation, real captures, complex materials (glass, hair, translucency), and sensor noise — all listed as untested. Real-time deployment is also flagged as future work.

Target Audience

Researchers and graduate students working on inverse rendering, relightable neural and Gaussian representations, and physically based rendering, as well as graphics engineers evaluating whether a ray-traced visibility module is worth adding to an existing 3DGS pipeline. Readers should already understand 3D Gaussian Splatting, the rendering equation, microfacet BRDFs, and BVH ray traversal; the paper is a methods-and-benchmark contribution rather than an introductory treatment.

Authors’ abstract

We address inverse rendering with 3D Gaussian Splatting (3DGS) by introducing explicit visibility modeling for relighting as well as material and light decomposition. Our approach augments the standard rasterized 3DGS pipeline with a sparse visibility oracle that estimates transmittance-weighted occlusion for Gaussian primitives from sampled lighting directions. This oracle is implemented using a point-based ray tracer accelerated by a bounding volume hierarchy (BVH) and queried sparsely within the inverse-rendering objectives, while the original 3DGS loop is retained for training efficiency, interactive visualization, and tooling compatibility. Shading follows a physically based rendering (PBR) formulation with a principled microfacet bidirectional reflectance distribution function (BRDF) and low-resolution high-dynamic-range (HDR) environment maps. We evaluate our method on the TensoIR synthetic dataset against baselines based on neural radiance fields (NeRFs), tensor representations, and 3DGS. The results show competitive performance and improvements over several earlier baselines for novel-view synthesis (NVS), albedo reconstruction, and relighting. Ablations indicate that explicit visibility modeling is an important contributor to relighting and material-illumination disentanglement, while pure PBR supervision can favor appearance fitting in NVS. Overall, the findings suggest that sparse, point-based visibility queries effectively complement rasterized splatting for inverse rendering while preserving the practical advantages of the 3DGS ecosystem.