IBGS: Image-Based Gaussian Splatting

AI summary

Overview

• Research area: Computer vision, specifically novel view synthesis (NVS) and 3D scene reconstruction using 3D Gaussian Splatting (3DGS) and image-based rendering.
• Technical level: Advanced. The paper assumes familiarity with 3DGS rasterization, spherical harmonics (SH), alpha blending, volume rendering notation, and camera exposure models.
• Scope: The paper introduces IBGS, an image-based Gaussian Splatting pipeline that adds a learned color residual (predicted from neighboring training images) on top of standard Gaussian-rasterized base color, plus an exposure correction module, and evaluates it on Tanks and Temples, Deep Blending, Mip-NeRF360, and the Shiny dataset.

What This Paper Is About

3DGS renders novel views quickly and with high quality, but each Gaussian stores only a single color per viewpoint using low-degree spherical harmonics, so it struggles to reproduce high-frequency texture and view-dependent effects such as specular highlights and reflections. Prior fixes either learn a global texture map (which fails on complex multi-object scenes) or per-Gaussian texture maps (which grow storage quadratically with texture resolution and still do not handle view dependence). The goal of this work is to recover fine detail and view-dependent color by borrowing information from neighboring training images while keeping the storage footprint small.

Key Contributions

1. An image-based Gaussian Splatting pipeline that combines a base color from standard 3DGS SH rendering with a color residual inferred from neighboring training images, capturing both high-frequency detail and view-dependent effects. The authors state this is, to the best of their knowledge, the first image-based Gaussian splatting method that maintains fast rendering.
2. A color residual prediction module, consisting of a PointNet-style per-pixel feature extractor (two linear layers of 32 output dimension with ReLU, followed by max-pooling over source views) and a nine-layer convolutional decoder with 3×3 kernels, which produces a residual color map added to the Gaussian-rasterized image.
3. An exposure correction strategy that estimates an affine transformation matrix for a novel view by mimicking the exposure setting of the nearest source view, generalizing exposure correction to images rendered at any viewpoint (unlike prior methods that only correct training views).
4. A multi-view color consistency loss (using warped colors from source views) plus a normal consistency loss, which the authors report yields more precise Gaussian parameters, allowing more low-opacity Gaussians to be pruned while maintaining rendering quality.

Main Findings

• Best image quality across standard benchmarks: On Mip-NeRF360, Tanks and Temples (TNT), and Deep Blending, the method achieves PSNR gains of at least 0.64, 0.56, and 0.59 dB respectively over prior methods. Reported values are Mip-NeRF360 28.33 PSNR / 0.837 SSIM / 0.186 LPIPS, TNT 24.84 / 0.869 / 0.148, and Deep Blending 30.12 / 0.912 / 0.237.
• Large reductions in Gaussians and storage: On Mip-NeRF360 and TNT, the method reduces the number of Gaussians and storage by at least 62% and 42% compared to existing Gaussian Splatting methods (3DGS, SuperGauss, TexturedGauss) while still outperforming them. On Mip-NeRF360 it uses 1.59M Gaussians / 291 MB, versus 3DGS at 3.22M / 764 MB and SuperGauss at 3.04M / 1021 MB.
• Storage advantage over per-Gaussian texture maps: On Deep Blending the method uses slightly more Gaussians than TexturedGauss but consumes 70% less storage, because TexturedGauss must store texture maps for all Gaussians whereas this method stores source images.
• Strong gains on view-dependent effects: On three Shiny dataset scenes (Guitars/specular highlight, Lab/reflection, CD/diffraction), the method achieves at least a 5.22 dB PSNR gain over 3DGS and SuperGauss with fewer Gaussians. Examples: Guitars 35.65 PSNR, 0.18M Gaussians, 46 MB versus 3DGS 29.37, 0.41M, 97 MB; Lab 35.06 versus 3DGS 29.17; CD 35.23 versus 3DGS 29.10.
• Ablation: the residual module is essential. Removing the residual ("Base color only") drops TNT to 23.06 PSNR / 0.836 SSIM / 0.202 LPIPS and Mip-NeRF360 to 27.08 / 0.814 / 0.227, versus the full model at 24.84 / 0.869 / 0.148 and 28.33 / 0.837 / 0.186.
• Ablation: other components matter but less. Without the color consistency loss, TNT is 24.70 / 0.866 / 0.152 and Mip-NeRF360 28.31 / 0.833 / 0.192. Using source colors directly instead of the difference from the base color gives TNT 24.61 / 0.867 / 0.150 and Mip-NeRF360 28.21 / 0.837 / 0.187. Without exposure correction, TNT is 24.28 / 0.866 / 0.152 (Mip-NeRF360 not reported for this ablation).
• Robustness to aggressive pruning: With a high opacity threshold of 0.05, 3DGS degrades (e.g., Mip-NeRF360 27.51 PSNR versus 27.69 at threshold 0.005), while the proposed method retains most quality (28.33 at 0.05 versus 28.42 at 0.005) with 1.59M Gaussians and 291 MB.
• Qualitative decomposition: In zoomed-in results on the Guitars scene, the method decomposes color into a diffuse component (base color) and a specular component (predicted residual); 3DGS and SuperGaussian fail to capture specular highlights and reflections.
• Reported limitations: The method may struggle in sparse-view settings where dense pixel correspondences for residual prediction are hard to obtain, and it achieves lower rendering speed and higher runtime memory than 3DGS.

Methodology in Plain English

• Two-part color model. Every rendered pixel is the sum of a base color and a residual. The base color comes from ordinary 3DGS rasterization using second-degree spherical harmonics (SH degree set to l = 2). The residual is learned and supplies the fine texture and view-specific effects that SH cannot represent.
• Borrowing pixels from nearby views. For a target pixel, the authors shoot a ray and intersect it with the planes defined by candidate Gaussians (each Gaussian now also stores a learnable normal vector). Following 2DGS, only the K = 4 median intersection points — those near the surface where accumulated transmittance is close to 0.5 — are projected into the neighboring source images and sampled with bilinear interpolation. Blending these samples with the same weights used for Gaussian colors produces a "warped color" per source view.
• Residual computation. For each source view, the warped color minus the base color gives a color feature, and the camera position/orientation difference gives a camera feature. A PointNet-style extractor embeds each view's features into 32 dimensions, max-pooling aggregates an arbitrary number of views, and a nine-layer 3×3-kernel CNN decoder predicts the residual map from the base image, the ray-direction map, and the aggregated features. The final image is base image plus residual map.
• Source view selection. From the S = 4 nearest candidate views, only views passing a depth-consistency check (relative depth error ≤ τ = 0.001) are used, yielding up to M = 3 visible source views.
• Exposure correction. Instead of learning one affine transform per training view, the method solves a least-squares problem for an affine 3×4 matrix that maps the base color to the warped color of the nearest source view, then applies that matrix to correct the exposure of rendered images at any viewpoint. If enabled, the corrected color replaces the raw rendered color in feature computation, residual prediction, and the final image.
• Training. The loss combines a color rendering loss on both the base and final images (with weight γ, starting at 1 and decreasing to 0.5 over the last 20,000 of 30,000 iterations; L1/SSIM mix with β = 0.8), a multi-view color consistency loss, and a normal consistency loss. Photometric and normal losses are activated only after the first 7,000 iterations, with λ₁ = 0.3 and λ₂ = 0.03. Gaussians with opacity below 0.05 are pruned. Training uses the Adam optimizer with initial learning rate 0.001, halved at iterations 18,000 and 25,000, on a single RTX 4090 GPU. Every 8th image is held out for evaluation.

Why This Matters

• Impact on research: The paper shows that high-frequency detail and view-dependent appearance can be recovered without expanding per-Gaussian storage, offering an alternative axis of improvement to texture-map approaches, higher-degree SH, or Gaussian compression. Because the authors note the method is orthogonal to geometry, densification, and compression work, it can be combined with those lines of research.
• Real-world applications (grounded in the paper's discussion and NVS setting):
  • 3D reconstruction pipelines, which the authors explicitly list as a downstream application.
  • Controllable human modeling, also explicitly named as a downstream application.
  • Photorealistic novel view synthesis for scenes with reflective or specular surfaces, the target capability demonstrated on the Shiny dataset.
  • Rendering at arbitrary novel viewpoints with corrected exposure, enabled by the exposure correction strategy that generalizes beyond training views.
• Industry relevance: The method runs on a single consumer-grade RTX 4090 GPU, uses a lightweight residual network, and reports substantially lower memory footprints than 3DGS and SuperGauss, which matters for deployment where storage and GPU memory are constrained. The involvement of an NVIDIA affiliation and the explicit framing of fast rendering as a design goal point toward graphics and rendering pipelines.
• Broader impacts and ethics (as stated): The authors state there are no immediate societal impacts, but caution that downstream uses such as 3D reconstruction or controllable human modeling could be abused for unauthorized reconstructions or identity fraud.

Future Directions

• Sparse-view robustness: The authors identify sparse-view settings as a failure case, since residual prediction depends on dense pixel correspondences between neighboring images.
• Rendering speed and runtime memory: The authors report lower rendering speed and higher runtime memory than 3DGS due to the additional computation, and defer details to the supplementary material; closing this efficiency gap is a natural next step.
• Combination with orthogonal techniques: The paper states the method can be integrated with geometry-improving work (flat-Gaussian constraints), compression/quantization, improved densification strategies, and Markov Chain Monte Carlo-based densification. Testing these combinations is a logical extension.
• Extending exposure correction: The exposure correction is demonstrated on the TNT dataset and the method is described as mimicking the nearest source view; whether this generalizes to more complex or rapidly varying lighting conditions is left open.

Target Audience

Researchers and practitioners working on novel view synthesis, 3D Gaussian Splatting, and neural rendering who already understand 3DGS fundamentals; engineers building photorealistic reconstruction or rendering systems with GPU memory and storage constraints; and readers interested in hybridizing image-based rendering with primitive-based scene representations.

Authors’ abstract

3D Gaussian Splatting (3DGS) has recently emerged as a fast, high-quality method for novel view synthesis (NVS). However, its use of low-degree spherical harmonics limits its ability to capture spatially varying color and view-dependent effects such as specular highlights. Existing works augment Gaussians with either a global texture map, which struggles with complex scenes, or per-Gaussian texture maps, which introduces high storage overhead. We propose Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling. Specifically, we model each pixel color as a combination of a base color from standard 3DGS rendering and a learned residual inferred from neighboring training images. This promotes accurate surface alignment and enables rendering images of high-frequency details and accurate view-dependent effects. Experiments on standard NVS benchmarks show that our method significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.