InfiniSplat: Implicit Gaussian Decoding for Large-Baseline Monocular View Synthesis

AI summary

Overview

Research area: Computer vision, specifically single-image feed-forward 3D Gaussian Splatting (3DGS) for novel view synthesis (NVS); listed under CCS "Computing methodologies — Image-based rendering."

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, monocular depth estimation, implicit neural representations, and DINO-style vision backbones.

Scope in one sentence: The paper proposes a single-image feed-forward 3DGS framework that replaces pixel-aligned Gaussian placement with geometry-guided, surface-aligned supports decoded by a query-conditioned implicit decoder, and evaluates it on four real-scene datasets in a cross-dataset zero-shot setting.

What This Paper Is About

Existing single-image feed-forward 3DGS methods predict Gaussians from fixed image-grid locations, so the resulting primitives behave more like locally expanded splats than a coherent 3D scene. Such renderings look good near the input view but show tearing, surface breaks, loose structure, and geometric distortion when the camera moves to large lateral baselines. InfiniSplat's goal is to make single-image 3DGS produce a surface-aligned Gaussian layout by placing Gaussian supports according to depth-induced local surface structure and predicting Gaussian attributes with a shared implicit decoder rather than at fixed pixel centers.

Key Contributions

1. A surface-aligned Gaussian representation for large-baseline NVS. A geometry-guided support sampling strategy places 2D supports according to depth-induced local surface structures instead of fixed image-grid locations, so predicted primitives can better assemble into coherent surfaces under large viewpoint changes.
2. Implicit Gaussian decoding for single-image 3DGS. Gaussian attribute prediction is formulated as query-conditioned implicit decoding over sampled supports and image features queried at those supports, allowing one shared prediction function to operate on support sets with different densities and spatial arrangements.
3. Two model variants under one framework. InfiniSplat-RGB takes only a single RGB image with DepthPro as the geometric scaffold; InfiniSplat-LiDAR additionally takes a sparse depth or LiDAR prompt and uses InfiniDepth-Metric for stronger geometric conditioning.
4. State-of-the-art cross-dataset zero-shot results. The paper reports the best PSNR, SSIM, and LPIPS among the compared single-image feed-forward baselines across ETH3D, ScanNet++, Tanks and Temples, and DL3DV, demonstrating zero-shot generalization from Hypersim indoor synthetic training to open-world scenes.

Main Findings

• Pixel alignment is identified as the bottleneck. The authors argue that a central limitation of current feed-forward single-image 3DGS is the pixel-aligned representation, in which Gaussians are generated from fixed image-grid locations; this leads to cracks, holes, and unstable geometry under large viewpoint changes.
• InfiniSplat-RGB leads all four RGB-only datasets. Table 1 reports InfiniSplat-RGB at PSNR/SSIM/LPIPS of 20.531/0.895/0.220 on ETH3D, 22.240/0.864/0.270 on ScanNet++, 17.118/0.695/0.306 on Tanks-and-Temples, and 21.685/0.772/0.310 on DL3DV, with a four-dataset average of 20.394/0.806/0.277.
• Average gains over the strongest pixel-aligned baseline (SHARP). InfiniSplat-RGB improves over SHARP by +1.919 PSNR and +0.048 SSIM while reducing LPIPS by 0.022 on the four-dataset average (SHARP averages 18.475/0.758/0.299).
• Average gains over other RGB-only baselines. Versus Flash3D, Flash3D-DepthPro, and LagerNVS, InfiniSplat-RGB improves average PSNR by +3.050, +4.082, and +2.934 respectively, and reduces average LPIPS by 0.051, 0.083, and 0.173.
• Per-dataset PSNR gains over SHARP. +1.485 on ETH3D, +1.439 on ScanNet++, +1.368 on Tanks-and-Temples, and +3.380 on DL3DV.
• Per-dataset PSNR gains over Flash3D. +1.985, +4.865, +1.166, and +4.183 on ETH3D, ScanNet++, Tanks-and-Temples, and DL3DV respectively.
• One weaker spot is acknowledged. The paper notes that the LPIPS improvement on Tanks-and-Temples is small, although InfiniSplat-RGB still maintains higher PSNR and SSIM there.
• The LiDAR-conditioned variant is far ahead in its setting. Against ADGaussian, InfiniSplat-LiDAR reports 25.880/0.946/0.151 on ETH3D, 24.148/0.896/0.237 on ScanNet++, and 17.617/0.712/0.287 on Tanks-and-Temples, averaging 22.548/0.851/0.225 versus ADGaussian's 12.249/0.639/0.418. Neither method is evaluated on DL3DV (marked "–" in the table).
• Qualitative behavior. Figures 3 and 4 report that InfiniSplat-RGB produces fewer visible cracks and more coherent scene structures than feed-forward baselines, especially on large planar regions and object boundaries, and yields cleaner normal maps with fewer fragmented regions than SHARP under side-view rendering.
• Zero-shot transfer works. All evaluation datasets are excluded from training; both variants are trained on Hypersim, and Figure 6 reports coherent reconstructions on in-the-wild images with complex geometry and varying lighting.

Methodology in Plain English

The pipeline has two parallel preprocessing branches and a two-stage core.

Preprocessing. A frozen monocular depth model produces a dense depth map and estimated camera intrinsics. In parallel, a trainable dual-branch encoder produces a DINO semantic feature map and a CNN texture feature map, giving the system both high-level semantics and low-level appearance cues.

Stage one: geometry-guided sampling. Instead of assuming Gaussians sit on a regular pixel grid, the method back-projects the depth map into 3D and measures the 3D area of small local triangles formed by neighboring pixels. Triangles whose vertices differ sharply in depth are discarded so that a patch does not span a foreground-background discontinuity. The remaining triangle areas are normalized into sampling weights, so larger or more strongly slanted surface patches receive proportionally more sampled supports. Query coordinates are then drawn inside the corresponding 2D triangle regions. Each sampled coordinate becomes a "base Gaussian," initialized by bilinearly sampling the input image and depth map at that coordinate.

Stage two: implicit Gaussian decoding. For each support, a shared MLP bilinearly queries the DINO and CNN feature maps at that coordinate. The CNN feature is projected to the DINO feature dimension, then a channel-wise sigmoid gate adaptively fuses semantics and texture into one descriptor. An MLP maps that descriptor to a 14-dimensional bounded residual covering image-plane displacement, depth displacement, scale, quaternion rotation, color, and opacity. Applying these residuals to the base Gaussians yields the final Gaussians. Position updates are expressed as constrained image-plane and depth displacements that are then back-projected, and scale, color, and opacity updates are applied in the unconstrained space of their activations (following SHARP), keeping values valid. Rotation is normalized in the subsequent Gaussian transformation.

Why the two stages are coupled. Sampling escapes the pixel grid but produces an irregular layout that per-pixel regression cannot handle; implicit decoding handles arbitrary layouts but would have no surface awareness without the geometric scaffold. The paper states each stage necessitates the other.

Training. Both variants train on Hypersim with one context view and three target views per sample, supervising novel-view renderings against ground truth. The objective combines an L1 RGB reconstruction loss, a perceptual loss combining VGG feature-space distance and Gram-matrix distance (with the Gram term applied only to target views), and Gaussian regularization that bounds log-scale values and encourages scale and opacity to vary smoothly between neighboring sampled supports. Geometry priors stay frozen; the dual-branch image encoder, fusion module, and Gaussian MLP are trained.

Evaluation protocol. Cross-dataset zero-shot evaluation on ETH3D, ScanNet++, Tanks and Temples, and DL3DV, none used for training. Source-target pairs come from trajectory windows of 10 consecutive views per scene or sequence, require more than 60% source-target frustum overlap, and filter out excessive rotations and backward-facing views. Each dataset contributes 512 pairs — 128 in each of four baseline ranges: [0, 0.5), [0.5, 1), [1, 2), and [2, ∞) meters — chosen with a fixed random seed and a scene-level sampling cap. Metrics are computed after aligning to dataset-specific target sizes: 2016 × 1344 for ETH3D, 1536 × 1024 for ScanNet++, and 1920 × 1080 for Tanks-and-Temples and DL3DV. A frustum-visible mask derived from source-view geometry is applied identically to all methods.

Baselines and implementation details. RGB-only comparisons are SHARP, official Flash3D, Flash3D-DepthPro (Flash3D with its depth model replaced by DepthPro, as a control), LVSM, and LagerNVS; the RGB+DepthSensor comparison is ADGaussian. InfiniSplat-RGB and SHARP both use DepthPro. Each input image samples 1.5M Gaussian supports. The image branch uses a DINOv3 ViT-L/16 backbone and a 128-channel CNN (7 × 7 stride-2 convolution, eight residual blocks with 3 × 3 kernels, 3 × 3 fusion convolution, 1 × 1 projection to C = 128). Loss weights are λ_rgb = 1, λ_perc = 1, λ_reg = 0.1, with γ = 10 for the Gram term and λ_scale = 0.1, λ_smooth = 0.1 for regularization; log-scale bounds are [−8, −3]. Both variants train on 8 NVIDIA H20 GPUs for about 100K steps with AdamW, learning rate 5 × 10⁻⁵, and per-GPU batch size 1. For the LiDAR variant, 1,500 sparse depth samples are drawn per source image as prompts for InfiniDepth-Metric.

Why This Matters

Impact on research. The paper reframes a design choice — where Gaussians come from — as the key obstacle to large-baseline single-image NVS, and shows a concrete alternative that keeps a feed-forward, explicit, real-time-renderable output. It also connects feed-forward 3DGS to implicit decoding ideas from continuous image and depth representations (LIIF, InfiniDepth), and argues that freely learned query tokens (as in C3G and TokenGS) lack geometric anchors in the under-constrained single-image setting.

Real-world applications:

• Interactive spatial photo browsing, where a single photograph becomes a walkable 3D scene.
• Natural head motion in AR/VR, where view-dependent structure must stay coherent as the user moves.
• Spatial content display on handheld devices, where a renderable 3D representation must be produced without multi-view capture or per-scene optimization.
• Depth-sensor-equipped capture (phones or LiDAR devices), where InfiniSplat-LiDAR uses sparse depth prompts to substantially improve rendering quality.

Industry relevance. The method removes the multi-view capture and per-scene optimization cost that limits 3D content creation, and its LiDAR variant suggests a path for depth-sensing hardware. Reported industry involvement comes from the Udeer.ai co-authors.

Future Directions

• Ablation results are not present in the provided content. The paper states that ablations use the same training setting and evaluation protocol and Section 4.2 ends mid-sentence in the supplied text, so the contributions of geometry-guided sampling versus implicit decoding are not quantified here.
• Efficiency is not reported. The provided content gives no runtime, memory, or Gaussian-count trade-off analysis, despite 1.5M supports per image and a feed-forward claim — a natural next step is measuring inference cost on the devices the motivation describes.
• Extending beyond a single still image. The setting is one RGB image (or one image plus sparse depth); extending the surface-aligned support formulation to video or multi-view input is an open question, particularly since the paper argues multi-view cues are what make learned query tokens effective.
• Closing the remaining quality gap. LPIPS on Tanks-and-Temples improves only marginally over SHARP, and the LiDAR variant is not evaluated on DL3DV, leaving questions about robustness across scene types and about how much the LiDAR prior is contributing.

Target Audience

Researchers and practitioners working on novel view synthesis, 3D Gaussian Splatting, single-image 3D reconstruction, and feed-forward scene generation, as well as engineers building AR/VR, mobile spatial content, or depth-sensor-based capture pipelines. Readers should be comfortable with Gaussian Splatting rendering, monocular depth priors, and standard NVS metrics (PSNR, SSIM, LPIPS).

Authors’ abstract

Single-image feed-forward 3D Gaussian Splatting (3DGS) aims to directly generate a renderable 3D scene representation from one input image, avoiding the cost of multi-view capture and per-scene optimization. However, existing methods are often constrained by a pixel-aligned representation, where Gaussians are predicted from fixed image-grid locations. Such pixel-aligned primitives can produce promising nearby-view renderings, but they remain weakly coupled to underlying scene surfaces and struggle to preserve coherent structures under large viewpoint shifts. We present InfiniSplat, a feed-forward single-image 3DGS framework that moves from a pixel-aligned representation toward a surface-aligned representation. InfiniSplat constructs this representation by first using geometry-guided sampling to place 2D supports according to depth-induced local surface structure, and then applying a query-conditioned implicit decoder to predict Gaussian attributes from the image features queried at these supports.By grounding support locations in geometry while decoupling Gaussian prediction from fixed pixel centers, InfiniSplat produces Gaussian layouts that better follow scene surfaces and reduce scattered primitives caused by grid discretization.Across multiple cross-dataset NVS evaluations, InfiniSplat achieves state-of-the-art performance compared with single-image feed-forward baselines, and demonstrates zero-shot generalization from Hypersim indoor synthetic training to complex open-world scenes.Project page: https://zju3dv.github.io/InfiniSplat.