InfiniSplat v2: Analyzing and Improving Feed-Forward 3DGS from Any Views

AI summary

Overview

Research area: Computer Vision — feed-forward 3D Gaussian Splatting (3DGS) and novel view synthesis from sparse, unposed or posed image sets.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, multi-view geometry foundation models (DUSt3R/VGGT/Depth Anything 3), DPT-style decoders, and perceptual metrics (LPIPS, DISTS).

Scope (one sentence): The paper diagnoses why feed-forward Gaussian reconstruction on top of predicted multi-view geometry produces blur, ghosting, and flying-point artifacts, and proposes architectural and training fixes (frozen geometry branch, GT-camera scale alignment, large-angle target supervision, plus a learned render-feedback refiner) that lower mean LPIPS and DISTS across six datasets in both pose-conditioned and pose-free settings.

What This Paper Is About

Feed-forward Gaussian methods build a renderable 3D scene directly from input images without per-scene optimization, usually by back-projecting depth predicted by a geometry foundation model (here, Depth Anything 3, or DA3) into a point cloud and then predicting Gaussian attributes for those points. The predicted point clouds are imperfect — they contain spatially isolated clusters and clusters that disagree across views — so rendering from large viewpoint changes still shows blur, ghosting, and missing detail, as the authors show for DA3-GS and other baselines. The goal of InfiniSplat v2 is to learn Gaussian attributes on top of this imperfect geometry such that the resulting Gaussian scene renders high-quality novel views even at extreme viewpoints.

Key Contributions

1. Architectural decoupling of geometry and appearance. InfiniSplat v2 uses two independently parameterized branches initialized from the same pretrained DA3 weights: a frozen geometry branch (Φ_geo, including its depth and camera heads) that predicts depth and cameras, and a trainable attribute branch (Φ_attr) in which only the last K Transformer blocks plus a DPT Gaussian decoder are fine-tuned. Gaussian centers are fixed at back-projected point positions with no learned positional offsets.

2. Scale-aligned GT-camera training. Instead of back-projecting with predicted poses, the method back-projects normalized predicted depth with GT intrinsics and GT poses whose translations are rescaled by a scale s_C obtained from an Umeyama similarity fit between GT and normalized predicted input-camera centers. The same scale is applied to input and target cameras so their GT relative poses are preserved in a common reconstruction frame.

3. Large-angle target view supervision. Target cameras are sampled with a bias toward large angular separation a(v, C) between their optical axis and the nearest input-view optical axis (the large-angle pool is chosen with probability 0.7, the small-angle pool with 0.3), which constrains Gaussian scale and orientation through projections from different directions.

4. A learned rendering-feedback refiner. Starting from the pruned initial scene, a shared recurrent KNN Transformer updates Gaussian attributes using input-view rendering gradients (detached, fed as inputs rather than used as direct gradient-descent steps) while centers and counts stay fixed; Ours* uses three such refinement steps.

5. An empirical analysis of learned opacity. The authors show that opacity trained with only image reconstruction supervision preferentially suppresses geometrically inconsistent points, linking opacity to geometric reliability.

Main Findings

• State-of-the-art mean perceptual scores in both settings. Across six evaluation datasets (DL3DV, ETH3D, NRGBD, Mip-NeRF 360, RE10K, DTU), the feed-forward outputs of the same trained model achieve the lowest mean LPIPS and DISTS among the evaluated methods in both pose-conditioned and pose-free settings.

• Pose-conditioned gains. Mean LPIPS falls to 0.2472 (vs. 0.3333 for ReSplat, the strongest evaluated baseline on that metric) and mean DISTS to 0.1163 (vs. 0.1734 for WorldMirror 2.0) — decreases of 25.8% and 32.9% respectively. With three refinement steps (Ours*) these become 0.2392 LPIPS and 0.1159 DISTS.

• Pose-free gains. Mean LPIPS of 0.3026 and DISTS of 0.1341, corresponding to 19.5% and 30.7% reductions relative to WorldMirror 2.0 (0.3757 and 0.1935). Refinement gives 0.2970 LPIPS and 0.1379 DISTS.

• Freezing geometry matters most among the ablations. On ETH3D and DTU, removing geometry freezing (w/o G) worsens LPIPS from 0.4339/0.2070 to 0.5204/0.2695; the authors report that freezing geometry instead of fine-tuning reduces LPIPS by 16.6% and 23.2% on the two datasets. Learned positional offsets (w/ O) are similarly harmful (ETH3D 0.4998, DTU 0.2826), producing local blur and depth distortions in an NRGBD example.

• GT-camera back-projection is critical. Dropping it (w/o C) yields ETH3D 0.5017 and DTU 0.3237 LPIPS; GT-camera back-projection reduces DTU LPIPS by 36.1% relative to predicted cameras.

• Fine-tuning the attribute backbone helps unevenly. Fine-tuning the last K = 10 blocks versus K = 0 (w/o A) reduces LPIPS by 1.1% on ETH3D and 20.8% on DTU, with lower DISTS on both.

• Large-angle supervision beats local interpolation views. w/o V gives ETH3D 0.4533 and DTU 0.2295 LPIPS versus 0.4339 and 0.2070 for the full model. The sampled target views have a mean minimum angle to input views of 17.97° versus 12.34° for local sampling.

• Opacity encodes geometric consistency without explicit supervision. Using GT-depth labels on held-out Hypersim validation scenes, pruning at opacity threshold 0.01 removes 51.8% of geometrically inconsistent points while removing only 7.5% of all points and retaining 93.3% of surface-consistent points.

• Qualitative improvements. The method preserves fine textures and structural details at both object level and scene level; ablations show cracks on a tabletop and blurred door frames for variants such as w/o A, w/o V, w/o G, and w/o C.

Methodology in Plain English

The pipeline starts from a pretrained DA3 model, which takes the input images (and optionally camera poses) and outputs a depth map and camera for each view. The authors keep this geometry prediction completely frozen — it is never updated by rendering losses — because they found that letting geometry move lets the network "cheat": positions drift to compensate for attribute mistakes, which creates flying points, blur, and distorted depth.

A second copy of the same network, with only its last ten Transformer blocks and its Gaussian decoder trainable, reads the same images and produces per-pixel Gaussian attributes: opacity, three-axis scale, rotation, and a zeroth-order spherical-harmonic color. Each pixel's depth is back-projected into 3D to give a Gaussian center; these centers are never moved.

To avoid a mismatch between where points are placed (predicted cameras) and how they are scored (GT target cameras), the authors place points using GT cameras whose translation scale has been fitted to the predicted reconstruction. Concretely, predicted depths and translations are divided by a mean radial distance factor, and an Umeyama fit on input-camera centers gives a scale that puts the GT cameras into the same unit.

Supervision is then applied on renderings from the input views and from four sampled target views. Targets are deliberately chosen far from the input viewing directions (large angular separation) so that Gaussian shapes and orientations get constrained from several sides; a GT-depth warp mask restricts the loss to regions with depth-consistent support from at least one input view. Losses are equally weighted masked L1 and LPIPS. Before rendering, Gaussians are merged with the voxel strategy of AnySplat and WorldMirror 2.0 (voxel side length 0.004 in the normalized frame) and pruned below opacity 0.01.

A second stage trains a small refiner. It renders the current scene into the input views, collects gradients of a reconstruction loss with respect to the Gaussian attributes, detaches and normalizes them, and feeds them along with projected DPT features into a four-block KNN Transformer that predicts attribute residuals and a recurrent hidden state. Centers stay fixed, so the neighbor graph can be reused across steps. Training uses up to three unrolled updates with step weights 0.9^(K_r − k) plus an opacity-change regularizer; a first-order surrogate loss supplies gradients through the unrolled steps without second-order differentiation.

Training uses Hypersim, Raw3D, and ScanNet++ sampled at 60%, 10%, and 30%. Stage I runs 25,000 steps in about 48 hours on 16 H20 GPUs with a global batch size of 16; Stage II runs 5,000 steps in about 13 hours. Input view counts during training range from 4 to 12; evaluation uses 4, 8, and 16 input views with four held-out target views per example. At inference the same trained model handles both pose-conditioned input (supplied intrinsics and poses, rescaled to match normalized predicted depths) and pose-free input (the geometry branch's predicted cameras, normalized by the same factor).

Why This Matters

Impact on research. The paper reframes feed-forward Gaussian splatting as an attribute-learning problem on top of fixed, imperfect geometry rather than a joint geometry-and-appearance problem. The ablation showing that geometry fine-tuning and learned positional offsets both hurt perceptual metrics is a concrete, transferable lesson for anyone building on geometry foundation models. The opacity analysis — that reconstruction-only supervision implicitly learns a geometric-consistency filter — offers a cheap alternative to explicit outlier removal, and the large-angle target sampling result gives a data-curation principle that is independent of the architecture.

Real-world applications:

• Simulation and content creation: rebuilding scenes from casually captured photos for games, film previsualization, and virtual production, where per-scene optimization is too slow.
• Autonomous driving and robotics: turning a short clip of onboard camera frames into a renderable scene for closed-loop simulation and perception testing.
• Real estate and e-commerce: generating navigable 3D previews from a handful of listing photos, including at viewpoints that were never photographed.
• AR/VR capture and telepresence: fast scene reconstruction where a user's device cannot run minutes-long per-scene optimization.

Industry relevance. The method is a thin, trainable head on an existing frozen geometry foundation model, which makes it cheap to retrain when the underlying geometry model improves. Inference-time refinement runs with fixed network weights and three update steps, and the same checkpoint serves both posed and unposed input — attractive properties for deployment pipelines that receive heterogeneous capture data.

Future Directions

• Extending the evaluation to reconstruction metrics and other metrics. The main table reports LPIPS and DISTS; the authors state that results for all four metrics and a comparison of learned refinement against Adam are given in the appendix, leaving open how the approach trades off against photometric or geometric accuracy.

• Better handling of residual geometry errors. Because geometry is frozen, the method inherits DA3's failure modes; the paper's own framing suggests investigating whether the learned opacity filter can be complemented by explicit outlier rejection or by selectively unfreezing geometry where it is provably wrong.

• Refinement beyond three steps and beyond input-view feedback. The refiner is trained with K_r sampled from {1, 2, 3} and uses only input observations; whether more steps, or feedback from synthesized intermediate views, improves extreme-view rendering is not established here.

• Generalization to longer sequences and larger scenes. Training samples up to 12 input views and evaluation up to 16; the related work it cites (VGGT-Long, VGG-T3, Scal3R, Long-LRM, tttLRM) targets long sequences, and whether the fixed-center assumption holds at that scale is untested.

Target Audience

Researchers and advanced engineers working on novel view synthesis, 3D Gaussian Splatting, and feed-forward 3D reconstruction — particularly those building on multi-view geometry foundation models such as DA3, VGGT, or DUSt3R. It is also useful for practitioners in AR/VR, robotics simulation, and autonomous driving who need fast, pose-flexible scene reconstruction, and for students who want a clear worked example of how training-signal design (camera alignment, target-view sampling) can matter as much as architecture in feed-forward 3D. Beginners will find the qualitative figures and the ablation narrative accessible, but the method section assumes working knowledge of Gaussian splatting parameterizations and camera conventions.

Authors’ abstract

Feed-forward Gaussian reconstruction builds 3D scenes directly from input images for novel view rendering, but learning Gaussian attributes from predicted geometry for high-quality rendering under large viewpoint changes remains challenging. We analyze limitations in Gaussian attribute learning and introduce InfiniSplat v2, which improves rendering quality through architectural and training changes. Our method decouples geometry and attribute prediction with a frozen geometry branch, aligns predicted Gaussians using ground-truth camera poses during training, and adopts large-angle target view supervision. Across six datasets, the feed-forward outputs of the same trained model achieve state-of-the-art mean LPIPS and DISTS among the evaluated methods in both pose-conditioned and pose-free settings. In the pose-conditioned setting, mean LPIPS and DISTS decrease by 25.8% and 32.9%, respectively, compared with the strongest evaluated baselines. We further find that opacity learned with only image reconstruction supervision suppresses geometrically inconsistent points. Project page: https://zju3dv.github.io/InfiniSplat2/