SplatAnything: Gaussian Spacing Normalization for Feed-forward 3DGS across Arbitrary Scenes

AI summary

Overview

Research area: Computer Vision — feed-forward 3D Gaussian Splatting (3DGS) and novel-view synthesis, specifically the design of the coordinate/normalization space in which Gaussian attributes are predicted. Technical level: Advanced (assumes familiarity with 3DGS, pinhole camera geometry, depth foundation models, and rendering losses such as LPIPS/DISTS). Scope: The paper diagnoses a scale-supervision ambiguity caused by radial-distance normalization in feed-forward 3DGS and replaces it with a Gaussian-spacing-based normalization, evaluated on a new seven-dataset cross-scene, cross-view benchmark.

What This Paper Is About

Feed-forward 3DGS systems first predict depth with a geometry foundation model and then predict per-pixel Gaussian attributes inside that model's normalized coordinate space. That space is built by dividing the scene by the mean radial distance of 3D points to the reference camera, which does not control the spacing between neighboring Gaussian centers — so visually similar patches can require very different Gaussian scales across scenes and camera setups, making the scale targets ambiguous. The authors propose to normalize by a robust scene-level statistic of that spacing instead, so that one model can predict Gaussians consistently across arbitrary scene scales, camera configurations, and viewpoints.

Key Contributions

1. Problem diagnosis: The paper identifies that Radial Distance Normalization (RDNorm) leaves Gaussian spacing — the distance between adjacent per-pixel Gaussian centers — varying across scene scales and camera configurations, which induces systematic shifts in Gaussian-scale targets and ambiguous supervision. It reports up to 4.5× variation in median Gaussian spacing across scene categories under RDNorm (Fig. 4).
2. Gaussian Spacing Normalization (GSNorm): A normalization that sets the scene-level scale factor to the median characteristic Gaussian spacing over all valid observations, computed from the relation d ≈ D / sqrt(f_x f_y) under a locally constant-depth approximation. It canonicalizes the median spacing to 1 while preserving relative local variation.
3. SplatAnything framework: A feed-forward 3DGS pipeline that uses the frozen WorldMirror 2.0 geometry model, rescales depths and camera translations with GSNorm, initializes a Gaussian Backbone from WorldMirror 2.0's geometry backbone and fine-tunes it, adds a lightweight GSHead trained from scratch, and applies a voxel-merge strategy (following AnySplat) to control primitive density from overlapping views.
4. A new cross-dataset benchmark: Evaluation on seven datasets (ScanNet, NRGBD, DL3DV, Tanks & Temples, Mip-NeRF 360, ETH3D, DTU), spanning 4/8/16/32/64 input views plus a single-view monocular setting, resolutions from roughly 480p to 1K, pose-free and pose-conditioned protocols, and explicitly extrapolated target views.

Main Findings

• Spacing varies where radial distance does not: Under RDNorm, median Gaussian spacing differs by up to 4.5× across scene categories; GSNorm equalizes it by construction (Fig. 4).
• Best overall multi-view results: In the 4–64-view average (Table 1), SplatAnything leads on most datasets. Pose-free: ScanNet DISTS 0.184 / LPIPS 0.207, NRGBD 0.195 / 0.183, DL3DV 0.221 / 0.156, Tanks & Temples 0.252 / 0.299, Mip-NeRF 360 0.233 / 0.463, ETH3D 0.147 / 0.049, DTU 0.229 / 0.483. The strongest pose-free baseline, Mirror2-GS, is higher on all seven DISTS values (e.g., ScanNet 0.193, DTU 0.347).
• Pose-conditioned gains are larger still: SplatAnything (Pose Cond.) reaches DL3DV 0.215 / 0.148, Mip-NeRF 360 0.219 / 0.440, and DTU 0.184 / 0.364, improving on Mirror2-GS (Pose Cond.) at 0.239 / 0.180, 0.272 / 0.506, and 0.315 / 0.532 respectively.
• Monocular setting is mixed: With a single input view (Table 2), SplatAnything is best on DL3DV (0.279 / 0.220 vs SHARP 0.335 / 0.280), Tanks & Temples (0.342 / 0.430 vs 0.374 / 0.470), and ETH3D DISTS (0.195 vs 0.200), but SHARP is better on ScanNet (0.223 / 0.244 vs 0.238 / 0.274), NRGBD (0.238 / 0.174 vs 0.265 / 0.284), Mip-NeRF 360 (0.328 vs 0.410 DISTS), and DTU (0.433 vs 0.327 — where SplatAnything is better).
• GSNorm beats both RDNorm variants: In the normalization-space ablation (Table 3), GSNorm outperforms RDNorm at every view count — e.g., 8 views 0.196 / 0.237 vs 0.223 / 0.272, and 64 views 0.217 / 0.229 vs 0.224 / 0.250.
• Post-hoc scale correction does not fix the problem: "RDNorm + scale calibration" is worse than plain RDNorm at every view count in Table 3 (e.g., 4 views 0.249 / 0.309 vs 0.223 / 0.278), indicating the ambiguity must be resolved before Gaussian prediction.
• Failure modes are scale-dependent: RDNorm tends to produce holes in small object-centric scenes and over-smoothed renderings in large outdoor scenes; scale calibration turns the holes into over-smoothing while outdoor scenes remain blurry (Fig. 7).
• Efficiency: The paper states that inference timing was evaluated under different input-view settings, but the timing numbers are deferred to the appendix and are not included in the provided content.

Methodology in Plain English

The system takes any number of images, plus optional camera intrinsics and extrinsics, and outputs a 3D Gaussian scene in seconds.

1. Borrow geometry, then rescale it. A frozen geometry foundation model (WorldMirror 2.0) predicts multi-view depths, intrinsics, and poses. Because each pixel becomes one Gaussian center, the gap between neighboring centers follows directly from depth and focal length: roughly depth divided by the square root of the focal lengths. The authors compute this gap for every valid pixel across all views and take the median as the scene's scale factor. Dividing the depths, 3D points, and camera translations by that median forces the typical spacing to be 1 in every scene, regardless of whether it is a small object or a large outdoor environment — while genuine local variations in spacing are preserved.
2. Predict Gaussians. A Gaussian Backbone (initialized from WorldMirror 2.0's geometry backbone and fine-tuned) plus a lightweight GSHead predicts opacity, rotation, scale, and spherical-harmonic coefficients for every pixel. Scale is predicted in log space with an exponential activation and clamped to a maximum of 0.3; rotations are normalized quaternions; opacity uses a sigmoid.
3. Remove redundant primitives. Because overlapping views place multiple Gaussians at nearly the same 3D location, a per-pixel weight map and voxel merging (following AnySplat) consolidate them — GSNorm fixes the sampling scale, voxel merging fixes density accumulation.
4. Train with rendering supervision. Supervision uses MSE plus LPIPS on rendered target views, with both weights set to 1. Rendering requires the predicted geometry to be aligned to the ground-truth camera scale, so the authors estimate a global scale factor as the median ratio of corresponding camera baselines. Training runs 20,000 iterations on 64 NVIDIA H20 GPUs, with 4 to 32 input views, input resolution dynamically varied from 504 to 1024, a 500-step warm-up, and learning rates of 5×10⁻⁶ (backbone) and 2×10⁻⁵ (GSHead).
5. Evaluate carefully. The benchmark samples source views from a compact trajectory arc (fixed span across view counts, so only density changes), selects target views by occlusion-aware symmetric co-visibility restricted to 0.20 ≤ C_sym < 0.60, evaluates up to 40 candidates per scene, and keeps up to four targets. Pose-free methods are aligned to ground truth by a post-hoc similarity transform so all methods are scored on identical viewpoints.

Why This Matters

The paper reframes a modeling problem as a coordinate-space problem: rather than adding capacity or data, it argues that the supervision target itself is inconsistent, and shows a cheap, principled fix. This is directly useful for anyone building general-purpose 3D reconstruction systems, because it explains why single-model feed-forward 3DGS has struggled to transfer across scene scales and camera rigs. The benchmark is arguably as valuable as the method, since prior protocols concentrated on training-adjacent domains and interpolated views.

Real-world applications:

• Large-scale scene capture for AR/VR and digital twins, where scenes range from tabletop objects to city blocks within the same product.
• Robotics and embodied AI, where a robot must reconstruct an environment from a handful of views and reason from viewpoints it never observed.
• Content creation and e-commerce, turning casual phone photos into renderable 3D assets without per-scene optimization.
• Simulation and asset pipelines, where fast, scale-consistent Gaussian reconstructions feed downstream rendering or training data generation.

Industry relevance: the method is a drop-in normalization change on top of existing geometry foundation models, so it is cheap to adopt; seconds-level reconstruction across arbitrary scales is the prerequisite for making feed-forward 3DGS commercially viable outside curated capture rigs. The paper also documents AI assistance for writing and code, with authors retaining responsibility for all claims and results.

Future Directions

• Robustness under degenerate geometry: SplatAnything inherits the failures of its frozen geometry backbone and may degrade on extremely sparse or otherwise difficult inputs; stronger or self-correcting geometry estimation is an open problem.
• Completing unobserved regions: The method reconstructs mainly what the input views see, leaving holes in unobserved areas — the authors propose generative priors as a route forward.
• Generalizing the normalization idea: Whether spacing-based (rather than radial-based) canonicalization helps other feed-forward 3D representations, or interacts better with learned density control than the current voxel merge, is untested here.
• Beyond the current benchmark: The evaluation covers static scenes and fixed pose/pose-free splits; extending the controlled extrapolation protocol to dynamic scenes, rolling-shutter or unknown-intrinsics capture, and efficiency-constrained deployment remains open. The timing study is promised in the appendix rather than the main text.

Target Audience

Researchers and engineers working on 3D Gaussian Splatting, feed-forward reconstruction, and novel-view synthesis who already understand camera models and rendering losses; practitioners building reconstruction pipelines on geometry foundation models who need one model to work across scene scales and camera setups; and benchmark designers interested in controlled extrapolation and sparsity protocols. Readers new to 3D vision will find the core intuition (match Gaussian size to the spacing between Gaussian centers) accessible, but the equations and experimental protocol assume intermediate-to-advanced background.

Authors’ abstract

We identify a key limitation of existing feed-forward 3DGS methods: Gaussian attributes are typically predicted in a radial-distance-normalized space that canonicalizes a global radial statistic of the reconstructed geometry, while the spacing between neighboring Gaussian centers can still vary substantially across scene scales and camera configurations. This mismatch induces systematic shifts in Gaussian-scale targets, leading to ambiguous supervision. To address this, we…