Splat-SAP: Feed-Forward Gaussian Splatting for Human-Centered Scene with Scale-Aware Point Map Reconstruction

AI summary

Overview

Research area: Computer vision — feed-forward novel view synthesis and 3D reconstruction of human-centered scenes, combining Gaussian Splatting with point-map geometry prediction from sparse binocular inputs.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, multi-view stereo and cost volumes, DUSt3R/MASt3R-style point maps, camera intrinsics/extrinsics, and self-supervised rendering losses.

One-sentence scope: Splat-SAP reconstructs scale-aware (metric) point maps from only two widely separated input views and uses them to anchor a Gaussian "plane" on the target view, producing free-viewpoint renderings of moving people without per-scene optimization or 3D geometry supervision.

What This Paper Is About

Existing feed-forward Gaussian Splatting methods need input views with large overlap so that multi-view stereo or binocular stereo matching can supply a geometry prior, which makes them unreliable when the two cameras are far apart. Splat-SAP instead predicts point maps for each input view independently (using a DUSt3R/MASt3R-style foundation model), then converts those canonical-space point maps into real metric space. The goal is stable, temporally consistent free-viewpoint video of human-centered scenes from a sparse binocular pair plus camera calibration.

Key Contributions

1. A feed-forward pipeline that reconstructs scale-aware point maps and renders free-view video of human-centered scenes, trained in a self-supervised manner without any 3D geometry supervision.
2. A 2D-coarse-to-3D-fine registration strategy with a learnable affinity (scaling plus pixel-wise translation) that estimates scale-aware point maps, followed by stereo-constrained 3D refinement.
3. A Gaussian plane representation anchored on the target view, initialized from the scale-aware point maps and equipped with both 2D and dense 3D features to preserve rendering efficiency and completeness.
4. A new multi-view dataset of human-centered scenes — over 10,000 frames of motion sequences collected for training, including portable GoPro captures of large-scale, multi-person sports movement in 1080P 30FPS.

Main Findings

• Rendering quality (Table 1). On the Camera dataset, Splat-SAP reaches 32.220 PSNR / 0.957 SSIM / 0.079 LPIPS; on GoPro, 31.640 / 0.955 / 0.096; on Mobile, 25.721 / 0.827 / 0.244. Baseline values include NoPoSplat (25.035 / 0.866 / 0.173; 26.128 / 0.889 / 0.121; 21.594 / 0.591 / 0.272), 4D-GS (27.814 / 0.906 / 0.150; 27.244 / 0.907 / 0.205; 25.655 / 0.825 / 0.284), MVSplat (27.899 / 0.902 / 0.148; 29.942 / 0.934 / 0.157; 26.545 / 0.805 / 0.314), MVSGaussian (29.326 / 0.957 / 0.069; 27.413 / 0.926 / 0.151; 19.927 / 0.683 / 0.272) and ENeRF (28.272 / 0.943 / 0.084; 29.906 / 0.943 / 0.108; 20.579 / 0.640 / 0.302). The authors state the method generally outperforms the others, especially on the Camera and GoPro datasets.
• LPIPS resolution caveat. The authors report that because LPIPS is sensitive

Authors’ abstract

We present Splat-SAP, a feed-forward approach to render novel views of human-centered scenes from binocular cameras with large sparsity. Gaussian Splatting has shown its promising potential in rendering tasks, but it typically necessitates per-scene optimization with dense input views. Although some recent approaches achieve feed-forward Gaussian Splatting rendering through geometry priors obtained by multi-view stereo, such approaches still require largely overlapped input views to establish the geometry prior. To bridge this gap, we leverage pixel-wise point map reconstruction to represent geometry which is robust to large sparsity for its independent view modeling. In general, we propose a two-stage learning strategy. In stage 1, we transform the point map into real space via an iterative affinity learning process, which facilitates camera control in the following. In stage 2, we project point maps of two input views onto the target view plane and refine such geometry via stereo matching. Furthermore, we anchor Gaussian primitives on this refined plane in order to render high-quality images. As a metric representation, the scale-aware point map in stage 1 is trained in a self-supervised manner without 3D supervision and stage 2 is supervised with photo-metric loss. We collect multi-view human-centered data and demonstrate that our method improves both the stability of point map reconstruction and the visual quality of free-viewpoint rendering.