SurfSplat: Conquering Feedforward 2D Gaussian Splatting with Surface Continuity Priors

AI summary

Overview

Research area: Computer vision, specifically sparse-view 3D scene reconstruction and novel view synthesis using Gaussian Splatting.

Technical level: Advanced. The paper assumes familiarity with 3D/2D Gaussian Splatting, feedforward reconstruction networks, cost volumes, and differentiable rasterization.

Scope: The paper introduces SurfSplat, a feedforward network that predicts 2D Gaussian surfels from sparse images using a surface continuity prior and forced alpha blending, plus a new high-resolution evaluation metric called HRRC (arXiv:2602.02000v2, Shanghai Jiao Tong University and Tianyi Shilian Technology Co., Ltd).

What This Paper Is About

Feedforward reconstruction models can turn a couple of sparse images into a 3D scene in milliseconds, but the scenes they produce often degrade into discrete, color-biased point clouds with holes and discontinuous surfaces. These flaws are hard to see in standard low-resolution renders near the input viewpoints, but become obvious in close-up or off-axis views. SurfSplat's goal is to force such models to reconstruct smooth, continuous 3D surfaces rather than scattered blobs, and to introduce a metric (HRRC) that actually exposes the difference.

Key Contributions

1. SurfSplat, a feedforward network that reconstructs 3D scenes from sparse inputs using 2D Gaussian Splatting (2DGS) surfels as the representation primitive, combined with a surface continuity prior and forced alpha blending.
2. A surface continuity prior that derives each 2D Gaussian's rotation and scale from predicted 3D positions rather than regressing them directly, binding orientation and shape to local surface geometry.
3. A forced alpha blending strategy that clips predicted opacity below an upper bound to prevent opacity collapse and preserve depth-ordered 3D alignment.
4. High-Resolution Rendering Consistency (HRRC), a new evaluation metric that renders reconstructions at 2× and 4× resolution (for example, 512×512 and 1024×1024 from 256×256 inputs) and compares against bicubically upsampled ground truth using PSNR, SSIM, and LPIPS.

Main Findings

• RealEstate10K results: The largest model (Ours-L) reaches 27.537 PSNR, 0.892 SSIM, and 0.112 LPIPS at 256×256 standard resolution, versus 27.504 / 0.890 / 0.112 for DepthSplat. At 1024×1024 HRRC, Ours-L reports 24.897 PSNR / 0.842 SSIM / 0.320 LPIPS, while DepthSplat drops to 16.385 / 0.635 / 0.491 and MVSplat to 17.966 / 0.755 / 0.425. Averaged over standard, 512×512, and 1024×1024, Ours-L reaches 26.255 / 0.867 / 0.216.
• Model scaling: On RealEstate10K, the small model (Ours-S) averages 25.842 / 0.859 / 0.222, the base model (Ours-B) 26.157 / 0.865 / 0.217, and Ours-L 26.255 / 0.867 / 0.216, showing consistent gains with a larger Depth Anything V2 backbone.
• ACID results: "Ours" reports 28.336 / 0.845 / 0.144 at 256×256 and 21.253 / 0.690 / 0.457 at 1024×1024, averaging 25.486 / 0.783 / 0.294. In Table 2, pixelSplat averages a higher PSNR (27.478) and HiSplat 25.700, while both collapse at the 1024×1024 setting relative to their standard-resolution numbers.
• Cross-dataset generalization (trained on RE10K, tested elsewhere): ScanNet 20.305 / 0.731 / 0.313 for SurfSplat versus 20.201 / 0.735 / 0.305 for DepthSplat; DL3DV 27.384 / 0.890 / 0.106 for SurfSplat versus 28.141 / 0.905 / 0.083 for DepthSplat; DTU 15.544 / 0.488 / 0.329 for SurfSplat versus 16.019 / 0.671 / 0.277 for HiSplat. Over the three datasets combined, SurfSplat averages 21.078 / 0.703 / 0.249 against DepthSplat's 20.978 / 0.688 / 0.275.
• Ablation without forced alpha blending: Standard PSNR falls from 27.001 to 26.481 and the 1024×1024 HRRC PSNR falls from 24.535 to 17.576, with the average dropping from 25.842 / 0.859 / 0.222 to 21.700 / 0.770 / 0.316. Removing both forced alpha blending and the surface continuity prior gives 26.925 at standard resolution but only 18.563 at 1024×1024 (average 22.346 / 0.800 / 0.283).
• Standard metrics hide the failure mode: The ablated variant without the surface continuity prior still achieves competitive 256×256 novel view synthesis scores while producing visually noisy, discontinuous surfaces. Only the HRRC metrics drop sharply, which the authors present as evidence that conventional NVS metrics miss geometric flaws.
• HRRC transfers to native high-resolution data: On a sampled subset of the high-resolution DL3DV dataset, the relative rankings stayed consistent with HRRC evaluation without any bicubic upsampling. SurfSplat scored 24.411 PSNR / 0.788 SSIM / 0.252 LPIPS, against pixelSplat 24.082 / 0.755 / 0.250, HiSplat 22.780 / 0.765 / 0.237, TranSplat 19.545 / 0.679 / 0.257, MVSplat 17.966 / 0.645 / 0.301, and DepthSplat 16.066 / 0.600 / 0.424.
• Hyperparameter robustness: Varying the opacity threshold and alpha normalization threshold produced small changes. With τ_α = 0.3 the average is 25.889 / 0.861 / 0.220 and with τ_opa = 0.4 it is 25.829 / 0.859 / 0.221, compared with 25.842 / 0.859 / 0.222 for the default settings.
• Qualitative geometry: Visualizations show that increasing resolution exposes dark unfilled gaps in image space and unnatural yellow depth regions in competing methods, while the full model yields continuous surfaces. A normal map and mesh comparison against DepthSplat is also reported.

Methodology in Plain English

SurfSplat takes a handful of posed images and predicts, in one forward pass, one 2D Gaussian "surfel" per pixel. It uses a dual-path encoder: a single-view branch built on the pretrained Depth Anything V2 monocular depth backbone (upsampled bilinearly), and a multi-view branch that converts images into low-resolution feature maps, processes them with 6 stacked self- and cross-attention layers in a multi-view Swin Transformer, and builds plane-sweep cost volumes from candidate depths. The two feature sets are concatenated and passed through a 2D U-Net, which predicts depth, scale multipliers, higher-order spherical harmonics, and opacity; a DPT head upsamples outputs to full resolution.

The distinguishing step is what happens next. Instead of regressing each Gaussian's rotation and scale directly, the model derives them from the predicted 3D positions of neighboring pixels. For a pixel at (h, w) with 3D position p₀, rightward and downward Sobel filters over a 3×3 neighborhood produce two virtual neighbors, p₁ and p₂, whose differences give two tangent vectors. Their normalized cross product gives a local surface normal, and Rodrigues' rotation formula converts that normal into the surfel's rotation matrix, aligning the canonical normal (0,0,1) with the estimated surface. For scale, the model computes variances from the image-space distances between neighbors and multiplies these coarse estimates by network-predicted multipliers clamped to [1/3, 3]; the depth-axis scale is fixed to zero because the primitives are 2D.

The second component addresses a training pathology. When the continuity prior is applied, the model tends to learn nearly fully opaque splats that saturate each pixel, so under standard alpha blending the deeper Gaussians contribute almost nothing and 3D alignment suffers. Forced alpha blending clips each predicted opacity to an upper bound of 0.6 so all splats contribute regardless of depth order. Because the final alpha is then below 1 by design, rendered color is divided by alpha when alpha ≥ 0.1 during training (0.001 during evaluation) to remove transparency-induced bias, and RGB is initialized directly into the DC component of the SH basis.

Training uses only an image-level loss, MSE plus LPIPS with λ = 0.05, summed over the batch. The models were trained in PyTorch with AdamW and a cosine learning rate schedule on an NVIDIA A100 GPU: 300K iterations at batch size 16 for Ours-S and 600K iterations at batch size 8 for Ours-B and Ours-L, with the paper also stating a total of 4800K iterations following DepthSplat. The pretrained backbone uses a learning rate of 2×10⁻⁶ and all other layers 2×10⁻⁴. Training is at 256×256 by default, with an appendix experiment at 256×448.

Why This Matters

Impact on research: The paper argues that the standard benchmark setup for feedforward 3D reconstruction rewards models that look good at low resolution and near input viewpoints while hiding broken geometry. SurfSplat shifts the representation primitive from 3D Gaussians to 2D surfels in the feedforward setting, and HRRC offers a metric that requires no new annotations and can be computed from existing datasets. The ablation in which standard metrics stay high while HRRC collapses is a direct challenge to how the field measures progress.

Real-world applications:

• Immersive VR and gaming environments, where users move close to and around objects, exposing holes and faceted surfaces that are invisible in a benchmark render.
• Digital content creation and asset generation, where reconstructed 3D geometry needs to be clean enough to edit or export.
• Real estate and interior visualization, the domain captured by the RealEstate10K training data.
• Robotics or embodied agents that need geometrically trustworthy scene geometry rather than view-dependent appearance, though the current method requires known camera poses.

Industry relevance: Feedforward models reconstruct a scene in milliseconds rather than the hours required by per-scene optimization pipelines that depend on COLMAP preprocessing and hundreds of posed views. That speed enables real-time and scalable applications, and the HRRC metric gives practitioners a way to screen reconstruction quality under the close-up viewing conditions that end users actually encounter.

Future Directions

• Removing pose dependence: The stated limitations note that SurfSplat still relies on known camera poses, so joint pose elimination is an open problem.
• Compact representations: Predicting one Gaussian per pixel can produce redundant representations; adaptive or compact alternatives are left for future work.
• Reducing the gap at high resolution: On ACID at 1024×1024, SurfSplat reports 21.253 PSNR and 0.457 LPIPS, well below its standard-resolution numbers, indicating that high-resolution fidelity remains an unsolved problem even for this approach.
• Extending HRRC and higher-resolution training: The appendix explores training and evaluating at higher input resolution (256×448), and the authors propose HRRC as a general metric that could be adopted and stress-tested across more datasets and reconstruction paradigms.

Target Audience

Researchers and graduate students working on 3D reconstruction, novel view synthesis, and Gaussian Splatting, particularly those focused on feedforward or generalizable methods; engineers building VR, gaming, or content-creation pipelines who need fast, geometrically coherent scene capture; and benchmark designers interested in evaluation metrics that expose artifacts hidden by standard PSNR, SSIM, and LPIPS protocols. Readers without a background in differentiable rendering and Gaussian primitives will find the method sections demanding.

Authors’ abstract

Reconstructing 3D scenes from sparse images remains a challenging task due to the difficulty of recovering accurate geometry and texture without optimization. Recent approaches leverage generalizable models to generate 3D scenes using 3D Gaussian Splatting (3DGS) primitive. However, they often fail to produce continuous surfaces and instead yield discrete, color-biased point clouds that appear plausible at normal resolution but reveal severe artifacts under close-up views. To address this issue, we present SurfSplat, a feedforward framework based on 2D Gaussian Splatting (2DGS) primitive, which provides stronger anisotropy and higher geometric precision. By incorporating a surface continuity prior and a forced alpha blending strategy, SurfSplat reconstructs coherent geometry together with faithful textures. Furthermore, we introduce High-Resolution Rendering Consistency (HRRC), a new evaluation metric designed to evaluate high-resolution reconstruction quality. Extensive experiments on RealEstate10K, DL3DV, and ScanNet demonstrate that SurfSplat consistently outperforms prior methods on both standard metrics and HRRC, establishing a robust solution for high-fidelity 3D reconstruction from sparse inputs. Project page: https://hebing-sjtu.github.io/SurfSplat-website/