CompactSplat: Spatially Adaptive Gaussian Distribution for Feedforward Scene Reconstruction

AI summary

Overview

Research area: Computer vision — sparse-view 3D scene reconstruction and novel-view synthesis using generalizable (feed-forward) 3D Gaussian Splatting.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, plane-sweep cost volumes, U-Net decoders, DPT heads, and differentiable rasterization.

Scope: CompactSplat is a feed-forward network that replaces rigid one-Gaussian-per-pixel prediction with texture-driven quadtree partitioning, producing compact Gaussian sets whose sparsity can be adjusted at inference time with a single threshold.

What This Paper Is About

Feed-forward 3D Gaussian Splatting methods reconstruct a scene from a few images in one pass, but they normally assign one or several Gaussians to every pixel. This wastes thousands of primitives on flat, textureless regions such as walls and floors, inflating storage and producing artifacts when renders are magnified. CompactSplat instead decides where detail is needed, placing dense Gaussians on textured/complex regions and single, enlarged, scale-modulated Gaussians over homogeneous ones — while allowing the user to trade rendering quality against primitive count at test time without retraining.

Key Contributions

1. A compact feed-forward reconstruction framework. CompactSplat combines a Texture-Aware Partitioner (TAP) with a Hierarchical Pyramid Sampler (HPS) so that 3D Gaussians are allocated according to image texture rather than pixel grid position, reducing spatial redundancy.
2. A controllable-inference paradigm. A single user-set partition threshold τ modulates the primitive count on the fly using one set of pretrained weights, with no sparsity-specific retraining or auxiliary modules.
3. A scale-aware mapping from 2D partitions to 3D geometry. Each Gaussian's scale is explicitly multiplied by its quadtree block footprint, letting one primitive tile an entire planar surface instead of leaving gaps.
4. Evaluation on standard fidelity metrics plus high-resolution rendering consistency (HRRC) at 2× and 4× magnification, alongside representation-specific metrics (Gaussian count, Gaussians Per Pixel), on RealEstate10K and ACID.

Main Findings

• Fewer primitives, competitive or better quality on RealEstate10K. On RE10K, Ours-L averages PSNR 26.392 / SSIM 0.874 / LPIPS 0.198 using 72.6k Gaussians at 0.554 GPP, versus SurfSplat at 26.255 / 0.867 / 0.216 with 131.1k Gaussians at 1.000 GPP, pixelSplat at 25.584 / 0.869 / 0.204 with 393.2k at 3.000 GPP, and HiSplat at 25.575 / 0.877 / 0.188 with 229.4k at 1.750 GPP.
• The advantage grows with resolution. At 1024×1024 on RE10K, Ours-L reaches 25.252 PSNR versus 24.897 for SurfSplat, 24.785 for YonoSplat, 24.920 for pixelSplat, and 16.385 for DepthSplat, while pixelSplat and HiSplat remain ahead on SSIM/LPIPS at that scale (0.877 and 0.878 SSIM respectively vs. 0.858 for Ours-L).
• Mixed results on ACID. Ours averages 26.146 / 0.810 / 0.268 with 91.4k Gaussians at 0.625 GPP, above SurfSplat's 25.486 / 0.783 / 0.294 at 1.000 GPP and pixelSplat's 27.478 / 0.849 / 0.244 at 3.000 GPP on average PSNR. However, SurfSplat scores higher PSNR at 256×256 (28.336 vs. 27.992) and at 512×512 (26.868 vs. 26.384), while CompactSplat wins clearly at 1024×1024 (24.063 vs. 21.253).
• Baselines collapse at high magnification; CompactSplat does not. Methods such as MVSplat, TranSplat, and DepthSplat drop from roughly 26–27.5 PSNR at 256×256 to roughly 16–21 PSNR at 1024×1024, which the authors attribute to unfilled gaps registering as spurious near-field depth artifacts in magnified renders.
• Small, base, and large variants behave similarly. Ours-S (27.086 / 0.884 / 0.123 at 256×256 on RE10K) trails Ours-B and Ours-L (27.523 and 27.567 PSNR) only slightly, while all three share the same 72.6k Gaussian count and 0.554 GPP, suggesting the compactness gain is architectural rather than capacity-driven.
• The quality–compactness trade-off is asymmetric. Sweeping τ to reduce GPP produces only marginal degradation in fidelity metrics over a reasonable operating range, while low-GPP settings yield smoother, watertight planar surfaces and high-GPP settings recover sharper high-frequency edges.
• Not reported in the supplied content: quantitative cross-dataset results on DTU, DL3DV, and ScanNet are stated to be in the supplementary material; the abstract's claim of outperforming prior methods on RealEstate10K, DL3DV, and ScanNet is not backed by tables in the text provided. Training compute, inference latency, and the specific τ values used are also not given.

Methodology in Plain English

CompactSplat keeps the standard feed-forward pipeline but changes where Gaussians come from.

First, two encoders look at the input views. One branch builds a plane-sweep cost volume through self- and cross-attention so that the different views agree geometrically. The other branch uses a pretrained monocular depth backbone (Depth Anything V2) to supply structural cues where views do not overlap. These features are concatenated and passed through a 2D U-Net.

The Texture-Aware Partitioner then splits each image using a top-down quadtree. A block starts at a maximum size, and the network measures how much pixel intensity varies inside it — the maximum channel-wise spread between the block's brightest and darkest pixel. If that spread exceeds a user-chosen threshold τ, the block is split into four; otherwise it stops. Flat regions therefore become a few large blocks, and busy regions become many small ones. A binary mask discards blocks in invalid or boundary areas.

The Hierarchical Pyramid Sampler builds a pyramid of Gaussian attribute maps by repeatedly average-pooling a full-resolution attribute field predicted by a DPT head. Each quadtree leaf is sampled (via grid sampling) from the pyramid level matching its subdivision depth, and its predicted relative scale is multiplied by the block's footprint. That multiplication is the key trick: one Gaussian sampled from a coarse level is physically stretched to cover the whole flat block, so no gaps appear.

Training runs in two stages to keep this stable. In stage one τ is fixed at 0, so the quadtree splits everywhere and the network learns dense geometry under unambiguous supervision. In stage two τ is drawn randomly from a uniform distribution between 0 and a maximum, forcing the sampler to learn correct scale multipliers across pruning levels. Supervision is direct: MSE plus 0.05 × LPIPS between rendered and ground-truth views.

Why This Matters

Impact on research. The paper attacks a structural assumption rather than a metric: the 1:1 pixel-to-Gaussian mapping inherited from Splatter Image-style pipelines. It shows that compactness can be a design property of the architecture and a runtime control knob, not just a post-hoc compression step — relevant to the line of work including TokenGS, C3G, VolSplat, and Off The Grid. It also reframes high-resolution rendering consistency as a first-class evaluation axis, since uniform-allocation baselines degrade sharply under magnification.

Real-world applications:

• Immersive VR and gaming content pipelines, where a scene must fit within a fixed memory budget on a headset.
• Digital content creation and e-commerce, where a few phone photos must be turned into a usable 3D asset without per-scene optimization.
• Mobile or embedded 3D capture, where a single model must serve devices with different memory ceilings by changing τ.
• Large-scale scene archives or streaming, where storing far fewer primitives per scene (72.6k vs. 393.2k on RE10K) reduces storage and bandwidth.

Industry relevance. The inference-time sparsity knob is commercially attractive: one deployed checkpoint can serve a high-fidelity cloud renderer and a low-memory mobile client, avoiding separate models. CompactSplat assumes known camera intrinsics and extrinsics, unlike uncalibrated methods such as VGGT and MV-DUSt3R, so it fits rigs and capture setups where poses are already available.

Future Directions

1. Recovering the missing cross-dataset evidence. The abstract claims DL3DV and ScanNet results, but the tables cover RealEstate10K and ACID only; a full report of the DTU, DL3DV, and ScanNet evaluations would show whether texture-based partitioning holds on non-planar or outdoor geometry.
2. Replacing the intensity-spread heuristic. The splitting criterion is a maximum channel-wise intensity range, which the authors acknowledge is only a conservative proxy for geometric complexity. Learned or depth-aware complexity measures could recover the fine textures that currently receive unnecessary subdivision.
3. Absolute vs. relative sparsity control. τ is an image-space threshold whose mapping to actual Gaussian count depends on scene content; a predictor that converts a target memory budget directly into τ would make the knob usable in deployment.
4. Combining with pose-free pipelines and compression. Merging content-aware allocation with uncalibrated reconstruction (VGGT-style) or with explicit Gaussian compression schemes could push memory further down, and end-to-end latency comparisons against the baselines remain to be measured.

Target Audience

Graduate students and researchers working on generalizable 3D Gaussian Splatting, sparse-view novel-view synthesis, or neural scene representations who already understand the feed-forward 3DGS pipeline and want an alternative to pixel-aligned primitive prediction. It is also useful for graphics and XR engineers evaluating whether a compact, budget-adjustable scene representation is practical for deployment, and for reviewers or practitioners tracking how the field is moving from dense per-pixel output toward content-adaptive, controllable representations.

Authors’ abstract

Reconstructing 3D scenes from sparse views without per-scene optimization remains highly challenging, especially for recovering accurate geometry and fine textures. While recent feedforward approaches leverage generalizable 3D Gaussian Splatting (3DGS) for scene generation, they typically assign one or multiple Gaussians to each pixel. Such uniform allocation not only produces highly redundant representations by wasting primitives in homogeneous areas, but also fails to exploit the inherent geometric priors of planar surfaces, often resulting in sub-optimal structural modeling. To address these limitations, we present CompactSplat, a novel feedforward framework that enables a content-aware allocation of 3D Gaussians. By integrating texture-guided spatial partitioning with hierarchical sampling, CompactSplat adaptively distributes primitives - concentrating dense Gaussians in complex regions while tiling flat areas with expanded, scale-modulated primitives. Moreover, our framework inherently supports dynamic sparsity control at inference time, enabling seamless trade-offs between rendering fidelity and memory footprint without requiring network retraining. Extensive experiments on RealEstate10K, DL3DV, and ScanNet demonstrate that CompactSplat consistently outperforms prior methods on both standard metrics and high-resolution rendering consistency, achieving high-fidelity reconstructions with significantly fewer primitives.