RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting

AI summary

Overview

Research area: Computer vision — feed-forward 3D Gaussian Splatting (FF3DGS), with a focus on inference-time resource control and reinforcement-learning-based allocation.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, transformer cross-attention, k-nearest-neighbor search, and policy-gradient methods (GRPO/GSPO).

Scope (one sentence): The paper introduces RLSplat, a feed-forward 3DGS framework in which the number of Gaussians, the inference latency, and the peak memory all shrink linearly as the user-specified Gaussian budget is reduced.

What This Paper Is About

Feed-forward 3DGS reconstructs a 3D scene from images in a single forward pass, and recent "budget-controllable" variants let a user request a specific number of Gaussians at inference time. However, those methods first predict a dense set of candidate Gaussians and then prune or select down to the target count, so the cost of inference stays nearly constant no matter how small the budget is. RLSplat instead learns to allocate exactly the requested number of Gaussians up front and decodes only those, so that representation size, latency, and peak memory all decrease together.

Key Contributions

1. Problem identification and resource-linear framework. The authors show that existing budget-controllable FF3DGS methods (EcoSplat, F⁴Splat) control only the size of the output, not the cost of producing it, and propose RLSplat, in which Gaussian count, inference latency, and peak memory all decrease linearly as the budget B is reduced.

2. Gaussian allocation policy trained with policy optimization. Gaussian generation is reformulated as an allocation problem rather than pruning or selection: a policy predicts how many Gaussians each anchor region receives, summing exactly to B, and is trained with Group Sequence Policy Optimization (GSPO) using rendering quality as the reward.

3. Resource-linear decoder. Gaussian tokens are refined through local cross-attention over only K_gs nearest Gaussian tokens found by a hierarchical neighbor search (K_anc nearest anchors, then K_gs nearest Gaussians inside them), reducing decoding from O(B²) to O(B·K_gs).

4. Consistent gains across budgets. RLSplat outperforms prior budget-controllable methods at every tested budget, and at 2K Gaussians it improves PSNR by 0.9 dB over F⁴Splat while using 9.6× less peak memory and running 4.1× faster.

Main Findings

• Best quality at every budget. On ACID (zero-shot from RE10K), RLSplat achieves the best LPIPS, SSIM, and PSNR at 1K, 2K, and 4K Gaussians per context view across 8, 16, and 24 views. With 8 views and 4K per view (32K total), it reaches 25.24 dB PSNR versus 24.50 dB for F⁴Splat and 23.29 dB for EcoSplat.

• Advantage grows as the budget shrinks. With 8 context views, the PSNR margin over F⁴Splat grows from 0.74 dB at 4K Gaussians per view to 0.99 dB at 2K and 1.22 dB at 1K; the same trend holds at 16 and 24 views.

• Fewer Gaussians than budget-fixed methods. With 24 views, RLSplat reaches 25.58 dB PSNR using only 24K Gaussians, compared with 25.00 dB for SPFSplat using 1536K Gaussians — 64× fewer Gaussians.

• The allocation policy targets high-frequency structure. At an identical 8K total budget on RE10K, RLSplat places 5,388 Gaussians on the thin stair balusters versus 3,026 for F⁴Splat and 2,521 for EcoSplat. Local PSNR in that region rises from 17.79 dB (EcoSplat) and 20.00 dB (F⁴Splat) to 23.51 dB (RLSplat); full-view PSNR is 21.82, 23.42, and 26.51 dB respectively.

• Policy beats learning-free allocation rules. At B = 8,192 on ACID with 8 views, the learned policy reaches 24.83 dB PSNR versus 24.76 dB for an edge-based Sobel heuristic, 24.53 dB for a uniform split, and 24.18 dB for the base model trained with random allocations.

• Both locality regularizers matter. Removing the offset loss costs 0.53 dB (22.31 vs. 22.84 dB); removing the Voronoi loss as well costs 2.51 dB (20.33 dB). Without the projection loss, training fails entirely (5.80 dB PSNR, 0.007 SSIM).

• Local refinement matters. Removing the refinement module drops PSNR by 0.96 dB (21.88 vs. 22.84 dB).

• Hierarchical search is accurate and much cheaper. At 65,536 Gaussians, Table 5 reports the neighbor search at 8.1 ms versus 220.8 ms exact, and end-to-end latency of 328.3 ms versus 539.2 ms (the main text quotes 8.1 ms vs. 219.6 ms and 595.0 ms → 388.0 ms, described as 27× faster search). With the Voronoi loss, the hierarchical search retrieves exactly the same K_gs neighbors as exhaustive search for 99.0% of Gaussians; without it, that rate falls to 60.2%.

• K_gs can be tuned at inference without retraining. Larger K_gs improves quality at higher memory cost; the quality gap between K_gs settings narrows as the budget grows, so users can shrink K_gs at large budgets and increase it at small ones.

• GSPO is robust to its hyperparameters. All tested settings of group size C and rollout temperature τ are within 0.11 dB PSNR of each other; C = 5 and τ = 1.0 perform best (24.68 dB, 0.268 LPIPS, 0.722 SSIM in the shortened training schedule used for ablations).

Methodology in Plain English

Starting point. Given a handful of uncalibrated images and a target Gaussian budget B, a geometry backbone predicts the camera parameters and encodes the images into N "anchor tokens," each standing for a local 3D region of the scene.

Allocation instead of pruning. Rather than producing a dense cloud and throwing Gaussians away, RLSplat predicts a probability p_n for each anchor — its expected share of the budget — and samples an integer allocation from a multinomial distribution. Because "how many Gaussians go where" is a discrete, non-differentiable decision, the authors treat it as a reinforcement-learning problem: they draw a group of C = 5 candidate allocations, render each one, score it by reconstruction loss (reward = negative loss), and update the policy with GSPO using group-normalized advantages and a clipped, length-normalized importance ratio. At inference, sampling is replaced by a deterministic systematic-rounding rule that converts expected counts into integers summing exactly to B.

Decoding only what was allocated. Each anchor spawns exactly the number of Gaussian tokens it was granted. Each token then refines itself through L layers of cross-attention — but only over its K_gs nearest Gaussian neighbors, not all B tokens. To find those neighbors cheaply, the method searches hierarchically: first the K_anc nearest anchor centroids, then the K_gs nearest Gaussians within those anchors.

Keeping the shortcut honest. Two regularizers make the hierarchical search valid. A Voronoi loss keeps each Gaussian closer to its own anchor's centroid than to neighboring anchors' centroids, and an offset loss caps how far the residual refinement can move a Gaussian relative to the local anchor spacing (δ = 0.5). A projection loss ties each anchor's Gaussians to the image patch it came from, with a cheirality penalty.

Training recipe. Two stages: base training (40,000 steps) trains the backbone and decoder with random allocations over budgets sampled from [2048, 16384V]; policy training (5,000 additional steps) adds the allocation head, initialized to uniform output via zero-initialized final layers, and fine-tunes the whole model with the policy loss, sampling budgets with a skew toward the low end. No KL penalty is used because base training has no reference policy.

Evaluation setup. Trained on RealEstate10K with 2 to 24 context views, evaluated zero-shot on ACID (249 test scenes) at 256×256, reporting PSNR, SSIM, LPIPS, and Gaussian count, with latency and peak VRAM measured for Gaussian generation only on a single RTX A6000 at batch size one.

Why This Matters

Research impact. The paper reframes budget control in FF3DGS from "how many Gaussians do we keep" to "how many do we generate," and shows that inference cost — not just output size — should be part of the controllability contract. It also demonstrates a working use of sequence-level policy optimization (GSPO) for a non-language, spatially structured discrete decision.

Real-world applications:

• AR/VR and mobile 3D capture, where storage, memory, and latency budgets vary by device and scene.
• Robotics, where on-the-fly scene representations must fit within onboard compute limits.
• Digital content creation pipelines that need predictable per-asset memory footprints.
• 3D vision-language models that consume Gaussian splats as a scene representation and have their own resource caps.

Industry relevance. Deployment teams can trade quality against latency and memory with a single knob and no retraining, and the near-linear scaling makes capacity planning for streaming or edge inference tractable in a way that constant-cost pruning pipelines are not.

Future Directions

• Removing the fixed encoding cost. Image encoding does not scale with B, which bounds the achievable speed-up at very small budgets; making the backbone itself budget-aware is an open problem.
• Broader evaluation. The authors note that experiments are at 256×256 and in a single cross-dataset setting (RE10K → ACID), leaving higher resolutions and more transfer settings untested.
• Downstream integration. Using RLSplat as the scene representation inside 3D vision-language models and robot learning, where downstream tasks bring their own resource limits, is proposed as the natural next step.
• Allocation under richer objectives. Since the reward is currently rendering quality alone, extending the policy to optimize latency or memory directly — rather than only through the budget — is a natural extension the framework's design permits.

Target Audience

Researchers and engineers working on novel-view synthesis, feed-forward 3D reconstruction, and efficient 3D representations; practitioners deploying Gaussian splatting under memory or latency constraints on edge and mobile hardware; and anyone interested in applying policy-gradient optimization to discrete, spatially structured allocation problems in vision. Readers should already be comfortable with 3DGS fundamentals, attention mechanisms, and the GRPO/GSPO family of policy-optimization methods.

Authors’ abstract

Feed-forward 3D Gaussian Splatting (FF3DGS) reconstructs 3D scenes from images in a single forward pass, making it well-suited to applications that must operate within limited resource budgets. To further improve its practicality, recent budget-controllable FF3DGS methods allow the number of Gaussians to be adjusted at inference, so that the output meets a given storage limit or rendering-latency constraint. However, these methods overlook the cost of inference itself: to produce a representation with the target number of Gaussians, they first predict a dense set of Gaussians and then prune or select from it, requiring nearly constant latency and peak memory regardless of the budget. We therefore propose RLSplat, a resource-linear budget-controllable FF3DGS framework whose representation size, inference latency, and peak memory all decrease linearly as the Gaussian budget is reduced. Instead of pruning or selection, RLSplat learns a Gaussian allocation policy that distributes the budget across local scene regions, so that exactly the requested number of Gaussians is generated. These Gaussians are then refined through resource-linear Gaussian decoding, in which each Gaussian attends only to nearby Gaussians found via an efficient hierarchical neighbor search, so that the decoding cost decreases linearly with fewer Gaussians. In zero-shot evaluation, RLSplat outperforms prior budget-controllable methods at every budget, surpassing the strongest baseline even with 4$\times$ fewer Gaussians, while running up to 4.1$\times$ faster with 9.6$\times$ less peak memory at low budgets.