ReSplat: Learning Recurrent Gaussian Splatting

AI summary

Overview

Research area: 3D computer vision — feed-forward 3D Gaussian Splatting for sparse-view novel view synthesis.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, neural rendering, transformers/attention, and the distinction between per-scene optimization and feed-forward inference.

Scope: ReSplat introduces a recurrent, gradient-free feed-forward model that iteratively refines 3D Gaussians using rendering error as a feedback signal, achieving state-of-the-art sparse-view synthesis while using 16× fewer Gaussians than per-pixel baselines.

What This Paper Is About

Feed-forward Gaussian splatting models predict 3D Gaussians in a single forward pass, which makes them fast but caps their quality at whatever a single network pass can produce. Optimization-based methods like 3DGS reach higher quality through thousands of iterative gradient updates, but at enormous computational cost. ReSplat bridges the two: it learns a weight-sharing recurrent network that mimics the iterative refinement of optimization, but runs feed-forward and computes no gradients. The key discovery is that the rendering error — the difference between rendered and ground-truth input views — is a rich enough signal to tell the network exactly how to update each Gaussian.

Key Contributions

1. Gradient-free recurrent Gaussian refinement. A weight-sharing recurrent network iteratively predicts incremental updates to Gaussian parameters and hidden states from the rendering error of input views, without any explicit gradient computation.

2. Compact 16× subsampled initialization. An initial reconstruction model predicts per-view depth at 1/4 resolution and unprojects into a 16× subsampled 3D space, producing 16× fewer Gaussians than per-pixel models. kNN and global attention layers recover the lost detail by aggregating local and global 3D context.

3. Rendering error as an error-propagation mechanism. Pixel-space and feature-space (multi-scale ResNet-18) rendering errors are combined, then globally attended across all rendered pixels so every Gaussian receives information from the entire error field, not just its spatially aligned pixel.

4. Efficiency engineering for scalability. A pixel-unshuffle-based global attention scheme and a camera-guided local kNN variant (reducing complexity from O(N²) to O(N·C)) let the model scale to high resolutions and many input views.

Main Findings

• Large gains at 8 views on DL3DV (512×960). ReSplat reaches 27.70 dB PSNR after 4 iterations — 4.2 dB above optimization-based 3DGS and 3.5 dB above DepthSplat — while running roughly 100× faster than 3DGS and using 16× fewer Gaussians (246K vs. 3,932K), yielding 4× faster rendering.

• Iterations converge quickly and pay off consistently. Gains are monotonic across iterations (26.21 → 27.15 → 27.51 → 27.65 → 27.70 dB), converging after about 4 steps. The model is trained with a randomly sampled iteration count between 1 and 4, so a single model supports a flexible speed-accuracy trade-off at test time.

• Recurrence beats parameter scaling. ReSplat-Small (77M parameters) outperforms ReSplat-Large init (559M), AnySplat (886M), and WorldMirror (1263M), reaching 29.07 dB vs. 23.54 dB for the 16×-larger WorldMirror. The 14M-parameter recurrent module delivers more than brute-force capacity.

• Feed-forward refinement beats per-scene optimization from the same start. Starting from identical ReSplat initializations, the learned refinement reaches good quality in 4 iterations versus 80 for optimization-based refinement, a 53× speedup in reconstruction time, and is faster per iteration because no gradients are computed.

• Outperforms Long-LRM at 16 views (540×960). ReSplat achieves 23.51 dB PSNR with 518K Gaussians versus Long-LRM's 22.66 dB with 2,073K Gaussians — a 0.8 dB gain with 4× fewer Gaussians, despite a slightly slower reconstruction time due to the kNN operation.

• State-of-the-art on 2-view benchmarks. On RealEstate10K (256×256), ReSplat reaches 29.75 dB PSNR, beating DepthSplat, GS-LRM, Long-LRM, and LVSM's encoder-decoder variant, while offering roughly 20× faster rendering thanks to explicit 3D Gaussians.

• Strong out-of-distribution generalization. Trained only on DL3DV at 512×960 with 8 views, the recurrent model adapts to unseen datasets (RealEstate10K), unseen view counts, and unseen resolutions. Notably, it improves by 5 dB PSNR when tested at 320×640 instead of the training resolution, where single-step models degrade sharply.

• Works with plug-in initializations. The recurrent refinement improves monotonically regardless of starting point — MVSplat, ReSplat-Small, or ReSplat-Base — though MVSplat's per-pixel Gaussians make refinement 13× slower.

• Distinct from SplatFormer. Unlike SplatFormer's single-step, non-recurrent refinement of optimized 3DGS on object-centric data, ReSplat is recurrent, feeds on rendering error, initializes feed-forward, and targets complex scene-level benchmarks.

Methodology in Plain English

The pipeline runs in two learned stages.

Stage 1 — Compact initialization. Given N posed input images, a depth model predicts per-view depth maps at 1/4 resolution. These are unprojected into 3D using camera parameters, giving a point cloud with N·HW/16 points — 16× fewer than a per-pixel scheme. Each point carries a feature vector from the input images. Six alternating blocks of kNN attention (local neighborhoods, k=16) and global attention (all points) enrich these features with 3D context. A small MLP head then decodes each point into a Gaussian: position (the point itself), opacity, covariance, and spherical harmonics. The enriched feature also becomes the initial hidden state for the recurrent stage.

Stage 2 — Learned recurrent refinement. At each iteration, the current Gaussians render the input views. The difference between rendered and ground-truth views is measured two ways: directly in pixel space (downsampled 4× to match the Gaussian count) and in feature space using multi-scale features from a pretrained ResNet-18. These are added together. Because a single Gaussian can affect many rendered pixels, the error is passed through global attention so every Gaussian sees the full error field. The concatenation of the current Gaussian parameters, hidden state, and globally aggregated error is fed into four kNN attention blocks, which output small increments to the Gaussian parameters and hidden state. These are simply added to the current values — no gradients involved — and the loop repeats.

Training. Stage 1 trains on rendering loss plus a depth smoothness regularizer. Stage 2 freezes stage 1 and trains only the recurrent module end-to-end, supervising every intermediate prediction with exponentially increasing weights so later iterations matter more.

Efficiency details. Global attention is computed at 1/16 resolution via pixel unshuffle and upsampled back with pixel shuffle. For large point clouds, the exact O(N²) kNN is replaced by a camera-guided local search over a constant-size candidate set built from same-view spatial neighbors and cross-view projected neighbors. The global reference frame is set to the spatially central input view, which empirically balances Gaussian positions best.

Why This Matters

Impact on research. The paper reframes feed-forward 3D reconstruction as a learned optimization problem. It shows that unrolling iterative refinement into a weight-sharing recurrent network — an idea proven in optical flow, stereo, and SLAM — transfers effectively to Gaussian splatting, and that the rendering error is the right feedback signal for this domain. The result that a 77M-parameter recurrent model beats 1.2B-parameter single-step models challenges the assumption that feed-forward 3D reconstruction scales primarily through model capacity.

Real-world applications:

• AR/VR content capture from a handful of phone photos, where sparse views and fast rendering are both required.
• Robotics and autonomous navigation, where a robot must build a 3D scene representation from sparse, possibly out-of-distribution camera views in real time.
• E-commerce and real estate, generating novel views of products or properties from a few images without per-scene optimization.
• Telepresence and 3D video, where per-frame reconstruction must be fast and robust to varying resolutions.

Industry relevance. The combination of feed-forward speed, explicit 3D Gaussian output, and strong cross-resolution and cross-dataset generalization is directly useful for production systems that cannot afford per-scene optimization or retraining per deployment condition. The public release of code and pretrained models lowers the barrier to adoption.

Future Directions

• Scaling the recurrent stage. Only the initialization model was scaled to ViT-L; whether a larger or deeper recurrent module yields further gains, and where returns diminish, remains open.

• Reducing the kNN bottleneck. The authors attribute ReSplat's slower reconstruction time versus Long-LRM at 16 views to the kNN operation, explicitly leaving implementation-level optimization for future work.

• Beyond view synthesis. The gradient-free recurrent framework currently supervises only rendering loss; extending the error signal or auxiliary losses to improve underlying geometry (e.g., surface reconstruction) is a natural next step.

• Adaptive iteration counts. The model supports variable iterations, but a learned policy that decides when to stop per scene — rather than a fixed budget or a speed-accuracy knob — could improve efficiency further.

• Richer feedback signals. The paper found a pixel-plus-feature-space error combination works best; whether learned error metrics or other feedback (e.g., uncertainty) can further improve adaptation to unseen distributions is untested.

Target Audience

Researchers and engineers working on neural rendering, 3D reconstruction, and novel view synthesis will gain the most from this paper, particularly those already familiar with 3D Gaussian Splatting and feed-forward reconstruction baselines like MVSplat, DepthSplat, and GS-LRM. It is also valuable for practitioners interested in "learning to optimize" frameworks and weight-sharing recurrent architectures applied outside their traditional domains. The paper is not introductory — readers without background in Gaussian splatting, attention mechanisms, or sparse-view geometry will need to consult the cited works first.

Authors’ abstract

While existing feed-forward Gaussian splatting models offer computational efficiency and can generalize to sparse view settings, their performance is fundamentally constrained by relying on a single forward pass for inference. We propose ReSplat, a feed-forward recurrent Gaussian splatting model that iteratively refines 3D Gaussians without explicitly computing gradients. Our key insight is that the Gaussian splatting rendering error serves as a rich feedback signal, guiding the recurrent network to learn effective Gaussian updates. This feedback signal naturally adapts to unseen data distributions at test time, enabling robust generalization across datasets, view counts, and image resolutions. To initialize the recurrent process, we introduce a compact reconstruction model that operates in a $16 \times$ subsampled space, producing $16 \times$ fewer Gaussians than previous per-pixel Gaussian models. This substantially reduces computational overhead and allows for efficient Gaussian updates. Extensive experiments across varying number of input views (2, 8, 16, 32), resolutions ($256 \times 256$ to $540 \times 960$), and datasets (DL3DV, RealEstate10K, and ACID) demonstrate that our method achieves state-of-the-art performance while significantly reducing the number of Gaussians and improving the rendering speed. Our project page is at https://haofeixu.github.io/resplat/.