GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats

AI summary

Overview

Research area: Computer vision / neural rendering — specifically, learned (rasterizer-free) rendering of 3D Gaussian Splatting scenes with a transformer.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, transformer attention, rotary position embeddings, and standard novel-view-synthesis metrics (PSNR, SSIM, LPIPS).

Scope: A feasibility study at the object level showing that a transformer can render a set of 3D Gaussians directly into an image with no rasterizer in the renderer, plus an analysis of the remaining gap to rasterization.

What This Paper Is About

Every image of a 3D Gaussian Splatting scene today is produced by a hand-written rasterizer: an algorithm that sorts Gaussians by depth, projects them onto the image plane, and alpha-blends them per pixel. This paper asks whether a neural network can instead take a set of Gaussians plus a camera and output the image directly, learning the visibility and blending computation from data rather than executing it as an algorithm. The authors build GaussianRenderFormer by adapting RenderFormer — a transformer that renders triangle meshes — to take Gaussians as input, and then diagnose how far it is from the rasterizer it replaces.

Key Contributions

1. GaussianRenderFormer, described by the authors as the first feed-forward neural renderer for 3D Gaussian splats with no rasterizer in the renderer. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns patches of camera rays into pixels by attending to those tokens.

2. Three decoder changes that adapt a mesh renderer's view decoder to Gaussians: a 2D rotary embedding of where each Gaussian lands in the image (Eq. 4), an embedding of each Gaussian's projected footprint added to the attention keys (Eq. 5), and smaller 4×4-pixel patches instead of 8×8. Each change is ablated at moderate scale (1,000-object harness) and again at full scale, alongside a training schedule that trains the new input layer first.

3. An analysis of the remaining gap to rasterization, showing it is not a generalization gap and that the diagnostics are consistent with a bottleneck in the view decoder — while explicitly leaving under-training as an open possibility.

4. Public release of code, trained models and videos at the project page and on GitHub.

Main Findings

• Headline numbers: On 300 fresh held-out objects over all 28 views, GaussianRenderFormer reaches 38.08 dB PSNR against renders of the full, unpruned splat, while rasterizing the same pruned 20,000-Gaussian input reaches 44.02 dB. SSIM is 0.9814 versus 0.9954 and LPIPS 0.0113 versus 0.0046.

• The gap is not generalization: On the development held-out objects the model scores 38.02 dB, essentially the same. On 300 training objects over their own training views it scores only 0.17 dB higher than on unseen objects — and still stays 5.9 dB below rasterizing that input. Measured against a rasterization of its own input rather than the unpruned render, the model scores only 0.3 dB higher (4 views at distance 1.7).

• The gap varies with camera distance: from 3.9 dB at distance 1.15 to 6.8 dB at 1.7. Rasterizing the input is best at 1.7, the distance of the 64 views the pruned input was fine-tuned on, while the model is nearly flat across distances. Masked PSNR (over pixels where the full splat's opacity exceeds 0.5) is about 5 dB lower for both renderers, with roughly the same gap.

• Each decoder change matters: On the 1,000-object ablation harness, removing the footprint costs 1.71 dB, removing the 2D rotary embedding costs 5.20 dB, and using 8×8 patches instead of 4×4 costs 3.05 dB. Two runs of the same recipe with a different seed differ by only 0.02 dB on held-out objects.

• The footprint is the most direct evidence for a decoder bottleneck: each Gaussian's footprint follows from information the decoder already receives, yet giving it explicitly adds 1.4 dB at full scale on held-out and training objects alike, and the model loses 1.4 dB on the fresh held-out set without it.

• Smaller components barely matter: the 0.3-square-pixel isotropic term taken from gsplat accounts for only 0.08 dB, and restricting cross-attention to Gaussians that reach a patch's tile (windowed attention) adds only 0.20 dB.

• The footprint in values as well as keys adds a further 0.68 dB on the harness, but the authors keep it in the keys only because the values variant is more sensitive to denser inputs and blurs the split between computed geometry and learned function.

• Denser inputs hurt the model: at 40,000 Gaussians, rasterization gains 4.6 dB while the model loses 3.1 dB; the same recipe without the footprint loses 1.6 dB, so dense cross-attention is part of the cause.

• Errors concentrate on fine detail: the 10% of pixels with the strongest reference edges hold 71% of the model's squared error, and the model trails the rasterizer by more off the edges than on them (8.1 vs 6.2 dB). Above 1/4 cycles per pixel it reproduces 85% of the reference's content versus 93% for the rasterizer, and it also adds detail not present in the reference.

• Not data-limited at their compute: fine-tuning the end of stage 2 on 1, 10, 100, 1,000 and all 26,820 objects shows the model fitting a single object 7 dB above rasterizing its input while forgetting others, with the two curves meeting at the full set about 12 dB below rasterizing the input.

• Training was still improving: the final run gained about 0.9 dB with each doubling of steps from 94,000 to 375,000, with no slowdown, so under-training is not ruled out.

• Cost: one view of 20,000 Gaussians takes 275 ms on an NVIDIA L40S (151 ms scene encoder, 124 ms view decoder) versus 0.7 ms for gsplat; 14 views in one call take 2,012 ms total (144 ms per view) and 13.0 GB peak memory. The model has 195M parameters (113M encoder, 82M decoder). One view of 640,000 Gaussians takes 75 s and 23.7 GB, while gsplat stays under 3 ms and 0.1 GB.

• Prior-weight initialization is essential: training the new input layer alone first gives 3.1 dB more than training the whole model from the start in the same number of steps, and training from random weights collapsed to a constant black image.

Methodology in Plain English

The authors take RenderFormer, a transformer that renders triangle meshes and is trained on scenes of up to 4,096 triangles, and keep its two-stage layout and pretrained weights.

In the first stage, each triangle token becomes a Gaussian token. A Gaussian is described by 14 numbers — center (3), scales (3), rotation quaternion (4), color (3) and opacity (1) — which a linear layer plus RMSNorm turns into a 768-dimensional token, with a learned embedding shared across all Gaussians. Twelve self-attention layers over these tokens, plus 16 register tokens placed at the mean center of the Gaussians, let the Gaussians exchange information. A 3D rotary position embedding of each center replaces RenderFormer's triangle-vertex embedding. This stage runs once per object, regardless of how many views are requested.

In the second stage, the image is split into small patches, each patch becomes a token built from the directions of the camera rays through its pixels, and six layers of cross-attention to the Gaussian tokens followed by self-attention among patches feed a DPT head that produces pixels. Three adaptations are made here. First, each Gaussian's center is projected into the image (in patch units), and a 2D rotary embedding of that projected position — 8 channel pairs per image axis per head, at 8 frequencies from 0.25 to 1.75 radians per patch — is applied to the keys and to the patch queries, so attention depends on image-space distance. Second, patches shrink from 8×8 to 4×4 pixels, with ray directions upsampled by nearest neighbor so the pretrained 8×8 ray embedding can be reused. Third, each Gaussian's 2×2 image-space covariance is computed in closed form from its scale, rotation and the projection Jacobian, described by four numbers, and added to its key through a small linear layer initialized to zero. Crucially, everything per pixel — which Gaussians cover a pixel, which are visible, how they blend and what color results — is left to the network.

Data and training: training uses Objaverse_Splats, filtered to splats with PSNR of at least 32 dB, LPIPS at most 0.06 and exactly 50,000 Gaussians, giving 26,820 training and 1,806 validation objects. Ground truth is 28 gsplat renders per object at 512×512 with a 45° field of view and a black background (14 cameras at distance 1.7, 7 each at 1.15 and 2.45). The model's input is a pruned 20,000-Gaussian version — top Gaussians by an importance score adapted from LightGaussian, then fine-tuned for 1,500 steps to match full-splat renders from 64 views, about 10 seconds per object — so the model must reproduce full-splat renders from a pruned input. The loss is an L1 term in the log10(1+I) encoding (background weighted 0.05) plus LPIPS on a VGG backbone with λ=0.5. Training runs in four stages: 33.5k steps training only the new input layer at 256, 402k steps for the whole model at 256, 3k steps with the new decoder changes switched on, and 385k steps at 512 with LPIPS. Optimizer is AdamW with weight decay 0.01, batch of 8 views on 8 GPUs, learning rate 5×10⁻⁵ (10⁻³ in the first stage). Total training cost was 1,327 GPU-hours.

Evaluation discipline: of the 1,806 validation objects, 100 monitor validation loss, 300 form a development held-out set used for all design decisions, and 300 further objects never used during development form the fresh held-out set for the main result. Every metric is computed on an evaluation crop around the object, resized to 512×512, with masked PSNR reported on the whole image.

Why This Matters

The paper is the first test of whether the rasterizer — the one hand-written component that every Gaussian-splatting pipeline still depends on — can be replaced by a learned function. That matters because a learned renderer could in principle learn light transport from data, render Gaussians alongside meshes and lights in one model, avoid the popping caused by depth sorting as the camera moves, and learn a prior that fills in what its input lacks. The authors are explicit that none of these benefits is demonstrated here; what they show is that the network renders Gaussians at all, and where it currently falls short.

Potential real-world applications (as motivations, not demonstrated results):

• Relightable capture: shading captured splats without a hand-written shading model or an approximation of indirect light.
• Unified scene rendering: rendering captured splats together with meshes and lights in a single model, as RenderFormer-V2 does for other primitives.
• Real-time viewers and VR/AR: eliminating the sorting step that causes popping as the camera moves, if the cost gap can be closed.
• Robotics and simulation: fast differentiable-style rendering of explicit 3D representations where a learned prior could complete missing detail.

Industry relevance: The result is currently far from practical — 275 ms per view versus 0.7 ms for gsplat, and quadratic growth to 75 s at 640,000 Gaussians — so the near-term value is as a research baseline and a diagnostic target rather than a replacement for gsplat. The finding that the view decoder, not data or generalization, is the binding constraint gives the field a specific place to work.

Future Directions

• Widen or deepen the decoder, not the encoder. The evidence points to a bottleneck in the view decoder, but the authors note that the evidence against more encoder layers and a wider network comes from an earlier model on 100 objects, so this needs retesting at scale.
• Longer training. The final run was gaining about 0.9 dB per doubling of steps at 94,000 to 375,000 steps with no slowdown; the paper states that longer training would narrow the gap by an amount its data cannot predict.
• Footprint in values. Giving the footprint to values as well as keys added 0.68 dB on the harness but was more sensitive to denser inputs; the authors leave this variant to future work.
• Handling inputs denser than training. Since 40,000-Gaussian inputs cost the model 3.1 dB while helping the rasterizer 4.6 dB, and windowed cross-attention lost nothing but gained nothing at that density, scaling to real captured scenes (which are far denser) remains open.

Target Audience

Researchers and graduate students working on neural rendering, 3D Gaussian Splatting, or transformer-based scene representations who want a rigorous baseline for rasterizer-free rendering and a candid accounting of the remaining gap. It is also useful to practitioners deciding whether a learned renderer is ready to replace a rasterizer (it is not, on the evidence here) and to anyone interested in a well-instrumented negative-ish result: the ablations, the data-scaling study, and the explicit admission that under-training is not ruled out make it a useful methodological template.

Authors’ abstract

Can a neural network learn to render 3D Gaussian splats on its own, with no rasterizer in the renderer? We present GaussianRenderFormer, a transformer that takes a set of Gaussians and a camera and outputs the image directly. Each Gaussian becomes one token in a view-independent encoder, and a view-dependent decoder turns small patches of camera rays into pixels by attending to those tokens. Trained on 26,820 Objaverse objects pruned to 20,000 Gaussians each, GaussianRenderFormer renders 300 unseen objects at 38.08 dB PSNR against the full, unpruned splats, while rasterizing the same pruned input reaches 44.02 dB. The gap is not one of generalization, and our diagnostics are consistent with a bottleneck in the view decoder: telling it the projected footprint of each Gaussian, which it could in principle infer, adds 1.4 dB. Under-training is not ruled out. Code, trained models and videos: https://svlwoof.github.io/gaussianrenderformer