Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting

AI summary

Overview

Research area: Computer Vision — feed-forward 3D Gaussian Splatting (3DGS), novel view synthesis, and 3D scene reconstruction from unposed images.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, transformer-based 3D reconstruction backbones, and keypoint detection via heatmaps.

One-sentence scope: The paper proposes a "Off-The-Grid" decoder that detects 3D Gaussian primitive locations at sub-pixel precision with content-adaptive density, replacing the rigid pixel- and voxel-aligned grids used by prior feed-forward 3DGS methods.

What This Paper Is About

Feed-forward 3D Gaussian Splatting models predict a 3D scene directly from a set of images in a single network pass, but they almost always tie one Gaussian primitive to each input pixel (or voxel), which is both wasteful and limiting. The authors ask whether a regular grid is the right way to distribute primitives at all, and instead let the network learn where to place them. Their goal is a pose-free feed-forward model that produces photorealistic 3DGS scenes in seconds while using far fewer primitives than input pixels.

Key Contributions

1. Sub-pixel primitive placement. A decoder that detects Gaussian centers at continuous 2D coordinates inside image patches — rather than snapping them to the pixel grid — using a differentiable heatmap/soft-argmax (DSNT) approach inspired by 2D keypoint detection.
2. Adaptive density mechanism. A multi-density scheme that assigns 16, 32, or 64 primitives per 14×14 patch based on a Shannon entropy measure of patch content, so detailed regions receive more primitives than homogeneous ones.
3. Confidence-based multi-view aggregation. Per-Gaussian confidence values, multiplied with opacity, let the model implicitly perform multi-view reasoning and prune redundant primitives; at test time primitives with α·c < 0.1 are discarded.
4. A pose-free feed-forward 3DGS model. The decoder is combined with a pre-trained VGGT backbone (VGGT-1B) and fine-tuned end-to-end with only photometric supervision and no 3D annotation, achieving state-of-the-art novel view synthesis among feed-forward methods.

Main Findings

• Novel view synthesis lead. On averages over 3, 6, 9, and 12 views, the method reports PSNR 21.21, SSIM 0.6470, LPIPS 0.3532, versus AnySplat (voxel-aligned, also built on VGGT) at PSNR 17.71, SSIM 0.5075, LPIPS 0.3937, and DepthAnything3's GS decoder (pixel-aligned) at PSNR 18.83, SSIM 0.5428, LPIPS 0.3834.
• Per-dataset results. The method reports the best average scores on every evaluation set: 7Scenes (PSNR 25.13 / SSIM 0.8119 / LPIPS 0.2629), Charge (21.93 / 0.6928 / 0.3997), DL3DV (20.48 / 0.6489 / 0.3163), SCRREAM (22.00 / 0.7171 / 0.3550), Tanks and Temples (19.37 / 0.5982 / 0.3348), and MipNeRF360 (18.36 / 0.4131 / 0.4503).
• Large compression. The method uses a primitives-to-pixels ratio of 0.1431 after confidence-based pruning — an 86% reduction, or roughly 7 times fewer primitives than pixel-aligned approaches (ratio 1). AnySplat's voxel aggregation reduces primitives by only 19% (ratio 0.8141).
• Primitive placement ablation. Replacing the pixel-aligned baseline with Off-The-Grid placement yields +0.3 dB PSNR, +0.014 SSIM, and a 13% LPIPS reduction on average (full model 18.63 / 0.5215 / 0.3796 vs. pixel-aligned 18.33 / 0.5076 / 0.4341). A SplatterImage-style 3D-offset variant performs similar to or worse than pixel-aligned (18.33 / 0.5020 / 0.4744) and shows isolated point artifacts.
• Both adaptive density and confidence matter. On DL3DV, removing adaptive density gives PSNR 17.80 / SSIM 0.4223 / LPIPS 0.4459; removing confidence pruning gives 18.72 / 0.5652 / 0.3898; using both gives 19.09 / 0.5998 / 0.3379.
• Geometry improves over the backbone. On average across Charge and SCRREAM, the fine-tuned encoder reports AbsRel 0.1433, AUC@30 0.9278, and FoV angular error 0.96, improving on the un-tuned VGGT (0.14905 / 0.8957 / 1.49). AnySplat's fine-tuning degrades its encoder (0.159 / 0.8333 / 3.225). DepthAnything3-Giant is best for depth and camera pose (0.1339 / 0.9339) but poor for intrinsics (ang_err 3.66).
• Learned heatmaps specialize. Visualization of the detection heatmaps shows low-density channels acting locally, while high-density channels specialize for borders and corners or spread out to handle detailed areas dynamically.
• Confidence encodes multi-view reasoning. The model assigns low confidence to primitives that are better observed in other views — for example, discarding Gaussians from a side view of a cube once a frontal view is available.
• Cleaner extrapolated views. Under highly extrapolated viewpoints, the method avoids the scan-line artifacts and visible voxel grid seen with voxel-aligned primitives in AnySplat, and avoids the blurriness/half-transparency seen in DepthAnything3.
• Self-rendering helps. Training with a "self-rendering" pass (rendering each image from only its own predicted primitives) improves LPIPS in the ablation: full model 0.3796 vs. 0.4142 average without self-rendering.
• Not reported: the paper states scenes are generated "in seconds" but does not give a specific inference time, nor does it report training wall-clock time or iteration count.

Methodology in Plain English

The system takes N unposed, uncalibrated images of a static scene and outputs a 3D Gaussian model.

Backbone. The authors start from VGGT-1B, a large multi-view transformer that already reconstructs depth maps and camera parameters from unposed images. Each image is encoded into 14×14 patches with DINOv2 and processed by 24 transformer blocks. Rather than using VGGT's point-map head, the method computes 3D positions by combining predicted depth with predicted camera parameters, then fine-tunes the backbone so its latent features carry useful multi-view information and its geometry improves under photometric supervision.

Decoder. A small U-Net (Pytorch-UNet implementation, 13 input channels, 32 output channels) takes the unpatchified tokens (8 channels per 14×14 patch), the input images, and the predicted depth maps. It produces detection features and description features through separate heads.

Detection instead of a grid. Detection features are reshaped into P channels per patch, where P is the number of primitives per patch — one channel per primitive. A softmax over the spatial dimensions (not channels), with temperature 0.2, turns each channel into a heatmap representing where that primitive's center lies inside the patch. Taking the expectation of pixel coordinates over the heatmap gives continuous (x, y) coordinates, so a primitive can sit exactly between pixels.

Adaptive density. Each patch is assigned a density level from a Shannon entropy measure of its grayscale intensity histogram. The 55% lowest-entropy patches get 16 primitives, the next 35% get 32, and the highest 15% get 64 — all well below the 196 pixels in a patch. Separate convolutional heads decode detection and description at each density level.

From 2D points to 3D Gaussians. For each detected point, the model bilinearly samples depth, RGB color, and descriptors (via PyTorch grid_sample with border padding), unprojects to a 3D center in camera coordinates, and predicts scale, orientation (quaternion), opacity, and confidence from the descriptors with a small MLP. Scale is predicted relative to depth rather than as absolute world scale, so an object that is n times larger and n times farther away gets the same scale parameter. Color is simply copied from the image rather than predicted, which the authors found performs similarly.

Aggregation and rendering. Primitives are transformed to world coordinates with the predicted extrinsics, and opacity is multiplied by confidence so the model can suppress redundant views. A customized 3DGS rasterizer renders depth and normals and supports backpropagation to camera pose parameters. Each image is rendered twice during training: once from its own primitives and once from the full aggregated model.

Training. Training uses only images, no 3D annotation, on a single GPU with 140 GB of VRAM. Each iteration processes up to 24 images, with 2 to 12 images per scene sampled from monocular video frames at random steps of 5 to 10. Training data is drawn from 10 datasets: DL3DV, Co3D-v2, WildRGBD, BlendedMVS, UnrealStereo4K, Real Estate 10k, ARKitScenes, DTU, ScanNet++, and KITTI360. The loss combines photometric terms (L1, SSIM, LPIPS), geometry consistency terms (depth L1 and normal-map agreement between rendered normals and normals derived from predicted depth), teacher geometry regularization against the original VGGT depth and camera poses, an opacity regularization pushing α·c toward 0 or 1, and a soft L2 penalty keeping intrinsics consistent within a scene. Notably, held-out target views are not used during training — only input images are rendered.

Evaluation. Six held-out datasets are used: the DL3DV benchmark (140 indoor and outdoor handheld scenes), Charge, SCRREAM, 7Scenes, Tanks and Temples, and MipNeRF360. Images are resized so the largest dimension is 518 pixels. Because pose-free methods predict their own coordinate systems, evaluation uses a two-pass alignment strategy similar to AnySplat's, with no test-time optimization of camera parameters through rendering.

Why This Matters

Impact on research. The paper challenges a default design assumption of nearly all feed-forward 3DGS work — that primitives must be pixel- or voxel-aligned. It shows that decoupling primitive count from pixel count improves both quality and efficiency, and that many of the artifacts of existing methods (visible voxels, scan-line artifacts, half-transparency) are consequences of the grid rather than of the reconstruction backbone. Because the method maintains or improves the encoder's geometry while adding a decoder, it also suggests a general recipe for turning large 3D foundation models into renderable scene models.

Real-world applications.

• Real-time 3D capture and viewing from casual phone or handheld video, without a separate Structure-from-Motion step.
• AR/VR and interactive content creation, where a photorealistic scene model is needed within seconds.
• E-commerce and real-estate visualization, where spaces or products are scanned from a handful of frames.
• Robotics and simulation, where fast, compact scene reconstructions feed downstream planning or rendering.

Industry relevance. The work comes from Huawei Noah's Ark Lab, and the efficiency gains matter commercially: using roughly 7 times fewer primitives per image directly reduces memory and rendering cost, and the ability to work on high-resolution images (rather than the typical 256×256) removes a long-standing scalability barrier for feed-forward 3DGS. Removing the pose-estimation dependency also simplifies the capture pipeline for consumer products.

Future Directions

1. Completing unseen regions. The method only reconstructs visible areas, leaving holes where the scene was never captured — for instance, the top face of a cube never shown in the input. The authors suggest either using video diffusion models to produce final renderings (as in MVSplat360) or learning 3D inpainting of Gaussian models.
2. View-dependent appearance. The model does not represent view-dependent color, so quality drops in scenes with lighting variation. The authors suggest color harmonization as one option.
3. Handling people. Human subjects are absent from the training data, so the model does not perform well on them, even though it handles indoor, outdoor, and object-centric scenes robustly.
4. High-resolution feed-forward 3DGS. Since the number of primitives is no longer tied to the number of input pixels, the authors argue this opens the door to feed-forward processing of high-resolution images — a setting previously out of reach.
5. Combining with view aggregation. The compression achieved per image could potentially be stacked with existing view-aggregation methods for further reduction.

Target Audience

Researchers and engineers working on 3D reconstruction, novel view synthesis, Gaussian Splatting, and feed-forward scene generation — particularly those already familiar with methods such as PixelSplat, MVSplat, NoPoSplat, and AnySplat who want to understand alternatives to grid-aligned primitive placement. It is also relevant to practitioners building real-time 3D capture pipelines for AR/VR, robotics, or content creation, who care about primitive count, memory footprint, and the ability to work without pre-computed camera poses. Readers without a background in differentiable rendering or keypoint detection will find the method section demanding.

Authors’ abstract

Feed-forward 3D Gaussian Splatting (3DGS) models enable real-time scene generation but are hindered by suboptimal pixel-aligned primitive placement, which relies on a dense, rigid grid that limits both quality and efficiency. We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level, replacing the pixel grid with an adaptive, ``Off-The-Grid" distribution. Inspired by keypoint detection, our decoder learns to locally distribute primitives across image patches. We also provide an Adaptive Density mechanism by assigning varying number of primitives per patch based on Shannon entropy. We combine the proposed decoder with a pre-trained 3D reconstruction backbone and train them end-to-end using photometric supervision without any 3D annotation. The resulting pose-free model generates photorealistic 3DGS scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models. It outperforms competitors while using far fewer primitives, demonstrating a more accurate and efficient allocation that captures fine details and reduces artifacts. Project page: https://arthurmoreau.github.io/OffTheGrid/.