Anchor3DGS: Feed-Forward 3D Gaussian Splatting with Compact Anchor-Based Representation

AI summary

Overview

Research area: Computer Vision — feed-forward 3D reconstruction and novel view synthesis with 3D Gaussian Splatting (3DGS).

Technical level: Advanced. The paper assumes familiarity with 3DGS rasterization, multi-view geometry, camera intrinsics/extrinsics, attention mechanisms, and VGGT-style 3D foundation backbones.

Scope: A pose-free, feed-forward 3DGS framework that replaces pixel-aligned Gaussian prediction with a fixed budget of entropy-placed 3D anchors, each decoded into a small set of local Gaussians.

What This Paper Is About

Most feed-forward 3DGS methods predict exactly one Gaussian per input pixel, so the number of Gaussians is dictated by image resolution and view count rather than by the scene itself. This wastes primitives on flat, textureless areas, leaves holes in poorly observed regions, and makes the representation grow with every extra view. Anchor3DGS instead places a fixed number of 3D anchors on a geometry-aware scaffold, concentrating them where visual complexity is high, and decodes each anchor into a handful of local Gaussians — achieving better or comparable rendering with far fewer primitives.

Key Contributions

1. Anchor-based paradigm replacing pixel-alignment. A pose-free feed-forward 3DGS framework in which the Gaussian count is decoupled from input resolution and view count, controlled instead by a fixed anchor budget.
2. Entropy-guided anchor sampling. A training-free Shannon-entropy measure of local visual complexity drives anchor placement, combined with coarse grid subsampling for broad spatial coverage; a local spacing value per anchor modulates the spatial extent of its decoded Gaussians.
3. Occlusion-aware multi-view aggregation with cross-anchor interaction. Each anchor is projected into all input views, features are weighted by a depth-discrepancy-based occlusion score, and anchors exchange information through self-attention so weakly observed anchors borrow context from well-observed neighbors.
4. Feature-based rendering refinement. Each Gaussian carries an extra feature vector that is rasterized alongside color into a feature map, which a lightweight UNet decodes into an RGB residual to restore high-frequency detail while inheriting geometry from the 3D representation.

Main Findings

• State-of-the-art sparse-view synthesis: With 3 input views, Anchor3DGS reaches PSNR 23.69 / SSIM 0.764 / LPIPS 0.320 on VR-NeRF, 21.36 / 0.678 / 0.307 on MipNeRF360, and 21.22 / 0.683 / 0.280 on DL3DV — best on all three metrics on all three datasets against NoPoSplat, FLARE, DA3, and AnySplat.
• Six-view results: PSNR 24.03 on VR-NeRF, 22.39 on MipNeRF360, and 20.70 on DL3DV, again best across all metrics. The paper reports PSNR gains of over 2 dB over the best baseline on MipNeRF360 and DL3DV in these sparse-view settings.
• Constant Gaussian budget: All runs use roughly 0.26M Gaussians, fixed at N = 32,768 anchors × K = 8 Gaussians. Baselines scale with views — for example AnySplat uses 0.49M at 3 views on VR-NeRF and 1.03M at 6 views, while NoPoSplat and FLARE use 0.60M and 1.20M respectively.
• Dense-view parity at far lower cost: At 32 views on VR-NeRF, Anchor3DGS scores PSNR 24.56 / LPIPS 0.297 with 0.26M Gaussians and 35MB storage, versus AnySplat's 24.05 / 0.309 with 4.75M Gaussians and 779MB. AnySplat is slightly higher on SSIM there (0.819 vs 0.815). On DL3DV at 32 views, Anchor3DGS reaches PSNR 19.52 versus AnySplat's 18.76 and DA3's 19.38, the latter using 6.42M Gaussians and 1.03GB.
• Faster than per-scene optimization: The VGGT+3DGS reference requires 6 minutes (VR-NeRF) and 8 minutes (DL3DV) per scene, while Anchor3DGS runs in 2.9s and 3.1s on a single H800 GPU — comparable to AnySplat's 2.8s.
• Pose estimation is not degraded: On Co3Dv2, the jointly trained geometry backbone reaches AUC@10 of 54.8, AUC@20 of 74.0, and AUC@30 of 81.5, above AnySplat (53.1 / 73.1 / 80.8) and VGGT (50.4 / 67.2 / 74.9).
• Ablations confirm every component matters (VR-NeRF): Removing the enhancement UNet drops 3-view PSNR to 22.50; removing self-attention gives 23.03; replacing occlusion-aware aggregation with mean pooling gives 23.28; replacing entropy-guided sampling with uniform grid subsampling gives 22.16, a drop of 1.53 dB at 3 views. The full model is 23.69. The same ordering holds at 32 views (full model 24.56).
• Qualitative behavior: NoPoSplat renders overly blurry results and AnySplat leaves holes in poorly observed regions; Anchor3DGS avoids both and also avoids the fragmented, needle-like artifacts common in optimization-based 3DGS.

Methodology in Plain English

The pipeline has four stages.

1. Build a geometric scaffold. A VGGT-based backbone takes the uncalibrated images and predicts, per view, a camera extrinsic, a camera intrinsic, a depth map, and a feature map. Unprojecting the depth maps with the predicted cameras produces a cloud of 3D scene points — the candidate locations where anchors may live.

2. Place anchors by entropy. For each image, the authors convert to grayscale, build a local histogram in a sliding window, and compute Shannon entropy at every pixel; textured edges and structures score high, walls and sky score low. Anchors are drawn in two complementary ways: coarse grid subsampling over the 3D points for even coverage, and sampling with probability proportional to entropy so complex regions get more. Each anchor also records a spacing value — the mean distance to its k nearest anchors — which later controls how widely its Gaussians spread.

3. Encode anchors from all views. Each anchor is projected into every input view and its feature is bilinearly sampled. To avoid trusting views where the anchor is hidden, the method compares the anchor's camera-space depth against the depth map at that projected location; a large positive discrepancy means occlusion and yields a low weight through a sigmoid. These weights enter a softmax over views, combined with an MLP relevance score, to produce one aggregated feature per anchor. Self-attention across all anchors then lets poorly observed anchors draw context from better-observed ones.

4. Decode to Gaussians and refine. Each anchor embedding is combined with K shared learnable slot embeddings to predict, per slot, an offset (scaled by the anchor's local spacing) and the Gaussian attributes: opacity, covariance, base RGB, and a 16-dimensional feature vector. The Gaussians are rasterized with standard differentiable 3DGS, producing both a coarse RGB image and an aligned feature map from the same compositing weights. A three-level UNet turns the feature map into an RGB residual added to the coarse image.

Training. The model is trained end-to-end on DL3DV, ScanNet++, Hypersim, BlendedMVS, and Waymo with mixing probabilities 0.4, 0.2, 0.2, 0.1, 0.1, using MSE, LPIPS, and a depth-consistency loss (weights 1.0, 0.05, 0.1). Context views are chosen either by index gap (sequential datasets) or by camera-pose-distance ranking (non-sequential), with randomized window size so the model sees both small and wide baselines. Training runs 30,000 iterations on 8 NVIDIA H20 GPUs (about 2–3 days) with AdamW at a peak learning rate of 2×10⁻⁴, the geometry backbone fine-tuned at 0.1× that rate with its patch-embedding layer frozen. Inputs are resized to 448 pixels on the longer side.

Why This Matters

Research impact. The paper challenges an assumption baked into nearly all feed-forward 3DGS work — that Gaussian count should track pixel count — and shows the scene, not the camera, should determine representation size. The entropy-guided allocation is a simple, training-free signal that other compact-representation methods could adopt, and the constant 0.26M budget across 3, 6, and 32 views makes an unusually clean efficiency argument.

Real-world applications:

• Robotics and drone navigation, where a scene must be captured from a handful of uncalibrated frames and turned into a usable 3D map within seconds.
• Mobile and AR/VR content capture, where a 35MB scene representation versus 779MB matters for on-device storage and streaming.
• Autonomous driving and street-scale reconstruction from fleet cameras, one of the training domains (Waymo).
• Digital content creation and gaming, where fast feed-forward reconstruction replaces minutes of per-scene optimization.

Industry relevance. The combination of pose-free operation, second-scale inference, and order-of-magnitude smaller assets directly targets deployment constraints — bandwidth, memory, and latency — that have limited 3DGS in production. The paper notes funding from NSFC and an NSFC-RGC Joint Research Scheme, and the work was done partly during an internship at Ant Group, indicating commercial interest.

Future Directions

• Scene-adaptive anchor budgets. The anchor count is fixed regardless of scene scale; the authors explicitly flag very large or unbounded scenes as needing an adaptive budget.
• Reducing dependence on the geometry backbone. Incorrect poses or noisy depth maps degrade anchor placement and hence rendering — improving robustness here, or making anchor placement less sensitive to backbone errors, is an open problem.
• Compact representations as world models. The authors suggest the anchor set could serve as a "global geometric memory" and a substrate for future video world models.
• Scaling to 64 views and beyond. The appendix extends evaluation to 64 context views on VR-NeRF and DL3DV, reporting that the method stays competitive with roughly one to two orders of magnitude fewer Gaussians; the numeric table values are not included in the excerpt provided, so the exact 64-view figures are not reported here.

Target Audience

Researchers and graduate students working on feed-forward 3D reconstruction, Gaussian Splatting, and pose-free novel view synthesis who already understand 3DGS rasterization and multi-view geometry. It is also relevant to practitioners in AR/VR, robotics, and autonomous driving who need fast, memory-efficient scene capture, and to anyone interested in compact or token-based scene representations — the paper cites concurrent work on learnable-token alternatives. Beginners will find the pipeline conceptually clear but will need background on camera models, depth unprojection, and attention to follow the method section.

Authors’ abstract

Recent feed-forward 3D Gaussian Splatting (3DGS) methods have enabled effective multi-view 3D reconstruction by predicting Gaussian primitives in a single forward pass. However, existing approaches predominantly adopt a pixel-aligned paradigm that predicts one Gaussian per input pixel, coupling the Gaussian count to the input resolution and number of views. This also leads to redundant primitives in textureless regions and insufficient coverage in geometrically complex areas. We present Anchor3DGS, a pose-free feed-forward 3DGS framework that replaces pixel-aligned prediction with a compact anchor-based representation. Guided by information entropy, our method places a set of 3D anchors in the scene, concentrating them in regions of higher visual complexity while maintaining broad spatial coverage. Each anchor performs occlusion-aware aggregation of multi-view image features, exchanges information with other anchors, and is decoded into a set of local Gaussians. A lightweight feature enhancement module further refines the coarse RGB output from Gaussian splatting and improves rendering fidelity. Extensive experiments on diverse benchmark datasets demonstrate that our Anchor3DGS achieves state-of-the-art novel view synthesis quality while only using an order of magnitude fewer Gaussian primitives.