VGG-T$^3$: Offline Feed-Forward 3D Reconstruction at Scale

AI summary

Overview

Research area: Computer vision — large-scale multi-view 3D reconstruction, feed-forward Transformer models, test-time training, and visual localization.

Technical level: Advanced. The paper assumes familiarity with Transformer attention, Key-Value (KV) representations, and test-time optimization, though its core idea can be grasped in plain terms.

Scope: The paper introduces VGG-T³ (Visual Geometry Grounded Test Time Training), a method that converts a quadratic-complexity multi-view 3D reconstruction model (VGGT) into a linear-time model by compressing its scene representation into a fixed-size MLP at test time.

What This Paper Is About

Offline feed-forward 3D reconstruction models predict scene geometry, camera poses, and depth directly from unposed image collections, but their compute and memory grow quadratically with the number of input images. This happens because these models store scene geometry in a variable-length Key-Value (KV) space that must be queried with global softmax attention. The paper's goal is to replace that variable-length KV scene representation with a fixed-size MLP optimized at test time, so that reconstruction scales linearly with input views while keeping the global scene aggregation that makes offline methods accurate.

Key Contributions

1. An offline feed-forward 3D reconstruction model, VGG-T³, whose cost scales linearly (O(n)) rather than quadratically (O(n²)) with the number of input views, by replacing global softmax attention with a test-time-trained fixed-size MLP.
2. A demonstration that models representing scene geometry with a variable-length implicit representation (the KV space) can be "converted" into linear-time models via a fixed-dimensional implicit state representation.
3. Inference strategies enabled by this representation change: processing arbitrarily large image collections on a single GPU via mini-batch off-loading to host memory, and distributed multi-GPU inference via gradient synchronization in the global layers.
4. A proof-of-concept unified model that performs both mapping (optimizing the MLP) and feed-forward visual localization (querying the frozen MLP with unseen images) within a single architecture.

Main Findings

• Linear scaling replaces quadratic scaling. The global attention operation changes from O(n²) to O(n). The abstract reports reconstructing a 1k image collection in 54 seconds with an 11.6× speed-up over softmax-attention baselines; Section 4.2 reports 1k images in 58 seconds, versus over 11 minutes for VGGT (11.6× slower) and more than 4 minutes for FastVGGT (4.3× slower). Elsewhere the paper reports a 2k image collection in 48.5 s, a 33× improvement over VGGT (27 min).
• Point map accuracy beats other linear-time methods. On 7scenes-D, 7scenes-S, DTU, ETH3D, NRGBD-D and NRGBD-S, VGG-T³ outperforms the O(n) baseline TTT3R on every benchmark except Chamfer Distance on 7scenes-D, where it is marginally worse. It reduces error by 2–2.5× on DTU, ETH3D, and NRGBD-D. Against O(n²) baselines it remains competitive and surpasses them on DTU.
• Video depth is competitive. On Bonn, KITTI and Sintel, VGG-T³ outperforms TTT3R on two of three datasets by a substantial margin and performs on par with O(n²) methods on KITTI.
• Camera pose estimation is the weak point. The authors report that their TTT-linearized model struggles on camera pose estimation (TUM-RGBD, ScanNet, Sintel). They hypothesize this relates to VGGT's dedicated camera token, which creates two input "modalities" that the MLP may struggle to memorize. VGG-T³ does support both ordered and unordered input sequences, whereas TTT3R degrades on unordered inputs.
• Test-time scaling helps at large sequence lengths. Reconstruction error increases about 5× when extending from N = 100 to N = 1k images of the same scene. Logging the top-performing optimizer step at 20 images (in-distribution) versus 1k images (out-of-distribution, roughly a 50× increase) showed one step suffices in-distribution but more steps help at 1k. The paper uses 2 steps unless otherwise noted.
• Distributed inference gives linear speed-up. For 2,000 images, latency is 230.7 s on 1 GPU, 74.8 s on 2 GPUs, and 48.5 s on 4 GPUs. VGGT runs out of memory on 1 GPU at 1,500 images and needs 1779.3 s on 2 GPUs (913.6 s on 4) for 1,500 images. TTT3R is not compatible with multi-GPU inference due to its autoregressive processing.
• Feed-forward visual localization improves over TTT3R. On 7Scenes, VGG-T³ reaches e_r = 6.71°, e_t = 0.16 m, localizing 40.69% of queries within 10 cm/10° and 73.00% within 20 cm/20°, versus TTT3R's 7.18°, 0.17 m, 34.59% and 70.21%. On Wayspots the gap is much larger: 32.04°/1.90 m with 13.41% and 30.64% localized, versus TTT3R's 74.45°/4.38 m with 0.69% and 2.94%. The paper notes that explicit-mapping pipelines such as Reloc3R achieve e_r = 1.02°, e_t = 0.04 m on 7scenes.
• Ablations confirm the design. Against a softmax-attention upper bound (CD 0.061, NC 0.844, mAA(30) 76.33), training from scratch with TTT reaches 0.262/0.727/52.95, T2R reaches 0.137/0.804/66.27, LoLCats reaches 0.097/0.804/62.87, the proposed linearization reaches 0.074/0.833/72.16, and adding ShortConv2D reaches 0.066/0.838/74.14. Linearizing a pretrained softmax-attention model is key; training with TTT from scratch gets stuck in a local optimum.

Methodology in Plain English

The authors start from VGGT, a pretrained multi-view Transformer that tokenizes images and alternates image-wise and global self-attention to predict depth, camera poses, and intrinsics. That global attention is what stores the scene: every image token is projected into query, key, and value vectors, and the variable-length set of keys and values acts as the scene's memory.

Instead of reading from that memory with softmax attention — an operation that compares every query against every key — the authors learn a small MLP that maps keys to values. This MLP is trained at test time with a simple self-supervised dot-product objective, so its weights become a fixed-size compressed version of the scene. To retrieve information for a given query, the model simply applies the MLP to that query, an operation that is linear in the number of images because it no longer depends on the full sequence of keys.

Two refinements make this work. First, the authors replace the LayerNorm in the query/key projections with L2 normalization, because LayerNorm's extra learnable parameters distort the input space the MLP must learn and slow test-time convergence. Second, they apply a 2D convolution ("ShortConv2D") to the value vectors before training, so the MLP must predict a spatially aggregated target from a single token's key — creating a nontrivial objective rather than the near-linear key-to-value relation that would otherwise admit a trivial solution.

Training starts from the public VGGT checkpoint, freezes all original VGGT parameters, and fine-tunes only the global attention layers for 100k steps on 8 NVIDIA A100-80GB GPUs, about 12% of the cost of training VGGT from scratch. The TTT layer uses a SwiGLU MLP, the Muon optimizer, and a dot-product loss.

Because the TTT objective is a sum over local per-token losses, its gradient is a sum of local gradients. This lets the authors split the work into mini-batches — sharding across GPUs for distributed inference, or off-loading mini-batches to host memory to handle arbitrarily large collections on one GPU. After reconstruction, the frozen MLP can be queried by a new image to estimate its pose relative to the scene, which is how the paper performs visual localization.

Why This Matters

Impact on research. This is a post-training linearization technique for multi-view 3D reconstruction, extending an idea previously explored mainly for large language models. It shows that the quadratic bottleneck of feed-forward reconstruction models stems from the KV scene representation itself, not from the encoder or decoder — and that replacing only the global attention layers preserves accuracy while changing the asymptotic complexity. It also offers a unified mapping-and-localization model, whereas those tasks traditionally require separate solutions.

Real-world applications:

• Large-scale structure-from-motion from tourist-sourced photo collections; the paper reconstructs Rome landmarks (Colosseum, Castel Sant'Angelo, Pantheon, Trevi Fountain) within about a one-minute time budget from unposed images.
• Visual localization for augmented reality or robotics, where a device queries a previously reconstructed scene with a new photo.
• Video depth estimation for autonomous driving and scene understanding (evaluated on KITTI, Bonn, Sintel).
• Single-GPU processing of very large image sets, which matters where multi-GPU infrastructure is unavailable.

Industry relevance. The method achieves its speed-up while reducing training cost to roughly 12% of training VGGT from scratch, and supports standard distributed data parallel inference instead of hand-engineered context-parallel implementations such as ring attention. Both lower the practical barrier to deploying large-scale feed-forward reconstruction.

Future Directions

• Closing the remaining accuracy gap to softmax attention, particularly in wide-baseline settings, by reconciling the fixed expressivity of the MLP scene representation with the accuracy of quadratic attention.
• Improving camera pose estimation under the TTT-linearized model, which the authors suspect requires handling VGGT's dedicated camera token and its two input "modalities" more effectively.
• Extending the feed-forward visual localization proof-of-concept toward the accuracy of explicit-mapping pipelines such as Reloc3R, which the authors present as an open direction rather than a solved problem.
• Further exploring test-time scaling — how many optimizer steps to use at different sequence lengths — since the paper's analysis shows the optimal number of steps depends on how far the input is out of distribution.

Target Audience

Researchers and engineers working on 3D reconstruction, structure-from-motion, visual localization, and efficient Transformer architectures. It is most useful to readers already familiar with attention mechanisms and multi-view geometry, as well as practitioners who need to scale reconstruction to thousands of images on limited hardware. Readers looking for an introduction to feed-forward 3D reconstruction may find the attention and test-time-training details challenging without background preparation.

Authors’ abstract

We present a scalable 3D reconstruction model that addresses a critical limitation in offline feed-forward methods: their computational and memory requirements grow quadratically w.r.t. the number of input images. Our approach is built on the key insight that this bottleneck stems from the varying-length Key-Value (KV) space representation of scene geometry, which we distill into a fixed-size Multi-Layer Perceptron (MLP) via test-time training. VGG-T$^3$ (Visual Geometry Grounded Test Time Training) scales linearly w.r.t. the number of input views, similar to online models, and reconstructs a $1k$ image collection in just $54$ seconds, achieving a $11.6\times$ speed-up over baselines that rely on softmax attention. Since our method retains global scene aggregation capability, our point map reconstruction error outperforming other linear-time methods by large margins. Finally, we demonstrate visual localization capabilities of our model by querying the scene representation with unseen images.