tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction

AI summary

Overview

• Research area: Computer vision — feedforward 3D reconstruction, novel view synthesis, and efficient sequence modeling (Test-Time Training / linear-complexity architectures).
• Technical level: Advanced. The paper assumes familiarity with transformers, attention complexity, 3D Gaussian Splatting, Neural Radiance Fields, and state-space/linear-attention models.
• Scope (one sentence): The paper introduces a large 3D reconstruction model built on Test-Time Training blocks that compresses posed input images into the fast weights of a neural memory, then decodes that memory into explicit 3D formats (Gaussian Splats or triplanes) with linear complexity and support for streaming, autoregressive reconstruction.

Authors: Chen Wang, Hao Tan, Wang Yifan, Zhiqin Chen, Yuheng Liu, Kalyan Sunkavalli, Sai Bi, Lingjie Liu, Yiwei Hu (Adobe Research, University of Pennsylvania, UCI). arXiv:2602.20160v2 [cs.CV], 02 Mar 2026, CC BY 4.0.

What This Paper Is About

Existing feedforward 3D reconstruction models are limited to a small number of input views (for example, four), and the one method that reaches 32 views, Long-LRM, relies on bidirectional attention, which blocks further scaling and prevents streaming or causal use. Purely implicit latent-space 3D models render well but are slow to render and hard to control.

tttLRM's goal is to reconstruct explicit 3D representations — usable for real-time rendering and downstream tools — from long or continuously arriving streams of images, using an architecture whose cost grows linearly with the number of input tokens.

Key Contributions

1. The first large reconstruction model that uses Test-Time Training for both feedforward long-context and autoregressive 3D modeling with linear complexity.
2. A unified, scalable 3D modeling framework that interprets TTT fast weights as observable and controllable explicit 3D representations, rather than treating them as an opaque internal state.
3. State-of-the-art results on object-level and scene-level datasets, with superior quality and efficiency against feedforward and optimization-based baselines.
4. Practical machinery for scale: sequence parallelism for distributed feedforward reconstruction, and an online/autoregressive inference variant that refines the reconstruction as new view batches arrive.

Main Findings

• Object-level results (Google Scanned Objects, 100 objects): At 256×256 with 8 input views, tttLRM reaches 33.14 PSNR / 0.972 SSIM / 0.024 LPIPS at 0.1s, versus GS-LRM at 31.55 / 0.964 / 0.028 at 0.1s. At 512×512, 8 views: 34.02 / 0.974 / 0.025 at 0.3s versus GS-LRM at 32.83 / 0.969 / 0.029 at 0.7s.
• Speed and quality at higher resolution: The authors report that at 512×512 their model runs twice as fast as attention-based models while achieving over a 1 dB PSNR improvement; GS-LRM encounters out-of-memory issues under high-resolution training, while tttLRM scales to 1024×1024.
• Generalization across view counts (object-level): Trained with 8 input views, the model is applied directly at 16 and 24 views (using 10 virtual views to query the fast weights), reaching 34.67 and 34.80 PSNR respectively, outperforming GS-LRM's 33.55 and 33.26 while running in 0.8s and 1.1s versus 2.5s and 5.5s.
• Scene-level results (DL3DV-140, 16 views): tttLRM (single model) reaches 23.60 PSNR / 0.784 SSIM / 0.255 LPIPS in 3.6s, versus Long-LRM (16v model) at 22.66 / 0.740 / 0.292 in 0.4s, and Scaffold-GS at 22.13 / 0.738 / 0.250 trained for 30k iterations in 16 minutes.
• Scene-level results (DL3DV-140, 32 views): tttLRM reaches 25.07 PSNR / 0.822 SSIM / 0.215 LPIPS in 7.2s; Long-LRM (32v model) reaches 24.10 / 0.783 / 0.254 in 1s, and Long-LRM with post-optimization reaches 24.99 / 0.809 / 0.243 in 12s. The autoregressive variant of tttLRM reaches 24.31 / 0.803 / 0.237 in 7.5s.
• Scene-level results (DL3DV-140, 64 views): tttLRM reaches 25.95 PSNR / 0.844 SSIM / 0.195 LPIPS in 14.8s, versus Long-LRM (64v model) at 24.63 / 0.799 / 0.243 in 3.7s. The autoregressive variant reaches 24.81 / 0.814 / 0.225 in 15.2s.
• Approximately 1 dB PSNR improvement over Long-LRM is reported consistently across different numbers of input views on scene data, and tttLRM is reported to outperform Long-LRM even when Long-LRM is combined with additional post-optimization.
• One model, many sequence lengths: Long-LRM trains a separate model for each input view count, whereas tttLRM is a single model across all input views; it also transfers to Tanks&Temples (18.15/0.613/0.360 at 16 views; 19.22/0.662/0.305 at 32 views; 20.31/0.700/0.274 at 64 views).
• Pretraining transfers from novel view synthesis: Initializing from TTT-LVSM pretrained weights accelerates convergence and improves final quality. For Gaussian Splatting, 32.77 PSNR without pretraining versus 33.14 with pretraining; for triplanes, 26.40 versus 27.87 PSNR, with SSIM improving from 0.903 to 0.925 and LPIPS from 0.093 to 0.075.
• Autoregressive reconstruction works progressively: With only 4 input views the model already produces reasonable 3D Gaussians, and quality and scene coverage improve as 8 and 32 views arrive. The autoregressive model consistently outperforms Long-LRM and stays competitive with or superior to optimization-based baselines.
• Full reconstruction beats "Predict & Merge": Reusing previously predicted Gaussians and merging new ones degrades quality (21.50 PSNR / 0.891 SSIM / 0.318 LPIPS) versus full reconstruction (23.63 / 0.904 / 0.259), compared on 32 views under 1K iterations of finetuning, because accumulated errors cannot be corrected.
• Optimizer and regularization matter: On 32 views at 256×144, using Muon improves PSNR from 20.44 to 20.68 and reduces the fraction of Gaussians with opacity > 0.001 from 96% to 97%; adding depth and opacity regularization raises PSNR to 20.76 and cuts that fraction to 47%.
• Selective updating of fast weights in the autoregressive setting helps: A training-free mechanism that uses an exponential moving average of squared gradients as a Fisher-information estimate to anchor less important parameters improves 24.81 PSNR / 0.814 SSIM / 0.225 LPIPS to 24.95 / 0.818 / 0.223.
• Scaling to hundreds of views: Finetuning the full model with more iterations on 128 input views (more than 1M tokens) achieves 26.80 PSNR.
• Fixed-size memory is a real constraint: The authors' analysis on DL3DV scene labels shows degraded performance with higher scene complexity — outdoor versus indoor scenes (PSNR 24.45 vs 24.96) and high- versus low-frequency scenes (24.20 vs 25.97). Earlier inputs may be gradually forgotten as more tokens are processed.
• Attention would bottleneck the design: The authors deliberately avoid attention blocks because attention costs O(N²d) versus O(Nd²) for LaCT blocks; their measurement shows even 3 attention layers become slower than 24 layers of LaCT blocks from 2M tokens (256 views).

Methodology in Plain English

The core object: fast weights as a 3D memory. Test-Time Training replaces the usual attention mechanism with a set of "fast weights" — a fixed-size neural memory that is updated at inference time from the input itself. Each input token provides a key and value pair, and the memory is updated with a gradient step while a separate set of tokens queries it. In tttLRM, this memory becomes the implicit 3D representation of the scene.

Architecture. Posed input images are concatenated channel-wise with ray embeddings (9 channels) for positional information, split into non-overlapping patches of size 8×8, and tokenized by a lightweight linear layer. The tokens flow through 24 LaCT (Large Chunk Test-Time Training) blocks with a hidden dimension of 768. Each block applies window attention to capture local relationships within a view, then updates the fast weights, then applies them back to the tokens. The update and apply operations are linear in sequence length. The window attention heads are 64-dimensional with QK-normalization for stability, and the feedforward layer is a two-layer MLP with a 4× intermediate expansion ratio. The Muon optimizer performs the fast-weight updates.

Querying with virtual tokens. To read the 3D scene out of the memory, the model introduces "virtual tokens" that only participate in the apply step and never update the weights. For Gaussian Splatting these are virtual views, whose decoded outputs give per-patch Gaussian color, scale, rotation, and opacity, plus a depth per pixel converted to real depth with a range function (object-centric for objects, linear for scenes) and then to 3D positions using the known ray directions. For triplane NeRFs, the virtual tokens are learnable triplane features, and the queried triplanes feed a NeRF decoder.

Streaming and autoregressive inference. Instead of processing all views jointly, the streaming variant processes one mini-batch of views at a time (for example, four images), updates the fast weights causally, and immediately predicts Gaussians for the corresponding query views. This turns inference into an RNN-like process where the internal state evolves as observations arrive. A variant of the fast-weight update can also consider historical gradients and weights to mitigate drifting.

Distributed training. Input tokens are sharded along the sequence dimension across devices. Each GPU predicts Gaussians for its assigned virtual views after the fast weights are synchronized, the Gaussians are gathered into the full scene, each GPU renders its own novel views and computes photometric losses, and gradients are all-reduced. Because the LaCT fast-weight updates are linear, gradients synchronize cleanly through PyTorch Distributed Data Parallel. At inference, the same scheme accelerates reconstruction with more GPUs.

Training objective. No explicit 3D supervision is needed. The model renders the reconstructed Gaussians at target views and minimizes an MSE plus VGG-19-based perceptual loss. Scene-level training adds a scale-invariant depth loss aligned with pseudo ground truth from a monocular depth estimator (chosen over feedforward multi-view stereo methods like VGGT because the latter gave less detailed depth), plus an opacity regularization to reduce Gaussian count, following Long-LRM.

Data and training setup. Object-level training uses Objaverse: 730K objects, each normalized into a [-1, 1] bounding box, 32 rendered views per object at 512×512 under uniform lighting, with cameras at distances uniformly sampled from [1.5, 2.8]. Scene-level training uses DL3DV-10K (10,510 high-resolution videos, up to 500 keyframes with COLMAP poses) and evaluates on DL3DV-140 (140 test scenes) and Tanks&Temples. Scene training proceeds through a curriculum of three resolutions — 144×256, 288×512, and 540×960 — with specific learning rates, batch sizes, and iteration counts described in the appendix (peak learning rates of 3e-4, 5e-5, and 1e

Authors’ abstract

We propose tttLRM, a novel large 3D reconstruction model that leverages a Test-Time Training (TTT) layer to enable long-context, autoregressive 3D reconstruction with linear computational complexity, further scaling the model's capability. Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer, forming an implicit 3D representation in the latent space that can be decoded into various explicit formats, such as Gaussian Splats (GS) for downstream applications. The online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations. We demonstrate that pretraining on novel view synthesis tasks effectively transfers to explicit 3D modeling, resulting in improved reconstruction quality and faster convergence. Extensive experiments show that our method achieves superior performance in feedforward 3D Gaussian reconstruction compared to state-of-the-art approaches on both objects and scenes.