Research
TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Overview Research area: Computer vision — feed-forward novel-view synthesis with Transformer-based "Large View Synthesis Models" (LVSMs), and specifically the efficiency of decoder-only variants at…

- arXiv
- 2611.17331
- Published
- 2026-11-12
- Authors
- Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
AI summary
Overview
Research area: Computer vision — feed-forward novel-view synthesis with Transformer-based "Large View Synthesis Models" (LVSMs), and specifically the efficiency of decoder-only variants at render time.
Technical level: Intermediate. The paper assumes familiarity with attention, KV-caching, and the idea of Test-Time Training (TTT) layers, but its central idea (baking a mapping into a small MLP) is explained clearly enough for readers who know standard Transformer rendering pipelines.
Scope: The paper proposes Test-Time Baking (TTB), a method that replaces cross-view attention's linear-cost KV-cache lookup at render time with a constant-cost fast-weight MLP trained during context prefilling, and evaluates it on DL3DV, Re10K, and three in-the-wild datasets.
What This Paper Is About
Decoder-only view synthesis models such as DVSM produce high-quality novel views by storing all context views in a KV-cache, but every novel-view token must attend to that cache, so rendering cost grows linearly with the number of context views. The fast alternative, LaCT's Test-Time Training (TTT) layers, renders in constant time but visibly degrades quality. This paper asks whether the inner-loop optimization of a fast-weight MLP can be retargeted to learn from cross-view attention's output rather than replace attention, recovering most of the lost quality at the same constant rendering cost.
Key Contributions
-
Test-Time Baking. The paper retargets the TTT fast-weight MLP's inner-loop objective from raw value tokens (the
W_v Xtarget used in LaCT) to the output of cross-view attentionCrossAttn(X). The MLP thus collaborates with attention instead of substituting for it, and only the updated fast weights — not the KV-cache — need to be stored. -
Diagnosis and fix for modality misalignment. Context tokens used during prefilling contain both camera and image content, while novel-view tokens at render time are camera-only. The authors disentangle each context view into a context ground-truth token (pose + RGB) that supplies the baking target and a context rendering token (pose only, matching novel-view modality) that serves as the MLP input, so the baked MLP sees the same modality at baking and rendering.
-
A light-field interpretation of the inner loop. The paper frames TTB as few-step latent-space light-field optimization — mapping ray-patch latent coordinates to appearance latents — which motivates a convergence-oriented recipe using MSE and a small number of inner steps (
T = 3). -
Fast-weight local-duplicate inference. Context views are clustered into
K = V/Bspatially local groups by k-means on camera poses, each group gets its own copy of the per-layer fast weights, and each novel view queries only the nearest group's copy, keeping renderingO(1)per view while letting the state grow linearly withK.
Main Findings
-
Large speed-quality trade-off win on DL3DV (960p). TTB renders at a constant 23 FPS regardless of context-view count, described as up to 23x faster than DVSM and 2.3x faster than LaCT. With 16 input views, DVSM reaches 26.86 PSNR at 4.4s (4 FPS) while TTB reaches 27.00 PSNR at 5.4s (23 FPS). With 64 views, DVSM reaches 31.11 PSNR at 60s (1 FPS) versus TTB at 30.20 PSNR at 65s (23 FPS).
-
Quality well above the TTT baseline. LaCT reports 24.70 / 26.90 / 28.30 PSNR at 16 / 32 / 64 views (10 FPS); TTB reports 27.00 / 28.92 / 30.20 at the same view counts.
-
Duplicate inference closes the residual gap. With local-duplicate inference enabled (
B = 16), "Ours (duplicate)" reaches 29.26 PSNR at 32 views and 31.33 PSNR at 64 views, described as approaching DVSM quality at the same rendering speed. -
The attention-free variant is competitive and cheaper. "Ours–", which replaces the cross-view attention target with a linear projection
W_v X^(ctx-gt), reports 26.26 / 28.12 / 29.29 PSNR at 16 / 32 / 64 views with 23 FPS and lower reconstruction time (e.g., 7.5s at 64 views), comparable to LaCT with fewer parameters and faster rendering. -
Near-DVSM quality on Re10K with two views (512x512). DVSM reports 31.17 / 0.924 / 0.129 and TTB reports 31.28 / 0.926 / 0.127 (PSNR / SSIM / LPIPS); the authors note this two-view, low-resolution setting is not the main target scenario because the rendering-time gain from cross-view attention becomes marginal there.
-
Stronger parameter efficiency than LaCT in the in-the-wild table. On 64 views at 960p, LaCT has 379M parameters while TTB has 271M, with TTB reporting higher PSNR on MipNeRF360 (24.49 vs 23.01), Free (26.05 vs 24.61), and Hike (23.77 vs 22.39) at 23 FPS versus 10 FPS.
-
Clear margin over LagerNVS at lower resolution. At 64 views x 512 pixels on the longer side, TTB reports 23.62 / 24.92 / 23.52 PSNR on MipNeRF360 / Free / Hike at 67 FPS, versus LagerNVS at 20.98 / 19.50 / 18.54 at 7 FPS, described as a large quality margin with 3.1–9.5x faster rendering.
-
Ablation decomposes the TTT-to-TTB gap. At
V = 32, starting from a LaCT-style TTT replacement at 23.12 PSNR: adding context pairing gives 23.78, adding cross-view attention as target gives 24.13, adding MSE withT = 3gives 24.22, and combining both additions gives 24.84, while retaining roughly 50 FPS rendering. DVSM in the same ablation setup is 24.68. -
MSE beats dot-product for the inner loop, and TTB benefits more. Switching from LaCT's unnormalized dot-product loss to MSE improves PSNR by +0.27 for TTT and +0.44 for TTB–; TTB– improves monotonically but with diminishing returns from
T = 1(23.78) toT = 5(24.29). -
Residual learning and feedback each help. Learning the residual to the target and feeding the baked MLP output back to the context ground-truth tokens give +0.43 and +0.44 PSNR in isolation and +0.52 combined (24.32 without either, 24.84 with both).
-
Wider effective view field. At TTB block 7 with
V = 16, input-gradient saliency shows theW_v Xtarget peaks sharply at the top-1 context view and decays, while theCrossAttn(X)target flattens that peak and sustains a heavier mid-rank tail.
Methodology in Plain English
The starting point is DVSM, a decoder-only model with 12 blocks where each block does intra-view attention, a feed-forward network, cross-view attention, and another feed-forward network. Context views are prefilled into a KV-cache at every cross-view attention layer; novel-view tokens then read from that cache.
TTB keeps the architecture but swaps each cross-view attention layer for a TTB layer built on a fast-weight MLP (a single-headed SwiGLU-MLP with hidden dimension 3072). During prefilling, the model runs a small inner optimization loop: it takes pose-only "context rendering" tokens as input, computes what cross-view attention would have produced for "context ground-truth" tokens that also carry RGB, and updates the MLP weights with a few Muon steps to map the former to the latter (specifically to the residual between that output and the input token). The learned weights are the scene representation. At rendering, novel-view tokens — which are also pose-only, matching the input modality of the baked MLP — pass through the same baked MLP in a single cost-independent step instead of attending over a growing cache.
Training is end-to-end: the rendering loss is backpropagated through the inner-loop baking process, and models are initialized from the corresponding DVSM checkpoint. Configuration: 768 latent channels, patch size 8, AdamW with cosine annealing, perceptual loss weight 0.2, trained on DL3DV-10K on 64 A100 GPUs (100K iterations at 340p at learning rate 4x10^-4, then 10K iterations each at 480p, 720p, and 960p at 1x10^-5), sampling 16 or 32 context views per scene. Evaluation uses 140 held-out DL3DV test scenes. Ablations use a smaller 8-A100 setup at patch size 16 truncated to 720p, with efficiency metrics still reported at patch size 8.
Why This Matters
The paper reframes an architectural either/or — attention or TTT — as a both/and: pay more at context-prefilling time (which happens once per scene) to make rendering cheap and high-quality. That framing matters for any system where a scene is captured once but viewed thousands of times interactively.
- Real-world applications:
- Interactive AR/VR and headset rendering — a scene is captured once, then rendered continuously as the user moves; constant 23 FPS at 64 views is directly relevant to frame budgets.
- Telepresence and volumetric video calls — many cameras at the sender, a single low-latency renderer at the receiver.
- Digital twins and real-estate/product capture — quick multi-view capture of a physical space or object for walkthrough viewing on modest hardware.
- Simulation for robotics and autonomous driving — re-rendering a logged drive from new viewpoints at interactive rates.
- Industry relevance: the work builds directly on NVIDIA's DVSM line and was supported by the NVIDIA Taiwan AI Research & Development Center, so its efficiency claims speak to deployable inference cost rather than only benchmark scores. The paper also reports parameter counts (e.g., 271M for TTB versus 379M for LaCT), which matters for on-device and batch-serving budgets.
Future Directions
- Learned duplication routing. Fast-weight local-duplicate inference currently clusters views with k-means on camera poses; the authors explicitly call this a heuristic and leave learning the routing during training as future work.
- Breaking the fast-weight capacity ceiling. Quality saturates as the number of context views
Vgrows because the MLP has fixed capacity — the core limitation the duplication trick only mitigates. - Removing the quadratic reconstruction cost. Because cross-view attention remains inside the inner loop, prefilling scales quadratically with
V, which the authors flag as a drawback for single-shot scenarios; the linear-projection "Ours–" variant is a partial answer. - Beyond interpolation. TTB is built on reconstruction rather than generation, so it interpolates between context views and does not extrapolate to unobserved viewpoints — extending the baking idea to generative or extrapolative settings is an open question.
Target Audience
Graduate students and research engineers working on novel-view synthesis, feed-forward 3D reconstruction, or efficient Transformer inference. It is most valuable to readers who already know the LVSM/DVSM family or Test-Time Training layers and want to understand where the quality gap in TTT-based rendering comes from and how to close it. Practitioners concerned with rendering latency and memory in production view-synthesis systems will also find the FPS, parameter, and reconstruction-time tables directly useful.
Authors’ abstract
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key…
The full abstract on arXiv