Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention

AI summary

Overview

Research area: Computer vision — feedforward novel view synthesis (NVS), specifically transformer-based Large View Synthesis Models that render new camera views of a 3D scene from a set of posed input images without any explicit 3D representation.

Technical level: Advanced. The paper is about transformer attention architecture design, complexity analysis, and training/inference efficiency, and assumes familiarity with self-attention, cross-attention, KV-caching, and PSNR/SSIM/LPIPS metrics.

Scope: The paper diagnoses two flaws in the LVSM monolithic full-self-attention design and proposes a decoupled dual-stream, co-refining encoder–decoder replacement, then benchmarks it for quality, speed, memory, and generalization.

What This Paper Is About

Feedforward NVS models such as LVSM concatenate all input-view tokens and target-view tokens into one sequence and run full self-attention over them. The authors argue this is suboptimal for two reasons: cost grows quadratically with the number of input views, and the same parameters are forced to serve two very different jobs — content-rich input tokens that must understand scene semantics and geometry, and pose-only target tokens that must render a new image. The goal of the paper is to decouple those two roles into separate streams so that the model is simultaneously faster, cheaper to train, and more accurate.

Key Contributions

1. A dual-stream architecture. An Input Encoder applies intra-view self-attention so each input view is processed independently, and a Target Decoder applies self-then-cross attention so target tokens refine themselves and then query input-view features through cross-attention. This reduces complexity with respect to the number of input views from quadratic O(N_in²) to linear O(N_in).
2. Co-refinement between encoder and decoder. Instead of using only the encoder's final layer, each decoder layer queries the corresponding intermediate encoder layer, so fine-grained early-layer detail and high-level late-layer semantics are both available to the renderer.
3. KV-cache and incremental inference. Because input views are encoded independently, their keys and values can be cached; adding a new input view costs one view's processing, and adding a new target view reuses the cache, giving near-constant latency and memory.
4. State-of-the-art results with a smaller training budget, including conditional distillation from a frozen DINOv3 teacher via REPA, which the paper reports helps its dual-stream design much more than it helps LVSM.

Main Findings

• Scene-level state of the art: Efficient-LVSM reaches 29.86 dB PSNR on RealEstate10K at res-512, which the paper reports as surpassing LVSM by 0.2 dB and GS-LRM by 1.7 dB, corresponding to a 4.5% reduction in Mean Squared Error. In the paper's Table 2 the comparable numbers are Ours (res-512) 29.86 PSNR / 0.905 SSIM / 0.147 LPIPS versus LVSM Dec-Only (res-512) 29.53 / 0.904 / 0.141 and GS-LRM 28.10 / 0.892 / 0.114.
• Res-256 nuance: In the same table, Ours (res-256) scores 28.93 PSNR / 0.895 SSIM / 0.102 LPIPS, below LVSM Dec-Only (res-256) at 29.67 / 0.906 / 0.098. The headline gain appears at res-512.
• Object-level gains: On ABO at res-512, Ours reaches 32.65 PSNR / 0.951 SSIM / 0.042 LPIPS versus LVSM Dec-Only 32.10 / 0.938 / 0.045 and GS-LRM 29.09 / 0.925 / 0.085. On GSO at res-512, Ours reaches 32.92 / 0.973 / 0.021 versus LVSM Dec-Only 32.36 / 0.962 / 0.028 and GS-LRM 30.52 / 0.952 / 0.050.
• Training cost halved: The paper states the model was trained for 3 days on 64 A100 GPUs, half the training time required by LVSM, i.e. 50% of the training budget, with roughly 2× faster training convergence and 4.4× faster inference speed; elsewhere it describes 2–4× speed acceleration in both training iteration and inference.
• Scaling efficiency with view count: With 16 input views, the model is approximately 14.9× faster and consumes 50% less memory than LVSM Decoder-Only. Latency in the efficiency table is 24.78 ms for Ours (199M parameters, 1325 GFLOPS) versus 109.37 ms for LVSM Dec-Only (177M, 8523 GFLOPS) and 88.24 ms for GS-LRM (307M, 5047 GFLOPS).
• Incremental inference: Feeding input views one at a time yields near-constant latency and memory for Efficient-LVSM, while both LVSM baselines show clear growth.
• Zero-shot generalization: Trained with 4 input views and tested with varying numbers of input views, Efficient-LVSM consistently outperforms LVSM at every setting, with the gap gradually narrowing as more input views make reconstruction easier.
• Ablation — architecture: With a smaller 6+6 configuration, cross-attention only scores 24.18 PSNR, self-then-cross ("Self-Cross Attention") scores 24.97 (a 0.79 dB gain), and full co-refinement scores 26.25 (a further 1.28 dB gain).
• Ablation — REPA distillation: Applying REPA gives Efficient-LVSM a 0.8 dB gain (26.02 → 26.81 PSNR) but gives LVSM Dec-Only only 0.16 dB (25.52 → 25.68). Smooth L1 loss (26.81 PSNR) beats L2 (26.39) and cosine (26.30); distilling both input and target tokens (26.60) beats either alone (26.35 / 26.27); and DINOv3 layer 8 (26.60) beats layers 10 (26.28) and 12 (26.11).
• Ablation — model size: Enc(12)+Dec(12) at 199M parameters reaches 28.32 PSNR, Enc(6)+Dec(6) at 101M reaches 27.77, and Enc(3)+Dec(3) at 53M reaches 26.43.
• Alternative designs underperform: In the comparison of alternatives, Co-Refinement (101M) scores 26.02 PSNR at 17.58 ms latency and 647 GFLOPS, while LVSM scores 25.24 at 74.45 ms / 3487 GFLOPS, LVSM w/ mask scores 24.13 at 125.13 ms, LVSM w/ MMDiT scores 24.37 at 78.58 ms, and LVSM w/ mask+MMDiT scores 23.24 at 130.76 ms.
• Feature evolution: PCA visualizations show features becoming progressively more semantic from layer 1 to layer 6, then becoming similar to the output RGB images from layer 6 to layer 12.

Methodology in Plain English

The authors start by taking LVSM's feedforward formulation as given: input images and Plücker ray embeddings are patchified into tokens, target camera rays are patchified into target tokens, and a transformer stack maps them to rendered RGB patches through a linear layer and a sigmoid, followed by unpatchifying.

They then replace the single monolithic attention stack with two cooperating streams. The encoder looks only at each input view on its own (intra-view self-attention plus a feed-forward network), which keeps one view's encoding independent of the others and makes the cost linear in the number of views. The decoder holds the target tokens: at each layer the target tokens first self-attend among themselves to build scene-level context, then cross-attend to the encoder's tokens of that same layer to pull in scene content, then pass through a feed-forward network. Cross-attending to the matching layer, rather than only the last layer, is what the authors call co-refinement — it lets the decoder use both fine details and high-level semantics.

Because input keys and values are produced independently of the targets, they can be stored in a KV-cache and reused for any number of target views, and a newly supplied input view only needs its own encoding appended.

For extra accuracy, the authors add REPA distillation: a frozen DINOv3-ViT-B/16 teacher encodes the clean image, and a small trainable 3-layer MLP projects the student's hidden tokens (input or target, at a chosen layer) to be aligned with the teacher's patch features using Smooth L1 loss. The teacher and projector are discarded at inference, so speed is unaffected.

Evaluation covers two regimes: scene-level RealEstate10K (80K video clips from 10K YouTube videos, following LVSM's split) with 2 input and 3 target views, and object-level training on Objaverse (730K objects rendered, 32 random views each) with 4 input and 8 target views, tested on GSO (1099 objects) and ABO (1000 objects) with 4 structured input views and 10 random target views. The main model uses 8×8 patches, 24 transformer layers (12 encoder, 12 decoder), and hidden dimension 1024, trained with AdamW at a constant learning rate of 4e-4 with 2500 warmup iterations and weight decay 0.05 on LayerNorm parameters.

Why This Matters

The paper reframes a quality-versus-cost trade-off in feedforward 3D reconstruction as an attention-design question, and shows that matching the attention pattern to the role of each token beats simply scaling a monolithic transformer. If the reported efficiency holds, it puts multi-view latent rendering in the latency range where interactive use becomes plausible, and it gives a concrete template for decoupling asymmetric token types in other multimodal transformer pipelines.

Real-world applications:

• Interactive 3D content creation and scene editing, where a user adds photos and instantly previews new camera angles.
• Virtual tours for real estate and architecture, which is exactly the domain of the RealEstate10K benchmark used.
• E-commerce product visualization, matching the object-level ABO and GSO evaluations, where 3D previews are rendered from a handful of product shots.
• AR/VR and simulation content pipelines that need low-latency view synthesis on constrained hardware.

Industry relevance: the paper's latency (24.78 ms), parameter count (199M), and GFLOPS (1325) figures matter for deployment budgeting, and the KV-cache behavior matters for streaming or incremental-capture workflows where views arrive over time. The public code/model release pledged in the reproducibility statement lowers the barrier for adoption.

Future Directions

• Scaling behavior: The paper confirms larger models reconstruct better (53M → 101M → 199M parameters giving 26.43 → 27.77 → 28.32 PSNR in the ablation setting), but how far this scales, and whether the dual-stream advantage persists at much larger sizes, is not established.
• Video and long-sequence capture: The near-constant incremental inference cost suggests streaming or on-the-fly capture scenarios, but no streaming or video experiment is reported.
• Wider distillation and backbone choices: REPA with DINOv3 was studied only in terms of loss function, token target,

Authors’ abstract

Feedforward models for novel view synthesis (NVS) have recently advanced by transformer-based methods like LVSM, using attention among all input and target views. In this work, we argue that its full self-attention design is suboptimal, suffering from quadratic complexity with respect to the number of input views and rigid parameter sharing among heterogeneous tokens. We propose Efficient-LVSM, a dual-stream architecture that avoids these issues with a decoupled co-refinement mechanism. It applies intra-view self-attention for input views and self-then-cross attention for target views, eliminating unnecessary computation. Efficient-LVSM achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed. Efficient-LVSM achieves state-of-the-art performance on multiple benchmarks, exhibits strong zero-shot generalization to unseen view counts, and enables incremental inference with KV-cache, thanks to its decoupled designs.