NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis

AI summary

Overview

Research area: Computer vision — zero-shot novel view synthesis (NVS) and 3D-aware generative modeling, specifically sparse-view multi-view image generation from posed source images.

Technical level: Advanced. The paper assumes familiarity with autoregressive transformers, vector-quantized autoencoders (VQ-VAE), rotary positional embeddings, diffusion-based view synthesis, and camera geometry in SE(3).

Scope: The paper proposes and evaluates NAMVIS, a diffusion-free, geometry-conditioned next-scale autoregressive framework that directly generates posed target views from one or more source images, benchmarked against representative diffusion-based baselines on three object datasets.

What This Paper Is About

Zero-shot novel view synthesis asks a model to produce photorealistic images of an unseen object from new camera viewpoints, given only one or a few posed input photographs and no per-scene optimization. The dominant approach fine-tunes large 2D diffusion models on relative camera pose, which achieves strong visual quality but requires iterative denoising that makes multi-view inference expensive. NAMVIS instead reformulates the task as geometry-conditioned next-scale autoregression over discrete visual tokens, generating all tokens within a scale and across all target views in parallel through a small number of coarse-to-fine steps, with camera geometry injected directly into the attention mechanism.

Key Contributions

1. A next-scale autoregressive NVS framework. NAMVIS predicts target views over coarse-to-fine scale blocks of discrete visual tokens rather than through iterative denoising, producing a diffusion-free alternative for sparse-view multi-view synthesis conditioned on a variable number of source images, their poses, and desired target poses.

2. Multi-scale Projective Pose Encoding (multi-scale ProPE). The paper extends projective camera-based pose encoding to the varying spatial resolutions of next-scale autoregression, injecting source and target camera transformations into both target-view self-attention and source-to-target cross-attention at every resolution.

3. A dual-path conditioning mechanism. Global source-view conditioning (a pooled start-of-sequence token derived from VQ-VAE source features, also used for adaptive layer normalization) is combined with dense, geometry-aware cross-attention over full-resolution source features, designed to preserve both semantic appearance and fine spatial detail.

4. Empirical evaluation and released data. NAMVIS is evaluated on Objaverse, GSO, and OmniObject3D against representative diffusion baselines, reporting improved PSNR, SSIM, and LPIPS plus faster inference under the same setting, and the authors release a filtered scene list, rendered RGBA images, camera poses, masks, point maps, captions, and preprocessing scripts for approximately 200K objects.

Main Findings

• Best perceptual and fidelity metrics on all three datasets. On in-domain Objaverse, NAMVIS 1B reaches 22.485 PSNR, 0.861 SSIM, and 0.091 LPIPS, versus 19.097/0.816/0.152 for Zero-1-to-3, 19.341/0.827/0.136 for Zero-1-to-3 XL, 15.801/0.821/0.209 for Wonder3D, 19.368/0.826/0.156 for SyncDreamer, and 19.078/0.816/0.147 for EscherNet.

• Strong zero-shot generalization. On GSO, NAMVIS reaches 21.715 PSNR, 0.843 SSIM, and 0.111 LPIPS; on OmniObject3D it reaches 21.098 PSNR, 0.845 SSIM, and 0.104 LPIPS. All methods degrade moving from Objaverse to GSO and OO3D, but the paper reports NAMVIS degrades more gracefully than the evaluated baselines.

• Inference efficiency. In the 1-to-1 setting, NAMVIS requires 0.6 seconds per target view, while the fastest evaluated diffusion baseline, Wonder3D, requires 2.0 seconds. The paper states NAMVIS is approximately 3.3× faster than the fastest diffusion baseline under the same 256×256 resolution setting.

• Scaling to larger view counts. On Objaverse, 6→6 joint generation achieves 26.12 PSNR, 0.9015 SSIM, 0.0473 LPIPS with 1.56 s inference time and 5.0 GB peak VRAM. The 2→6 setup reaches 24.49 PSNR, 0.8845 SSIM, 0.0654 LPIPS at 1.53 s and 4.3 GB, and the 6→2 setup reaches 26.07 PSNR, 0.9017 SSIM, 0.0457 LPIPS at 0.84 s and 4.1 GB.

• Better multi-view geometric consistency. Using COLMAP reconstructability, NAMVIS reconstructs an average of 221 sparse points, compared with 214 for EscherNet, 172 for Wonder3D, and 151 for SyncDreamer. Zero-1-to-3 and Zero-1-to-3 XL are excluded because they do not natively generate jointly conditioned multi-view sets.

• Better camera registration coverage. Across 30 scenes, NAMVIS registers 162 of 240 views (67.5%) versus 145 of 240 (60.4%) for EscherNet, approaching the 70.0% coverage (168/240) obtained with ground-truth images.

• ProPE beats Plücker ray conditioning. In ablations on a 10K-object Objaverse-XL subset trained for 200 epochs, the full model scores 21.360 PSNR, 0.907 SSIM, 0.115 LPIPS, while replacing ProPE with Plücker rays gives 20.260/0.899/0.142.

• Dense cross-attention is critical. Removing dense cross-attention (relying on the [SOS] token alone) drops performance to 17.891 PSNR, 0.865 SSIM, 0.248 LPIPS, and replacing the dual-attention design with prefix conditioning drops it to 14.539 PSNR, 0.797 SSIM, 0.427 LPIPS.

• VQ-VAE features outperform alternatives for conditioning. Using CLIP for the [SOS] token yields 20.613/0.901/0.126 and learned dense features yield 20.886/0.901/0.134, both below the full model.

• Full fine-tuning beats LoRA. LoRA tuning with rank r = 256 scores 18.912 PSNR, 0.883 SSIM, 0.185 LPIPS, substantially worse than full fine-tuning, which the authors interpret as evidence that adapting a pretrained autoregressive image prior to geometry-conditioned multi-view synthesis requires updating core transformer weights.

• Caching matters for geometry-aware attention. In a 3-to-3 setup, the model runs in 0.96 s; removing KV caching increases runtime to 1.06 s, and removing ProPE caching increases it to 1.28 s.

• Attention shows geometry-consistent behavior. Visualizations suggest self-attention emphasizes target-target consistency at finer scales, while cross-attention focuses on source-target alignment at coarser scales, with high weights often assigned to visually and geometrically corresponding regions across views.

Methodology in Plain English

Reframing synthesis as token prediction. NAMVIS is not a diffusion model. It builds on next-scale autoregressive image generation (VAR and Infinity), where an image is encoded by a multi-scale VQ-VAE into a sequence of residual token maps whose spatial resolution increases with each scale. The transformer predicts these tokens scale by scale, from coarse to fine, where the continuous feature at scale k is the sum of the upsampled residuals from scales 1 through k. NAMVIS keeps Infinity's bitwise representation, where each token is quantized to a binary code and the transformer predicts each bit independently through binary classifiers, with a bitwise self-correction strategy for training stability.

Multi-view sequence layout. To generate N target views concurrently, token sequences are interleaved scale by scale into a combined sequence. A block-causal mask enforces the autoregressive property: tokens at scale k attend to all tokens at scales up to and including k, across all views, but are masked from future scales. Within a scale block, tokens from different views attend to each other freely, so views communicate at every resolution. A learned scale-level embedding is added before the first transformer block so the model can distinguish resolutions.

Injecting camera geometry. Standard 2D rotary embeddings capture image-plane position but not 3D camera relationships. NAMVIS uses a multi-scale projective positional encoding based on ProPE: each attention head is split into one projective part and two local rotary parts. The projective part multiplies by branch-specific camera matrices — P_v transpose for queries, P_v inverse for keys and values, and P_v for outputs — while the two RoPE parts preserve local 2D layout at the current autoregressive resolution. Because generation spans multiple scales, separate coordinate grids and rotary coefficients are constructed per scale. This operator is applied in two routes: target self-attention, where queries, keys, values, and outputs all use the target-view camera, and source-to-target cross-attention, where queries and outputs use the target camera while keys and values use the source cameras.

Conditioning on reference images. Source images pass through two complementary paths. The global path uses a learnable query that cross-attends to source features to produce a pooled [SOS] vector, which is prepended to the target sequence (duplicated per target view) and also mapped to scale and shift parameters for AdaLN modulation of every transformer block. The dense path performs cross-attention in every block against unpooled VQ-VAE source features, modulated by ProPE. Crucially, source keys and values stay at full single-scale resolution even while target queries progress through scales, so even the coarsest generation stages are anchored by high-fidelity source detail.

Training. The VQ-VAE is frozen; only the autoregressive transformer is trained, with a scale-reweighted cross-entropy loss over the independent bit predictions. Training samples random reference and target views from the same scene, computes multi-scale residual token maps only for targets, and drops source conditioning with probability 10% to enable classifier-free guidance at inference. The optimizer is AdamW with base learning rate 1×10⁻⁴, global batch size 512, 20 warm-up epochs followed by cosine decay, and 500 total epochs. The model is a 1B-parameter transformer initialized from the Infinity 2B checkpoint with every other block kept (16 blocks, hidden dimension 2048, 16 attention heads). The training data is a filtered Objaverse-XL subset of approximately 200K objects, each rendered as 100 posed RGBA views with intrinsics, extrinsics, masks, point maps, and captions; filtering removes failed renders, low foreground occupancy, missing or predominantly white textures, geometry artifacts, low aesthetic-quality renders, and degenerate captions.

Evaluation. Metrics are computed at 256×256 resolution, averaged across nine source–target configurations (1-to-1, 1-to-2, 1-to-3, 2-to-1, 2-to-2, 2-to-3, 3-to-1, 3-to-2, 3-to-3). Baselines use official pretrained weights; single-view baselines are evaluated per available source view with the best PSNR reported. Wonder3D is evaluated at 0° elevation while other methods are evaluated at 30° elevation, per their supported protocols; NAMVIS is evaluated at both angles.

Why This Matters

Impact on research. The paper tests whether the next-scale autoregressive paradigm — which has matched or surpassed diffusion quality in text-to-image generation — transfers to 3D-aware multi-view synthesis. If the reported results generalize, it opens an alternative axis for organizing NVS research: instead of the usual split between iterative (optimization-based, diffusion-based) and single-pass (feed-forward) methods, NAMVIS positions discrete next-scale autoregression as an efficient generative route that avoids per-scene optimization without iterative denoising. It also shows that adapting a pretrained autoregressive image prior to this task requires full fine-tuning rather than LoRA, and that injecting geometry into attention (ProPE) outperforms concatenating explicit ray features (Plücker raymaps) in this setting.

Real-world applications (as implied by the paper's framing):

• Augmented reality, where posed views of an object must be synthesized quickly on constrained hardware.
• Robotics, where generating consistent novel views supports perception and manipulation planning.
• 3D content creation pipelines that need multiple consistent views of an unseen object from one or a few photographs.
• Multi-view asset generation feeding downstream reconstruction: the reported COLMAP reconstructability and registration-coverage results suggest the generated views support stable feature matching, camera registration, and triangulation, which matters for photogrammetry-style workflows.

Industry relevance. Inference latency is a practical constraint on deployed generative 3D tools, and the paper reports 0.6 s per target view in the 1-to-1 setting, with a 6→6 configuration at 1.56 s and 5.0 GB peak VRAM, versus 16.6 s for SyncDreamer in the comparison table. The release of a filtered 200K-object dataset with rendered RGBA images, poses, masks, point maps, captions, and preprocessing scripts lowers the barrier for groups without large rendering pipelines. Together, the speed and consistency results target production use cases where per-scene optimization is too slow.

Future Directions

1. Higher-resolution generation. The current checkpoint is trained and evaluated only at 256×256, so direct 512×512 inference would require additional training or fine-tuning, as is common for transformers trained at a fixed image resolution.

2. Cheaper attention over scales. The authors point to work such as Switti suggesting that next-scale generation may not need full causal attention to all previous scales, since the current-scale representation already accumulates lower-scale information. An analogous NAMVIS design would replace full block-causal target-target attention with scale-local or windowed attention while preserving geometry-aware source-to-target cross-attention, reducing memory and KV-cache cost at high resolution.

3. Beyond object-centric data. The model is trained on object-centric Objaverse-XL data, and its learned geometric priors may not transfer directly to complex real-world scenes; the architecture itself is not restricted to object-centric inputs, so larger, more diverse multi-view datasets combining object-centric and scene-centric data are identified as an important direction.

4. Robustness and sequence-length bottlenecks. Future work is stated to focus on reducing sequence-length bottlenecks for higher resolutions and denser target-view generation, improving robustness to pose noise, and extending training to more complex scene-level multi-view settings.

Target Audience

This paper is most useful to graduate students and industrial researchers working on generative 3D vision, novel view synthesis, and autoregressive image generation, particularly those already familiar with diffusion-based NVS baselines such as Zero-1-to-3, SyncDreamer, and EscherNet, and with next-scale autoregressive models such as VAR and Infinity. Practitioners building fast multi-view generation or 3D asset pipelines will find the inference-time and VRAM figures (0.6 s per target view at 1-to-1; 5.0 GB peak VRAM at 6→6) and the COLMAP consistency results most directly relevant. Readers without a background in transformer attention mechanisms, vector quantization, or camera projective geometry will find the method sections demanding.

Authors’ abstract

Sparse-view novel view synthesis is a central problem in 3D content creation, but diffusion-based approaches remain limited by iterative denoising, making multi-view generation expensive at inference time. We introduce NAMVIS, a diffusion-free framework that reformulates multi-view image synthesis as geometry-conditioned next-scale autoregression. Instead of generating target views through repeated denoising, NAMVIS predicts discrete visual tokens through a small number of coarse-to-fine scale steps, while sampling all tokens within each scale and across target views in parallel. To anchor this generation process to explicit camera geometry, we propose Multi-scale Projective Pose Encoding, which injects source and target camera transformations into both target-view self-attention and source-to-target cross-attention at every resolution. NAMVIS further combines global conditioning with dense geometry-aware cross-attention, enabling the model to preserve source-view appearance while maintaining target-view consistency. Across Objaverse, GSO, and OmniObject3D, NAMVIS outperforms diffusion-based baselines in PSNR, SSIM, and LPIPS, while running over 3 times faster than the evaluated diffusion baselines under the same evaluation setting. These results suggest that geometry-conditioned next-scale autoregression is a promising and efficient alternative to diffusion for sparse-view multi-view synthesis. Additional qualitative results, videos, and resources are available at https://corl-team.github.io/namvis/