Sparse auto-regressive modeling for scene generation from multi-view images

AI summary

Overview

Research area: Computer vision — 3D scene generation and completion from images, sitting at the intersection of neural scene representation (3D Gaussian Splatting), generative modeling, and autoregressive sequence modeling.

Technical level: Advanced. The paper assumes familiarity with latent generative models, transformers, voxel grids, and differentiable rendering.

Scope: The paper introduces SPAR3S, a sparse voxel-aligned latent generative model that completes 3D scenes from a few unconstrained input views without ever needing ground-truth 3D data during training.

What This Paper Is About

Given only a handful of photographs of a scene, standard feed-forward reconstruction methods can only reproduce what those photos show — the geometry behind the camera, inside a closet, or around a corner simply does not exist in the output. The goal of this work is to "fill in the blanks": produce a complete, spatially consistent 3D scene that plausibly extends beyond the observed content, while staying computationally affordable.

The core difficulty is that most existing 3D generative approaches either operate on dense volumetric grids (prohibitively expensive) or require large datasets of ground-truth 3D scans (which are scarce and hard to capture at scale). SPAR3S sidesteps both by generating in a compact sparse latent space learned directly from ordinary multi-view photographs.

Key Contributions

1. A voxel-aligned sparse 3D latent space in which only occupied voxels are represented, keeping the representation compact and structured while remaining spatially aligned to a regular grid so that generation is tractable.
2. Learning that latent space from multi-view images alone, using photometric supervision through differentiable 3D Gaussian Splatting — eliminating the need for any ground-truth 3D data during training.
3. A masked autoregressive transformer that jointly models voxel occupancy (where geometry exists) and latent token values (what that geometry looks like), treating scene completion as prediction of missing latent tokens and their spatial support.
4. Demonstrated gains in novel-view synthesis quality over prior work on synthetic indoor scenes, plus validation of generalization on RealEstate10k as a real-world dataset.

Main Findings

• Feed-forward reconstruction is fundamentally limited to observed content. The authors frame scene completion as a distinct problem from reconstruction: existing methods cannot hallucinate plausible unseen regions because they have no generative prior over what lies outside the input views.
• Sparsity is the enabling trick. Restricting the latent space to occupied voxels makes volumetric generation computationally feasible, in contrast to dense grid-based 3D generative models whose cost scales with empty space.
• Photometric supervision substitutes for 3D ground truth. Differentiable 3D Gaussian Splatting lets the model learn its 3D latent structure purely from rendered-image losses, removing the data bottleneck that constrains most 3D generative work.
• Joint occupancy-and-appearance modeling produces spatial consistency. Because the autoregressive transformer predicts occupancy and token values together rather than in separate stages, the generated unseen regions stay coherent with observed geometry.
• Higher novel-view quality than prior work on synthetic indoor scenes, indicating that the completed geometry and appearance are accurate enough to re-render from new viewpoints.
• Generalization to real-world data. Results on RealEstate10k show the approach is not overfit to synthetic training distributions.

Methodology in Plain English

The researchers first build a compact 3D description of a scene: a voxel grid where only the voxels that actually contain something are kept, each holding a learned latent vector rather than raw color or density. This sparse latent space is not hand-designed — it is trained end-to-end by rendering the scene into images with differentiable 3D Gaussian Splatting and comparing those renders against the real input photos.

Once that space exists, completing a scene becomes a sequence problem. From a few input views, the model encodes the voxels it can actually observe. The remaining empty slots in the grid are the unknowns. A masked autoregressive transformer is then trained to fill them in one token at a time, deciding both whether each voxel should be occupied and what latent value it should carry — the same way a language model predicts the next word in a sentence, except the "sentence" is a spatial arrangement of voxels.

Because generation happens over a sparse set of meaningful tokens rather than a dense grid, it stays efficient, and because every prediction is conditioned on both the observed voxels and previously generated ones, the resulting unseen regions remain spatially consistent with what was seen.

Why This Matters

This work reframes 3D scene understanding from reconstruction to completion, and shows that the data problem that has long constrained 3D generative modeling can be circumvented by learning structure from images alone. It also demonstrates that autoregressive sequence modeling — the dominant paradigm in language and 2D image generation — transfers usefully to sparse 3D structure, which suggests a route toward unifying 2D and 3D generative modeling.

Real-world applications:

• Virtual real estate and interior design: turning a handful of listing photos into a fully navigable 3D walkthrough, including rooms only partially captured.
• AR/VR and game content creation: generating complete environments from sparse capture instead of expensive full photogrammetry scans.
• Robotics and embodied AI: producing complete scene geometry for simulation and navigation planning, where unseen regions must be inferred rather than observed.
• Autonomous driving and digital twins: completing street-level scenes from limited camera coverage for simulation and testing.

Industry relevance: Any pipeline that converts photographs into 3D assets — real estate platforms, e-commerce 3D product views, VFX and game studios, and simulation providers — stands to benefit from removing the requirement for dense capture or ground-truth 3D scans. The use of Gaussian Splatting as the supervision signal also aligns the method with the fast-moving ecosystem of real-time neural rendering tools that industry has recently adopted.

Future Directions

1. Scaling beyond indoor scenes and the RealEstate10k distribution — outdoor, urban, and large-scale environments pose different sparsity and completion challenges that are not yet addressed.
2. Handling truly unconstrained input, including unknown or noisy camera poses and wide baselines, to test how robust the observed-voxel encoding is to imperfect input.
3. Improving controllability and diversity of generation, since a deterministic completion may not be desirable for design tasks where multiple plausible layouts should be offered.
4. Reducing the computational cost of autoregressive generation over voxel tokens — long token sequences remain a bottleneck, and parallel or hierarchical decoding could make interactive use practical.
5. Evaluating geometric fidelity directly, not only through novel-view rendering metrics, to determine whether completed regions are metrically correct or merely photometrically plausible.

Target Audience

Researchers and advanced practitioners in 3D computer vision, neural rendering, and generative modeling, particularly those working on Gaussian Splatting, neural scene representations, latent generative models, or autoregressive transformers applied to non-linguistic domains. It is also relevant to applied engineers building 3D content pipelines for real estate, AR/VR, robotics simulation, and digital twins who want to understand what is currently achievable from sparse photographic input. Readers without a background in volumetric representations or differentiable rendering will find the method section demanding.

Authors’ abstract

Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.