DistScene: Object-to-Scene Distillation for 3D Scene Generation

AI summary

Overview

Research area: Computer vision / 3D generative modeling — single-image compositional 3D scene generation, where a system must recover both the surrounding environment and the individual objects in it, with correct geometry and spatial arrangement.

Technical level: Advanced. The method builds on sparse-voxel representations, sparse VAEs, flow-matching generative models instantiated as sparse DiT transformers, and LoRA fine-tuning. Readers without a background in 3D generative models will find the mechanics dense, though the high-level idea is accessible.

Scope: One sentence — the paper proposes a framework that generates a decomposable 3D scene (environment plus independent objects) from a single image by jointly generating all components in a shared coordinate frame, refining objects locally, and training on synthetic scenes distilled from a pretrained object generator.

What This Paper Is About

Generating a full 3D scene from one image is hard because object decomposition, spatial placement, and 3D geometry must all be solved at once. Existing pipelines handle these in separate stages (segment, estimate layout, generate each object), which lets errors accumulate across stages, while newer feed-forward methods generate objects together but never model the surrounding environment as an explicit component. DistScene addresses this by generating the environment and the objects jointly as separate components in one shared scene frame, so the environment's geometry can act as the spatial context that anchors where objects go.

Key Contributions

1. DistScene framework. A compositional 3D scene generation framework that jointly generates the environment and multiple independent objects from a single image, preserving component decomposability.

2. Scene-Frame Generation. A formulation that allocates separate latents to N objects plus one environment latent, flattens and concatenates them into a single token sequence with learnable type embeddings, and denoises them together so the environment latent anchors object positions and spatial interactions.

3. Object-Centric Refinement. A refinement stage that takes each generated object, re-centers and rescales it into a normalized local coordinate frame, and refines it at higher effective spatial resolution conditioned on the input image, the scene latents, and an embedding marking that object's scene region — then places it back at its original location.

4. Object-to-Scene Distillation. A self-distilled data engine that synthesizes scalable scene training data: an LLM describes objects and environments, a pretrained object generator (TRELLIS.2) produces each as a complete mesh, objects are placed into environments with iterative physics-based collision and support checks, and multiple views are rendered as conditioning images.

Main Findings

• Outdoor geometry (UrbanScene3D). DistScene achieves the lowest CD-L1 (0.0772) and highest F@0.05 (0.722) among evaluated methods. Compared with Extend3D, CD-L1 drops from 0.0832 to 0.0772 and F-score rises from 0.680 to 0.722. The TRELLIS.2 baseline scores 0.0831 / 0.701, EvoScene 0.1120 / 0.543, TRELLIS 0.1397 / 0.556, and Hunyuan3D 2.1 0.1933 / 0.411.

• Indoor geometry (MIDI-test). DistScene lowers scene CD from 0.1295 to 0.0877 — stated as a 32.2% relative improvement over 3D-Fixer — and raises scene F-score from 65.08 to 71.59, a gain of 6.51 percentage points. It also reports the best bounding-box IoU (0.3688), object CD (0.1529), and object F-score (51.64) among compared methods.

• Indoor geometry (Gen3DSR-test). DistScene reaches scene CD 0.0958 and F-score 79.68%, versus 0.1027 and 77.97% for 3D-Fixer.

• Environment modeling matters. Ablating the explicit environment (SF w/o Env vs. SF w. Env) raises MIDI-test object F-score from 29.98 to 40.98 and IoU from 0.1798 to 0.2544, and on Gen3DSR-test lowers scene CD from 0.1367 to 0.1096 and raises F-score from 62.04 to 73.21.

• Distilled data generalizes. Training an object-only architecture on DistScene's distilled data (SF w/o Env) versus the MIDI training set (SF w/o Env (MIDI)) improves Gen3DSR-test scene CD from 0.1936 to 0.1367 and F-score from 46.68 to 62.04 — despite MIDI-test being in-domain for the MIDI-trained variant.

• Refinement needs scene context. Adding refinement improves MIDI-test object CD from 0.2037 to 0.1529, object F-score from 40.98 to 51.64, IoU from 0.2544 to 0.3688, and Gen3DSR-test scene CD from 0.1096 to 0.0958 with F-score from 73.21 to 79.68. Removing the scene-frame context (SF + Refine w/o Ctx) degrades all metrics, e.g. MIDI-test object F-score falls to 41.97 and IoU to 0.2964.

• Resolution trade-off. SF-512 + Ref-512 requires 5.7 s for scene generation and 3.3 s per refined object at 18.0 GB peak memory, while the full SF-1024 + Ref-1024 model achieves the best accuracy at 21.8 GB peak memory (36.4 s scene, 38.5 s per object).

• User study. With 43 participants on 20 randomly selected text-to-image-generated inputs, DistScene achieves the highest preference rate on all three criteria: image consistency 61.95%, geometry quality 67.80%, appearance quality 66.46%, and average 65.41%. For comparison, Extend3D averages 22.52%, TRELLIS.2 8.74%, 3D-Fixer 2.80%, and SAM3D 0.53%.

• Training scale. The distilled dataset contains about 125k synthetic training scenes built from 70k generated objects and 46k empty scenes, comprising 106k indoor and 18k outdoor scenes.

Methodology in Plain English

The starting point is an existing image-to-3D object generator (TRELLIS.2) that represents objects as sparse voxels — active cells on a 3D grid, each carrying geometry and material features — compressed into a compact latent space by a sparse VAE, with a flow-matching transformer generating new assets from image conditioning.

DistScene extends this from one asset to a scene. Instead of generating one latent, it initializes N+1 noise latents: one per object plus one reserved for the environment. These latents are flattened and concatenated into a single token sequence, with learnable type embeddings distinguishing object tokens from environment tokens, and the transformer attends over the whole sequence. Because everything lives in one shared coordinate frame, the environment latent acts as a bridge anchoring where objects sit, so their relative scales and relationships are learned together rather than assembled afterward.

A problem follows: each object occupies only a small slice of the scene's voxel grid, so its effective spatial resolution is low and fine detail is lost. The fix is to pull each generated object out, recenter and rescale it into a normalized local space, and refine it there with a model conditioned on the input image and on the scene latents — with a marker embedding telling the refiner which region of the scene corresponds to this object. The refined object is transformed back into its original scene location. Refinement runs in the same sparse voxel domain, so it slots in without disturbing the global layout.

Training data is the other bottleneck: decomposed scene data with complete per-object meshes is scarce (the paper cites 3D-FUTURE as limited in scale and object diversity). The Object-to-Scene Distillation engine sidesteps this. An LLM writes structured descriptions of objects and their approximate sizes; the pretrained object generator produces each object — and crucially also the environment, so backgrounds are real geometry rather than empty space. Objects are then dropped into environments and the layout is iteratively adjusted by checking support-surface contact, axis-aligned bounding-box overlap, and mesh-level intersections, discarding configurations that float, penetrate, or collide. Accepted scenes are rendered from many viewpoints, keeping views where all objects are visible, yielding image-to-scene training pairs.

The scene generator is fine-tuned with LoRA (rank 32) while the pretrained backbone stays frozen, using the standard flow-matching objective applied separately to every object latent and the environment latent. The refinement model is trained on self-generated coarse-to-fine pairs: a high-quality object latent is downsampled, passed through the sparse VAE encoder and decoder, and upsampled back to produce a degraded latent that the model learns to recover.

Why This Matters

Impact on research. The paper argues against both dominant paradigms at once. Cascaded pipelines (segmentation plus layout estimation plus per-object generators) propagate errors between stages; existing feed-forward methods reduce that dependency but remain object-centric and never model the environment as a generated component, so object layouts are not grounded in surrounding geometry. DistScene's contribution is showing that making the environment an explicit, jointly generated component improves object-level metrics as well as scene-level ones. The distillation engine also offers a recipe for training scene generators without any existing scene dataset or human annotation.

Real-world applications (applications listed in the paper's introduction):

• Autonomous driving
• Virtual reality
• Robotics
• Simulation

Industry relevance. The efficiency numbers matter for deployment: the 512-resolution variant with refinement runs at 5.7 s per scene and 3.3 s per object with 18.0 GB peak memory, a trade-off the authors describe as favorable. The LoRA-based adaptation means the approach reuses an existing pretrained object generator rather than requiring training from scratch, and the framework trains on synthetic data it generates itself — lowering the data-acquisition barrier for anyone building 3D content pipelines.

Future Directions

• Address the stated limitations. The supplement indicates a limitations discussion in Section A.5, but the details are not included in the provided content, so the specific constraints the authors acknowledge are not reported here.

• Reduce inference cost. At full 1024 resolution the model takes 36.4 s per scene and 38.5 s per refined object. Narrowing that gap without losing accuracy is an obvious next step, particularly for interactive or real-time applications.

• Improve out-of-domain object-level metrics. On MIDI-test, the MIDI-trained object-only variant retains an advantage on object-level metrics and bounding-box IoU, indicating room to improve object fidelity in domains the distilled data does not cover.

• Eliminate auxiliary inputs. The method requires a complete scene image including background; the authors had to re-render MIDI-test scenes with floor, walls, and objects, and additionally render aligned object masks and depth maps for baselines that need them. Handling images without full background context remains open.

Target Audience

Researchers and graduate students working on 3D generative models, image-to-3D reconstruction, and scene-level content generation will find the core contributions directly relevant, as will practitioners building 3D asset pipelines for simulation, robotics, or VR who need decomposable scene output rather than a single fused mesh. Readers interested in synthetic data generation and distillation from pretrained generative priors will also benefit from the data-engine design. The paper assumes familiarity with sparse voxel representations, flow matching, and transformer architectures.

Authors’ abstract

We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level spatial coherence over the evaluated baselines. Project page: https://coolbeam.github.io/DistScene/