Tetris3D: 3D Scene Generation With Objects That Fit Together

AI summary

Overview

• Research area: Computer vision / 3D generative modeling — specifically single-image, object-centric 3D scene reconstruction, amodal (occlusion-aware) shape completion, and physics-aware generative modeling.
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, sparse voxel latents (TRELLIS.2 / SLAT), vision-language models, and physics-simulation evaluation.
• Scope: The paper proposes Tetris3D, a generative framework that reconstructs a 3D scene from one image by generating each object conditioned explicitly on its neighbors' geometry and physical relationships, plus a new 1.2M-scene simulation dataset (ComOb) for training and evaluation.

What This Paper Is About

Reconstructing a 3D scene from a single image requires recovering each object as a complete 3D entity in a shared scene space, but objects in real scenes touch, support, contain, and occlude one another — often hiding exactly the regions where they interact. Existing methods either generate objects independently and assemble them, or couple them only implicitly through shared attention, giving weak fine-grained guidance about how neighboring objects should fit together. Tetris3D's goal is to generate each object in relation to its neighbors, so the resulting scene is geometrically and physically coherent even where the interacting regions are invisible in the input image.

Key Contributions

1. Tetris3D framework: An object-interaction-conditioned 3D generative framework built on TRELLIS.2 that explicitly conditions each object's shape and pose on (a) spatial context — a truncated unsigned distance field, the direction vector to the nearest neighboring surface point, and that point's surface normal — and (b) relation type — a learnable embedding from the set {stack, lean, contain, touch, none}. These conditions are defined on the same 3D grid used for generation and injected directly into the corresponding tokens at every DiT layer.
2. Pose-aligned generation in scene space: Instead of estimating object poses separately or post-aligning generated shapes, Tetris3D defines a per-object grid from the depth map and back-projects image features and a depth-occupancy embedding along camera rays, so objects are generated directly in the poses observed in the input scene image.
3. V2I (visible-to-invisible) cross-attention: An explicit amodal completion mechanism in which tokens in occluded regions act as queries and attend to visible tokens as keys/values, at every DiT layer of both generation stages, to complete hidden geometry.
4. ComOb dataset: A physics-simulation-based dataset of 1.2M scenes and 3.4M annotated samples with per-object meshes, rendered RGB images, object masks, depth maps, and pairwise physical relation labels, constructed around the interaction types {stack, lean, contain, pile, touch} and settled into stable configurations with a physics engine, using target objects from TRELLIS-500K.

Main Findings

• State-of-the-art on synthetic Toys4K scenes: On Toys4K-derived scenes composed with the ComOb generator (ground-truth depth), Tetris3D achieves CD-S 3.68, CD-O 0.97, F1-S 0.8407, F1-O 0.8084, IoU-B 0.7918, ICP-Rot 5.29, MMD 1.501, COV 83.11, P-FID 1.599, Uni3D 0.6884, ULIP 0.7182, PD 0.0036, D_mean 38.91, and E_peak 0.1721 — better than SAM-3D, ShapeR, WorldSculpt, Amodal3R + FoundationPose, and GENA3D + FoundationPose across reconstruction quality, generation quality, and physical stability.
• Large margins in physical stability: Penetration depth (PD) of 0.0036 versus 0.1177 (WorldSculpt), 0.5962 (Amodal3R), and 0.9106 (GENA3D) on Toys4K, and D_mean of 38.91 versus 103.5–192.4 for baselines.
• Generalization to real-world scenes: On MessyKitchens, Tetris3D reaches CD-S 0.11, F1-S 0.9620, IoU-B 0.7113, PD 0.0011, and D_mean 102.0; on Picasso, CD-S 0.27, F1-S 0.9053, IoU-B 0.7264, PD 0.0016, and D_mean 434.4 — outperforming scene generation methods (SAM-3D, ShapeR, WorldSculpt, MIDI, SceneGen, SceneMaker) and the amodal + pose-estimation baselines.
• Real-world evaluation uses estimated depth and VLM-inferred relations: The MessyKitchens and Picasso results are obtained via the autoregressive inference pipeline with depth from MoGe3 and physical relations inferred by a VLM.
• Interaction conditioning matters most for physical plausibility: Removing the interaction condition (Table 3, ablation setting trained 100K iterations) degrades results from CD-S 3.90 to 8.56 and PD from 0.0192 to 0.4980, with D_mean rising from 39.22 to 69.24.
• V2I attention helps, and beats an alternative design: Removing V2I attention gives CD-S 4.46 and PD 0.0270; replacing it with Amodal3R's occlusion-aware attention gives CD-S 4.29 and PD 0.0281 — both below the full model in the same ablation setting (CD-S 3.90, PD 0.0192).
• Qualitative results: Tetris3D reconstructs plausible shapes and poses under occlusion, preserves interactions without penetration, and remains stable after physics simulation on real scene images.

Methodology in Plain English

The pipeline works object by object, in an order determined by physics.

• Deciding the order: A vision-language model reads the scene image and infers a directed "physical dependency" hierarchy — an edge from object k to object i means i relies on k for support or stability. Topologically sorting this hierarchy gives the generation order and a physical relation label per edge. So a table is generated before a cup resting on it, and the table's reconstructed surface becomes the cup's context.
• Placing each object in the scene: From the depth map and the target object's mask, the method lifts pixels into a partial 3D point cloud and defines a voxel grid around it. Image features from DINOv3 are back-projected into that grid along camera rays, together with a learnable embedding marking observed surface occupancy. Because the object is generated inside this grid in the camera's coordinate system, no separate pose-estimation or post-alignment step is needed.
• Describing the neighbors: For every voxel, the method computes the truncated unsigned distance to the nearest point on any already-generated neighbor's surface, the direction to that point, and the surface normal there. It also looks up which neighbor that nearest point belongs to and attaches that neighbor's relation type embedding. These are combined by small MLPs into a per-voxel interaction condition.
• Injecting the conditions: Because the image features, depth occupancy, and interaction condition all live on the same grid, they are simply added — through per-layer MLPs and learnable gates initialized to zero — to the matching tokens at every layer of the two diffusion transformer stages (sparse structure, then structured latent). Zero-initialized gates mean training starts from the pretrained behavior and learns to use the new signals.
• Filling in hidden regions: Tokens lying along camera rays through pixels occupied by other objects are marked invisible and cross-attend to all visible tokens, so occluded interior geometry is inferred from what can be seen.
• Training data: Since existing datasets cover narrow environments or few interaction types, the authors built ComOb: objects are arranged around simple primitives (cylinders, cones) according to a target interaction type and then settled by a physics engine into stable configurations, producing 1.2M scenes and 3.4M annotated samples with meshes, masks, depth, and relation labels.
• Training setup: The backbone and image-conditioning components are initialized from TRELLIS.2 and Pixal3D; each stage has a 30-block DiT; the sparse structure and structured latent models have about 1.83B and 1.68B parameters. Training uses AdamW with learning rate 1×10⁻⁴ and weight decay 0.01, batch size 16 on two NVIDIA H200 GPUs — 200K iterations (about 7 days) for sparse structure and 100K iterations (about 9 days) for structured latent. The ablation trains sparse structure for 100K iterations with the structured latent model unchanged. Condition dropout (0.1 jointly, 0.05 per group) supports classifier-free guidance, and a two-stage curriculum mixes complete images with primitive-occluded and synthetic-occlusion images.

Why This Matters

• Research impact: The paper argues that inter-object physical relationships should be an explicit conditioning signal on geometry, not something left implicit in shared attention or confined to pose prediction — and it provides a dataset (ComOb) specifically built to train that signal, addressing a gap the authors identify in existing indoor- or tabletop-restricted datasets.
• Robotics and manipulation: Physically stable, penetration-free reconstructed scenes with per-object meshes are directly usable as simulation assets for training and testing manipulation policies.
• AR/VR content creation: Converting a single photograph into a set of individually editable, correctly posed 3D objects supports scene editing and immersive content pipelines.
• Physical simulation and scene editing: Because objects are recovered as separate complete entities with relation labels, they can be dropped into a physics engine and expected to remain stable rather than drifting or interpenetrating.
• Industry relevance: The work sits where 3D content generation, embodied AI simulation, and digital-twin reconstruction overlap, and its training budget (two H200 GPUs, days rather than months per stage) makes the recipe reproducible by well-resourced labs.

Future Directions

• Robustness to imperfect inputs: The authors report results with depth estimated by MoGe3 and relations inferred by a VLM in the appendix, but the main-table Toys4K protocol uses ground-truth depth — how much the guarantees degrade with noisy depth and relation inference remains an open question.
• Beyond the fixed relation vocabulary: Relation types are limited to {stack, lean, contain, touch, none}; extending to richer or articulated interactions (hinges, fluids, deformables) is a natural next step.
• Resolving generation-order errors: Because generation is autoregressive in a VLM-inferred topological order, an incorrect dependency edge early in the sequence could propagate; error recovery or joint re-optimization is unaddressed in the presented content.
• Comparison against agentic and LLM-driven generation: The paper notes a comparison with agentic generation using GPT-6 Astra in Appendix D.3 and additional analyses in Appendix D, leaving the question of how explicit conditioning compares with agent-orchestrated pipelines as an actively developing thread.

Target Audience

This paper is most useful to computer vision and graphics researchers working on 3D generative models, single-image 3D reconstruction, and occlusion-aware (amodal) generation; to researchers building scene datasets and physics-grounded generative pipelines; and to practitioners in robotics simulation, AR/VR, and 3D content creation who need physically coherent object compositions rather than isolated asset reconstructions. Readers will need background in diffusion transformers, sparse voxel representations, and physics-based evaluation metrics to follow the method and tables.

Authors’ abstract

We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibility between neighboring objects that interact with one another. To address this, we explicitly condition the generation of each object on the geometry of surrounding objects and their physical relationships, guiding its shape and pose to remain geometrically and physically plausible within the scene. Moreover, we introduce ComOb, a physics simulation-based dataset of 1.2M scenes featuring physical interactions across diverse object categories, with per-object meshes and pairwise physical relation annotations. Comprehensive experiments on synthetic and realworld scenes show that Tetris3D recovers coherent object shapes and poses even when interacting regions are occluded, and achieves state-of-the-art performance in both generation quality and physical stability.