Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction

AI summary

Overview

Research area: Computer Vision — single-image 3D scene reconstruction and generative 3D modeling, specifically compositional scene generation where object geometry and object placement (layout) are produced from a single RGB image.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, rectified flow models, voxel latents/VAEs, canonical object spaces, similarity transformations in SO(3), and robust estimation methods such as RANSAC and the Umeyama algorithm.

Scope (one sentence): The paper proposes replacing sparse, unbounded object pose regression in single-image 3D scene reconstruction with a dense, bounded, pixel-aligned correspondence representation called the Canonical Coordinate Map (CCM), then recovering object placement by aligning CCMs against a monocular scene point cloud.

Authors are affiliated with The University of Hong Kong and VAST; the project page is listed as https://sunyangtian.github.io/Mira-Scene-web/.

What This Paper Is About

Single-image 3D object generators now produce high-fidelity assets, but they operate in a canonical object space and do not know where objects belong in a scene. The core problem the paper attacks is how to represent object layout: holistic scene generators blur object detail into one scene-level process, while compositional methods keep object fidelity but represent layout as sparse, unbounded pose parameters (translation, rotation, scale) that are hard to learn and generalize from scarce scene-level 3D supervision. Mira-Scene's goal is a layout representation that is dense, bounded, and learnable from abundant object-level 3D data, so that accurate placement can be recovered geometrically rather than regressed directly.

Key Contributions

1. CCM with PCM as a layout representation. A pixel-aligned field (CCM) maps each visible object pixel to a surface coordinate inside the object's bounded canonical space. Paired with a scene-space Point Cloud Map (PCM) from monocular geometry estimation, it yields dense canonical-to-scene correspondences, so object transformations are recovered by robust geometric alignment rather than neural pose regression.

2. A geometry-layout co-generation model. A multimodal Mixture-of-Transformers (MoT) diffusion design with modality-specific expert streams (a Geometry Expert producing 3D voxel latents, a Layout Expert producing CCMs in pixel space) that exchange information through shared self-attention.

3. A shared geometry-layout positional embedding. Geometry tokens and layout tokens are embedded into a common 3D positional basis — layout tokens are treated as points on a designated 3D plane at z = z_layout with an offset (N, N) — to improve cross-modal consistency without assuming image-plane positions coincide with 3D surface positions.

4. Data-efficient compositional reconstruction. Demonstrated across indoor, outdoor, synthetic, and in-the-wild scenes, achieving substantially better layout accuracy than strong baselines trained with larger-scale supervision, using only 60K open-source object assets.

Main Findings

• Layout accuracy improves sharply over SAM3D. On BlendSwap, Mira-Scene raises 3D-IoU from 0.520 to 0.727 and 2D-IoU from 0.672 to 0.783 relative to SAM3D. The abstract reports relative gains of 39.8% in 3D-IoU and 16.5% in 2D-IoU over SAM3D; Section 3.2 reports 3D-IoU improvements of 39.8% on BlendSwap and 16.4% on 3D-Future Scene relative to SAM3D.

• Object geometry is competitive, not dominant. On BlendSwap, Mira-Scene reports CD 0.021 and FS@0.1 0.843 versus SAM3D's CD 0.027 and FS@0.1 0.817, while using far fewer objects (60K open-source assets versus SAM3D's much larger-scale data engine). On 3D-Future Scene, SAM3D has the better FS@0.1 (0.866 vs 0.845) though Mira-Scene reports the better 3D-IoU (0.694 vs 0.596) and ICP-Rot (5.485 vs 6.272).

• Densifying prediction alone is not enough. In the layout-representation ablation with identical architecture, Raw pose gives 3D-IoU 0.365 / 2D-IoU 0.358, Coord Cube gives 0.379 / 0.381, and CCM with PCM gives 0.727 / 0.783. Coord Cube's small gain shows that densifying unbounded scene-space targets does not resolve the learning difficulty.

• The representation benefit survives matched data. With matched training data, CCM with PCM reaches 3D-IoU 0.537 / 2D-IoU 0.662 versus Coord Cube's 0.379 / 0.381 and Raw's 0.365 / 0.358; adding object-level supervision lifts CCM further to 0.727 / 0.783.

• Joint attention matters most in the architecture. Removing joint attention drops 2D-IoU from 0.757 to 0.535 and raises CD from 0.017 to 0.070. Removing the shared positional embedding gives a smaller but consistent drop (2D-IoU 0.747, CD 0.019, FS@0.01 0.349, FS@0.05 0.925), while the full model reaches 2D-IoU 0.757, CD 0.017, FS@0.01 0.383, FS@0.05 0.940.

• CCM is a reverse-rendering formulation. Unlike CUPID, which stores for each 3D voxel center the pixel it projects to (many-to-one along camera rays), CCM stores a 3D coordinate per 2D pixel, giving a one-to-one mapping. CUPID obtains a slightly higher 2D-IoU (0.802 vs 0.795 in the Mesh+GT setting), but Mira-Scene obtains substantially better CD (0.023 vs 0.047) and F-scores (FS@0.01 0.456 vs 0.414; FS@0.05 0.885 vs 0.727).

• CCM and generated mesh are internally consistent. Ours-(CCM+Mesh) reaches CD 0.012, FS@0.01 0.546, and FS@0.05 0.977; the paper explicitly states this reflects consistency between the two generated outputs rather than fidelity exceeding the ground-truth mesh.

• Correspondence errors are long-tailed, motivating robust alignment. On BlendSwap, correspondence error has a median of 0.046, mean 0.097, p75 0.103, p90 0.205, p95 0.395, and p99 0.882. RANSAC+Umeyama improves 3D-IoU by approximately 13.4% relative to Umeyama without RANSAC; IRLS with a Huber loss adds a further 1.0% relative improvement, but RANSAC+Umeyama is retained for simplicity.

• Occlusion degrades layout but does not break it. Grouped by occlusion rate, 3D-IoU is 0.752 for objects below 0.2 occlusion (41.8% of evaluated objects), 0.724 for [0.2, 0.6] (35.5%), and 0.635 above 0.6 (22.7%).

• A small visual-quality edge. Using SceneGen's PSNR evaluation with RGB normalized to [0, 1], Mira-Scene reports PSNR 18.8 versus SAM3D's 18.6.

• Baseline behavior differs by domain. SceneGen produces plausible indoor results near its training distribution but degrades on outdoor and stylized inputs; MIDI has object priors but no reliable placement mechanism; SAM3D and SceneMaker both show misalignment attributed to sparse layout representations.

Methodology in Plain English

Given one image and a set of object masks (obtainable via a VLM-guided agentic segmentation pipeline built on SAM3 if not supplied), the system reconstructs a scene as a set of posed object assets (S_k, T_k), where S_k is canonical geometry and T_k = (s_k, R_k, t_k) is an isotropic scale, a rotation in SO(3), and a translation into the camera coordinate frame.

Rather than asking a network to output those pose numbers directly, Mira-Scene asks the network to output two things per object: the object's canonical 3D geometry, and a Canonical Coordinate Map — a three-channel image in which each visible object pixel stores the canonical xyz coordinate of the surface seen there (background and invalid pixels are excluded by a validity mask). Because those coordinates live in a bounded canonical object space, they are a stable, bounded prediction target that can be supervised from rendered object assets without any scene-level layout annotation at all.

The second ingredient is a scene-space Point Cloud Map, a dense point map where each pixel gives the 3D scene point observed there, obtained from a depth camera or monocular geometry estimation. Pasting the crop-space CCM back into the full image yields, for every valid pixel, a correspondence between a canonical object coordinate and a scene point. Placement then becomes a geometry problem: find the similarity transform minimizing the squared distance between transformed canonical points and their scene points, solved with RANSAC over the dense correspondences followed by the closed-form Umeyama algorithm on the inlier set to resist noisy CCM predictions and depth outliers.

Generation itself is a Mixture-of-Transformers: a Geometry Expert denoises a noisy latent feature grid (a lightweight VAE compresses a binary N×N×N voxel grid into G_k ∈ R^{D×D×D×C_G}, serialized into tokens with 3D positional embeddings), while a Layout Expert denoises the CCM directly in pixel space (tokenized by a strided convolution), motivated by CCMs' smooth spatial variation. Geometry is conditioned via cross-attention on DINOv2 features from the full image, the object mask, and the cropped object image; the layout expert concatenates the cropped RGB with the noisy CCM and also cross-attends to full-image and mask features. Both experts are rectified flow models trained with two losses, L_CFM on the geometry tokens and L_CFM on the CCM tokens, with λ1 = λ2 = 1.

Training is two-stage. Pre-training uses isolated 3D object assets: 60K objects and 1M object-centric rendered views, plus 20K photo-realistic object views with generated backgrounds to shrink the appearance gap with real scenes. Fine-tuning uses 20K 3D-FRONT scene views by sampling occluded object instances, adapting the object-level prior to amodal reasoning under partial visibility.

Why This Matters

This work reframes a learning problem as a geometry problem. Instead of forcing a network to regress unbounded scene-space poses from scarce scene-level 3D data, it predicts bounded canonical coordinates that can be learned from the abundant supply of object-level 3D assets and then solves for placement analytically. That decoupling is what allows object fidelity and layout accuracy to be pursued independently, and it is the paper's distinctive claim: data scaling alone (as in SAM3D's artist-assisted data engine) cannot fix a hard layout representation.

Real-world applications named in the paper:

• Editable 3D content creation — removing, repositioning, reconfiguring, rigging, or animating individual reconstructed objects while preserving coherent spatial layout.
• Embodied simulation and embodied AI — exporting object assets to simulators for perception, planning, and manipulation.
• Physical simulation — exporting objects to physics engines for physical simulation.
• AR/VR and robotic interaction — the paper lists these as capabilities supported by single-image compositional scene reconstruction.

Industry relevance: The compositional output format (separate object assets plus transforms) is directly usable by interactive editing tools, simulators, and physics engines, which matters for content pipelines, robotics, and simulation-driven industries. The paper also emphasizes data efficiency — achieving better layout accuracy with 60K public object assets than a baseline using "Million+" 3D data scale and private data — which lowers the barrier for groups without large proprietary 3D datasets.

Future Directions

• Scaling object-level pre-training. The paper states that learning from abundant object-level 3D assets "provides strong potential for further scaling," suggesting that increasing the number and diversity of object assets (beyond the 60K used here) is an open avenue.

• Heavily occluded objects. Layout accuracy falls from 3D-IoU 0.752 for mostly visible objects to 0.635 for objects with occlusion above 0.6, which make up 22.7% of the evaluated objects — an explicit remaining weakness that amodal reasoning must address.

• End-to-end pipelines without external segmentation. Masks are treated as an input; when unavailable they come from a VLM-guided agentic segmentation pipeline built on SAM3. Folding segmentation into the framework or characterizing its error propagation is a natural next question.

• Quantifying downstream applications. The applications section (scene editing, embodied AI, physical simulation) is presented qualitatively through figures; rigorous evaluation of task success in simulators or physics engines is not reported. More broadly, the paper refers readers to Appendix F for discussion of limitations and future work, and that appendix is not included in the provided content, so its specific list of limitations is not reported here.

Target Audience

Researchers and engineers working on 3D generative modeling, single-image scene reconstruction, and diffusion-transformer architectures will get the most from this paper, particularly those interested in representation design for layout, correspondence-based geometric alignment, or training strategies that exploit object-level 3D data in place of scarce scene-level supervision. It will also interest practitioners building editable 3D content pipelines, embodied-AI simulators, and physics-based simulation tools, though the dense technical content (rectified flow, MoT, RANSAC/Umeyama alignment, voxel VAEs) makes it most accessible to readers with prior background in generative 3D vision.

Authors’ abstract

Single-image 3D object generation can now produce high-fidelity assets, yet accurately placing them into a coherent scene layout remains an open challenge. A central difficulty lies in how object layout is represented. Holistic methods absorb placement into a scene-level generation process, sacrificing object-level detail. Compositional methods preserve object fidelity by decoupling geometry from layout, but typically parameterize layout as sparse, unbounded pose variables that are difficult to learn and generalize poorly under scarce scene-level supervision. We present Mira-Scene, a compositional 3D scene reconstruction framework that replaces sparse pose regression with dense, bounded correspondence recovery. At its core is the Canonical Coordinate Map (CCM), a pixel-aligned field that maps each visible object pixel to a surface coordinate in the object's bounded canonical space. When paired with a scene-space Point Cloud Map (PCM) from monocular geometry estimation, CCM induces dense canonical-to-scene correspondences from which object transformations are recovered through robust geometric alignment. Because CCM operates in bounded canonical space, it provides a stable prediction target that can be trained from scalable object-level 3D data without requiring scene-level layout annotations. Mira-Scene further introduces a multimodal diffusion transformer that jointly generates object geometry and CCMs, using modality-specific expert streams with shared attention and positional encoding to promote geometry-layout consistency. Experiments on indoor, outdoor, synthetic, and in-the-wild scenes show that Mira-Scene substantially outperforms strong baselines in layout accuracy, achieving relative gains of 39.8% in 3D-IoU and 16.5% in 2D-IoU over SAM3D, using limited open-source training data.