SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

AI summary

Overview

• Research area: Computer Vision / 3D scene generation for simulation and embodied AI, combining image-to-3D generative priors with vision-language-model (VLM) agents.
• Technical level: Advanced. The paper assumes familiarity with 3D scene layouts, 6D pose parameters, rigid-body simulation, orthographic projection, VLM agent pipelines, and quaternion-based orientation metrics.
• Scope: The paper proposes SceneMosaic, a hybrid framework that initializes 3D indoor scenes from image-based layout priors and refines them through local, agentic layout evolution to produce a base scene plus diverse, physically valid layout variants.

What This Paper Is About

Generating indoor 3D scenes at scale for simulation is hard: agentic text-to-3D pipelines produce high-quality, physically valid scenes but take hours per scene, while parametric image-to-3D models are fast but yield imprecise, physically invalid poses. Both paradigms also return only one deterministic layout per input, which does not reflect how real rooms are rearranged by human activity. SceneMosaic aims to get both efficiency and physical/semantic validity by starting from an image-based layout prior and then evolving it with VLM agents, while decomposing the scene into independent local units so that variant layouts can be generated combinatorially.

Key Contributions

1. A hybrid generation framework. SceneMosaic combines fast image-based layout priors for scene initialization with agentic iterative evolution, targeting scenes that are simultaneously efficient to generate and physically and semantically valid.
2. Locality-driven agentic layout evolution with combinatorial diversity. Each scene is decomposed into independent local units; layout variants are generated and refined per unit and then composed via a Cartesian product, turning one evolution pass into a combinatorial number of candidate scenes.
3. A novelty-aware scene metric with dynamic representative selection. A quantitative layout novelty metric (relative position, absolute distance, and rotation differences) plus dynamic greedy search distills a large candidate pool into a compact, diverse set of layouts.
4. Empirical validation on SceneEval-100 and via user study. SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24× speedup, substantially reduces physical violations, and receives the highest human ratings.

Main Findings

• Semantic quality on par with the strongest agentic baseline. On SceneEval-100, SceneMosaic base scenes reach POS 84.3 ± 2.0 and ROT 78.4 ± 3.4, versus SceneSmith at POS 83.5 ± 2.7 and ROT 80.9 ± 3.1. Variant scenes reach POS 84.5 ± 2.4 and ROT 79.9 ± 3.6.
• 24× speedup. SceneMosaic base scenes take 0.14h on average (and variants 0.03h), while SceneSmith takes 3.43h, SAGE 7.56h, SceneWeaver 1.07h, and HoloDeck 0.05h. The paper states this as matching the strongest agentic baseline in semantic layout quality with a 24× speedup.
• Physical violations eliminated on the reported metrics. SceneMosaic reports COL 0.0 ± 0.0 and OOB 0.0 ± 0.0 for base scenes (NAV 99.2 ± 0.4) and COL 0.0 ± 0.0, OOB 0.0 ± 0.0 for variants (NAV 99.4 ± 0.3). Image-to-3D baselines report much higher collisions: SceneGen COL 25.6 ± 6.9, SAM3D COL 20.2 ± 6.1 and OOB 8.9 ± 4.2, MIDI COL 19.8 ± 5.7 (OOB not applicable because MIDI and SceneGen do not recover room boundaries).
• Agentic baselines trade speed for latency or quality. HoloDeck is fast (0.05h) but lower in semantic quality (POS 49.8 ± 6.2, ROT 47.2 ± 7.6); SAGE has the longest reported runtime (7.56h) with POS 59.8 ± 7.9.
• Highest human ratings. In a user study with 48 participants rating 15 scenes (3 scenes per method × 5 methods) on a 5-point Likert scale, SceneMosaic scored 4.33 ± 0.89 on semantic plausibility and 4.47 ± 0.82 on physical plausibility, versus SceneSmith at 4.01 ± 0.99 and 4.16 ± 0.97.
• Strong inter-rater agreement. Kendall's coefficient of concordance was W = 0.648 (semantic) and W = 0.633 (physical), with p < 0.001.
• Image initialization matters. Replacing image-based initialization with text-only initialization degrades semantic scores (base scenes POS 80.7 ± 3.2, ROT 74.2 ± 3.5) and raises base-scene generation time from 0.14h to 1.25h.
• Physics tools matter for validity. Removing physics-based pre-stabilization and simulation tools produces COL 18.9 ± 4.8 and OOB 8.5 ± 3.3 on base scenes, despite lower runtime (0.13h).
• Orthographic 2D abstraction beats perspective 3D operations. Perspective 3D operations with local decomposition drop to POS 78.1 ± 4.1 and ROT 71.8 ± 4.4 and raise time to 0.31h; global evolution with perspective 3D operations is worse still (POS 72.3 ± 4.8, ROT 64.9 ± 5.6, 0.68h).
• Local decomposition is what makes variants cheap. With global evolution, variant generation time is 0.65h versus 0.03h with the full framework; the paper states global evolution "completely fails to accelerate variant scene generation."
• Diversity ablation is not reported in the provided content. The paper states that because no baseline generates multiple layouts from a fixed asset set, layout diversity is ablated in the supplementary material; the provided content does not include those numbers.

Methodology in Plain English

SceneMosaic works in four stages.

1. Reconstruct and structure the scene. All input signals are unified into a scene image. A perception agent registers objects (semantic labels, text descriptions, 2D bounding boxes), each object is segmented with SAM3 using its text prompt and box, and an iterative refinement pass resolves missing instances, duplicate masks, ambiguous boundaries, and nested regions. SAM3D then reconstructs each object mesh and its initial layout parameters (position, rotation, scale) from the image and verified mask.
2. Infer relations and build a scene tree. Given the initialized layouts, masks, and multi-view renderings, the system recovers a canonical room boundary with wall primitives and infers object relations such as attach (floor, wall, ceiling, or object support) and contain dependencies, estimated per object along with simulation attributes. Relation extraction is organized as a directed acyclic graph: a manager agent schedules specialized task agents for support reasoning, containment analysis, wall recovery, wall attachment, semantic refinement, and related subtasks. The result is a hierarchical scene tree where each local unit is a non-leaf anchor node plus its immediate children. A small number of functional relations span unit boundaries (for example, floor-standing chairs facing a desk), so cross-unit constraints are extracted and injected as unit-local guidance.
3. Stabilize and evolve layouts with agents. Physics-based stabilization first performs containment correction in orthographic projection, then runs gravity-based simulation so objects settle, resolve collisions, and stop floating. For each local unit, the system spawns several candidate variants from the original layout; the original and the variants are evolved in parallel by a Critic-Actor loop. The Critic uses visual evidence and simulation tools to diagnose layout validity and returns qualitative, coordinate-free suggestions plus a quality score in [0, 100]; the Actor converts those into symbolic pose expressions that a sandboxed evaluator resolves against the current layout, so alignment and spacing edits are exact rather than approximated. Agents reason over 1024 × 1024 orthographic 2D renderings with coordinate grids, bounding boxes, and facing arrows, which reduces 3D pose editing to 2D center translations and in-plane rotations. Each agent's context is scoped to its own local unit, and a per-round layout memory records poses, suggestions, expressions, and collisions so the Critic can detect repeated failure patterns such as left-right oscillation and pivot to a different fix.
4. Compose and select diverse scenes. Valid local variants are filtered (those with residual collisions or out-of-bounds objects are discarded), then combined via a Cartesian product across local units to produce k globally complete candidate scenes. Variants are expressed in their own anchor's coordinate frame, so a child unit's variant can be transplanted under any variant of its parent. Pairwise novelty is measured with three normalized terms — relative position difference (pairwise displacement vectors projected into object i's local frame), absolute distance difference, and rotation difference computed from quaternion inner products — combined as N = w_p·D_pos + w_u·D_dist + w_r·D_rot with w_p + w_u + w_r = 1. A dynamic max-min greedy search keeps the original scene as the first seed, repeatedly adds the candidate maximizing minimum distance to the selected set, and prunes candidates below a dynamic threshold τ = α·d_max.

Why This Matters

• Impact on research. The work reframes the efficiency–fidelity trade-off in 3D scene generation as something a hybrid pipeline can resolve, and it treats layout diversity from a fixed asset set as a first-class objective rather than a sampling artifact. The local-unit decomposition plus Cartesian composition is a reusable idea for any agentic pipeline whose cost scales with scene size.
• Real-world applications.
  • Robot training and evaluation in simulation, where dense, physically valid, and varied indoor environments are needed at a scale real-world data collection cannot support.
  • Interactive entertainment and game level design, where a single reference image can yield a base layout plus alternatives.
  • Embodied AI benchmarking, where structured rearrangements of the same room test generalization to different object configurations.
  • Synthetic data generation for perception or navigation models that require physically plausible indoor layouts.
• Industry relevance. The reported reduction from multi-hour agentic pipelines to 0.14h per base scene and 0.03h per variant, with COL and OOB reported at 0.0, is directly relevant to anyone paying for VLM API calls or GPU time at content-generation scale. The public code release at https://github.com/rxjfighting/SceneMosaic and project page at https://rxjfighting.github.io/SceneMosaic lower the barrier to adoption.

Future Directions

• Scaling the benchmark. All quantitative results are on SceneEval-100 (100 prompts), with 5 variant scenes per base scene. Whether the conclusions hold at larger scene counts and in more varied room types is not established in the reported content.
• Broadening diversity evaluation. The paper defers layout-diversity ablation to the supplementary material; a standardized diversity metric for scenes generated from a fixed asset set remains an open question, since no evaluated baseline produces multiple layouts.
• Generalizing beyond indoor rooms. The scene tree, local units, and anchor-dependent gravity (downward for floor, inverted for ceiling, outward wall normal for wall-anchored units) are tailored to indoor layouts. Extension to outdoor or multi-room scenes is not reported.
• Reducing dependence on the underlying models. The pipeline depends on specific components (SAM3 for segmentation, SAM3D for reconstruction, GPT-Image-2 for text-to-image, CoACD for convex decomposition, and a single underlying VLM instantiated with role-specific prompts, plus GPT-5.5 for POS/ROT evaluation). Sensitivity to these choices is not reported.

Target Audience

Researchers and engineers working on 3D content generation, scene synthesis for robotics simulation, and embodied AI, particularly those already familiar with agentic VLM pipelines or image-to-3D models. It is also useful for practitioners who need simulation-ready indoor environments at scale and want to understand the practical cost and validity trade-offs between agentic text-to-3D and parametric image-to-3D approaches. The paper is not beginner-friendly: it presumes comfort with 3D pose representations, rigid-body simulation, and agentic system design.

Authors’ abstract

Diverse and simulation-ready indoor scenes are essential for interactive entertainment and embodied AI, yet their scalable generation remains challenging. Recent agentic text-to-3D scene pipelines that rely on vision-language models (VLMs) can generate scenes of high fidelity but require costly iterative object placement and refinement. Another mainstream paradigm, parametric image-to-3D scene models, produces scenes efficiently from strong priors learned from 2D images but often leads to imprecise and physically invalid scenes. More importantly, both paradigms struggle to output diverse scenes for a single input, making it hard for them to reflect the dynamically changing nature of real scenes. In this paper we propose \textbf{SceneMosaic}, a framework that combines the merits of both paradigms. It obtains the initial candidate from the learned image-based prior, and subsequently evolves the result through VLM agents, ensuring both efficiency and physical validity. Within the evolution process, SceneMosaic exploits the locality of natural scenes and decomposes a scene into independent local units, allowing separate evolution within each unit before composing the global scene via Cartesian product. On SceneEval-100, SceneMosaic matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Our code is publicly available at https://github.com/rxjfighting/SceneMosaic.