SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes

AI summary

Overview

Research area: Robotics simulation, indoor scene synthesis, LLM/VLM multi-agent systems, and 3D asset generation.

Technical level: Advanced. The architecture description involves hierarchical agent orchestration, SE(2)/SE(3) pose reasoning, articulated object libraries, and physics post-processing; the evaluation is metric-dense.

Scope: The paper introduces SceneSmith, a hierarchical agentic framework that turns a natural-language prompt into a dense, physically feasible, simulation-ready indoor scene with generated assets, and demonstrates its use for automatic robot policy evaluation.

What This Paper Is About

Simulation is a key tool for training and testing home robots at scale, but existing simulated indoor environments are sparsely furnished and physically simplistic — they lack the dense clutter, articulated furniture, and physical properties that real homes exhibit and that robotic manipulation actually requires. SceneSmith aims to close that gap by taking a natural-language description of an environment and automatically building a room- or house-level scene that is immediately simulatable, matching the prompt, and physically valid.

Key Contributions

1. A hierarchical agentic framework (SceneSmith) for constructing simulation-ready indoor environments from natural language, designed to support scalable robot training and evaluation. Scene construction proceeds through stages — architectural layout, furniture placement, wall-mounted objects, ceiling fixtures, and manipuland population — organized as a tree where rooms and supporting surfaces form independent branches.

2. An integrated asset generation and routing pipeline that combines text-to-3D synthesis for static objects, retrieval from an articulated object library (ArtVIP) for objects with movable parts, and thin coverings for flat decorative elements, augmenting every asset with collision geometry and estimated physical properties.

3. Demonstrated improvements over all baselines in human studies and automated metrics, producing denser, collision-free, and physically stable scenes.

4. An end-to-end robot policy evaluation pipeline, from natural-language task descriptions through scene generation and robot execution to automatic success verification by an evaluator agent.

Main Findings

• Object density: SceneSmith generates 3–6x more objects than prior methods, averaging 71.1 ± 13.0 objects per scene versus 11.2–23.0 for baselines. For house-level scenes it averages 214.1 ± 60.9 objects versus Holodeck's 81.3 ± 18.3 (a 2.6x difference).

• Physical quality: SceneSmith achieves under 2% inter-object collisions (1.2% ± 0.6 in the automated evaluation) versus 3–29% for baselines, and 95.6% ± 1.7 static equilibrium stability versus 8–61% for baselines. Remaining collisions are slight penetrations with 3.8 mm mean depth, described as 3–12x shallower than baselines. House-level results show 0.9% collisions and 79.8% stability versus Holodeck's 3.8% and 17.9%.

• Human preference: In a pairwise study with 205 crowdsourced participants and 3,051 valid responses, SceneSmith achieved a 92.2% average realism win rate and 91.5% average prompt faithfulness win rate against all room-level baselines (all p < 0.001). Against Holodeck on house-level scenes, it achieved 80.3% realism and 84.7% faithfulness win rates.

• Automated metrics: Using SceneEval plus two added physics metrics, SceneSmith performs best on CNT (object count), ATR (attributes), OOR (object-object relationships), OAR (object-architecture relationships), COL (collision rate), and STB (static equilibrium). It achieves a 2.2x improvement on OOR over the best baseline. Its ACC (accessibility) and NAV (navigability) scores are lower, which the authors attribute to the 3–6x higher object density reducing free space.

• Ablation ranking: The most impactful ablations were NotGenerated (63.8% realism, 67.0% faithfulness), NoAssetValidation (63.0%, 62.2%), and NoObserveScene (61.5% realism), all statistically significant. NoSpecializedTools (54.8% realism), NoAgentMemory (53.4%), and NoCritic (51.8% realism; 47.5% faithfulness) showed smaller effects that did not reach significance; the authors state detecting these would require 6–18x more comparisons.

• Cost trade-off: NoCritic achieves similar preference scores while being 70% cheaper, though it produces 24% fewer objects.

• Policy evaluation works: Across 100 generated scenes and four pick-and-place tasks (three room-level, one house-level), a standard model-based policy achieved 16% success versus 12% for a deliberately degraded variant. Of 300 evaluator judgments (100 scenes × 3 states), 99.7% agreed with human labels; the single disagreement was an ambiguous case where a fruit landed on the edge of a plate.

• Qualitative robot demonstrations: A teleoperated RB-Y1 humanoid manipulated an articulated cabinet, and a text-conditioned policy from prior work (Lin et al., 2026) executed zero-shot pick-and-place in a SceneSmith scene despite having no SceneSmith rendering, physics, scene, or asset training data.

Methodology in Plain English

SceneSmith builds a scene in stages, like a contractor working from general to specific. A natural-language prompt first produces a floor plan specifying how many rooms exist and their geometry. Each room is then furnished independently — furniture, wall-mounted items, then ceiling fixtures — using room-specific prompts derived from the global description.

Every stage is a conversation between three VLM agents. A designer proposes changes using structured tools (placing, adjusting, or fetching objects). A critic scores the result from 0–10 on things like semantic plausibility and physical feasibility, and writes feedback. An orchestrator decides whether to accept, refine further, or stop — and can roll back to an earlier checkpoint if scores drop. Agents can invoke as many tools as they like per turn, and the critic can request rendered images to look at rather than relying only on symbolic state. Critic access is restricted to observation tools so it cannot modify the scene it judges.

After furniture is placed, selected supporting surfaces — shelves, tables, floors — become their own branches that get populated with small manipulable objects, again under their own prompts. Object placement uses SE(2) poses within a surface's coordinate frame, which are lifted into full SE(3) poses via the surface's known pose.

Assets are produced on demand rather than retrieved from a fixed library, which the authors note also avoids training-data contamination so robot policies can be evaluated on genuinely unseen objects. A text-to-image model generates a reference image, SAM3 segments the foreground, and SAM3D reconstructs a textured mesh, which is canonicalized, scaled, and augmented with collision geometry and VLM-estimated mass, center of mass, inertia, and friction. Articulated items like cabinets and drawers come from the ArtVIP library, because the authors found current text-to-3D methods do not reliably produce usable articulation. Flat items like rugs and posters become "thin coverings" using materials from ambientCG.

Finally, because agents are not required to satisfy physics exactly, a post-processing step resolves inter-object penetrations with nonlinear optimization that preserves orientations, then lets the scene settle under gravity in Drake so unstable objects reach static equilibrium.

Why This Matters

Impact on research: The paper argues these results mark a point where environment generation is no longer the primary bottleneck for scalable robot training and evaluation in simulation. It also connects two lines of prior work that were largely separate — asset generation and scene assembly — into one pipeline, and shows that physics-level metrics (collision rate, static stability) can differentiate scene generators in ways visual or VLM-judged metrics alone do not. The authors report that baselines do not produce simulation-ready scenes at all and had to be augmented with collision geometry and physical properties for fair comparison.

Real-world applications:

• Training and benchmarking home robot manipulation policies in dense, cluttered, articulated environments that resemble real homes.
• Automatic evaluation of robot policies on open-ended, natural-language tasks without hand-crafted success predicates.
• Generating diverse, task-relevant test scenes on demand for a given task description, supporting evaluation across varied layouts and clutter conditions.
• Interactive robot simulation for teleoperation and data collection, demonstrated with an RB-Y1 humanoid.

Industry relevance: The work is directly motivated by companies such as 1X and Sunday explicitly targeting deployment of robots into arbitrary human homes. Affiliations include MIT and Toyota Research Institute, and the scenes export to major robotics simulators including Drake (used for demonstration), MuJoCo, Isaac Sim, and Genesis.

Future Directions

1. Reliably generating articulated objects. The paper states that current text-to-3D approaches do not yet reliably produce articulated structure and kinematics suitable for robotics simulation, which is why SceneSmith falls back to the ArtVIP library. Closing this would broaden what can be generated on demand.

2. Cheaper agent configurations. NoCritic produces 24% fewer objects but is 70% cheaper with statistically similar preference scores. Which stages actually need a critic, and how to trade cost against density, remains open.

3. More reliable automated evaluation. The authors note that VLM-based SceneEval metrics have false positives and negatives, and the evaluator agent avoids fixed success predicates "at the cost of determinism." Whether the evaluator's performance holds on other policies, tasks, or embodiments is not reported.

4. Generalization of the policy-serving demonstration. The end-to-end evaluation used a single model-based pick-and-place policy family across four tasks and 100 scenes. The paper does not report how the pipeline performs across a wider range of policy classes, longer-horizon tasks, or non-pick-and-place manipulation.

The paper also contains an appendix titled "Limitations and Failure Analysis" (Appendix S), but its contents are not included in the provided material, so the authors' own enumerated limitations are not reported here.

Target Audience

Researchers and engineers working on robot learning, simulation environments, embodied AI, and 3D scene synthesis; practitioners who need large volumes of physically valid, cluttered indoor scenes for policy training or benchmarking; and readers interested in multi-agent VLM systems where designer, critic, and orchestrator roles are used to iteratively refine structured outputs. Readers seeking a gentle introduction to scene generation should note the density of architecture and metric detail.

Authors’ abstract

Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stages$\unicode{x2013}$from architectural layout to furniture placement to small object population$\unicode{x2013}$each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with &lt;2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.