EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation

AI summary

Overview

Research area: Computer Vision / Embodied AI — specifically simulation-based generation of robot training data, 3D scene and asset synthesis, and robotic task (demonstration) generation.

Technical level: Advanced. The paper assumes familiarity with vision-language models, robot learning (VLA and world-action models), physics simulation, motion planning (CuRobo), and recursive self-improvement (RSI).

Scope: One sentence: the paper presents EmbodiedSmith, an agentic simulation framework that couples scene generation, asset generation, and robot task generation into a recursive refinement loop in order to produce validated robot demonstrations at scale across diverse embodiments and physical phenomena.

Paper metadata: arXiv:2610.07969v1 [cs.CV], 06 Oct 2026. Keywords listed: Generative AI, Robotic RSI. Affiliations include The Hong Kong University of Science and Technology (Guangzhou), Xiaomi Robotics, Southeastern University, Tsinghua University, Zhejiang University, Westlake University, Symbiosis Robotics, Peking University, Xi'an Jiaotong University, and Beijing Academy of Artificial Intelligence.

What This Paper Is About

Scaling robot foundation models needs both large, diverse training data and reliable environments for evaluation, and simulation is the natural place to get both. Existing simulation data pipelines, however, are held back by predefined assets and skills, by scene generation that is decoupled from task generation (so planned actions are never verified as feasible), and by weak support for complex physics (deformables, fluids) and complex bodies (mobile manipulators, humanoids, dexterous hands). EmbodiedSmith's goal is to unify asset, scene, and task generation in one pipeline where scenes and tasks iteratively improve each other, producing validated demonstrations across many embodiments and richer physics.

Key Contributions

1. A unified, high-diversity generation pipeline. The authors design 3D generation methods that can produce novel objects, layouts, and scenes, plus agents that autonomously create and compose new skills, all integrated into one pipeline supporting both autonomous generation and language-driven customization.

2. Richer physics simulation. The framework supports assets and tasks involving deformable objects and fluids, which the paper identifies as gaps in existing data diversity and a major source of the sim-to-real gap.

3. Recursive self-improvement (RSI) data generation. Scene generation anticipates downstream task requirements while task generation can revise the scene through targeted edits; this agentic loop lets the two recursively refine one another and is credited with substantially raising task generation success, including on long-horizon tasks.

4. Human-level dexterity across embodiments. Task generation is extended to mobile manipulators, humanoids, and dexterous hands through embodiment-specific adapters, enabling data with human-like dexterity.

Main Findings

• Scene quality beats five baselines. Against Holodeck, LayoutVLM, SAGE, EmbodiedGen V2, and SceneSmith on 20 indoor scene specifications with three independent generations per scene, EmbodiedSmith achieves state-of-the-art on CF, SL, PP, VQ, COL, OOB, and OPN. Content/Function reaches 7.13 ± 1.48 versus 5.57 ± 1.68 for the strongest baseline, a 28.0% relative improvement; Spatial Layout reaches 7.27 ± 1.41 and Physical Plausibility 7.68 ± 0.83, beating the strongest baselines by 17.6% and 16.5%; Visual Quality reaches 7.32 ± 0.79 versus 5.95 ± 0.98 for EmbodiedGen V2, an improvement of 23.0%.

• Text fidelity is not the top score. SceneSmith obtains the highest TF score of 8.92 ± 2.33, while EmbodiedSmith yields a competitive 8.33 ± 2.88.

• Geometric usability is preserved. EmbodiedSmith achieves the lowest Collision score of 0.29 ± 0.35, produces no out-of-bounds objects (0.00 ± 0.00), maintains Navigability of 0.9984 ± 0.0130 (comparable to the top baseline result of 0.9988 ± 0.0052), and achieves the highest Openness score of 0.9875 ± 0.0481. The paper stresses this matters because richer scenes often introduce collisions, blocked openings, or fragmented traversable space.

• Demonstration generation success rises sharply. On 16 tasks (10 fixed-base manipulation, 6 mobile manipulation, evenly split between short- and long-horizon), EmbodiedSmith averages 67% ASR, 77% Top-5 ASR, and 40,204 tokens. SceneSmith + Planner averages 17% ASR, 23% Top-5, and 137,062 tokens; EmbodiedGen V2 + Planner averages 21% ASR, 27% Top-5, and 91,062 tokens.

• Per-task gaps are large. For fixed-base Open Drawer, EmbodiedSmith reaches 78% ASR and 80% Top-5 versus 32%/32% for SceneSmith + Planner and 40%/40% for EmbodiedGen V2 + Planner; EmbodiedGen V2 is marked as not supporting the articulated assets required. For Basket to Tray, EmbodiedSmith reaches 70% ASR versus 0% for EmbodiedGen V2 + Planner. For mobile Block Order, EmbodiedSmith reaches 93% ASR and 100% Top-5 versus 0% for both baselines.

• Mobile and fixed-base settings differ. The paper notes that for EmbodiedSmith the mobile setting performs better on Cup to Tray but worse on Open Drawer, suggesting navigation may improve reachability while grasp-feasible placement may still be unsuitable for joint-constrained opening motions.

• Execution feedback is essential. In ablations, removing execution feedback ("w/o Feedback") reduces ASR to almost 0% across all task categories (Fixed-Short 4.0%/8.0% at 3,721 tokens; Fixed-Long, Mobile-Short, Mobile-Long all 0.0%/0.0%), despite lower token consumption.

• Placement validation matters most for long-horizon fixed-base tasks. "w/o Placement Validation" yields 48.6% ASR / 56.8% Top-5 (63,819 tokens) on Fixed-Short, 13.2% / 22.4% (208,316 tokens) on Fixed-Long, 67.3% / 79.6% (13,294 tokens) on Mobile-Short, and 47.6% / 64.7% (40,898 tokens) on Mobile-Long, versus 69.4% / 75.2% (15,003), 71.6% / 79.2% (78,183), 70.0% / 86.0% (8,921), and 54.0% / 65.3% (50,187) for the full method. The paper reports smaller drops on mobile tasks, suggesting navigation partially compensates for suboptimal initial placement.

• Data diversity improves generalization. Two equal-duration datasets were built: clean (20 hours of fixed-layout demonstrations without randomization) and mixed (10 hours of those plus 10 hours of randomized demonstrations with varying tabletop layouts). Averaged across ten tasks, π0.5 trained on clean scores 61.2% in the simple setting, 7.2% in the random setting, and 34.2% on average; trained on mixed it scores 50.8%, 40.2%, and 45.5%. FastWAM scores 25.5%, 1.0%, and 13.3% on clean versus 20.4%, 10.2%, and 15.3% on mixed.

• The mixed data changes the trade-off rather than uniformly improving everything. Simple-setting accuracy drops for both models on mixed data (61.2% → 50.8% for π0.5; 25.5% → 20.4% for FastWAM) while random-setting accuracy rises substantially (7.2% → 40.2% for π0.5; 1.0% → 10.2% for FastWAM).

• Prompt-driven task generation worked without manual debugging. For the policy-training data, the authors state all task plans for the study room and kitchen were generated directly from prompts and executed successfully without manual debugging or intervention.

• Reported limitation. For the evaluated deformable configuration, incomplete post-release recovery is documented as a known limitation rather than an acceptance criterion, and the evaluated implementation adopts a simplified procedural representation with uniform material properties.

Methodology in Plain English

The system takes a task specification and a robot embodiment as input and outputs validated robot demonstrations. It works in three stages:

1. Building the environment. A vision-language model parses the natural-language description into a typed relational graph describing functional zones, static fixtures, orientations, support surfaces, and robot approach directions. Assets come either from a persistent pool of validated open-source assets — canonicalized by a VLM using multi-view renderings and metadata, then checked for semantic correctness, geometric integrity, and scene compatibility — or from on-demand generation: a scene-consistent reference image is refined in image space and reconstructed into a textured 3D asset with SAM3D, then scaled, repaired, given collision geometry via adaptive convex decomposition, assigned physical parameters by a VLM, and tested through resting, contact, and perturbation checks. Physical assets (articulated, deformable, fluid) go through construction, physics validation, and task validation: articulated assets follow an Articraft-based loop where an LLM repeatedly writes and revises construction programs against an SDK using compiler and validation feedback; physics checks test inertial consistency, joint motion, grasp keypoints and enclosure geometry; CuRobo then assesses collision-free motion feasibility before simulator execution. Failures trigger targeted revisions, non-improving changes are reverted, and only episodes passing scene inspection, motion planning, and data export are kept.

Scene layout is solved hierarchically: ground fixtures are organized into local compositions and jointly optimized with Beam-DFS search, followed by wall fixture placement and parent-relative task-object positioning, enforcing boundary limits, collision avoidance, clearances, connectivity and workspace bounds, with reachability validated by CuRobo. The result is described as an initial geometry- and reachability-validated scene, not a guaranteed executable environment.

2. Generating the robot task. Three components cooperate: a Task Plan Agent that interprets tasks and proposes repairs, a Harness that schedules the workflow and triggers independent reviews, and simengine that runs simulation validation and records execution. The agent maps task references to scene entities, binds atomic skills, and builds a directed acyclic graph of skills (a Skill DAG) with robot placement, action parameters, execution order, and checkable success conditions. Five tools are exposed: build, run, inspect, finish, and revise. The agent builds a candidate configuration, runs it in physics simulation, receives programmatic acceptance checks plus failure evidence, and revises object bindings, dependencies, working poses, grasp poses, approach directions, or motion parameters as needed. Even when a first run passes, the agent must request an independent VLM review before calling finish for final delivery checks.

3. Adapting to difficult bodies. A shared skill DAG sits above execution adapters so atomic skills stay reusable across morphologies. For the Unitree G1 humanoid, locomotion targets become planar velocity and heading references for learned balance-walking policies while arm goals are resolved via G1 kinematics and cuRobo; at 50 Hz the adapter merges 15-DoF lower-body and 14-DoF arm commands into a 29-DoF body action, dispatching hands separately. For dexterous manipulation, Sharpa Wave hands are deployed on the Split ALOHA platform (6-DoF arm and 22-DoF hand per side), with a task-specific contact solver mapping interaction contracts into collision-checked hand poses, cuRobo planning arm trajectories, explicit finger-joint commands, and PhysX recording per-link contact telemetry, each task enforcing a fail-closed contact contract.

Evaluation approach. Environment generation is scored by GPT-5.5 on a 1–10 scale from four rendered views per scene, alongside geometry metrics. Demonstration generation runs 10 independent agent sessions per task, each evaluated with the same 10 simulation seeds not used during generation or refinement. Policy training compares two equal-duration datasets on π0.5 and FastWAM.

Why This Matters

Impact on research. The paper reframes simulation as infrastructure for recursive self-improvement in robotics, arguing that as RSI techniques advance, simulation environments become an essential component of robotics infrastructure. It offers a concrete mechanism — joint scene/task refinement — for one of the field's practical bottlenecks: scene generation decoupled from task generation means planned actions are never verified, which the paper says becomes more pronounced as tasks grow more complex or longer-horizon. It also provides a way to systematically analyze how task, environment, and embodiment diversity affect pretrained models, and to act as an evaluator supplying metrics and feedback for iterating foundation models.

Real-world applications.

• Generating pretraining data across single-arm, dual-arm, humanoid, and dexterous-hand configurations for general-purpose robot foundation models.
• Household service robotics, using the study-room and kitchen scenes and tasks such as drawer opening, bottle righting, bowl nesting, and sponge wiping.
• Logistics and mobile manipulation, where the framework's mobile tasks (cup to tray, lift cover, book stow) exercise navigation-plus-manipulation behavior.
• Simulation-based evaluation and regression testing of robot policies at low cost, quickly, and reproducibly, as the introduction argues simulation enables.

Industry relevance. Two of the listed affiliations are robotics industry entities (Xiaomi Robotics, Symbiosis Robotics) alongside academic and national research institutes, indicating direct industrial interest in scalable embodied data engines. The paper's results on token cost — 40,204 mean VLM output tokens per session for EmbodiedSmith versus 137,062 for SceneSmith + Planner and 91,062 for EmbodiedGen V2 + Planner — are directly relevant to the operating cost of such pipelines.

Future Directions

• Integrating physical asset generation into the scene generator. The paper states explicit that this integration remains downstream work, marking it as an unfinished piece of the pipeline.
• Improving deformable fidelity. Incomplete post-release recovery for the evaluated deformable configuration is currently documented as a known limitation rather than an acceptance criterion, and the evaluated implementation uses a simplified procedural representation with uniform material properties.
• Raising success on the hardest tasks. Even with the full pipeline, several tasks remain far from saturated (mobile Book Stow at 17% ASR, fixed-base Open Cabinet at 35% ASR, mobile Open Drawer at 43% ASR), and long-horizon categories remain the weakest.
• Balancing the diversity/generalization trade-off. Mixed data improved random-setting performance substantially but reduced simple-setting performance, raising the open question of how to combine fixed and randomized data without sacrificing in-distribution accuracy.

Target Audience

Robotics and embodied-AI researchers working on data generation, simulation, and policy pretraining; practitioners building simulation data engines for manipulation and mobile manipulation; researchers in 3D scene and asset generation who care about simulator-ready output rather than visual fidelity alone; and engineers at robotics companies evaluating whether generated data can improve downstream VLA or world-action model performance. Readers without a background in robot learning, motion planning, or agentic LLM pipelines will find the methods section demanding, though the experimental results are presented with enough context to be interpretable.

Authors’ abstract

Scaling robotic foundation models requires diverse training data and reliable evaluation environments. Simulation offers a scalable solution, yet existing generation pipelines remain constrained by predefined assets and skills, a disconnect between scene generation and task generation, and limited support for complex embodiments and physics. We introduce EmbodiedSmith, a framework for scalable embodied data generation through recursive self-improvement (RSI). EmbodiedSmith unifies asset, scene, and task generation in a pipeline that supports autonomous creation and language-driven customization. Its core is an agentic refinement loop: scene generation anticipates downstream task requirements, while task generation guides targeted scene edits, allowing scenes and tasks to iteratively improve one another. This joint refinement improves task generation success, including for long-horizon tasks. The framework further supports mobile manipulators, humanoids, and dexterous hands, as well as interactions involving deformable objects and fluids, broadening the range of behaviors and physical phenomena represented in generated data. Together, these capabilities provide a flexible simulation engine for both robot pretraining and evaluation. Extensive experiments validate the quality, diversity, and generation efficiency of the resulting data, while downstream policy experiments demonstrate that increased data diversity improves generalization.