IGen: Scalable Data Generation for Robot Learning from Open-World Images

AI summary

Overview

Research area: Robotics — specifically scalable data generation for visuomotor robot policy learning, combining 3D scene reconstruction, vision-language model reasoning, and point-cloud rendering.

Technical level: Advanced. The paper assumes familiarity with visuomotor policy learning, SE(3) end-effector poses, point clouds, vision-language models, and sim-to-real transfer, though the core idea is explained in plain terms below.

Scope: IGen is a framework that turns a single open-world image plus a natural-language task instruction into large-scale paired visual-and-action training data for robot manipulation, without any human teleoperation.

What This Paper Is About

Robot policies that map what a camera sees to what a robot should do need enormous amounts of matched visual-and-action data, but collecting that on real robots is slow, expensive, and tied to specific environments. Meanwhile, the internet contains vast numbers of open-world images covering diverse real-world scenes — but those images carry no robot actions, so they cannot be used directly. IGen closes this gap by converting unstructured 2D images into structured 3D scenes, reasoning out executable robot behavior with vision-language models, and rendering temporally coherent visual observations that form complete training data.

Key Contributions

1. A scalable data generation framework that produces visual-action datasets from open-world images, integrating cross-scene generalization, instruction diversity, and long-horizon task applicability.

2. A pipeline that turns unstructured images into actionable 3D scene representations — point clouds and spatial keypoints — enabling robotic task reasoning and motion planning that generates task-consistent behaviors aligned with the physical world.

3. A simulation-free point cloud synthesis approach that produces realistic visual observations and supports large-scale robot experience generation, rather than relying on physics-based scene reconstruction.

4. Real-world demonstration that policies trained on IGen-generated data alone can successfully perform manipulation tasks with no additional data collection, suggesting real-world images can serve as an effective source for robot policy training.

Main Findings

• Better visual fidelity than Real-to-Sim reconstruction. On scenes from the Simpler dataset, IGen reconstructed scenes scored higher on PSNR, SSIM, and three LPIPS variants than the Real-to-Sim method in Simpler. Averaged across four tasks (Pick Carrot on Plate, Put Eggplant in Basket, Put Spoon on Towel, Stack Cubes), IGen reached PSNR 27.0040 versus 17.2649, SSIM 0.8522 versus 0.6833, and LPIPS scores of 0.0630 / 0.1139 / 0.0446 versus 0.3235 / 0.3959 / 0.2302. The paper reports up to a 5.13× improvement in perceptual similarity compared with the Real-to-Sim baseline.

• Better instruction following and physical plausibility than video-generation baselines. On the DreamGen Bench, IGen outperformed Cosmos-Predict2 and TesserAct on both Instruction Following and Physics Alignment, as judged by GPT-4o, Qwen-3-VL-Plus, and GLM-4.5V. Each method generated 40 videos with their prompts, and the reported metric is the proportion of videos scoring 1. Under Qwen-3-VL-Plus evaluation, IGen produced nearly twice as many Instruction Following successes as the baseline.

• Substantially cheaper and faster generation. IGen required only 8.3 GB of GPU memory and approximately 18.6 seconds per sample. Under the same GPU memory conditions, it achieved roughly 30× and 200× higher data generation efficiency than TesserAct and Cosmos-Predict2, respectively.

• Significant real-world task success from a single image. Using π₀ as the base vision-language-action model fine-tuned with LoRA, and a Franka Research 3 arm with a Microsoft Kinect RGB-D camera (RGB stream only), IGen-generated data raised success on the Place Bottle task from 0.0% to 75.0%.

• Synthetic data beats human teleoperation under equal time budgets. With 100 and 1,000 IGen-generated samples and no human-collected fine-tuning data, the policy achieved success rates of 44.5% and 66.7%, surpassing the 19.4% and 58.3% achieved by human teleoperation under the same data collection time budget.

• Zero-shot transfer is near-zero, fine-tuning is essential. Without any fine-tuning, π₀ achieved an almost zero success rate across the tasks, indicating limited zero-shot capability in this experimental setting.

• Large-scale automated generation per image. From a single captured scene image, IGen automatically generated 1,000 task demonstrations with spatial randomization, which were used to train and deploy a visuomotor policy evaluated on real-world tasks including "Water Flowers," "Hit Box," and "Place Toy."

Methodology in Plain English

IGen runs a three-stage pipeline that starts from one image and one task description.

Stage 1 — From pixels to structured 3D. A monocular geometry foundation model (Metric3Dv2) estimates depth. A vision-language model reads the prompt to find task-relevant objects, and Segment Anything Model produces masks for them. DINOv2 extracts scene features, and K-means clustering over both features and 3D coordinates yields K spatial keypoints with 3D coordinates. An inpainting model removes the target object and rebuilds the background, which is lifted into a dense colored 3D point cloud using the depth map and camera intrinsics. Because a single view cannot model a manipulated object's full geometry, a 3D generative reconstruction model (TRELLIS) completes the object, 6D pose estimation puts it back in place, and candidate placement points are extracted from the supporting surface to allow spatial randomization.

Stage 2 — Planning behavior. The VLM receives the annotated keypoints, their 3D coordinates, and the instruction, and decomposes the task into sub-stages. It then writes low-level Python control functions based on SE(3) end-effector poses, with parameters like translation distance and lift height inferred from the 3D keypoint coordinates. Grasp poses come from a grasp prior model, and gripper state is set from the point cloud width along the gripper's principal axis. A motion planner turns the sub-goals into feasible trajectories, which are executed in simulation (Isaac Sim, with Curobo as the solver) to produce a time-stamped action sequence sampled at a fixed frame rate; each action includes end-effector pose and joint positions.

Stage 3 — Synthesizing observations. The simulated robot is placed at the planned pose and a virtual camera at the reconstructed viewpoint, sampling RGB and depth at 30 fps. These frames are back-projected into a point cloud sequence of the robot's motion. The background is a static point cloud. Once the grasp is established at time t_g, the object's point cloud is rigidly transformed by the end-effector's relative motion, so the object follows the gripper during the grasp interval and stays put otherwise. Combining static background, object, and robot point clouds gives a composite sequence that is rendered frame by frame into visual observations, which pair with the action sequence to form training data.

Why This Matters

Impact on research. The work challenges the assumption that robot policy training requires expensive on-robot data collection. By treating in-the-wild images as a first-class data source and generating grounded actions rather than only plausible-looking video, it offers a path toward the data scale that generalist robot policies need — comparable to how web-scale images fueled progress in vision-language models. The reported result that synthetic data can outperform human teleoperation within equal time budgets is a meaningful signal for how robot datasets should be collected.

Real-world applications:

• Warehouse and logistics manipulation, where varied packages and shelf layouts would otherwise require bespoke data collection per site.
• Household service robots that must handle diverse kitchen, tabletop, and living-room scenes captured in ordinary photos.
• Agricultural and outdoor manipulation, where robots face highly variable lighting, plants, and terrain that are hard to reproduce in a lab.
• Rapid task onboarding in manufacturing, where a photo of a new workstation plus a text instruction could replace weeks of demonstration collection.

Industry relevance. The computational profile matters commercially: at 8.3 GB of GPU memory and roughly 18.6 seconds per sample, and with 30× to 200× efficiency gains over video-generation baselines, data synthesis becomes practical on ordinary single-GPU training budgets rather than only at large-scale compute clusters. The 1,000-demonstrations-per-image generation rate, without human annotation, directly attacks the cost bottleneck that keeps robot foundation models from scaling the way language and vision models have.

Future Directions

• Extending beyond rigid-body manipulation. The current synthesis models grasped objects as rigid point clouds following the end-effector; deformable objects, articulated tools, and fluids are not addressed and represent an obvious next target.

• Scaling and diversifying evaluation. Real-world tests cover three task types with 12 independent trials per task and a fixed hardware setup (a Franka Research 3 arm and one Kinect camera). Testing across more robot platforms, camera configurations, and longer task horizons would test how well the pipeline generalizes.

• Addressing reconstruction limits from a single view. The pipeline compensates for monocular incompleteness using a generative 3D reconstruction model and 6D pose estimation; how much residual error this introduces, and whether multi-view or video input would help, remains open.

• Quantifying the sim-to-real contribution of each component. The paper includes an ablation study on pipeline components and reports that IGen reduces the sim-to-real gap, but a fuller accounting of which design choices drive the real-world success rates would guide further work.

Target Audience

This paper is most valuable to robotics researchers and engineers working on manipulation, visuomotor policy learning, and sim-to-real transfer; to practitioners building generalist robot foundation models who need scalable data pipelines; and to graduate students and technical readers already comfortable with imitation learning, 3D scene representations, and vision-language models who want a concrete example of converting unstructured visual data into grounded robot experience. Researchers working on video-generation-based data synthesis will also find it relevant as a contrasting, action-grounded alternative.

Authors’ abstract

The rise of generalist robotic policies has created an exponential demand for large-scale training data. However, on-robot data collection is labor-intensive and often limited to specific environments. In contrast, open-world images capture a vast diversity of real-world scenes that naturally align with robotic manipulation tasks, offering a promising avenue for low-cost, large-scale robot data acquisition. Despite this potential, the lack of associated robot actions hinders the practical use of open-world images for robot learning, leaving this rich visual resource largely unexploited. To bridge this gap, we propose IGen, a framework that scalably generates realistic visual observations and executable actions from open-world images. IGen first converts unstructured 2D pixels into structured 3D scene representations suitable for scene understanding and manipulation. It then leverages the reasoning capabilities of vision-language models to transform scene-specific task instructions into high-level plans and generate low-level actions as SE(3) end-effector pose sequences. From these poses, it synthesizes dynamic scene evolution and renders temporally coherent visual observations. Experiments validate the high quality of visuomotor data generated by IGen, and show that policies trained solely on IGen-synthesized data achieve performance comparable to those trained on real-world data. This highlights the potential of IGen to support scalable data generation from open-world images for generalist robotic policy training.