Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface

AI summary

Overview

Research area: Robot learning / data-efficient imitation learning, specifically synthetic demonstration generation for Vision-Language-Action (VLA) and visuomotor policies.

Technical level: Advanced. The paper assumes familiarity with 3D reconstruction, point-cloud editing, video diffusion models, and VLA policy training.

Scope in one sentence: The paper presents a three-stage pipeline that reconstructs real robot demonstrations in 3D, edits them to create new object placements and trajectories, and re-renders them as realistic multi-view RGB videos to train robot policies far more cheaply than collecting new real data.

What This Paper Is About

Robot policies generalize poorly to new spatial arrangements of objects unless they are trained on many demonstrations covering those arrangements, but collecting such data by teleoperation is slow and expensive. Existing synthetic-data approaches either render in simulators (introducing a sim-to-real gap), require object assets or dense scans, or produce only point-cloud data incompatible with the dominant multi-view RGB VLA paradigm. Real2Edit2Real solves this by using depth as a bridge: 3D geometry is edited to create new trajectories, and a depth-conditioned video generator turns those edits back into photorealistic RGB demonstrations.

Key Contributions

1. A metric-scale geometry reconstruction model (Metric-VGGT) trained with a hybrid real-plus-simulated paradigm. Simulated data supervises camera poses and point maps (where ground truth is exact), while real depth-sensor data supplies true metric scale—overcoming the domain gap that causes vanilla feed-forward reconstructors like VGGT to produce noisy, scale-misaligned point clouds in robot scenes.

2. A depth-reliable spatial editing pipeline that decomposes demonstrations into "motion" and "skill" segments, applies object transformations to skill segments to preserve robot–object interaction, plans new motion segments, and critically performs robot pose correction using forward/inverse kinematics and URDF re-rendering so the resulting depth maps are kinematically valid rather than rigid-body artifacts.

3. A 3D-controlled multi-view video generation model conditioned primarily on depth, augmented with Canny edges, action maps, and ray maps. It uses a dual-attention design (intra-view plus cross-view) for multi-view consistency at lower cost than global attention, and a "smooth object relocation" scheme that converts object repositioning into an interpolated video transformation so first-frame editing is handled by generation rather than a separate image-editing step.

4. Demonstration that policies trained purely on generated data from 1–5 source demonstrations match or exceed policies trained on 50 real demonstrations, plus extensions to object height and background texture editing within one unified framework.

Main Findings

• 10–50× data efficiency gain: Across four real tasks (Mug to Basket, Pour Water, Lift Box, Scan Barcode), Go-1 and π0.5 policies trained on 200 generated demonstrations from only 1–5 source demos reached 65.0%–81.3% average success, versus 61.3% for 50 real demonstrations. From 5 source demos, Go-1 hit 78.8% and π0.5 hit 81.3%, beating the 50-real-demo baseline by 17.5 and 20 percentage points.

• Scarce real data generalizes poorly: With fewer than 20 real demonstrations, average success fell below 50% across both policies, confirming the spatial-generalization bottleneck the paper targets.

• Height generalization: A policy trained on 20 real tabletop demonstrations scored 0/5 on a raised platform (out-of-distribution height). Generating 20 tabletop plus 20 platform demonstrations lifted the combined success rate to 80%.

• Texture robustness: Generating demonstrations with varied desktop textures raised average success from 50% (50 real demos) to 68% when texture variation was included in generation; without texture-inclusive generation the gain was marginal (52%).

• Ablations confirm each module matters: Metric-VGGT produced the cleanest point clouds and most accurate camera poses versus raw real data or vanilla VGGT; removing robot pose correction produced blurry, kinematically inconsistent generations; removing smooth object relocation caused objects to land in wrong positions, making demonstrations unusable.

• Practical generation cost: A 20-second, 30-FPS episode generates in ~48.6 seconds on 8 H100 GPUs.

Methodology in Plain English

The pipeline runs in three stages.

Stage 1 — Reconstruct the scene in true 3D scale. Take the multi-view RGB video from a real demonstration (head plus two wrist cameras on a humanoid robot). Run a fine-tuned VGGT model that outputs per-frame depth maps and camera poses. Fine-tuning mixes two data sources: simulated robot data (perfect camera poses and point maps, but unrealistic object scales) and real robot data (noisy depth from sensors, but true metric scale). This teaches the model to be both geometrically accurate and correctly scaled in the real world.

Stage 2 — Edit the 3D scene to create new trajectories. Split the demonstration into segments where the arm moves freely (motion) and where it interacts with an object (skill). Pick a random new placement for the object, apply the same rigid transform to the object and to the robot's gripper point cloud during skill segments so the interaction relationship is preserved, and use motion planning to generate a new path connecting the segments. Because the wrist camera is attached to the end-effector, its pose is transformed too, and the edited point cloud is projected back to depth maps. A key fix: the naive approach treats the whole robot as rigid, which breaks kinematics — so the robot's arm is re-rendered from forward kinematics under the new joint states (computed via inverse kinematics), producing depth that is physically plausible. Background holes are filled by inpainting the first frame, reconstructing the background, and aligning its scale to the original table plane.

Stage 3 — Turn depth edits back into realistic RGB video. A Transformer-based video diffusion model (built on GE-Sim / Cosmos-Predict-2B) takes the first frame plus the edited depth sequence as the primary control signal, supplemented by Canny edges (sharper object boundaries), action maps (motion grounding), and ray maps (multi-view consistency). Attention is split into intra-view and cross-view so the three camera streams stay coherent without full global attention. Object relocation is expressed as a smooth interpolated motion at the start of the video, so the model itself performs the repositioning rather than a separate 2D editing step.

Why This Matters

Impact on research: The paper closes a specific gap — prior work either needed simulators, object assets, dense 3D scans, or produced point-cloud-only data incompatible with RGB-based VLA training. Real2Edit2Real is the first framework (per its own comparison table) to simultaneously require no simulator, work from RGB-only source data, remain VLA-compatible, and generate both novel textures and novel trajectories. It also demonstrates that depth is a sufficient and natural interface between 3D editing and 2D generation, which is a reusable design principle.

Real-world applications:

• Rapidly bootstrapping manipulation policies in warehouses or factories where teleoperation time is expensive and object placements vary widely.
• Home-service or elder-care robots that must handle the same object in many spatial configurations without exhaustive recording.
• Retail or logistics scanning tasks (as in the Barcode task) where objects appear at unpredictable positions and orientations.
• Domain randomization for robustness — varying table height or surface texture synthetically rather than physically rebuilding a workspace.

Industry relevance: Companies deploying VLA models (the paper uses AgiBot's Go-1 and Physical Intelligence's π0.5) can cut data-collection budgets by an order of magnitude while preserving or improving task success. The framework is also modular: the geometry model and the video generator can each be upgraded independently as better reconstruction or diffusion backbones appear.

Future Directions

• Scaling to more complex interactions: The four tasks cover grasping, pouring, lifting, and two-arm coordination; open questions remain about deformable objects, articulated tools, and long-horizon multi-step tasks where 3D editing assumptions may break down.
• Reducing reliance on real depth sensors: The hybrid training still uses 100K real depth frames; a fully RGB-only reconstruction path would further lower setup cost.
• Generation speed and fidelity: ~49 seconds per episode is workable for offline augmentation but far from interactive; faster distillation or streaming generation could enable on-the-fly data expansion during training.
• Better handling of occlusion and novel viewpoints: The current approach projects edited point clouds from the source camera configuration; generalizing to substantially new camera poses or heavy occlusion remains an open technical challenge.

Target Audience

Robotics and embodied-AI researchers working on imitation learning, VLA policies, or data augmentation; engineers building data pipelines for manipulation policies who want to reduce teleoperation cost; and graduate students interested in the intersection of 3D reconstruction, video diffusion, and robot learning. Readers without background in point clouds, kinematics, or diffusion models will find Sections 3.3 and 3.4 dense, but the experimental tables and figures are accessible to a broader technical audience.

Authors’ abstract

Recent progress in robot learning has been driven by large-scale datasets and powerful visuomotor policy architectures, yet policy robustness remains limited by the substantial cost of collecting diverse demonstrations, particularly for spatial generalization in manipulation tasks. To reduce repetitive data collection, we present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface. Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model. Based on the reconstructed geometry, we perform depth-reliable 3D editing on point clouds to generate new manipulation trajectories while geometrically correcting the robot poses to recover physically consistent depth, which serves as a reliable condition for synthesizing new demonstrations. Finally, we propose a multi-conditional video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation videos. Experiments on four real-world manipulation tasks demonstrate that policies trained on data generated from only 1-5 source demonstrations can match or outperform those trained on 50 real-world demonstrations, improving data efficiency by up to 10-50x. Moreover, experimental results on height and texture editing demonstrate the framework's flexibility and extensibility, indicating its potential to serve as a unified data generation framework. Project website is https://real2edit2real.github.io/.