AOMGen: Photoreal, Physics-Consistent Demonstration Generation for Articulated Object Manipulation

AI summary

Overview

• Research area: Robotics — robot manipulation data generation, 3D scene reconstruction and editing (3D Gaussian Splatting), and Vision-Language-Action (VLA) policy training.
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, part-level segmentation, forward/inverse kinematics, articulated object joints, and VLA model fine-tuning.
• Scope: AOMGen is a data-generation framework that takes one real scan, one real demonstration, and a library of same-category 3D assets, and produces large volumes of photorealistic, physically verified multi-view demonstration data for articulated object manipulation.

What This Paper Is About

Fine-grained manipulation of articulated objects (drawers, microwaves, cabinets, tool boxes) requires large amounts of precise demonstration data, but collecting such data from real robots is expensive, labor-intensive, and limited in scenario coverage. Existing alternatives either rely on physics simulators that look unlike the real world (hurting Sim-to-Real transfer) or on video world models that look realistic but do not guarantee physical consistency or action executability. AOMGen aims to generate demonstrations that are simultaneously visually realistic and physically consistent, by reconstructing a single real manipulation scene and then swapping in new same-category articulated objects and new poses.

Key Contributions

1. Single-demonstration scalability: From a single static scan video of an articulated object, AOMGen can generate manipulation data for any other object in the same category.
2. Physics plus photorealism: The framework enforces precise physical interactions (contact points, joint parameters, replayable trajectories) while maintaining high visual realism in all synthesized data, without relying on a physics simulator.
3. Pose generalization: The architecture supports arbitrary adjustment of the target object's pose, greatly expanding the diversity of generated configurations and pushing generalization boundaries.
4. Validated for VLA training: The synthetic data is shown to be effective for fine-tuning VLA policies, improving model performance on both seen replacement objects and unseen objects.

Main Findings

• Simulator replay is physically sound: Across five replacement objects — microwave oven (98%), tool box (96%), computer (96%), drawer (100%), cabinet (100%) — the average simulation replay Success Rate was 98%, using 50 generated data entries per replacement object with pose changes only. This covers both rotational-joint and prismatic-joint objects.
• VLA fine-tuning improves dramatically: Fine-tuning on AOMGen data increased success rate from 0% to 88.7% (abstract), tested on unseen objects and layouts. In the specific experiments, π0.5 reached 88.66 and OpenVLA reached 81.34, with performance improving as the number of generated demonstrations increased (50 vs. 150 samples).
• Scale robustness: At scales from 0.6 to 0.9 of the optimized USD asset, microwave oven success rates were 55/65/65/70 for OpenVLA and 65/75/80/85 for π0.5; drawer rates were 60/70/80/80 for OpenVLA and 80/90/90/90 for π0.5.
• Object replacement enables generalization to unseen objects: Training on mixed data from multiple replacement objects yielded a 65% success rate on an unseen object, versus only 15% when trained on single data.
• Environment robustness: Because 3DGS-rendered images differ from the IsaacSim environment, the successful results already indicate some robustness to environment mismatch, and models were also tested under varied initial object positions.

Methodology in Plain English

AOMGen works in three broad stages.

Stage one — reconstruct the scene. The researchers scan the manipulation scene from multiple viewpoints and use COLMAP to get camera poses and a sparse reconstruction. They then build a 3D Gaussian Splatting (3DGS) model in which every Gaussian point carries a feature learned from multi-view 2D masks produced by SAM2 (following SAGA). Tuning SAM2's granularity yields part-level masks, so the articulated object is split into a movable part and a static part, distinguished by where the robot end effector is over time. The 3DGS coordinate frame is then aligned to the real-world frame by matching sampled points from the robot URDF mesh to the segmented robot points with Iterative Closest Point (ICP).

Stage two — recover motion. The robot arm's motion is recovered directly: segmented robot Gaussians are transformed into the real frame and link rotations are computed by forward kinematics from the recorded joint states. For the articulated object, there is no recorded motion, so the authors design AOMotion, a supervised method with four submodules. Keyframes are found by comparing movable-part masks between frames (subtracting the robot mask to remove its influence), computing a Motion Score per frame, smoothing with a Savitzky-Golay filter, and applying a dynamic threshold (baseline from the 20th quantile plus three noise standard deviations) to label the first and last motion frames. Contact points are then detected by finding the nearest points between the robot and the movable part at the start frame. Joint modeling constructs bounding boxes for both parts, scores edge pairs using parallelism and distance ((1-parallelism)*0.8 + distance*0.2), and selects the pair via a combined criterion that differs for rotational (CLOSE) and prismatic (OPEN) joints; the joint direction comes from the chosen movable edge, and the joint center from the average of the midpoints of the K closest point pairs. Finally, movable-part motion is recovered by optimizing the rotation or translation parameter so that the rotated surface intersects the real contact-point trajectory.

Stage three — replace the object and generalize. A new same-category asset (from USD) is converted to 3DGS. A NOCS-inspired contact-point mapping normalizes both old and new movable-part point clouds into a unit cube, projects the contact point, and denormalizes it onto the new object. Two optimization stages then fit geometry: a coarse stage estimating scale, initial motion parameter, and XY offset under a constant-speed assumption, and a fine stage that accounts for non-uniform motion and end-effector sliding by using the actual intersection-point objective. Visually, lighting is extracted from the real scene with DiffusionLight and baked onto the object's material in Blender, and Gaussian Inpainting fills holes left by replacement. For pose generalization, the end-effector trajectory is split into three stages around the start and end frames; the middle stage is transformed by the object's pose transform, and the new trajectory is rebuilt with linear and spherical linear interpolation, with joint angles recovered by inverse kinematics.

Evaluation setup. Real demonstrations were collected with a Universal Robot UR5e with a 2F85 gripper, scanned by a mobile device with a fixed RGB camera recording manipulation. Replacement assets came from ArtVIP: microwave oven, tool box, and computer (rotational joints) and drawer and cabinet (prismatic joints). Pose generalization used translation range [-0.05m, 0.3m] × [-0.05m, 0.05m] and rotation range [-45°, 45°]. Training ran on an NVIDIA RTX4090; π0.5 was fine-tuned with LoRA (batch size 16, learning rate 5×10⁻⁵, cosine decay with 10K step warm-up, 30,000 steps, AdamW, gradient clipping 1.0, action prediction horizon 5 steps, bfloat16 frozen and float32 trainable), and OpenVLA was fine-tuned with LoRA for 50,000 steps at rank 32. IsaacSim was the simulation platform, with a background wall textured using coordinate-aligned 3DGS renders. Each evaluation used 30 trials; robustness scale tests repeated each configuration 20 times.

Why This Matters

This work targets the data bottleneck that limits VLA and world-model approaches on fine-grained articulated manipulation. Its distinctive claim is that it achieves both realism and physical validity without a physics simulator, by reconstructing reality once and transferring the manipulation to new objects and poses. Prior comparable methods (DemoGen, R2RGen) were limited to simple grasping and placement, fixed object appearances, and single-view inputs; AOMGen addresses all three limitations.

Real-world applications:

• Household and service robots that need to open drawers, cabinets, and microwaves across many different homes and layouts.
• Warehouse and logistics robots handling tool boxes and other containers with latches, drawers, or doors.
• Industrial assembly and maintenance involving articulated fixtures, panels, or equipment enclosures.
• Rapid policy prototyping for robot vendors, who can generate training corpora from one demonstration instead of collecting thousands of real trials.

Industry relevance: reducing reliance on expensive real-world data collection and on hand-built simulation assets directly lowers the cost of training manipulation policies, and the ability to synthesize data for same-category objects from readily available digital assets fits naturally into asset pipelines that already produce USD models.

Future Directions

• Extending the framework beyond the categories demonstrated (microwave oven, tool box, computer, drawer, cabinet) to broader and more complex articulated objects, including multi-joint and multi-part mechanisms.
• Closing the remaining sim-to-real gap: the paper notes a visual difference between 3DGS-rendered images and the simulator environment, so testing directly on real hardware is a natural next step.
• Reducing dependence on the demonstration quality — assumptions such as constant-speed motion and no end-effector sliding had to be corrected with a second optimization stage, suggesting more robust motion models could help.
• Scaling the evaluation: the unseen-object study used 20 trials and a small number of objects, so larger-scale studies on unseen categories, scales, and pose ranges would strengthen the generalization claims.

Target Audience

Robotics researchers and engineers working on manipulation policy learning, VLA models, and sim-to-real transfer; practitioners building robot data-generation pipelines; and graphics/3D vision researchers interested in 3D Gaussian Splatting editing, part-level segmentation, and articulated object modeling. Readers without background in 3DGS, kinematics, or VLA fine-tuning will find the method sections dense, making this an advanced paper for specialists rather than a beginner-friendly introduction.

Authors’ abstract

Recent advances in Vision-Language-Action (VLA) and world-model methods have improved generalization in tasks such as robotic manipulation and object interaction. However, Successful execution of such tasks depends on large, costly collections of real demonstrations, especially for fine-grained manipulation of articulated objects. To address this, we present AOMGen, a scalable data generation framework for articulated manipulation which is instantiated from a single real scan, demonstration and a library of readily available digital assets, yielding photoreal training data with verified physical states. The framework synthesizes synchronized multi-view RGB temporally aligned with action commands and state annotations for joints and contacts, and systematically varies camera viewpoints, object styles, and object poses to expand a single execution into a diverse corpus. Experimental results demonstrate that fine-tuning VLA policies on AOMGen data increases the success rate from 0% to 88.7%, and the policies are tested on unseen objects and layouts.