Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos

AI summary

Overview

Research area: Robotics — cross-embodiment visual imitation learning, task-oriented grasping, and manipulation policy learning from human hand-object interaction video.

Technical level: Intermediate. The paper assumes familiarity with imitation learning, 6-DoF object pose estimation, grasp generation, and robot simulation, though the core idea (use a simulator to check which grasps can actually complete a task) is explained concretely.

One-sentence scope: The paper proposes Perceive-Simulate-Imitate (PSI), a three-step framework that learns modular prehensile manipulation policies from RGB-D human videos alone — using a simulator to filter grasp-trajectory pairs and to generate supervision for task-compatible grasping — and validates it on four real-world manipulation tasks with a UFACTORY xArm7.

What This Paper Is About

Robots with non-anthropomorphic grippers (like parallel-jaw grippers) can learn the post-grasp part of a task by watching human videos, but they cannot easily learn how to grasp, because human hands and robot grippers are too different. A common workaround is a modular policy: let an off-the-shelf grasp generator pick a stable grasp, and learn only the post-grasp motion from video. The problem is that a grasp can be stable yet not task-compatible — the paper's example is gripping a door handle firmly with an underhand grip, which makes turning it clockwise very hard. PSI's goal is to learn, from human videos and simulation, which stable grasps also allow the downstream motion to be executed.

Key Contributions

1. A simulation-based grasp-trajectory filtering scheme. Each 6-DoF object trajectory extracted from a human video is paired with multiple candidate grasps and executed in simulation, producing binary success labels. Trajectories where all grasps fail are discarded entirely, removing pose-tracking failures and robot-infeasible motions from the training data.

2. A learned grasp-scoring model for task-oriented grasping. The simulation labels are distilled into a model that predicts success probabilities for K = 8 predefined anchor grasps, allowing task-compatibility to be evaluated at inference time without running simulation.

3. A modular execution scheme that combines stability and task-compatibility. At test time, any existing grasp generator proposes stable 6D grasps; each candidate is assigned to its nearest anchor grasp by rotation difference, and the highest-scoring candidate is executed.

4. Evidence that direct 6-DoF object pose prediction outperforms flow-based motion representation, and that PSI pretraining on HOI4D improves sample efficiency. Real-world evaluation covers four tasks with no robot data used for training.

Main Findings

• Simulation-based trajectory filtering substantially improves success rates. On the xArm7 with FoundationPose-based data, PSI scores 16/20 on pick-and-place (P&P), 13/20 pour, 20/20 stir, 12/20 draw, versus 6/20, 12/20, 16/20, 12/20 without trajectory filtering. With ICP-based data, PSI scores 15/20, 13/20, 18/20, 0/20 versus 10/20, 8/20, 8/20, 0/20 unfiltered.

• Filtering closes most of the gap between model-based and model-free pose tracking. Without filtering, the ICP pipeline produces notably worse policies than FoundationPose; with filtering, the gap largely closes — except on Draw, where the marker is too small to give ICP enough 3D points, leaving PSI (ICP) at 0/20.

• Task-oriented grasp selection matters greatly. Replacing policy-selected grasps with random candidate grasps drops FoundationPose results to 5/20 (P&P), 8/20 (Pour), 10/20 (Stir), 1/20 (Draw), and ICP results to 4/20, 7/20, 11/20, 0/20.

• Direct 6-DoF pose prediction beats flow-based prediction. Against General-Flow (which predicts 3D flow and solves for SE(3) via singular-value decomposition, using the official HOI4D-pretrained ScaleFlow-B model): General-Flow with FoundationPose data scores 7/20 P&P, 4/20 Pour, 1/20 Stir, 0/20 Draw, versus PSI's 16/20, 13/20, 20/20, 12/20. With ICP data, General-Flow scores 5/20, 0/20, 0/20, 0/20 versus 15/20, 13/20, 18/20, 0/20.

• PSI pretraining on HOI4D outperforms ImageNet and R3M initialization. Using 1580 HOI4D pick-and-place clips, PSI reaches 16/20, 13/20, 20/20, 12/20; ImageNet-1K ResNet18 weights give 10/20, 11/20, 6/20, 0/20; R3M gives 5/20, 13/20, 10/20, 0/20. The paper attributes the smaller benefit on Pour to that task being heavily rotation-focused, which HOI4D pick-and-place data does not teach.

• The framework transfers across robot embodiments. In simulation-based evaluation with real inputs: xArm7 scores 11/13, 10/14, 13/13, 13/14; Franka Panda 11/13, 6/14, 13/13, 12/14; Kinova Gen3 10/13, 7/14, 13/13, 13/14; UR5e 11/13, 11/14, 13/13, 10/14. Pour shows the greatest variance because its large rotations challenge kinematic limits differently across robots.

• Performance scales with demonstration count. With 15 training demos, simulation-based proxy results are 7/13 P&P, 5/14 Pour, 13/13 Stir, 10/14 Draw; with 25 demos, 9/13, 9/14, 13/13, 12/14; with 35 demos, 11/13, 10/14, 13/13, 12/14. Stir reaches full success with only 15 demonstrations, which the authors partly attribute to a narrower initial state distribution and possibly more lenient simulation success criteria (pot collisions are not modeled).

• Trajectory discard counts are reported per task. Out of 50 trajectories: FoundationPose discards 6 (P&P), 5 (Pour), 3 (Stir), 3 (Draw); ICP discards 4 (P&P), 2 (Pour), 3 (Stir), 3 (Draw). The primary causes are motion blur and object occlusion.

• Pose tracking quality limits the pipeline. FoundationPose is slightly more accurate overall; ICP struggles with significant occlusion (e.g., the hand occluding the ladle handle), and both methods drift around the axis of symmetry for near-symmetric objects like the bottle and can — motivating the filtering step.

Methodology in Plain English

The framework has three stages, hence the name PSI.

Perceive. From each RGB-D human video, the system segments the active object with Grounding SAM and tracks its motion as a sequence of 6-DoF rigid poses. It supports two tracking paths. The model-based path uses FoundationPose when a 3D model of the object is available. The model-free path propagates the object mask with Cutie, applies iterative closest point (ICP) to the object point clouds, and then runs pose-graph optimization to keep the whole trajectory consistent. The authors chose 6-DoF object pose rather than optical/scene flow because a pose can be converted to an end-effector action by a single rigid transformation, avoiding an error-prone depth-based conversion step.

Simulate. For each trajectory, the system samples K = 8 anchor grasps (four cardinal directions relative to the robot base, at elevation angles of 10 and 50 degrees) around the object's initial center. Each grasp-trajectory pair is executed in robosuite with a waypoint controller and the operational space controller, with the object assumed to become rigidly attached to the end-effector once the grasp pose is reached. This isolates task-compatibility from grasp stability, which the authors argue cannot be simulated accurately for in-the-wild videos. Each pair receives a binary success label; trajectories that fail for all K grasps are discarded.

Imitate. A simple open-loop policy takes an RGB image from the start of the video, the object's binary mask, and a 2D goal point in pixel coordinates (e.g., a pick-and-place destination). ResNet18 encodes image and mask; features are concatenated and fused with the goal point through an MLP; separate MLP heads predict the 6-DoF trajectory (MSE loss, poses parameterized as rotation and translation vectors relative to an object frame) and grasp success probabilities for the K anchors (BCE loss). Training is two-stage: first trajectory loss only, then — for HOI4D (1580 episodes) — joint training on both losses, while for small task-specific datasets (under 50 episodes) all layers are frozen except the grasp head, which is trained alone because the grasp labels are somewhat noisy and prone to overfitting.

Execution. At test time an external grasp generator proposes stable candidate 6D grasps. Each candidate is mapped to its nearest anchor grasp by rotation difference, the learned score is inherited, and the top-scoring grasp is executed. This avoids running simulation at inference time.

Experimental setup. Four real-world tasks: pick-and-place, pouring from a can to a bowl, stirring a pot with a ladle, and drawing on a whiteboard. Fifty human demonstrations per task were captured with a fixed Intel RealSense D455 RGB-D camera; 35 were used for training and 15 for validation. Collecting the 35 demonstrations takes less than one hour. The robot is a UFACTORY xArm7 with an xArm gripper, with camera-to-base calibrated by hand-eye calibration. Success rates in Tables 1–3 are over 20 episodes with novel starting states; Tables 4–5 use simulation-based proxy evaluation on held-out real observations.

Why This Matters

Impact on research. The paper targets a specific, well-known weakness of modular video-imitation policies: they abstract grasping away and therefore cannot reason about the dependency between the grasp and the motion that follows. PSI shows that this dependency can be learned without any robot demonstration data, by using simulation purely as a label generator. It also provides a direct comparison showing that 6-DoF object pose prediction outperforms flow prediction with least-squares alignment, and it introduces a reusable pretraining signal derived from HOI4D rather than generic visual features.

Real-world applications:

• Warehouse and logistics pick-and-place, where the correct grasp depends on where the object must be placed.
• Kitchen and food preparation robots, where pouring, stirring, and reorienting containers require grasps that leave the wrist free to rotate.
• Tool-use and whiteboard/writing tasks, where a stable but awkward grasp makes the required motion kinematically impossible.
• Multi-robot deployments, since PSI trains embodiment-specific policies for xArm7, Franka Panda, Kinova Gen3, and UR5e from the same underlying human video data.

Industry relevance. The method requires no teleoperation rigs, no kinesthetic teaching, and no robot data — only an RGB-D camera and roughly an hour of human demonstrations per task (35 episodes), which lowers the cost of commissioning a new manipulation skill. The grasp-scoring module is designed as a plug-in on top of existing off-the-shelf grasp generators, so it can be added to an existing grasping stack rather than replacing it.

Future Directions

• Scaling PSI into generalist foundation models. The conclusion frames the current work as task-specific and identifies large-scale training on PSI-produced trajectory and grasp labels as an important next step.
• Improving Real2Sim fidelity. The authors anticipate extending PSI with more advanced Real2Sim techniques to raise the quality of the extracted training signals.
• Closed-loop policies and the visual domain gap. The current model only sees the first, unobstructed frame; intermediate frames of human videos are heavily occluded by the body, and inpainting or insertion-rendering methods are suggested as remedies.
• Beyond rigid objects. The 6-DoF pose representation restricts the framework to rigid or approximately rigid grasped objects, leaving articulated and deformable object manipulation unaddressed.
• Better data collection practices. The authors explicitly recommend that the community attend to motion blur and object occlusion during capture, since these are the main causes of discarded trajectories.

Target Audience

Robotics researchers working on imitation learning from human video, cross-embodiment transfer, and task-oriented grasping will find the core contribution most useful. Engineers building manipulation systems with parallel-jaw grippers gain a practical recipe for combining an off-the-shelf grasp generator with a learned grasp-scoring model. Students at an intermediate level in robot learning, simulation, or 3D vision will benefit from the clear decomposition of prehensile manipulation into grasp selection and post-grasp motion, and from the paper's side-by-side comparison of 6-DoF pose versus flow representations and of model-based versus model-free pose tracking.

Authors’ abstract

The ability to learn manipulation skills by watching videos of humans has the potential to unlock a new source of highly scalable data for robot learning. Here, we tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions. Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors, especially for robots without human-like hands. A promising way forward is to use a modular policy design, leveraging a dedicated grasp generator to produce stable grasps. However, arbitrary stable grasps are often not task-compatible, hindering the robot's ability to perform the desired downstream motion. To address this challenge, we present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data processed by paired grasp-trajectory filtering in simulation. This simulation step extends the trajectory data with grasp suitability labels, which allows for supervised learning of task-oriented grasping capabilities. We show through real-world experiments that our framework can be used to learn precise manipulation skills efficiently without any robot data, resulting in significantly more robust performance than using a grasp generator naively.