RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

AI summary

Overview

Research area: Robot learning / imitation learning — specifically, generating robot training data from egocentric human videos (human-to-robot video translation, dexterous manipulation, vision-language-action policies).

Technical level: Advanced. The paper combines video diffusion/flow-matching generative models, inverse dynamics models, hand reconstruction and kinematic retargeting, and fine-tuning of large vision-language-action models on real bimanual hardware.

Scope (one sentence): The paper proposes RoboTryOn, a three-stage pipeline that converts an egocentric human manipulation video into a paired dexterous-robot demonstration with pseudo action labels, and shows on two real bimanual robots that policies trained on this synthetic data can learn tasks with no teleoperated demonstrations.

What This Paper Is About

Adapting a robot foundation model to a new task still depends on teleoperated demonstrations, which require robot hardware, a teleoperation rig, and heavy operator effort — a burden that is especially severe for high-DoF dexterous hands. Human egocentric videos are plentiful and capture how people grasp and manipulate objects, but they are separated from robot demonstrations by a visual gap (the policy sees a robot, not a human arm) and an action gap (no exact retargeting map from a reconstructed human hand to robot joint commands exists, and even a perfect map would not convey grip force). RoboTryOn's goal is to convert a human video into a robot video plus action labels in the same scene, so a policy can learn new tasks from human video alone.

Key Contributions

1. A virtual try-on framing of human-to-robot transfer. Just as try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot, using generative models trained on robot data alone (no paired human–robot demonstrations required).

2. A three-stage synthesis pipeline. An image model autoregressively translates selected human keyframes into robot keyframes using a reference robot image; a video model synthesizes motion between keyframes conditioned on the human hand-skeleton flow; an inverse dynamics model (IDM) with depth features labels the resulting video with joint-space actions.

3. Demonstration that synthetic data closes both gaps. Synthesized videos overlap with real robot videos in the policy's feature space (Fréchet SigLIP distance reduced from 125.5 to 32.7), and IDM labels replay successfully far more often than retargeted labels (69.7% vs 34.8% over 66 episodes), without requiring any test-time visual editing of the robot's camera stream.

4. Real-robot evidence across two embodiments and eleven tasks. A single image model, video model, and IDM serve both an OpenArm (two 7-DoF arms, two 6-DoF Inspire RH56F1 hands) and an RB-Y1 (two 7-DoF arms, two 20-DoF WujiHand2 hands), enabling learning of novel objects, goals, skills, and bimanual coordination with few or no teleoperated demonstrations.

Main Findings

• Large gains on tasks with no robot demonstrations. Averaged over the two embodiments, success on out-of-domain tasks rises from 2.1% to 60.0%. Per robot: OpenArm 3.1% → 73.0%, RB-Y1 1.0% → 46.9%. Both baselines (EgoMimic and Masquerade) largely fail on these tasks, with out-of-domain averages below 10% (5.1% and 8.2% on OpenArm; 5.1% and 7.7% on RB-Y1).

• Bimanual tasks learned from single-arm teleoperation. The policy reaches 45.9% average success on the three bimanual tasks (Box Convey, Air Fryer, PnP Ball in Cup), even though every teleoperated demonstration contains only single-arm manipulation.

• In-domain improvement too. Adding synthesized data raises success on the four pick-and-place tasks with teleoperated data from 47.3% to 75.9% on OpenArm and from 69.6% to 79.5% on RB-Y1.

• Works without any teleoperated data for those tasks. Using synthesized episodes alone, success is 48.5% (OpenArm) and 40.3% (RB-Y1) averaged over out-of-domain tasks; overall table values for this setting are 57.1% in-domain and 48.5% out-of-domain average on OpenArm, and 60.7% / 40.3% on RB-Y1.

• Cost efficiency. Recording a human video takes about a fifth of the time of a teleoperated episode, and training on translated human videos alone matches the success rate of 50 teleoperated demonstrations per task at approximately one third of the collection cost. Augmenting the teleoperated dataset with an equal number of human videos raises success from 47.3% to 71.4% for less than one additional hour of recording.

• Better action labels than retargeting. Open-loop replay in a simulated replica succeeds in 69.7% of episodes with IDM labels versus 34.8% with retargeted labels; the gap arises mainly at the grasp moment, where imprecise hand reconstruction and differing hand geometry prevent a secure grasp.

• Gradients align with teleoperated data. The cosine similarity between policy gradients from teleoperated in-domain data and from out-of-domain task data is 0.53 for RoboTryOn versus 0.17 and 0.14 for the baselines; within the action expert, retargeted data yields slightly negative similarity (−0.03 and −0.05).

• Execution speed is inherited. Human demonstrators move at roughly 1.25× the teleoperated speed. Retiming the skeleton flow to 1× raises in-domain success from 75.9% to 85.7% while keeping out-of-domain success similar (70.4%); training at 2× yields the fastest policy (7.4 s mean time to placement) but lowers success to 57.1% in-domain and 42.9% out-of-domain.

• Every component matters (ablations on OpenArm). Removing the autoregressive reference gives 68.8% / 66.3% (in-domain / out-of-domain); removing subsequent keyframes gives 42.9% / 39.3%; removing the hand skeleton flow gives 42.0% / 21.4% (largest drop); removing the IDM wrist correction gives 65.2% / 48.0%.

• Not tied to one policy architecture. Fine-tuning RLDX-1-Img on the same data improves it from 53.6% to 60.7% in-domain and from 0.0% to 40.3% out-of-domain.

Methodology in Plain English

The pipeline has three learned stages, all trained as conditional flow-matching models and all trained on robot data only.

1. Pick keyframes and swap in the robot. The system reconstructs the human hands and selects frames where both wrists slow to a local minimum and a wrist has moved far enough from the previous keyframe, always including the first and last frames. Each selected frame is masked over the arm and hand (enlarged and smoothed so the outline does not reveal the human arm) and annotated with a 2D hand skeleton. An image model then inpaints the masked region with the target robot in the pose indicated by the skeleton, using a reference image to set appearance, objects, and background. Keyframes are translated in temporal order, each conditioned on the previously translated robot keyframe, which keeps appearance and scene consistent.

2. Fill in the motion. A video model interpolates between the translated keyframes, guided by the human hand-skeleton flow for the whole clip. To keep the model robust to imperfect hand retargeting, keypoints are perturbed and randomly dropped during training, so the skeleton gives only coarse guidance and fine hand configuration comes from the learned prior over robot hands. Long videos are synthesized as consecutive clips split at keyframes.

3. Label the actions. An inverse dynamics model takes two synthesized frames K steps apart and predicts the chunk of joint-space actions between them, trained with teleoperated actions as targets. States are derived from the labels using the measured lag between commanded action and recorded state. Because the IDM estimates wrist positions less accurately than hand reconstruction, inverse kinematics is used to correct the wrist while keeping the IDM's hand joints.

The resulting demonstrations have exactly the same format as teleoperated ones, so the policy (π0.5, or RLDX-1-Img in the ablation) is fine-tuned on the union of real and synthetic data, or on synthetic data alone. Experiments use four in-domain pick-and-place tasks with 50 teleoperated demonstrations per robot and 50 human videos each, plus seven out-of-domain tasks with 50 human videos each and no robot demonstrations; policies are evaluated over 28 trials per task with randomized object positions.

Why This Matters

Impact on research. The paper reframes human-to-robot transfer as an appearance-swap problem rather than a kinematics problem, and shows that the resulting data can teach manipulation skills — including bimanual coordination — that were never teleoperated. It also provides a diagnostic account of why prior retargeting/overlay approaches fail: their gradient updates conflict with teleoperated data, and their visual gap is only closed by editing the deployment camera stream.

Real-world applications (bullets):

• Rapidly teaching dexterous robots new household or warehouse tasks from ordinary egocentric footage, without a teleoperation rig or a skilled operator per task.
• Lowering the cost of building robot foundation-model training sets where teleoperation is expensive, particularly for high-DoF hands.
• Cross-embodiment data sharing: one shared set of generative models plus a few hours of video per robot, amortized across tasks.
• Bootstrapping policies for tasks that are awkward to teleoperate but easy for people to demonstrate.

Industry relevance. The cost result — matching teleoperated performance at roughly a third of collection cost, with human video taking about a fifth of the recording time — speaks directly to the economics of robot data collection. The single shared model stack serving two different robots (different kinematics, hands, and camera placements) is relevant to companies maintaining heterogeneous fleets.

Future Directions

• Scale to large egocentric corpora. The authors propose translating datasets such as EgoDex into every embodiment they own, so a robot's training data grows with the supply of human video rather than operators.
• Improve the inverse dynamics model. Policy rollouts on the robot are recorded with executed actions, so they naturally become IDM training data; an IDM that also takes the reconstructed wrist position as input could fuse the two estimates instead of replacing one with the other.
• Handle moving cameras and in-the-wild video. The current pipeline assumes human videos recorded from the robot head-camera position, and the IDM always predicts fixed neck and base actions because it never saw them move.
• Extend beyond force-free, in-reach tasks. Pseudo labels cannot capture forces and are bounded by stereo video and hand reconstruction resolution, so force-dependent or fine-precision tasks such as tight insertion remain out of reach; the pipeline also cannot fix motions beyond arm reach or grasps the hand cannot form. The authors additionally translated videos into a GR-1 but could not evaluate policies without the physical robot.

Target Audience

Researchers and engineers working on robot learning, imitation learning, and vision-language-action models, particularly those concerned with data scalability and dexterous manipulation. It is also relevant to practitioners building human-video pipelines for embodied AI, and to readers interested in video generation, virtual try-on techniques, and inverse dynamics models, since the paper connects those areas to real-robot policy training. Readers need familiarity with diffusion/flow-matching generative models and policy learning to follow the method, but the experimental narrative and cost analysis are accessible to a broader robotics audience.

Authors’ abstract

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but overlaying rendered robots on human videos or retargeting human poses into robot actions leaves significant visual or action gaps for high-DoF robot embodiments. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn converts a human video into a video of the target robot specified by a reference image, while preserving the original task, objects, and background. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis with motion conditioned on the human hand skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthetic data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.