Human-to-Robot Transfer via Bridging Representation and Action Space in VLAs

AI summary

Overview

Research area: Robot learning — specifically cross-embodiment transfer for pretrained Vision-Language-Action (VLA) policies, combining human video demonstrations with robot data.

Technical level: Intermediate. Readers should know roughly what a VLA policy, a flow-matching action head, and a latent action space are, but the paper's core ideas are explained through geometric and architectural intuitions rather than dense math.

Scope: The paper introduces H2RB, a co-training framework that adapts a robot-pretrained VLA to new tasks using unpaired human and robot demonstrations, and reports real-world transfer results on an OpenArm humanoid and an RB-Y1 platform.

What This Paper Is About

Pretrained VLA policies are a strong starting point for robot deployment, but adapting them to a new robot or environment normally requires collecting fresh demonstrations on that specific robot — an expensive, slow process that needs the physical platform, a teleoperation pipeline, and repeated environment resets. Human demonstrations are far cheaper to collect with only a camera, but they are hard to use directly because human hands, viewpoints, and motion dynamics differ substantially from a robot's, especially when the pretrained policy has learned mostly from robot data. H2RB's goal is to make human demonstrations a usable supervision source for adapting a robot-pretrained VLA, without any paired human-robot trajectories or target-robot demonstrations.

Key Contributions

1. A two-sided bridging framework (H2RB) that closes the human-robot embodiment gap at both the perception interface and the action interface, rather than only aligning actions as prior work does.
2. Geometry-anchored VLM conditioning: a residual adapter plus a training-only readout that grounds the VLA's vision-language features in a shared physical reference — the current wrist pose and five wrist-local fingertip positions — using only each observation's own geometry target, so no paired data or observation synthesis is required.
3. A decoupled-decoder latent tokenizer: a variational action autoencoder with a shared temporal encoder and embodiment-specific decoders, trained with reconstruction, KL, position-wise centroid matching, and variance/covariance regularization, so human and robot actions share latent structure without being forced through one action mapping.
4. Real-world evidence on two transfer axes (within-skill object transfer and skill-level transfer) plus heterogeneous multi-embodiment training and robot-only downstream adaptation on a second robot, showing gains where shared-action co-training baselines fail entirely.

Main Findings

• Object transfer jumps from 8.0 to 51.8. With target objects introduced only through human pick-and-place demonstrations, the co-training baseline scores 8.0 on average across ID and OOD rollouts, while H2RB reaches 51.8, and it keeps strong in-domain performance on the robot-trained Doll task (49.0 versus 55.0 for the in-domain-only baseline).
• Skill transfer goes from total failure to 62.7. For opening and closing interactions absent from the source robot data, vanilla co-training, HaWoR-shared action, and Ego-Pi-shared action all score 0.0 on both target skills, whereas H2RB reaches 50.7 on Open and 74.7 on Close, averaging 62.7, while also scoring 81.0 on in-domain Box PnP.
• Out-of-distribution transfer improves from 6.0 to 45.3. Under object-placement shifts, H2RB beats the Ego-Pi-shared action baseline in both ID (30.2 to 55.0) and OOD (21.3 to 45.3) settings.
• Each architectural piece contributes measurably. Adding the action tokenizer lifts object transfer from 34.7 to 43.3 and skill transfer from 0.0 to 49.7; adding cross-domain latent regularization further reduces the normalized human-robot latent gap from 3.81 to 1.53 (a 59.8% reduction) and raises object and skill transfer to 51.8 and 62.7.
• Transferred policies can be faster, not just as good. After collecting 100 robot teleoperation episodes (50 per task) for the same Open/Close tasks, the human-transferred policy executes at 7.36 cm/s versus 2.52 cm/s for the robot-only policy (×2.92), with comparable task scores (62.7 vs. 65.3) — the embodiment-specific decoders preserve the faster human motion dynamics.
• Heterogeneous human-robot training adds further gains. Large-scale training across multiple embodiments before task-specific fine-tuning raises OpenArm object transfer from 51.8 to 57.2 (+5.4) and skill transfer from 62.7 to 81.7 (+19.0).
• Benefits carry over to robot-only adaptation on a different robot. On the RB-Y1 parcel-handling task (binary success over 50 trials), success rises from 64% with robot-only training to 72% with human-robot heterogeneous training and 78% with geometry-grounded heterogeneous training.
• Stronger instruction conditioning further helps skill transfer. Concatenating the instruction representation with the state-conditioned vision-language latent and re-injecting it through the expert layers improves skill transfer from 62.7 to 83.3 (+20.6).
• Failure modes are complementary and diagnostic. Human-only policies capture the interaction pattern but fail at contact due to inaccurate wrist alignment and grasp configuration; robot-only policies produce executable motion but fail to localize or reproduce unseen behaviors; shared-action baselines produce invalid grasp poses or revert to familiar robot pick-and-place motions in human task settings.

Methodology in Plain English

The starting point is an existing robot-pretrained VLA (RLDX-1-IMG), which predicts chunks of future actions from a camera image, a language instruction, and the robot's proprioceptive state using a flow-matching action head.

Step one: put both kinds of motion in the same physical language. Human videos are processed with a HaWoR-based reconstruction pipeline to recover wrist motion and hand articulation; that motion is converted into the robot's coordinate convention and retargeted to the robot hand. Both human and robot actions are then written as a common wrist-hand representation — a 3D wrist position, a 6D rotation, and D hand joint angles (15 values per hand; 30 for bimanual). At deployment the predicted wrist pose is turned into arm motion via inverse kinematics.

Step two: share a latent action space but not the decoding. A variational autoencoder is trained with one shared temporal encoder over action chunks and two embodiment-specific decoders. The encoder maps both human and robot action chunks to latents; each decoder reconstructs actions for its own domain. Beyond reconstruction and KL terms, the two latent distributions are pulled together by matching per-position centroids and by variance/covariance regularization. The encoder and both decoders are then frozen, and their latents become the VLA's prediction targets — so the policy learns to predict a shared latent, while each domain keeps its own trajectory structure.

Step three: align perception through hand geometry. Each observation — human or robot — comes with its own measurable hand geometry: the current wrist pose plus five wrist-local fingertip positions. A small residual adapter modifies the VLM conditioning tokens, and a training-only linear readout predicts geometry from their mean-pooled representation via a masked Smooth-L1 loss. The action expert is conditioned on the adapted tokens. Because every observation supplies its own geometry target, this needs neither paired human-robot data nor temporal correspondence, and no human-to-robot image synthesis.

Step four: train and deploy. The policy is optimized with a flow-matching latent prediction loss plus the geometry loss (weight 0.2). At deployment, the latent is generated by numerical integration and decoded with the human decoder, whose output is converted into robot commands via the prescribed inverse normalization and coordinate mapping.

Evaluation uses an OpenArm humanoid with Inspire hands for object transfer (20 ID and 10 OOD rollouts per target object, 20 for the Doll in-domain task) and skill transfer, with weighted task scores from 0 to 100. Baselines are matched co-training with separate embodiment-specific interfaces, plus HaWoR-based and Ego-Pi-based shared-action variants using the same adaptation data and training budget. A separate RB-Y1 parcel-handling experiment (box flipping and polybag handling) uses robot-only fine-tuning and binary success over 50 trials. Training scales are 100k tokenizer updates, 30k OpenArm fine-tuning updates at batch size 64, and 200k heterogeneous-training updates at batch size 128.

Why This Matters

Impact on research. The paper argues that aligning only the action space is insufficient: shared action formats still leave a perceptual mismatch, causing invalid grasp poses or reversion to robot-familiar behaviors. By showing that a cheap geometric anchor (hand keypoints) plus a shared-latent/separate-decoder split fixes both problems, it reframes cross-embodiment transfer as two separate alignment problems and offers a recipe that needs no paired demonstrations, no trajectory matching, no image synthesis, and no human-robot backbone pretraining.

Real-world applications:

• Rapidly teaching household or service robots new object manipulations from a person simply demonstrating with a camera, rather than teleoperating the robot.
• Warehouse and logistics manipulation, such as the RB-Y1 parcel-handling task where a robot flips boxes and polybags to expose shipping labels before pushing them onward.
• Factory or lab settings where a new end-effector or arm is deployed and adaptation data is scarce, letting human demos fill the gap while preserving pretrained skills.
• Lowering the cost of data collection pipelines generally, since human demos avoid platform access, teleoperation hardware, and repeated environment resets.

Industry relevance. The dominant cost in deploying robot foundation models is target-embodiment data collection. A method that converts widely available human video into usable policy supervision — and additionally improves robot-only adaptation on a different robot (RB-Y1, 64% to 78%) — directly addresses the scaling bottleneck that makes robot foundation models expensive to commercialize.

Future Directions

• Better sample-level human-robot correspondence. The paper explicitly notes that its centroid matching, variance, and covariance regularizers "oppose collapse but do not guarantee correct sample-level human-robot correspondence" — a natural target for improvement.
• Extending beyond wrist-hand geometry. The geometry anchor uses wrist pose plus five fingertips; whether richer contact or object-centric anchors would close the remaining perceptual gap, and whether the approach transfers to non-anthropomorphic end-effectors, is open.
• Stronger language/skill conditioning. Instruction re-injection gave the largest single skill-transfer jump (62.7 to 83.3), suggesting that how task semantics are carried across embodiments deserves more systematic study.
• Broader and more varied heterogeneous training. Gains from heterogeneous training on OpenArm (51.8 to 57.2 and 62.7 to 81.7) and RB-Y1 (64% to 72% to 78%) hint at scaling behavior, but the paper evaluates only two robots and a small set of tasks, leaving the scaling laws and failure conditions of large-scale human-robot co-training uncharacterized.

Target Audience

Robotics and embodied-AI researchers working on VLA policies, imitation learning, and cross-embodiment transfer; engineers at robotics companies building data-efficient adaptation pipelines for new hardware; and graduate students who want a concrete, real-world case study of how perception-side and action-side alignment interact — including a candid failure-mode analysis of why shared-action co-training breaks down. Readers without background in flow matching or latent variable models will find the method sections dense, but the problem framing and experimental results are accessible.

Authors’ abstract

Pretrained Vision-Language-Action (VLA) policies provide a strong starting point for deployment, but adapting them to a target robot and environment still requires costly embodiment-specific demonstrations. Human demonstrations are far easier to collect, yet directly leveraging them is challenging due to the substantial human-robot embodiment gap, particularly when the underlying VLA has been pretrained primarily on robot data. We introduce H2RB (Human-To-Robot Bridge), a human-robot co-training framework for adapting pretrained VLA policies from unpaired human and robot demonstrations. H2RB bridges the embodiment gap at both interfaces: geometry-anchored alignment grounds human and robot observations in a compatible representation, while a decoupled-decoder tokenizer shares action latents without forcing both domains through a single action mapping. This design allows human demonstrations to provide useful behavioral supervision while retaining the rich visual, semantic, and control priors of the pretrained VLA. Across real-world tasks, H2RB substantially improves object transfer performance from 8.0 to 51.8 and enables skill transfer to score 62.7, in settings where co-train baselines totally fail to transfer human-demonstrated skills. The benefit also persists under object-placement shifts, improving out-of-distribution transfer from 6.0 to 45.3. H2RB suggests a way for human demonstrations to serve as a scalable supervision source for adapting pretrained robotic policies, reducing the dependence on costly target-robot data collection.