Dexterous Manipulation Transfer via Progressive Kinematic-Dynamic Alignment

AI summary

Overview

Research area: Robotics — dexterous manipulation, human-to-robot skill transfer, reinforcement learning, and simulation-based data generation.

Technical level: Advanced. The paper assumes familiarity with multi-fingered robotic hands, kinematic retargeting, reinforcement learning, contact dynamics, and physics simulators (MuJoCo).

Scope: The paper presents PKDA (Progressive Kinematic-Dynamic Alignment), a hand-agnostic system that converts human hand manipulation videos into dexterous robot hand trajectories by combining kinematic retargeting with residual reinforcement learning, validated in MuJoCo across three robotic hands and in real-world experiments on a UR10 arm with a Leap Hand.

What This Paper Is About

Collecting manipulation data on multi-fingered robot hand hardware is difficult and hard to scale, which leaves dexterous manipulation policy learning short of data. The authors build a system that takes only human hand manipulation RGB videos as input and automatically produces executable dexterous hand manipulation trajectories, with no massive training dataset required.

The core problem is that human hands and robot hands differ in structure, that hand-object contact dynamics are complex, and that manipulation tasks are diverse — so pure kinematic retargeting breaks physical constraints and pure reinforcement learning explores inefficiently with task-specific rewards. PKDA's goal is to balance kinematic matching and dynamic optimization across different hands, objects, and tasks.

Key Contributions

1. A new system for transferring human hand manipulation from demonstration video to multi-fingered dexterous robot hands, described as showing transfer stability across different dexterous hand configurations, manipulation tasks, and object categories by leveraging their commonalities, and offering an easily implementable, efficient and scalable technical solution.

2. The PKDA manipulation policy learning framework, characterized by synergistic optimization of kinematics mapping and contact dynamics. Kinematics mapping guides imitation and restricts RL exploration, while action space rescaling and thumb-guided pre-grasp initialization improve the efficiency of dynamic deviation correction, shortening transfer time while ensuring successful transfer.

3. A four-module pipeline — Interaction Perceptor, Trajectory Proposer, ContactAdapt Optimizer, and Wrist Trajectory Planner — that models transfer as four stages: extracting hand-object priors, approaching the object, anthropomorphically grasping the object, and manipulating, without requiring task-specific parameter tuning.

4. Demonstration that the framework is hand-agnostic: mapping finger correspondences alone, without adjusting overall parameters, adapts the transfer to different dexterous hands.

Main Findings

• Average transfer success rate: The abstract reports an average transfer success rate of 73%, with the system automatically generating smooth and semantically correct dexterous hand manipulation that reproduces human intentions.

• Comparison against baselines (full-information scenario, Adroit Hand): On the 40-sequence TCDM task subset, PKDA-P achieved 80% SR Grasp, 80% SR Follow, Ep 0.058, Er 31.5, and 77.5% TSR. PGDM reached 72.5% SR Grasp, 72.5% SR Follow, Ep 0.005, Er 18.2, and 72.5% TSR. D-Grasp reached 62.5% SR Grasp, 60% SR Follow, Ep 0.067, Er 35.5, and 57.5% TSR. Anyteleop reached 12.5% SR Grasp, 7.5% SR Follow, and 7.5% TSR, with Ep and Er reported as N/A.

• Larger-scale evaluation on GRAB: PKDA-F, tested on 600 sequences from GRAB, reached 84.2% SR Grasp, 77.6% SR Follow, Ep 0.060, Er 34.8, and 73.3% TSR.

• Tracking accuracy trade-off: Tracking accuracy is better than D-Grasp but lower than PGDM. The authors state PGDM treats the object trajectory as a strong constraint, sacrificing transfer efficiency to exactly reproduce trajectories, whereas PKDA optimizes only the grasp phase with RL and uses a PD controller for the remaining motion, yielding markedly higher transfer efficiency.

• Robustness under perception errors (Adroit Hand): In the Model-Known scenario PKDA reached 75% SR Grasp, 70% SR Follow, Ep 0.034, Er 22.4, 70% TSR. In the Model-Unknown scenario it reached 80% SR Grasp, 80% SR Follow, Ep 0.033, Er 34.8, 80% TSR. Success rates were no less than 70% in both scenarios.

• Cross-hand performance (PKDA-P): Adroit 80% SR Grasp, 80% SR Follow, Ep 0.0584, Er 31.5, 77.5% TSR; Allegro 77.5% SR Grasp, 72.5% SR Follow, Ep 0.0569, Er 32.8, 72.5% TSR; Leap 70% SR Grasp, 67.5% SR Follow, Ep 0.0544, Er 31.7, 67.5% TSR. Larger hands such as Allegro and Leap face challenges on small or slender objects (e.g., hammers). Position errors (0.054–0.058) and rotation errors (31°–33°) remained consistent across hands despite differences in joint freedom, finger length, and kinematics.

• Retargeting ablation: Using the finger-wrist vector retargeting method reduced TSR by 7.5% compared with the authors' fingertip-position approach (70% vs 77.5%).

• Pre-grasp guidance ablation: Index-finger and middle-finger guidance reduced TSR by 7.5% and 10% respectively versus thumb guidance under Nearest triggering conditions. With thumb guidance, increasing preset distance thresholds lowered TSR (dropping to 67.5% at 0.1 m versus Nearest).

• Action space rescaling ablation: This had the most significant influence. Taking thumb-guided, Nearest triggering as an example, removing the mechanism dropped TSR to 37.5%. Without rescaling, the dexterous hand exhibited unintended overshooting behavior during grasping.

• Real-world validation: On a UR10 robot arm equipped with a Leap Hand, the authors performed Shake, Pour, and Stamp tasks with daily objects, executing simulation-generated trajectories directly in the real world via open-loop control and successfully reproducing the intended manipulations.

• Failure modes of baselines: The Retarget-only method Anyteleop struggles to resist inertial disturbances, often resulting in "lifting and slipping down"; it occasionally succeeds with objects like goblets due to favorable lifting postures. D-Grasp, guided by static poses, excels at relocation but falters in repetitive tasks like knocking a nail.

Methodology in Plain English

The system begins with human demonstration videos. The Interaction Perceptor extracts hand trajectories, object trajectories, and contact points. For datasets with known object models (DexYCB, TACO), hand and object poses are estimated with HFL-Net, and contact points are identified by computing minimum fingertip-to-object distances and keeping those below a 5 cm threshold. For raw videos without ground-truth object models, Hold is used to reconstruct 3D hand-object geometry, with convex decomposition optimization for collision detection and a lowered object center of mass for physical plausibility.

The Trajectory Proposer maps human hand motion onto dexterous hand joint angles. Rather than aligning fingertip-to-wrist vectors as prior work does — which introduces fingertip errors given hand size differences — the authors use fingertip positions in the world coordinate system as the main objective and palm orientation as an auxiliary constraint. This is formulated as a nonlinear optimization minimizing fingertip position error, palm orientation difference (minimum geodesic distance), and a temporal smoothing term. The joint angle sequence is then converted into a primary control sequence via an inverse-dynamics-based joint angle-to-control-signal conversion, with gravity correction for self-weight offset.

The ContactAdapt Optimizer applies reinforcement learning to fix grasp dynamics. An RL-Configurator extracts common features from object poses, primary trajectories, and fingertip contact points to build a unified RL setup across tasks. It defines a pre-grasp state as the state where hand and object are not in contact and the thumb tip is closest to its corresponding grasp point, and sets the goal to bring the object to a target pose (the pose where the object first deviates from its initial position by 0.1 m). Action space rescaling compresses wrist joint movement from the global workspace into a local neighborhood around the pre-grasp wrist angle while keeping full finger joint range. The reward is hierarchical and unified: an approach reward guiding fingertips toward contact points, a grasp reward activated when all fingertips enter the 0.06 m contact tolerance (combining a contact reward based on distal phalanx collisions below a 0.002 m threshold with an imitation reward based on cosine similarity to retargeted joint angles), and a lifting reward activated when the thumb and one other finger make contact, guiding the hand to bring the object to the target pose.

Finally, the Wrist Trajectory Planner models hand-object interaction as low-dynamic manipulation with no relative sliding after pickup. It extracts the object trajectory during manipulation and the wrist pose at stable grasp, computes a wrist trajectory from the relative hand-object relation at each time step, and drives the wrist with a PD controller. This preserves the operational semantics of the task — for example, holding a cup through "rise, tilt to drink, put down."

Why This Matters

Impact on research: The paper targets the severe data scarcity that impedes data-driven dexterous manipulation policy learning, caused by the difficulty and limited scalability of collecting data with multi-fingered robot hand hardware. It offers an easily implementable and scalable method for collecting robot dexterous manipulation data, and argues that prioritizing transfer of manipulation action intentions is more reasonable in practical terms than exact trajectory reproduction.

Real-world applications (drawn from the paper's examples and experiments):

• Robots using everyday tools to assist human life, which the authors cite as a motivation for transferring human skills into robotic productivity.
• Cup-handling tasks such as drinking, where the transfer must preserve the "rise, tilt, put down" semantics.
• Repetitive tool tasks such as knocking a nail, where D-Grasp was shown to falter but PKDA was evaluated.
• Household daily-object manipulation, demonstrated in real-world Shake, Pour, and Stamp tasks on a UR10 with a Leap Hand.

Industry relevance: Generating manipulation data facilitates the development of data-driven robotic systems. The system requires only human manipulation RGB videos, needs no online human involvement during data generation, and needs no task-specific parameter adjustment — properties relevant to scaling up robotic manipulation datasets. The authors also note that simulation-based transfer offers higher data efficiency, broader task scalability, and greater tolerance to failure compared with real-world transfer, and that models trained in simulation have shown strong potential for real-world deployment.

Future Directions

• Dynamic multi-contact manipulation. The stated limitation is that PKDA primarily handles stable contact patterns; future work will explore dexterous manipulation with dynamic multi-contact changes.
• Closing the loop on real-world deployment. The real-world experiments execute simulation-generated trajectories via open-loop control, leaving open how the system performs under closed-loop feedback and real-world disturbances.
• Broadening task and object coverage. Larger hands such as Allegro and Leap faced challenges on small or slender objects such as hammers, raising the question of how far the approach generalizes to difficult object geometries.
• Reducing dependence on perception quality. The Model-Known and Model-Unknown scenarios showed robustness to pose estimation errors and object reconstruction defects, but the paper notes that pose estimation error yields unsatisfactory pre-grasp poses and reconstructed objects retain only a crude shape with large detail error — leaving room to improve the perception front end.

Target Audience

This paper benefits robotics researchers and engineers working on dexterous manipulation, human-to-robot skill transfer, imitation learning, and reinforcement learning for contact-rich tasks. It is also relevant to practitioners interested in generating manipulation data at scale for data-driven robot systems, and to teams working on simulation-based training with real-world deployment, particularly those using multi-fingered hands such as Adroit, Allegro, or Leap.

Authors’ abstract

The inherent difficulty and limited scalability of collecting manipulation data using multi-fingered robot hand hardware platforms have resulted in severe data scarcity, impeding research on data-driven dexterous manipulation policy learning. To address this challenge, we present a hand-agnostic manipulation transfer system. It efficiently converts human hand manipulation sequences from demonstration videos into high-quality dexterous manipulation trajectories without requirements of massive training data. To tackle the multi-dimensional disparities between human hands and dexterous hands, as well as the challenges posed by high-degree-of-freedom coordinated control of dexterous hands, we design a progressive transfer framework: first, we establish primary control signals for dexterous hands based on kinematic matching; subsequently, we train residual policies with action space rescaling and thumb-guided initialization to dynamically optimize contact interactions under unified rewards; finally, we compute wrist control trajectories with the objective of preserving operational semantics. Using only human hand manipulation videos, our system automatically configures system parameters for different tasks, balancing kinematic matching and dynamic optimization across dexterous hands, object categories, and tasks. Extensive experimental results demonstrate that our framework can automatically generate smooth and semantically correct dexterous hand manipulation that faithfully reproduces human intentions, achieving high efficiency and strong generalizability with an average transfer success rate of 73%, providing an easily implementable and scalable method for collecting robot dexterous manipulation data.