CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

AI summary

Overview

Research area: Robot manipulation / cross-embodiment learning from human demonstrations (hand-to-gripper transfer, imitation learning, dataset construction).

Technical level: Advanced — the paper builds on diffusion transformers, point-cloud encoders, and differentiable trajectory optimization, though the high-level framing is accessible.

Scope: This paper introduces a large-scale paired human-hand / robot-gripper demonstration dataset focused on complex spatial movements, together with a two-stage data-driven baseline that transfers hand motions from monocular RGB-D video to 6-DoF gripper action sequences.

What This Paper Is About

Most methods that turn human hand demonstrations into robot actions work only for simple planar tasks such as pick-and-place or pushing, because the mismatch between a human hand and a two-finger gripper becomes severe when the motion involves rotation or flipping. This paper collects a paired hand-gripper dataset that deliberately emphasizes such complex spatial movements, and then proposes a learning method that predicts gripper poses from hand and object point clouds rather than from hand-crafted kinematic rules or tracked object trajectories alone.

Key Contributions

1. A hand-gripper paired dataset described by the authors as the first of its kind focusing on complex spatial movements, containing 6,189 manipulation episodes across 1,254 distinct object instances, with a reported orientation variation substantially exceeding prior datasets.
2. A scalable paired data acquisition pipeline built around a rigorous UMI-based (handheld gripper) collection protocol that prioritizes motion complexity, plus an annotation scheme that extracts 3D motion data from monocular RGB-D video rather than requiring repeated recalibration.
3. A two-stage data-driven transfer framework in which Stage I predicts sparse starting and terminal gripper poses and Stage II generates the full continuous orientation sequence conditioned on those keyframes, with the gripper translation instead derived from hand motion and post-optimized.
4. Comprehensive evaluation in both simulation and on a real robot against optimization-based and learning-based baselines, plus ablations on stage design, model conditioning, action representation, and the Stage-II generation strategy.

Main Findings

• The dataset is more spatially complex than prior benchmarks. Using rotation angle during manipulation as a metric, the authors report that orientation variation in their dataset substantially exceeds that of prior datasets such as Robosuite, MimicPlay, and RT-1, which they characterize as dominated by translational actions.
• Paired trajectories align closely. Pairs are retained only if a 3D trajectory similarity score exceeds 0.9, yielding a mean similarity of 0.957 and median of 0.958.
• The two-stage framework outperforms all baselines. In simulation, CosmoH2G achieves GOA 7.53°, SR 83.87%, TS 0.9672, and TOPA 10.27°, versus the strongest baseline Track2Act at 10.67°, 77.42%, 0.9015, and 12.48°.
• Real-robot performance is lower than simulation but still leads. On the real robot, CosmoH2G reaches GOA 7.53°, SR 70.43%, TS 0.9413, and TOPA 19.34°, versus Track2Act at 10.67°, 60.22%, 0.8732, and 20.58°. The authors attribute the gap to object slippage during large rotations and objects being moved before being fully lifted.
• Rule-based retargeting fails outright on this task. The authors report that hand-pose retargeting methods are unable to grasp objects under their tasks, which is why the comparison set consists only of trajectory-based methods.
• Single-stage learning is insufficient. Three single-stage variants were tested: a denoising Transformer (GOA 15.28°, SR 62.90%, TS 0.8542, TOPA 47.83°), Diffusion Policy (12.36°, 58.60%, 0.8157, 52.84°), and ACT (14.58°, 64.52%, 0.8732, 49.27°), all well below the two-stage result.
• Three-keypoint action representation matters. Representing the gripper by its center and two fingertip keypoints (7.53°, 83.87%, 0.9672, 10.27°) beats a homogeneous transformation matrix (13.17°, 69.89%, 0.8868, 28.47°) and a quaternion-translation pair (11.34°, 72.58%, 0.8984, 24.63°).
• The Stage-II split of learned orientation plus optimized position is important. Generating orientation while optimizing position (7.53°, 83.87%, 0.9672, 10.27°) beats directly generating both (8.50°, 81.72%, 0.9176, 13.23°) and computing both from geometric heuristics (16.83°, 67.20%, 0.8742, 26.78°).
• Conditions contribute incrementally. Removing the contact map slightly degrades results (9.58°, 80.11%, 0.9117, 11.96°), while removing the ending action hurts placement orientation notably (9.48°, 76.34%, 0.8932, 21.79°).
• Gains are not attributable to the contact map alone. The authors state that even without the contact map, CosmoH2G still outperforms all baselines, and note that no baseline takes a contact map as input.
• Failure modes of baselines are tracking-related. MimicFunc relies on CoTracker, whose tracking degrades under complex spatial movements; 3DFlowAction and Im2Flow2Act rely on 2D tracking; Track2Act's calculation-and-refinement framework produces suboptimal predictions when the calculation deviates significantly from desired actions.

Methodology in Plain English

Data collection. Collectors manipulate objects by hand while recording RGB-D video from a fixed viewpoint, deliberately varying hand gestures and contact areas. Each task includes complex orientation changes such as in-plane rotation and vertical flipping. A second pass uses a handheld gripper (UMI) to perform the identical task, imitating the contact area, grasping orientation, and motion. Pairs are kept only when the object poses at the starting and terminal frames match between hand and UMI demonstrations.

Turning video into 3D data. Object meshes are reconstructed from multi-view images with ReconViaGen. Objects, hands, and the UMI gripper are registered and tracked in a unified 3D frame using FoundationPose++ with RGB-D frames. Object masks come from SAM2; hand meshes come from Wilor, reconstructed per sampled frame rather than tracked.

The transfer model. Stage I is a denoising Transformer that generates a single 6-DoF gripper pose for the starting frame and one for the terminal frame, conditioned on hand points, object points, and a coarse contact map. The gripper is represented by three keypoints (center, left fingertip, right fingertip), so conditioning and generation live in the same 3D-point space. Stage II generates the orientation sequence over all frames, conditioned on object points, the hand point sequence, and the Stage-I starting and terminal poses.

Translation handling. The gripper position sequence is initialized from the hand trajectory under the assumption that hand and gripper contact regions are roughly aligned, then refined in three steps: optimizing the starting and terminal positions for stable grasping (contact loss plus penetration penalty), smoothing intermediate frames toward a reference curve, and enforcing kinematic feasibility through an inverse-kinematics solver.

Training setup. Both stages use Transformer-based diffusion models with separate Point Transformer encoders for object and hand clouds producing 256-dimensional features. Training uses AdamW at a learning rate of 10^-4, batch size 32, on a single NVIDIA A100 for 20,000 steps, with 1000 diffusion timesteps and a cosine schedule; inference uses DDPM sampling with 500 steps.

Evaluation. The dataset is split by object; the test set is 186 cases spanning 40 objects unseen during training, restricted to pick-and-place with unseen hand motions. Simulation uses GalaxeaManipSim with an R1 Lite embodiment, and the real-robot evaluation uses a Galaxea R1 Lite. Metrics are Grasping Orientation Accuracy (GOA, lower better), Success Rate (SR, higher better), Trajectory Similarity (TS, higher better), and Target Orientation Placement Accuracy (TOPA, lower better).

Why This Matters

Impact on research. The paper argues that two dominant paradigms — rule-based kinematic retargeting and trajectory-conditioned policies — break down under complex spatial movements, and offers paired data plus a data-driven formulation as an alternative. It also provides a concrete claim that object-centric 2D or flow-tracked 3D trajectories degrade under occlusion from large rotations, positioning hand-pose guidance as the missing ingredient.

Real-world applications:

• Robotic pick-and-place in cluttered shelves, where objects must be reoriented rather than simply slid or lifted.
• Tool use and re-grasping tasks that require rotating an object to maintain functional contact or to avoid collisions.
• Learning manipulation skills from casual human video captured in everyday environments instead of teleoperated lab setups.
• Data collection for warehouse, kitchen, or retail settings where the same object is manipulated in several different ways.

Industry relevance. The dataset, pipeline, and algorithm are stated to be released publicly. The pipeline uses a handheld gripper plus RGB-D capture rather than teleoperation, which the authors present as a cheaper and more scalable route to robot training data. The reported real-robot evaluation on a Galaxea R1 Lite ties the work to a specific commercial platform.

Future Directions

• Broader task types. The evaluation is explicitly limited to pick-and-place with unseen hand motions; the authors leave broader task categories as future work.
• Closing the simulation-to-real gap. The real-robot results are lower than simulation, attributed to object slippage during large rotations and premature movement before full lift; addressing contact and friction during reorientation is an open problem.
• Extending the pipeline to less structured settings. The current approach relies on fixed-viewpoint RGB-D capture, multi-view object reconstruction, and manual annotation of starting and terminal frames for UMI videos; scaling to in-the-wild human video would require relaxing these.
• Depth estimation as a substitute for capture. The method supports depth from either real capture or model prediction, leaving open how much performance depends on the quality of predicted depth.

Target Audience

Robotics researchers working on imitation learning, cross-embodiment transfer, and dexterous or reorientation-heavy manipulation; dataset and benchmark builders interested in paired human-robot demonstrations; and practitioners building data collection pipelines for real robot arms who want a documented alternative to teleoperation.

Authors’ abstract

Transferring human hand demonstrations to robotic grippers has recently emerged as a cost-effective solution for robot learning. However, existing methods are largely confined to simple, planar tasks and fail to handle complex spatial movements (e.g., intricate trajectories involving rotations or flips) that are essential for robot manipulation. Motivated by this gap, we adopt an implicit, data-driven approach guided by fine-grained hand-pose motions. To this end, we introduce a scalable acquisition pipeline to collect hand-gripper paired demonstrations, governed by a rigorous protocol that prioritizes motion complexity and leverages a handheld gripper for seamless action mimicry. This yields a large-scale paired dataset comprising 6,189 episodes across 1,254 unique objects, exhibiting significantly higher spatial complexity than existing benchmarks. However, learning such complex mappings remains challenging. We observe that naive end-to-end generation of full gripper pose sequences is insufficient, as minor trajectory deviations compound rapidly under intricate dynamics. To address this, we propose a two-stage framework: Stage I predicts sparse gripper keyframes (initial and terminal) to simplify the mapping objective, while Stage II generates the full continuous action sequence conditioned on these keyframes. Furthermore, to mitigate cumulative drift, we keep the gripper's orientation being learned while post-optimizing its translation based on the grasping heuristic and kinematic consistency. In both simulation and real-robot experiments, our framework enables stable and precise hand-to-gripper transfer of complex spatial manipulations, significantly outperforming traditional baselines. Project page: https://cosmoh2g.github.io.