HandsOn: A Large-Scale Simulation Benchmark for Generalizable Dexterous Manipulation

AI summary

Overview

Research area: Robotics — dexterous manipulation benchmarks, simulation-based policy learning, and vision-language-action (VLA) / world-action model (WAM) evaluation.

Technical level: Advanced. The paper assumes familiarity with imitation learning, diffusion policies, VLA architectures, simulation benchmarking, and resampling-based statistics.

Scope in one sentence: HandsOn is a composable simulated benchmark of 66 human-verified dexterous manipulation tasks backed by 18,000 curated human demonstrations, used to measure how well ten current policy families (including a prompted GPT-6 controller) execute and generalize across objects, skills, and contact-rich interactions.

What This Paper Is About

Evaluating dexterous manipulation requires more than knowing whether a robot eventually succeeds: a policy must handle how contacts change during an operation — clicking a mouse only after moving it, holding two knobs at their targets at the same time, keeping a lid aligned while releasing it. Existing dexterous benchmarks cover many tasks and embodiments but have small human-demonstration sets and rarely reserve tasks purely for testing generalization. HandsOn addresses this by building a large, re-renderable demonstration corpus in simulation and splitting it into a 50-task learning suite (Core-50) and a 16-task held-out generalization suite (Transfer-16), then benchmarking ten methods on both with task-native success checkers and stage-level diagnostics.

Key Contributions

1. A large human-demonstration dataset. 18,000 curated simulated trajectories (15,178 training / 2,822 validation), roughly 56 hours of interaction with three synchronized camera views, re-rendered with visual and language-prompt randomization. The authors state this is 5.7–16.4 times the human trajectory count of representative dexterous benchmarks under their counting convention (Table 1).

2. Two complementary task suites with native scoring. Core-50 (50 demonstration-backed tasks across primitive, functional, and contact-rich interactions) and Transfer-16 (16 evaluation-only tasks: 6 composition, 7 generalization, 3 deformable), each scored by task-specific executable success predicates rather than a shared proxy, with ordered-progress measurement for trained multi-task policies.

3. A broad policy comparison with diagnostics. A comparison of single-task Diffusion Policy, five VLA models (π0.5, DM0.5, LingBot-VLA, HY-Embodied-0.5-VLA, Spatial Forcing) and three world-action models (OpenWAM, FastWAM, Motus), plus a covariate-adjusted skill-group analysis and an offline language-conditioning probe, locating failures at specific hand transitions.

4. A 660-trial prompted-VLM baseline without benchmark finetuning. GPT-6 (gpt-6-astra) driving a move_eef palm-and-finger interface, with disclosed observation interface, call budgets, and trial counts, reported separately from trained-policy aggregates.

Main Findings

• Best trained multi-task policy reaches 29.1% on Core-50. OpenWAM leads the eight trained checkpoints; FastWAM follows at 26.4%, π0.5 at 24.9%, DM0.5 at 21.4%, LingBot-VLA at 19.7%, HY-Embodied at 18.9%, Motus at 18.1%, and Spatial Forcing at 12.4%. Choosing the best checkpoint per task would give 35.4%.

• Single-task Diffusion Policy scores 9.7% over 50 tasks. On the 11 tasks evaluated under a matched protocol it reaches 9.9%, against 22.7–34.5% for seven multi-task checkpoints and 9.3% for Spatial Forcing. The authors note this does not separate multi-task data sharing from pretraining or model scale.

• Success tracks what the fingers must do beyond closing. Pushing or flicking without a grasp (two tasks) reaches 51% pooled over checkpoints, articulated mechanisms 43%, and grasp–transport–release 35%; finger-actuated controls (trigger, button, knob, slider, shutter) fall to 14%, pouring to 13%, and in-hand regrasping or reorientation to 11%.

• Tool use and precision fits are nearly unsolved. Acting on a third object through a held tool succeeds in 137 of 4,800 episodes (2.9%), and precision fits such as peg insertion, stacking, or face alignment in 107 of 4,800 (2.2%). The best checkpoint on any of these twelve tasks reaches 21%; on ten of them nothing exceeds 14%. These gaps persist after adjusting for demonstration count and expert duration, staying 20–25 points below grasp–transport–release with task-bootstrap intervals excluding zero.

• Failures concentrate at three hand transitions. Diagnostics show policies fail at contact establishment (OpenWAM clicks before the mouse is ready in 16 of 100 episodes, π0.5 in 67), action coordination (holding both knobs within 5° simultaneously), and goal retention (keeping the laptop display at the requested angle: OpenWAM 67, π0.5 42; completing a handover: OpenWAM 41, π0.5 20). Composed tasks add step-order and completion errors.

• Two VLA checkpoints barely condition on the named object. HY-Embodied and LingBot-VLA score 55% and 61% on Place Cup on Plate (π0.5: 62%) but at most 22% on Place Ball in Bowl, Place Fruit in Bucket, and Place Largest Fitting Ball, where π0.5 scores 35–75%. In an offline probe, renaming the target ball moves π0.5's and DM0.5's 1-s palm endpoint by 15 and 17 cm versus 7 and 8 cm under resampling, but HY-Embodied and LingBot-VLA by only 7–9 cm; with the swapped name those two still head for the original ball in 57% and 71% of first frames (DM0.5: 10%).

• World-action models finish later but finish more. The three WAM checkpoints complete successes at a median of 1.13–1.27 expert p75 durations versus 0.88–0.99 for the four 30-step VLA checkpoints, and 14.5–18.7% of their time-outs are still making progress after 1× p75 versus 10.1–11.8%. Under a 1× p75 cutoff π0.5 leads at 14.6% and OpenWAM drops to 8.3%; 12 of 28 checkpoint pairs change order between the 1× and 3× limits.

• Transfer-16 is much harder than Core-50. Ranked by ordered progress, OpenWAM reaches 12.9%, exceeding HY-Embodied, LingBot-VLA, and Spatial Forcing by 6.0, 3.7, and 2.6 points with paired intervals excluding zero; the other four are within 1.7 points. DP is not run on Transfer-16 and the authors state this is not a measured zero-shot failure.

• GPT-6 without finetuning scores 5.2% on Core-50 and 1.9% on Transfer-16. Over 660 trials (ten per task) success appears on only 12 of the 66 tasks. Qualitative action traces are described as suggesting task-level planning potential, motivating richer interfaces between planning and high-dimensional dexterous execution.

• Feasibility is verified, so low scores reflect policy gaps. Every task was completed in person by data-collection operators in the released environment; for 47 of the 50 Core-50 tasks at least one evaluated multi-task checkpoint succeeds in at least one episode. The 95% interval for OpenWAM's 29.1% is 22.8–36.0% over tasks and episodes, and 28.0–30.1% over episodes alone, indicating the task mix dominates the uncertainty.

• Visual factor checks show invariance, not generalization. Within the trained-checkpoint Core-50 evaluation, which of the four table materials, three colour palettes, or two roughness levels is drawn does not measurably change success; the authors state those factors lie inside the training distribution.

Methodology in Plain English

Data collection. Demonstrations are recorded in NVIDIA Isaac Sim 5.1 by human operators wearing MANUS Prime I gloves (finger articulation) and HTC VIVE wrist trackers (hand pose). Retargeting converts this to simulated arm and hand commands — the arm branch produces feasible end-effector targets through robot kinematics, and Wuji retargeting maps a palm-centred 21-keypoint hand representation to 20 joint targets for the first-generation Wuji Hand. At least ten operators contributed more than 1,050 collection sessions. All training demonstrations are simulated; the physical robot is used only for qualitative rollout videos.

Curation and quality control. Trajectories are selected under task-group quotas of 500, 300, or 100, preferring shorter eligible demonstrations. A written review procedure separates recording from review: operators record and self-check, independent reviewers judge, and a third person arbitrates. New tasks are piloted with at least 20 operator episodes requiring no severe defect and at least 98% agreement between labels and objective evidence. About 20% of each batch is blind-reviewed, with a further 5–10% of passed episodes re-reviewed. A success flag without objective completion quarantines the task.

Re-rendering for visual variety. Recorded interactions are replayed with new appearance — four table materials, three background configurations, three ceiling styles, plus texture, roughness, tint, and illumination variation — at 32 samples per pixel, while state and action labels stay fixed. Camera extrinsics are perturbed within ±2 mm translation and ±0.3° rotation per axis, preserving intrinsics. The paper is explicit that geometry and dynamics changes cannot be obtained this way and would require valid re-execution.

Language variation. A frozen bank of 108 task–instruction entries (91 active in the final manifest) supplies 32 training and 8 held-out paraphrases per entry, giving 2,912 training and 728 held-out strings. The default training mixture is 20% canonical and 80% paraphrases. Paraphrases preserve object, numerical goal, direction, order, and constraints; swapping the object or changing order counts as a task variation, not a paraphrase.

Evaluation protocol. Actions execute at 30 Hz with each action advancing the simulation exactly 1/30 s. Each Core-50 task gets a fixed physical-time budget of three times the non-interpolated 75th-percentile demonstration length, converted to a maximum action count. The 13 non-Flex Transfer-16 tasks use a duration proxy built from predeclared supporting atomic tasks — for example, Open Laptop and Click Mouse uses (196 + 125) source frames, giving 32.1 s or 963 actions — while the three Flex tasks keep native budgets of 240 s (sponge compression) and 180 s each (towel folding, pillow packing). Rollouts stop at the first terminal outcome, including mid-chunk; success requires the full native predicate including any release, stability, or dwell requirement. Learned checkpoints use 100 episodes per task; GPT-6 uses ten.

Analysis. Results are reported as task-macro success with Wilson intervals for binary success and resampling-based aggregate intervals with the resampling unit stated. Ordered progress for a task with K stages is the longest valid ordered prefix divided by K. Skill-group comparisons use log demonstration count and log expert duration as covariates.

Why This Matters

Research impact. HandsOn separates "execution on demonstrated tasks" from "generalization to held-out tasks" in a single platform, and supplies stage-level diagnostics that distinguish initial progress from reliable functional completion. The finding that tool use and precision fits remain near 2–3% while grasp-and-place reaches 35% gives the field a concrete target rather than an aggregate score, and the language-conditioning probe offers a reusable way to test whether a VLA actually reads the instruction.

Real-world applications:

• Household and service robots that must operate buttons, knobs, triggers, and hinged appliances rather than only pick and place.
• Tool-mediated tasks such as hammering, scooping, or spreading, where the policy must control a held object's effect on a third object.
• Bimanual assembly and packaging, where both hands must hold targets simultaneously and handovers must complete reliably.
• Deformable-object handling such as folding towels, packing pillows, or compressing sponges — represented here only in the held-out suite, with no deformable-task demonstrations in the dataset.

Industry relevance. The platform's task and asset layer is designed to survive hardware substitution: the code supports arms from ROKAE, KUKA, UR, ABB, and FANUC, hands including first- and second-generation Wuji Hands and the Sharpa Hand, and two physics back-ends (NVIDIA PhysX and Newton). That makes it a candidate internal benchmark for teams comparing policy stacks across robot hardware without rewriting tasks. The reported diagnostic breakdown — where time-outs are still progressing, which transitions fail — is directly useful for prioritizing data collection.

Important caveat for adopters. All reported experiments use one configuration: ROKAE AR5-08 arms (7 DoF, 5 kg payload, 683 mm reach), first-generation Wuji Hands (20 actuated joints each), and PhysX. The authors state the numbers should not be read as performance on the other supported configurations.

Future Directions

• Interfaces between planning and execution. GPT-6's traces are described as showing task-level planning potential despite 5.2% success under a move_eef interface with no learned grasp primitive; the paper motivates richer interfaces connecting high-level planning to high-dimensional dexterous control.

• Contact-establishment and retention as first-class problems. The three failing transitions (closing a grasp that fires a button, coordinating two hands to hold targets together, keeping a reached goal) suggest targeted training objectives or reward/curriculum design beyond aggregate success maximization.

• Language grounding in VLA training. The probe showing two checkpoints barely move toward the named object indicates instruction conditioning as an open problem, separate from manipulation skill.

• Cross-embodiment and deformable transfer. Five arms, six hands, and two physics back-ends are supported but untested in the reported results; the three deformable Transfer-16 tasks have no deformable demonstrations, leaving open how such skills should be acquired. Real-robot evaluation is qualitative only — no physical success rate is reported — so quantitative sim-to-real transfer remains open.

Target Audience

Robotics researchers building or evaluating manipulation policies, particularly those working on dexterous hands, VLA and world-action models, and imitation learning from human demonstrations. Benchmark designers will find the task-card and review protocol, the duration-budget formula, and the reporting conventions (distinct counting units, separate GPT-6 reporting, explicit provenance) directly reusable. Industry teams selecting a simulation benchmark for internal policy comparison benefit from the portability design and the diagnostic breakdown, though they should note the single-configuration results and the absence of quantitative real-robot numbers. Readers seeking an introductory treatment of dexterous manipulation will find the paper dense; it assumes comfort with VLA architectures and resampling statistics.

Authors’ abstract

General-purpose manipulation requires reliable behavior across embodiments, objects, and task demands. We introduce HandsOn, a composable simulation platform and benchmark for cross-embodiment dexterous manipulation. It supports five robotic arms, six dexterous hands, and approximately 200 rigid, articulated, and deformable assets in randomized scenes. The benchmark pairs 18,000 curated human demonstrations with 66 human-verified tasks: Core-50 supports demonstration-based learning and execution tests, while Transfer-16 evaluates asset and relation transfer, skill composition, and deformable manipulation without target-task training demonstrations. Qualitative real-robot rollouts illustrate sim-to-real transfer. Across diffusion, vision-language-action, world-action, and prompted VLM-based methods, the best trained multi-task policy achieves 29.1% Core-50 success. Without benchmark finetuning, GPT-6 using move_eef for palm-and-finger control achieves 5.2% on Core-50 and 1.9% on Transfer-16 over 660 trials. Trained-policy stage diagnostics expose failures in contact establishment, action coordination, and maintaining achieved goals, with additional sequencing errors in composed tasks. HandsOn provides a shared basis for investigating manipulation generalization and the limits of reliable physical interaction.