HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing

AI summary

Overview

Research area: Robotics / Embodied AI, at the intersection of image editing and dexterous manipulation data generation.

Technical level: Advanced. The paper assumes familiarity with URDF robot descriptions, kinematic retargeting, diffusion-based image editors, and robot policy pre-training.

Scope: The paper introduces HandEdit, a large-scale dataset and benchmark for editing egocentric human hand and arm images into images of dexterous robotic embodiments specified by a target URDF.

What This Paper Is About

Training dexterous robot policies requires robot-centric manipulation data, but collecting it through teleoperation is expensive and tied to specific hardware. Egocentric human videos of hand-object interaction are abundant and diverse, yet they cannot be used directly because human hands and robot hands differ in appearance, geometry, articulation, and kinematics. HandEdit addresses this by building a dataset and evaluation protocol that asks image-editing models to replace a human hand or hand-arm with a specific robot embodiment inside a first-person scene, while keeping the object, the contact, and the background intact.

Key Contributions

1. A large-scale embodiment-aware editing dataset. HandEdit is built from five public egocentric hand-object datasets (EgoDex, ARCTIC, OakInk2, HOI4D, HO-Cap) and yields over 200M image-level editing instances derived from 300K+ clips, spanning 600+ scenes, 400+ tasks, and 1.1K+ objects. It covers 26 target URDFs: 13 hand-only and 13 hand-arm configurations.
2. The first unified benchmark for embodiment-aware dexterous hand editing. Two tracks, Hand-only and Hand-Arm, support URDF-conditioned evaluation. The official test set contains 1K images per track, each paired with 13 corresponding target embodiments.
3. A multi-dimensional metric suite. The suite combines generic visual similarity (PSNR, SSIM, LPIPS, FID measured over full images, background regions, and ROI crops), VLM-based judgment (Semantic Consistency and Perceptual Quality scored with GPT-4o), and embodiment-aware metrics (Human-hand Removal, Target-robot Fidelity split into structural and ID fidelity, and Interaction consistency). A blinded human evaluation on the test set is also reported for metric validation, with details in Appendix F.
4. A benchmark of 11 representative commercial and open-source image editors, evaluated under their native conditioning interfaces with fixed prompt templates per track.

Main Findings

• GPT-Image-2 is the strongest overall baseline. It achieves the best LPIPS(ROI) and FID(ROI) on the Hand-only track (0.482 and 99.71 respectively, against a next-best 0.501 from Nano-Banana-2 and 103.83 from Nano-Banana-2), and the highest structural fidelity and interaction scores (0.780 and 0.703 on Hand-only; 0.778 and 0.595 on Hand-Arm).
• VLM judgment is useful but not sufficient. GPT-Image-1.5 obtains the highest VLM score on both tracks (0.765 Hand-only, 0.856 Hand-Arm) but does not lead on structural fidelity or interaction preservation. The paper attributes this to VLM judgment capturing high-level semantics rather than fine-grained robot morphology and contact-level interaction.
• High perceptual quality does not mean the edit succeeded. FireRed-Image-Edit-1.1 achieves the highest PQ score on the Hand-only track (0.754) yet has low Semantic Consistency (0.228) and low embodiment-aware scores (structural fidelity 0.630, interaction 0.293).
• The hard part is embodiment correctness, not hand removal. Human-hand removal scores are consistently high across models (for example 0.904 to 0.955 on Hand-only), whereas structural fidelity, ID fidelity, and interaction preservation vary widely across models on both tracks.
• Hand-Arm is substantially harder than Hand-only. GPT-Image-2 drops from 19.85 PSNR (full image) on Hand-only to 15.07 on Hand-Arm, and its ID fidelity drops from 0.852 to 0.563. Background preservation degrades as well (GPT-Image-2 BG PSNR 22.82 to 18.70).
• Pseudo-GT is not a ceiling on every metric. The harmonized pseudo-GT reference scores 1.000 structural fidelity by construction, but on VLM judgment it scores 0.623 (Hand-only) and 0.644 (Hand-Arm), below several models; on Hand-Arm ID fidelity it reaches 0.962 while Interaction reaches 0.788.
• The benchmark landscape has a gap. Table 1 compares HandEdit against 12 editing benchmarks, none of which jointly support egocentric input, dexterous hands, URDF conditioning, and multi-embodiment evaluation. HandEdit reports 26 embodiments and 200M scale, with Sim./VLM/Embodiment metrics.

Methodology in Plain English

The team built the dataset as a staged pipeline that turns ordinary human hand videos into robot-hand images that can serve as reference targets.

1. Remove the human. For each egocentric hand-object clip, a segmentation model (SAM3 by default) cuts out the human hand or hand-arm region, and video inpainting (ProPainter by default) fills in the occluded background behind it.
2. Retarget the pose. The MANO or 3D hand pose is converted into robot joint states using embodiment-specific retargeting. A coarse alignment handles differences in kinematic trees, link lengths, joint limits, and contact geometry, then position-based retargeting refines the configuration under the target robot's constraints.
3. Place the arm. For hand-arm embodiments, the robot's physical base is not visible in a human video, so the authors define a camera-relative virtual base for each sequence and hold it fixed. The base is initialized from shoulder landmarks when available, otherwise from the wrist-trajectory center and the arm's nominal reach, then searched over horizontal translation and yaw. Candidate bases are scored using inverse-kinematics feasibility, joint limits, collision checks, and trajectory quality, and a human operator selects one of the top-three valid solutions; sequences with no plausible solution are dropped.
4. Render and composite. The target URDF is rendered under the matched egocentric camera view and placed into the restored background.
5. Harmonize. Because foreground and background are processed separately, a lightweight Harmonizer module (about 20 MB), trained on 10,000 natural egocentric hand images with self-supervised appearance augmentations covering lighting, color, contrast, and boundary appearance, refines the robot foreground and its boundary. It changes appearance only, not joint configuration, wrist pose, object state, or interaction geometry.
6. Filter. Automatic checks and human screening run at every stage, with stage-specific rejection rules (for example, segmentation requires a non-empty mask and SAM 3 confidence of at least 0.75).

Evaluation then compares editor outputs against these harmonized pseudo-GT targets using the metric suite, and against the requested embodiment's URDF render bank for identity fidelity.

Why This Matters

Impact on research. HandEdit reframes human-to-robot dexterous hand conversion as a systematic image-editing problem with a shared dataset, a URDF-conditioned evaluation protocol, and metrics that go beyond visual plausibility. It gives the embodied AI and image editing communities a common yardstick for a task that previously had no dedicated benchmark, and it shows that current editors get semantic plausibility right while failing on morphology and contact.

Real-world applications.

• Generating robot-centric visual observations from abundant human video for policy pre-training and co-training, reducing reliance on costly real-robot or teleoperated data collection.
• Evaluating and selecting commercial or open-source image editors for robotics pipelines before committing to a platform.
• Transferring manipulation demonstrations across different dexterous hand and arm platforms without collecting new hardware-specific data.
• Building controllable editing tools that respect robot-specific geometry, useful for simulation asset creation and embodiment visualization.

Industry relevance. The paper benchmarks commercial APIs (GPT-Image-2, Nano-Banana-2, GPT-Image-1.5, Seedream-4.5, Flux-2-Pro, Nano-Banana, Flux-Kontext-Max) alongside open-source models (Hunyuan-Image-3.0, Qwen-Image-Edit-2511, Omnigen2, FireRed-Image-Edit-1.1) under their native interfaces with fixed prompts, which gives robotics teams a directly usable comparison for data-generation tooling. The dataset also draws on a URDF roster spanning hand families (Ability, Allegro, DexHand(021), Inspire RH56DFX and RH5DG2, Leap, OrcaHand, Revo2, Schunk SVH, Shadow Hand, Sharpa, Wuji, RoHand) and arm platforms (Jaka, KUKA, Panda, RM65, RM75, UR5, xArm), which matters for teams choosing hardware.

Future Directions

• Develop purpose-built embodiment-aware editing models. The paper positions HandEdit as a foundation for such models; the benchmark shows no current editor consistently wins across structural fidelity, ID fidelity, and interaction preservation.
• Use the paired data for downstream policy learning. The authors state that the paired human-source and robot-embodied images can train editing models that generate robot-centric observations for policy pre-training, but the paper does not report policy-learning results, so this remains to be demonstrated.
• Close the gap between semantic judgment and geometric correctness. The divergence between high VLM scores and low structural or interaction scores raises the question of how to design evaluation and training signals that reward correct morphology and contact, not just plausible-looking images.
• Extend beyond the current scope. HandEdit is an image-level benchmark over 26 URDFs; scaling to more embodiments and to temporally consistent video editing for manipulation sequences are natural next steps not addressed here.

Target Audience

Researchers and engineers working on embodied AI, dexterous manipulation, and robot learning from human video; image and video editing researchers interested in controllable, condition-specific generation; builders of robot data pipelines who need embodiment-aware editing tools; and benchmark or evaluation researchers focused on metrics that capture physical and structural correctness rather than perceptual quality alone.

Authors’ abstract

Robotic manipulation with dexterous hands is a cornerstone of Embodied AI, yet its progress is stifled by the high cost of collecting embodiment-aware teleoperation data. While abundant egocentric videos of human hands offer a scalable alternative, the profound discrepancies in appearance, articulation, and camera viewpoints between human and robotic data raise significant challenges for co-training. Though existing general image-editing models demonstrate strong capabilities, they lack necessary embodiment-specific priors to fully bridge this gap. In this work, we present HandEdit, a unified large-scale embodiment-aware image-editing dataset and benchmark specifically designed to transform human hands and arms into various dexterous robotic embodiments within egocentric frames. HandEdit comprises over 200M editing instances derived from five diverse source datasets, covering 26 distinct URDFs, including 13 hand-only and 13 hand-arm configurations. Alongside the dataset, we establish a unified benchmark protocol with two tracks: Hand-only and Hand-Arm, supporting URDF-conditioned evaluation. We conduct extensive evaluations of 11 representative image-editing baselines using a multi-dimensional metric suite, including generic similarity metrics, VLM-based judgment, and embodiment-aware metrics. HandEdit serves as a critical resource at the intersection of image editing and robotics: it advances embodiment-aware editing models while enabling scalable dexterous robotic learning from abundant human video data, paving the way for more generalizable Embodied AI.