WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos

AI summary

Overview

Research area: Computer vision, specifically 3D/4D reconstruction of hand-object interaction from egocentric (first-person) video, combining diffusion-based generative motion priors, hand pose estimation, 6D object pose estimation, and vision-language models for contact labeling.

Technical level: Advanced. The paper assumes familiarity with diffusion models, DDPM/classifier guidance, MANO hand parameterization, SE(3) object pose, metric SLAM, and 6D pose metrics (ADD/ADD-S, MPJPE).

Scope in one sentence: The paper presents WHOLE, a method that jointly reconstructs both hands and a manipulated object in a persistent world coordinate frame from metric-SLAMed egocentric video plus a 3D object template, using a diffusion prior over hand-object motion that is guided at test time by video observations and VLM-derived contact cues.

What This Paper Is About

Egocentric manipulation videos are hard: hands and objects occlude each other constantly, and objects frequently leave and re-enter the camera view as the wearer moves, while the moving camera creates large egomotion even when the object is nearly still. Existing methods reconstruct either hands or objects in isolation, and their independent predictions often produce inconsistent, implausible hand-object relationships; they also fail when the object is truncated or out of view. WHOLE's goal is to recover both hands and the manipulated object as a single coherent, world-grounded 3D trajectory, by learning a generative prior over how hands and objects move together and then steering that prior with evidence from the video.

Key Contributions

1. A joint world-space hand-object reconstruction framework. WHOLE holistically reconstructs hand articulation (MANO parameters, joints, and velocities), object 6D trajectories, and binary left/right-hand contact labels in a single consistent world coordinate frame from metric-SLAMed egocentric video and a given object template, with a fixed temporal window of T = 120 frames and blending of overlapping windows for longer sequences.

2. A diffusion-based generative motion prior over hand-object interaction. The prior is conditioned on a roughly estimated hand trajectory and object geometry, operates in a gravity-aware local coordinate frame, and includes an "Ambient Sensor" feature measuring hand-joint displacement to the nearest object surface. It is trained with a conditional DDPM loss plus interaction, consistency, and temporal smoothness auxiliary objectives (after a 10k-step warm-up using only the DDPM loss).

3. Reconstruction formulated as guided generation. At test time, classifier guidance modifies the diffusion score with three objective categories — a reprojection term (contact binaries, 2D hand joints, object masks, with a one-way Chamfer loss for occlusion/truncation), an interaction term, and a temporal smoothness term — which the authors report is faster than score distillation sampling because it needs a single forward generation pass rather than thousands of optimization steps.

4. Robust VLM-based contact annotation. A vision-language model is prompted with masks overlaid on the image, indexed hands/objects, validation rules such as a "one-out-of-k" contact limit, and five annotated in-context examples; the authors report this raised contact detection F1 from 57% to 81%.

The paper also demonstrates a hand-guided HOI planner application, in which a coarse hand trajectory, picking/placing times, and an object template are enough to synthesize diverse interaction motions with no video input.

Main Findings

• Best overall hand motion among compared methods: On the 150-frame held-out HOT3D-CLIP sequences, WHOLE scores 3.26 cm WA-MPJPE, 10.41 cm W-MPJPE, 0.58 ACC-NORM, and 6.67 mm PA-MPJPE. HaWoR scores 3.76, 11.26, 4.15, and 8.99; HaMeR scores 16.93, 28.35, 32.31, and 12.76; FP+HaWoR-simple scores 3.34, 9.16, 0.95, and 8.99; FP+HaWoR-contact scores 5.85, 12.19, 1.26, and 8.99.

• Large gains in object motion, especially under difficult conditions: WHOLE reaches 51.1 ADD AUC and 69.9 ADD-S on the full test split (ACC 0.11), versus FoundationPose at 36.1 and 51.9, FP+HaWoR-simple at 30.4 and 45.0, and FP+HaWoR-contact at 31.5 and 46.9. On the out-of-view subset, WHOLE scores 28.8 ADD / 43.8 ADD-S compared with 4.8 / 19.6 for FoundationPose and 8.9 / 17.7 for FP+HaWoR-contact. On the truncated subset, WHOLE scores 42.5 / 60.2 versus 18.6 / 33.8 for FoundationPose.

• Stronger relative hand-object alignment: After aligning object poses using the predicted hand trajectory, WHOLE reports 53.5 ADD AUC, 72.3 ADD-S, and 0.11 ACC, compared with 36.8 / 53.8 and 0.572 for FP+HaWoR-contact and 29.0 / 44.0 and 0.896 for FP+HaWoR-simple.

• Guidance must be intertwined with diffusion, not applied afterward: A "Gen+Opt" variant that generates unguided and then optimizes post-hoc drops to 15.92 W-MPJPE (versus 10.41 for WHOLE), 44.3 / 61.0 object ADD/ADD-S (versus 51.1 / 69.9), and 45.0 / 62.1 interaction scores (versus 53.5 / 72.3).

• The interaction loss is necessary for object and interaction accuracy: Removing it yields 28.5 / 42.9 object ADD/ADD-S and 30.6 / 45.5 interaction scores, far below WHOLE's 51.1 / 69.9 and 53.5 / 72.3, although hand W-MPJPE is similar (10.47 versus 10.41).

• VLM contact labels approach ground-truth ceiling: Replacing VLM contact labels (3 fps) with ground-truth annotations (30 fps) gives 10.79 hand W-MPJPE, 53.7 / 72.2 object ADD/ADD-S, and 56.1 / 74.5 interaction scores, close to WHOLE's 10.41, 51.1 / 69.9, and 53.5 / 72.3.

• A trade-off exists between precise hand pose and hand-object consistency: Adding the interaction term to the combined baseline (FP+HaWoR-contact) improves object results over FP+HaWoR-simple but worsens hand accuracy (5.85 versus 3.34 WA-MPJPE; 12.19 versus 9.16 W-MPJPE).

• Zero-shot generalization is more robust than an RGB-conditioned baseline: WHOLE, trained on HOT3D, is additionally tested on the unseen H2O dataset and reports 44.7 ADD AUC, 64.9 ADD-S, and 0.23 ACC (with 51.1 / 69.9 / 0.11 on the in-domain HOT3D split). H2OTR, trained on H2O, reports 62.1 / 77.1 and 0.48 on H2O but collapses on HOT3D with 3.2 / 7.7 ADD/ADD-S and 25.4 ACC.

• Generation is diverse, not deterministic: From the same conditioning hand motion, sampled interactions differ in predicted contact timing, grasp/release moments, and hand-object relative poses (for example, a mixer picked up later in one sample, a bottle grasped differently in another).

• Efficiency: On a single NVIDIA RTX 6000 Blackwell GPU, a 150-frame clip takes an average of 59.34 seconds, dominated by the guidance step at 59.06 seconds, with the diffusion step at only 0.28 seconds; peak memory is 14 GB. The authors state this is orders of magnitude faster than prior works they cite at 30 hours and 1 hour. VLM queries average 18.6 seconds per image with GPT-5.

• Training and evaluation setup: The diffusion model is trained on 2,443 HOT3D-CLIP sequences, each 150 frames (3 seconds); evaluation holds out 50 dynamic object trajectories with displacement greater than 5 cm. Training contact labels are defined by proximity below 5 mm, while reconstruction contact labels come from prompting GPT-5.

Methodology in Plain English

The researchers treat reconstruction as a generation problem rather than a per-frame fitting problem. First they train a diffusion model that learns how hands and objects typically move together. The model takes two inputs: a rough hand motion estimate (produced at test time by an existing hand estimator, and during training simulated with injected trajectory-level noise, per-frame noise, and randomly dropped frames so the model does not overfit to one estimator) plus a shape descriptor of the object template. It outputs, for a 120-frame window, refined hand parameters, a 6D object trajectory, and binary left/right contact indicators. To keep the motion physically sensible, the loss includes not only denoising but also terms that encourage contact points to be transported rigidly while in contact, that keep hand features consistent with forward kinematics, and that penalize large accelerations. Every sequence is rotated into a gravity-aligned frame so the model learns relative motion rather than arbitrary global orientation, and these segments can be rotated back into world coordinates afterward.

At test time, the pretrained model is not used blindly. Instead, each denoising step is nudged with gradients from objectives that measure how well the current guess matches the video: 2D reprojection of hands and object masks (using a one-way Chamfer loss so occlusions and truncation do not mislead it), hand-object interaction constraints, and temporal smoothness. Contact information comes from a vision-language model that is shown the image with colored masks for each hand and candidate object, given strict rules that reaching or hovering is not contact, and asked to return JSON labels. Because contact detection tends to give false positives, five annotated examples are supplied for calibration, which the authors report raised F1 from 57% to 81%. Longer videos are handled by running overlapping 120-frame windows in parallel, blending the overlaps and shape parameters, refining under the same guidance, and continuing the diffusion process. The system is evaluated on HOT3D-CLIP against HaWoR for hands, FoundationPose for objects (with depth from Metric3D since FoundationPose needs RGB-D), and combined baselines that pair them and then optimize. Notably, the authors state that evaluating at 150-frame clips rather than the 60–100 frame segments used in prior work explains why some reported numbers differ from those in earlier papers, and they use a more permissive AUC threshold of 0.3 instead of the usual 0.1.

Why This Matters

Research impact: The paper argues that reconstructing hands and objects separately and then post-processing is fundamentally insufficient, and provides evidence that a joint generative prior over interaction outperforms that recipe. It also pushes egocentric 3D understanding beyond short, local hand-object clips into long, world-grounded sequences that survive occlusion, truncation, and out-of-view gaps, and it offers an evaluation protocol on 150-frame clips with subsets designated for contact, truncated, and out-of-view frames.

Real-world applications:

• Robot learning from human demonstrations, where a robot imitates manipulation skills from first-person human video.
• Immersive AR/VR environments that need consistent hand and object placement in a shared virtual world.
• Wearable-camera assistants that can track what a person picked up, where it went, and when it was released, even when it leaves the frame.
• Motion synthesis and planning, since the framework doubles as a hand-guided HOI planner that generates object trajectories from a coarse hand motion, picking and placing times, and an object template, with no video needed.

Industry relevance: The training data comes from Aria Glasses and the pipeline targets smart-glasses and wearable platforms, so the work is directly relevant to companies building XR headsets, egocentric AI assistants, and robotics platforms that learn from human video. The reported runtime (59.34 seconds for a 150-frame clip, 14 GB peak memory on one GPU) and the authors' statement that code and models will be public upon acceptance matter for practical deployment, though the dependency on GPT-5 queries averaging 18.6 seconds per image indicates an external model cost.

Future Directions

• Scene-level, multi-object reconstruction. The current framework reconstructs each hand-object pair independently; the authors propose joint diffusion with scene-level objectives as a next step.
• Removing the object-template assumption. Relaxing the requirement for a known template via LLM-based retrieval or template-free generation.
• Scaling the generative prior. The prior is trained on a single dataset, so training on recent large-scale hand-object datasets is proposed to improve generalization and robustness.
• Broader cross-dataset evaluation. The authors describe the zero-shot H2O result as encouraging but state that

Authors’ abstract

Egocentric manipulation videos are highly challenging due to severe occlusions during interactions and frequent object entries and exits from the camera view as the person moves. Current methods typically focus on recovering either hand or object pose in isolation, but both struggle during interactions and fail to handle out-of-sight cases. Moreover, their independent predictions often lead to inconsistent hand-object relations. We introduce WHOLE, a method that holistically reconstructs hand and object motion in world space from egocentric videos given object templates. Our key insight is to learn a generative prior over hand-object motion to jointly reason about their interactions. At test time, the pretrained prior is guided to generate trajectories that conform to the video observations. This joint generative reconstruction substantially outperforms approaches that process hands and objects separately followed by post-processing. WHOLE achieves state-of-the-art performance on hand motion estimation, 6D object pose estimation, and their relative interaction reconstruction. Project website: https://judyye.github.io/whole-www