Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery

AI summary

Overview

Research area: Computer vision — monocular 3D hand pose and motion estimation, combined with conditional generative modeling (flow matching).

Technical level: Advanced. The paper assumes familiarity with MANO hand models, diffusion/flow-matching generative models, transformer backbones, and camera projection geometry.

Scope: The paper proposes JoHan, a video-conditioned generative framework that jointly generates aligned local 2D and 3D hand motion sequences from monocular video, then recovers camera-space (global) hand pose from the generated 2D–3D correspondences, and evaluates it on HO3Dv2, HO3Dv3 and HOT3D.

What This Paper Is About

Recovering faithful 3D hand motion from video is hard because hands are frequently self-occluded, occluded by manipulated objects, or blurred by fast motion, so per-frame pose estimates are unreliable and jitter over time. Most existing methods either regress a single deterministic answer or use generative priors that still depend on noisy per-frame pose estimates as conditioning or initialization. JoHan instead learns to generate the hand's 2D and 3D motion jointly and directly from video, so that the strongly image-grounded 2D trajectory can guide 3D generation at every step, and their learned correspondence can then be used to place the hand in the camera's coordinate frame.

Key Contributions

1. A unified video-conditioned generative framework (JoHan) that learns a distribution over hand motion directly from video observations, trained from scratch and without relying on pretrained per-frame pose estimators as conditioning or initialization.
2. Joint 2D–3D local motion generation inside a single shared flow-matching process, where the evolving 2D prediction is encoded as guidance for the 3D branch ("2D-Guided Fusion") at every flow step, modeling temporal dynamics of both modalities while preserving joint-level 2D–3D correspondence.
3. Global pose recovery from generated correspondences: the generated local 2D poses are restored to image coordinates using refined, Kalman-smoothed detector bounds, and a PnP solver estimates the hand's global orientation and camera-space wrist translation, giving full camera-space hand motion.
4. Empirical validation across static and moving camera settings: state-of-the-art local pose accuracy on HO3Dv2 and HO3Dv3, markedly smoother motion on HOT3D, and substantially lower parameter count and inference time than the compared baselines.

Main Findings

• Best local pose and mesh accuracy on HO3D: JoHan reaches a PA-MPJPE of 7.4 mm and PA-MPVPE of 7.5 mm on HO3Dv2, and 7.0 mm and 6.9 mm on HO3Dv3 — the lowest among compared methods on both versions (HO3Dv3 comparison includes HaMeR at 7.4/7.3 mm and WiLoR at 7.2/7.1 mm).
• Lowest 2D joint error: 3.92 pixels on HO3Dv2 and 4.19 pixels on HO3Dv3, versus 6.37 px (HaMeR) and 6.53 px (WiLoR) on HO3Dv3, supporting the value of explicitly modeling 2D motion dynamics.
• Large camera-space gain after global recovery: MPJPE of 32.8 mm on HO3Dv3, compared with 53.5 mm for WiLoR and 54.7 mm for HaMeR.
• Best local pose accuracy and much smoother motion on HOT3D: PA-MPJPE of 4.75 mm and acceleration error of 2.69 m/s², improving over HaWoR by 21.5% and 62.3% respectively (HaWoR: 6.05 mm, 7.14 m/s²).
• World-space trajectory accuracy is not the strongest: W-MPJPE of 55.05 mm and WA-MPJPE of 20.36 mm trail HandFlow (43.00 mm and 16.17 mm); the authors attribute this to the deliberate focus on local motion modeling plus a simple geometric composition with estimated camera motion.
• Substantially more efficient: on a 128-frame HOT3D clip with one NVIDIA H200 GPU, total hand recovery takes 2.03 s (0.34 s detection, 1.69 s recovery) using 351M parameters — 27.0% of WiLoR's 7.52 s and 44.3% fewer parameters than WiLoR's 630M.
• Reprojection supervision matters most among auxiliary losses: removing it raises PA-MPJPE from 7.8 to 8.2 mm and 2D error from 5.21 to 5.96 px (ablation setting with DINOv3 ViT-B/16); removing the 3D joint loss changes 7.8 → 7.9 mm and 5.21 → 5.32 px.
• Full frames beat hand crops: conditioning on cropped hand videos instead of full frames degrades PA-MPJPE from 7.8 to 8.9 mm and 2D error from 5.21 to 15.68 px, attributed to fluctuating crop scale and center disrupting visual continuity.
• Masks help 2D more than 3D: training and evaluating without hand-mask conditions gives 8.0 mm PA-MPJPE and about 7.8 px 2D error, so accurate local 3D poses can still be recovered from full frames plus the hand-side label.
• Joint generation beats separate generation: a 3D-only generator gives 8.9 mm (vs. 7.8 mm), a 2D-only generator gives 5.64 px (vs. 5.21 px), and separately training the branches with ground-truth 2D conditioning gives 12.89 mm; noisy teacher forcing improves that to 9.49 mm but still trails joint generation.

Methodology in Plain English

Setup. The input is a monocular RGB video, a detector-produced binary hand mask per frame, and a label saying whether the left or right hand is the target. The output is a full sequence of hand poses (MANO parameters) placed in the camera's coordinate system. The system is trained from scratch on clips of up to 128 frames with RGB frames resized to 256 × 256.

Splitting local from global motion. Following the HMP line of work, the authors separate local hand motion (joint angles and shape, which are governed by hand anatomy) from global motion (where the hand is and how it is oriented relative to the camera, which varies with task, body and camera motion). Local motion is the part worth learning a prior over. Similarly, 2D joint coordinates are normalized per axis to remove image-plane location and scale, producing local 2D poses in [−1, 1], plus per-joint visibility labels.

One shared generation process for both modalities. The model uses flow matching: it starts from Gaussian noise at generation time s = 0 and integrates toward clean motion at s = 1, with the same s shared across both modalities and all frames. Two separate transformer-style generators predict clean motion for the 2D and 3D branches; their predictions define the velocity fields, and a shared Euler update advances both states.

2D guides 3D at every step. At each flow step the 2D branch first predicts clean local 2D motion. That prediction (Fourier-encoded, passed through an MLP, weighted by predicted visibility) is concatenated with the noisy MANO pose parameters of the corresponding joint, giving a "2D-Guided Fusion" token that conditions the 3D branch. Because the guidance is recomputed at every step, the 3D branch tracks the evolving 2D geometry throughout generation. This means the 3D branch is never conditioned on externally estimated per-frame poses.

Visual conditioning. Full-frame patch features come from a fine-tuned DINOv3 ViT-L/16 (ViT-B/16 in ablations), and CNN-encoded hand-mask features are added to them. Masks locate the hand; full-frame features keep context without crop-induced jitter. Individual frames' mask conditions are randomly dropped during training, classifier-free-guidance style, so the model can cope with missing or unreliable detections.

Training signals. Three losses are combined: a 2D loss (MSE on normalized 2D poses plus a BCE term on visibility labels), a 3D loss (MSE on pose, shape, and camera-space joints obtained using ground-truth global orientation and translation), and a reprojection loss that compares the generated 2D poses against projected 3D joints in the same normalized coordinate system, with gradients flowing to both branches.

Global recovery. After generation, the local 2D poses are mapped back to image coordinates using detector bounding boxes corrected by a lightweight bbox refiner and smoothed with a Kalman filter. A PnP solver then minimizes reprojection error between those 2D coordinates and the generated local 3D joints to estimate global orientation and camera-space wrist position; the MANO translation follows from the wrist offset. The full training setup, preprocessing and evaluation protocols are deferred to appendices not included in the provided content.

Why This Matters

Impact on research. The paper argues that per-frame pose estimation is a fragile bottleneck for generative hand motion recovery, and shows a working alternative: generate 2D and 3D motion together inside one flow so the 2D trajectory acts as a continuously updated, image-grounded constraint. It also reframes 2D hand trajectories as something to be generated and refined rather than merely consumed, and shows that joint generation improves both modalities relative to 2D-only or 3D-only generators. The reported speed and parameter efficiency suggest generative motion recovery need not be expensive.

Real-world applications:

• AR/VR input and avatar hands: low-latency, temporally smooth hand tracking from a single camera is the core requirement for gesture interfaces and embodied avatars.
• Robotics and imitation learning: recovering how a human hand moves relative to the camera provides demonstration data for dexterous manipulation.
• Human behavior analysis: smooth, consistent hand-motion traces support study of how people interact with objects and each other.
• Content creation and motion capture: markerless hand animation from ordinary video, including handheld/moving-camera footage, which the HOT3D experiments target.

Industry relevance. The efficiency numbers matter commercially: 2.03 s per 128-frame clip and 351M parameters on a single NVIDIA H200 GPU make this far more deployable than the 57.70 s reported for HaMeR or the 9.35 s for HaWoR in the same pipeline comparison. Handling moving cameras, occlusion by manipulated objects, and unreliable detections (via mask dropout) are exactly the conditions in consumer and industrial capture settings.

Future Directions

• Improving world-space trajectory accuracy. W-MPJPE and WA-MPJPE remain behind HandFlow and, for W-MPJPE, behind HaWoR. The authors explicitly frame this as a consequence of focusing on local motion and composing it with an off-the-shelf camera trajectory; explicitly modeling world-space hand motion, or jointly estimating camera and hand motion, is the obvious next step.
• Reducing dependence on camera-motion modules. JoHan uses HaWoR's camera motion estimation module for HOT3D, so errors there propagate into world-space results; an integrated approach is not reported.
• Extending beyond a single labeled hand side. The formulation targets one hand specified by a left/right label; bimanual and hand–object interaction settings are not addressed in the reported formulation, though interacting hands appear in the cited prior work.
• Testing the local-motion prior more broadly. The ablations use a smaller ViT-B/16 backbone and are run only on HO3Dv3; how the design scales to other backbones, longer sequences than the 128-frame training clips, and other capture conditions is not reported.

Target Audience

Researchers and graduate students in computer vision working on 3D hand or body pose estimation, generative modeling for motion, and video-based reconstruction. It is also relevant to practitioners building AR/VR hand tracking, markerless motion capture, or robot learning from demonstration, who care about the trade-off between per-frame accuracy, temporal smoothness, and inference cost. Readers without background in MANO parameterization, flow matching, or camera projection geometry will find the methodology sections demanding.

Authors’ abstract

Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D-3D correspondence further enables recovery of the hand's global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.