DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

AI summary

Overview

• Research area: Computer vision and human-computer interaction, specifically egocentric (head-mounted camera) 3D hand pose estimation for extended reality (XR).
• Technical level: Advanced. The paper combines MANO hand modeling, dense visual featurizers (DINOv3), motion-capture ground truth, and statistical occlusion analysis. The XR/HCI framing is approachable, but the method and evaluation assume familiarity with hand pose estimation literature.
• One-sentence scope: The paper introduces DeltaDorsal, a system that estimates 3D hand pose from the visible back-of-hand (dorsal) skin by comparing dense visual features against a neutral reference image, and shows it outperforms state-of-the-art pose estimators in self-occluded egocentric scenarios using a smaller model.

What This Paper Is About

Egocentric hand tracking fails when the fingers are hidden behind the palm, and the authors report that this happens often: in four egocentric datasets they analyzed, more than 20% of frames contain at least one occluded finger. Existing state-of-the-art models rely on whole-hand silhouettes or coarse geometry, so they lose accuracy exactly when the fingers disappear. The goal of DeltaDorsal is to instead read pose from dorsal skin deformation (tendons, knuckles, wrinkles), which remains visible even when the fingers cannot be seen, and to predict hand pose without any temporal input or additional wearables.

Key Contributions

1. An analysis of the prevalence and impact of self-occlusion across common egocentric hand datasets (ARCTIC, H2O, EgoExo4D, AssemblyHands), motivating the use of dorsal features.
2. DeltaDorsal, described as the first end-to-end system to use dorsal skin features for egocentric hand pose estimation, and an open-source pipeline that turns dorsal skin imagery into pose predictions and click detection without temporal dependencies.
3. A 12-participant evaluation against state-of-the-art baselines (HaMeR and HandOccNet), plus analyses with respect to occlusion, skin tone, image size, and backbone (note: the truncated text does not report the skin tone, image size, or backbone numbers).
4. A new 4K egocentric dorsal dataset of 172,222 annotated frames covering 17 gestures from 12 participants, and demonstrations of downstream uses: pinch, tap, and isometric "force click" detection.

Main Findings

• Self-occlusion is pervasive in egocentric data: Using a MANO-mesh, Z-buffer-based occlusion metric, the authors analyzed ARCTIC (184,346 frames analyzed, 25,242 occluded), H2O (26,529; 5,666), EgoExo4D (8,703; 1,873), and AssemblyHands (394,621; 140,669). More than 20% of frames across all evaluated datasets show at least one occluded finger, and over 5% contain two or more occluded fingers.
• The dorsum stays visible when fingers do not: With at least one finger over 90% occluded, median dorsal visibility is above 40% in every dataset, reaching 80% (H2O) at the highest and 43% (EgoExo4D) at the lowest, with distributions skewed toward high visibility.
• Baselines degrade with occlusion: Both baselines showed a statistically significant (p < .0001) negative correlation between finger visibility and pose error — slope m = −15.84°, R² = 0.212 for HaMeR and m = −9.11°, R² = 0.095 for HandOccNet (the Figure 4 caption lists R² = 0.95 for HandOccNet).
• DeltaDorsal reduces error under occlusion: Using only cropped dorsal images, the method reduces Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers ≥ 50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones.
• Smaller model, comparable or better accuracy: DeltaDorsal uses a DINOv3 ViT-L backbone of 300M parameters, while HaMeR uses a ViT-H backbone of 632M parameters. In the leave-one-subject-out study the system is competitive on several joints — for example Index MCP 6.81° ± 1.42 (vs. HaMeR 7.38° ± 2.07, HandOccNet 10.61° ± 3.25), Index DIP 6.92° ± 4.01 (vs. 8.39° ± 3.57 and 10.90° ± 3.29), and Ring DIP 5.16° ± 2.11 (vs. 4.58° ± 1.70 and 9.42° ± 2.31), where HaMeR is better on that joint. Results for the Pinky are cut off in the available text.
• HandOccNet collapses in these conditions: The authors report it was difficult to fine-tune or train end-to-end, often collapsed, and frequently defaulted to outputting a neutral hand position under self-occlusion.
• Dorsal feature changes scale with gesture type: In cosine-similarity maps between the reference and target frames, small gestures such as taps produce small changes concentrated around the knuckles and tendons; pinches produce large differences toward the bottom of the dorsal area; curls, bear claw, and fist produce large differences across the entire dorsal area.
• Robustness to skin tone: The system is reported as not meaningfully affected by skin color.
• Beyond visible motion: Dorsal features support detecting isometric "force click" — a surface press or pressing fingers together from an already-touching pose — where there is no discernible hand movement.

Methodology in Plain English

The researchers first measured how often fingers are hidden in egocentric views. They fit a MANO hand mesh to pose annotations from four public datasets, rendered the mesh, and used Z-buffering to work out which mesh faces a camera could actually see, grouping faces into index, middle, ring, pinky, thumb, palm, and dorsum. They then tested two strong existing models on their own new dataset to see how much accuracy is lost as fingers disappear.

For data, they recorded over 170,000 high-resolution frames (172,222 annotated frames) from 12 participants performing 17 gestures in three hand orientations (51 trials), using eight Vicon Vero motion-capture cameras for 3D ground truth alongside a 4K iPhone 12 Pro Max camera (3840 × 2160 at 30 FPS) positioned to mimic an egocentric view.

For the model, each user first provides a "reference" image of their relaxed, neutral hand. For any new frame, HaMeR predicts rough hand joints, which are used to align the reference and the current hand with a homography (estimated with RANSAC) and crop just the dorsum, resized to 384 × 384. Both crops go through the same frozen-mostly DINOv3 ViT-L featurizer with shared weights. A lightweight convolutional "change encoder" then combines four things: the current features, the reference features, their difference, and their per-patch cosine similarity. A regression head predicts only the 15 hand joint angles; shape comes from the reference capture and translation is either set to zero or taken from HaMeR. Training supervises joint angles and 3D joint positions, with random brightness, contrast, gamma, noise, and blur applied independently to the input and reference images for robustness. Evaluation uses leave-one-subject-out cross-validation (about 12,000 test frames per subject) with MPJAE and Procrustes-aligned mean per-joint position error (PA-MPJPE).

Why This Matters

Impact on research: The work reframes a signal that most pose estimators treat as noise — fine dorsal skin deformation — as a primary cue, and shows that an occlusion problem can be addressed by changing what the model looks at rather than by scaling up the model. It also contributes a 4K egocentric dorsal dataset and an open-source pipeline, giving other researchers a baseline for occlusion-robust hand tracking.

Real-world applications:

• XR headsets and smart glasses that need reliable free-hand input when the palm blocks the fingers.
• Gesture interfaces on wearables and mobile devices where a single forward-facing camera sees the back of the hand.
• Pressing a virtual surface or squeezing fingers together with no visible movement, enabling "force click" style input without extra hardware.
• Robust pinch and tap detection for confirming selections and issuing commands in spatial computing.

Industry relevance: Head-mounted and wearable camera tracking is central to AR/VR products, and the method achieves competitive accuracy with a 300M-parameter backbone versus a 632M-parameter one, which matters for on-device latency, power, and memory. Because it requires only a one-time reference capture per user and no wrist-mounted sensors, it fits existing egocentric camera setups rather than adding new hardware.

Future Directions

• Report and extend the promised analyses on skin tone, image resolution, and backbone choice, which the available text does not quantify.
• Reduce or remove the dependency on a separate hand pose estimator (HaMeR) for alignment and cropping, since the authors note any 2D/3D pose estimator or segmentor could serve the same role.
• Extend beyond pose to richer interaction — the paper demonstrates pinch, tap, and isometric force click, but accuracy figures for those downstream tasks are not reported in the available text.
• Address the limits the authors acknowledge: the gesture set is a limited subset of all possible hand poses, the camera was stationary during data collection, and occlusion from external objects or body parts was not modeled.

Target Audience

Researchers and practitioners in hand tracking, egocentric vision, and XR interaction design; engineers building gesture input for headsets, wearables, or mobile devices; and HCI researchers interested in on-body sensing and skin deformation as an interaction signal. Readers who want a self-contained introduction to occlusion challenges in egocentric hand pose estimation will also find the motivating study useful, though the modeling sections assume comfort with MANO, transformers, and pose error metrics.

Authors’ abstract

The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers &gt;= 50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface "click" without visible movement while minimizing model size.