EventEgoHands++: Event-based Egocentric 3D Hand Mesh Reconstruction with Real Dataset

AI summary

Overview

• Research area: Computer vision — event-based (neuromorphic) sensing, egocentric (first-person) vision, and 3D hand mesh reconstruction.
• Technical level: Intermediate. The architecture uses standard components (an instance segmentation detector, a CNN encoder, self/cross-attention, and the MANO parametric hand model), but the paper assumes familiarity with event cameras, event-frame representations, and hand pose benchmarks.
• Scope in one sentence: The paper proposes EventEgoHands++, a two-stage framework for reconstructing 3D hand meshes from egocentric event-camera data, and introduces a real-world egocentric event dataset (EEH-R) plus refined synthetic annotations on N-HOT3D.

What This Paper Is About

RGB and depth cameras struggle to capture hands in low light and during fast motion, so event cameras — which respond only to brightness changes and offer high dynamic range and high temporal resolution — are attractive. Applying them from an egocentric viewpoint is hard because the camera wearer's own motion floods the sensor with background events, and the previous approach (EventEgoHands) used a binary hand mask that could not tell left from right, so it always predicted both hands even when only one or none was visible. This paper fixes that by detecting hands as distinct left/right instances and by letting the attention mechanism adapt to which hands are actually present, and it adds the first real-world egocentric event dataset so the task can be evaluated on real sensor data, including in low light.

Key Contributions

1. EventEgoHands++ framework: an event-only 3D hand mesh reconstruction pipeline for egocentric video that explicitly handles hand visibility and left/right hand identity under severe camera-wearer motion.
2. Two methodological components: a Hand Detector that jointly estimates instance-level bounding boxes and masks for the left and right hands, and Adaptive Attention that dynamically activates inter-hand attention only when the corresponding hands are visible.
3. EEH-R dataset: described as the first and largest real-world event-based egocentric hand dataset, with approximately 1M annotated frames captured under both well-lit and low-light conditions, plus an extension of the synthetic N-HOT3D dataset with refined masks and newly added bounding-box annotations, enlarging it to approximately 480K annotated frames.
4. Evaluation against baselines on both synthetic and real data, demonstrating consistent improvements over existing methods.

Main Findings

• Synthetic benchmark gains: On the synthetic N-HOT3D dataset, EventEgoHands++ reduces mean MPJPE by 21.82 mm (33.7%) and MPVPE by 20.57 mm (34.0%) compared with the best existing method.
• Real-world gains: On the EEH-R dataset, it reduces MPJPE by 7.90 mm (18.8%) and MPVPE by 7.13 mm (18.1%).
• Instance-level detection matters: The authors argue that a binary mask (as in the earlier EventEgoHands) cannot encode left/right identity, which produces invalid two-hand outputs when one or neither hand is visible and degrades estimated inter-hand relative positions. Adopting an instance segmentation framework — jointly learning bounding boxes and masks rather than masks alone — is described as more robust because mask-only prediction tends to be unstable.
• Attention should depend on visibility: Unconditional cross-attention between the two hand branches forces feature exchange with absent or misidentified hands. The authors contrast this with masked attention, which still executes the cross-attention computation and can corrupt the features of the detected hand; Adaptive Attention instead switches which operations are applied.
• Dataset scale and composition: N-HOT3D contains 480,120 frames (334,190 training, 83,760 validation, 62,170 evaluation) from nine subjects, 136 sequences, and 4.4 hours, having grown from 447,704 frames in the earlier version. EEH-R contains 1,019,716 frames (636,433 training, 164,727 validation, 218,556 evaluation) from eight subjects, 85 sequences, and 2.36 hours.
• Synthetic–real domain gap: N-HOT3D involves more locomotion and head motion than EEH-R, producing a larger number of events per frame; the paper analyzes the per-frame event count and the temporal distribution of events within a frame to characterize this difference.
• Evaluation on real data in low light: EEH-R includes both well-lit (average illuminance of 457 lux) and dark (average illuminance of 3.5 lux) recordings, enabling the first egocentric evaluation on real event data under the low-light conditions that motivate event cameras.

Note: The paper content provided is truncated partway through Section IV-B, so the detailed results tables, the hand segmentation performance numbers, the ablation studies, and the failure case analysis referenced in Sections V and VI are not available in the supplied text and are therefore not reported here.

Methodology in Plain English

The pipeline has two stages.

Stage 1 — Hand extraction. An event frame is built from the raw event stream using LNES (Locally-Normalized Event Surfaces), a representation that preserves temporal information through temporal weighting. A hand detector — YOLO26, from the YOLO family — takes this frame and predicts instance-level bounding boxes and segmentation masks, along with hand-side labels (left/right) and confidence scores. The masks are applied to the event frame to keep only the events falling inside each hand region, producing separate masked event frames for the left and right hand. Validity indicators record whether each hand was detected.

Stage 2 — Hand reconstruction. A shared EfficientNetV2-S backbone encodes each masked hand frame into a spatial feature map. Adaptive Attention then behaves according to three conditions:

• Both hands detected: each hand's features are refined by self-attention (intra-hand relationships), then the two feature maps exchange information through bidirectional cross-attention (inter-hand dependencies), with each hand as query and the other as keys and values.
• Single hand detected: cross-attention is skipped and only self-attention is applied to the visible hand.
• Neither hand detected: processing for that sample is skipped.

The refined features are pooled into a vector and mapped by a linear layer to MANO parameters — a 45-dimensional pose vector, a 10-dimensional shape vector, a 3-dimensional translation, and a 3-dimensional rotation — which the MANO model decodes into 20 hand joints and 778 mesh vertices per hand. Training combines four losses: an L1 joint loss, an L2 inter-hand loss on relative left–right joint offsets, an L1 vertex loss, and an L2 MANO parameter loss, optimized with AdamW.

Data collection. For N-HOT3D, the event simulator v2e was applied to a subset of the Aria glasses data in the HOT3D dataset (nine subjects), with outputs at 346x260 pixels to match the DAVIS346 camera, and ground-truth masks and bounding boxes obtained by projecting the provided 3D meshes. For EEH-R, the authors built a capture rig combining a head-mounted DAVIS346 event camera (with synchronized grayscale frames at 30 fps), MoCap gloves with 16 IMU sensors per hand providing 16 joint positions per hand (wrist plus three joints per finger), and an OptiTrack system of 16 cameras (6 PrimeX22 and 10 PrimeX13) recording ground-truth hand and camera poses at 120 fps; the average 3D calibration error of the motion-capture system was below 0.1972 mm. Plain fabric gloves worn over the MoCap gloves conceal the sensors while keeping hand texture similar to bare hands. Two scenes (kitchen and workspace) and two lighting conditions were recorded with eight subjects (6 male, 2 female), with each sequence lasting approximately 2 minutes. MANO annotations were obtained by fitting the model to the MoCap joint positions; 2D masks were generated automatically with SAM3 on grayscale images for well-lit scenes (198,410 annotated frames) and manually for 1,000 event frames in dark scenes.

Why This Matters

The work pushes egocentric hand reconstruction into sensing conditions where conventional cameras fail, and it supplies the real-world data that was previously missing for this specific task. It argues that treating hands as distinct instances — rather than as a single binary mask — is necessary for correct behavior when hands enter and leave the field of view, which is the normal case in first-person capture.

Real-world applications:

• AR/VR and immersive interfaces: accurate hand meshes from a head-mounted device enable natural gesture interaction and rendering of the user's own hands.
• Human–robot interaction: reconstructing a wearer's hand pose supports teleoperation, shared manipulation, and safe collaboration.
• Low-light and fast-motion activity capture: recording hands during cooking, tool use, or assembly tasks where lighting is poor or motion is rapid.
• Edge and wearable platforms: event cameras offer low power consumption and memory-efficient sensing, which the paper frames as well suited to deployment on edge devices and wearable platforms.

Industry relevance: the results matter to XR headset and smart-glasses makers, event-camera sensor vendors, robotics companies building first-person interfaces, and wearable-device developers. The released datasets and code (https://ryhara.github.io/EventEgoHandsV2/) also give the community a benchmark for a task that previously had no real-world egocentric event data.

Future Directions

• Robustness of hand detection under egocentric motion: the detector is the gatekeeper of the whole pipeline, so improving its reliability under severe background event clutter and partial hand visibility is a direct open problem.
• Closing the synthetic-to-real gap: N-HOT3D shows a higher number of events per frame than EEH-R due to greater locomotion and head motion; understanding and modeling this distribution shift could improve transfer.
• Extending to hand–object interaction: EEH-R contains sequences with bowls, spoons, cups, bottles, cutting boards, knives, notebooks, keyboards, mice, pens, and glasses, but the method reconstructs hands alone, leaving joint hand–object reconstruction unexplored.
• Temporal and on-device modeling: the paper notes that 4D hand mesh reconstruction incorporating temporal information has advanced in image-based settings, and that event cameras suit edge deployment — both suggest natural extensions not evaluated here.

Target Audience

Researchers and practitioners in event-based vision, egocentric and first-person vision, and 3D hand pose and shape estimation; engineers building XR/VR hand tracking or robotics interfaces; and dataset builders interested in synchronized multi-sensor capture with motion-capture gloves and optical tracking. Readers looking for detailed benchmark tables or ablation numbers will need the full paper, since the supplied content is truncated before the results sections.

Authors’ abstract

3D hand mesh reconstruction is a challenging yet essential task for downstream applications, including human-robot interaction and AR/VR. Although conventional cameras have been widely adopted for this task, methods that rely on them struggle in low-light environments and under severe motion blur. To address these limitations, event-based cameras have recently attracted attention for their high dynamic range and high temporal resolution. However, applying event cameras to egocentric hand reconstruction remains challenging because camera wearer's motion produces dense background events that obscure hand-specific signals. Although the first egocentric event-based approach mitigates this issue using hand segmentation, its binary hand mask does not distinguish between left and right hands. As a result, the model lacks instance-level hand information and predicts both hands even when only one or neither hand is present. This limitation leads to incorrect inter-hand relationships and degraded reconstruction accuracy. In this paper, we propose EventEgoHands++, a framework for event-based 3D hand mesh reconstruction from an egocentric viewpoint. The proposed method incorporates a Hand Detector that estimates instance-level bounding boxes and masks for both the left and right hands. Moreover, we introduce Adaptive Attention, which dynamically gates the attention based on these detection results to accurately learn the spatial relationship and mutual interactions between the hands. To train and evaluate our framework, we extend the synthetic N-HOT3D dataset and newly construct EEH-R, the largest real-world event-based egocentric hand dataset to date, comprising approximately 1M annotated frames captured in environments including low-light conditions. Extensive experiments on both synthetic and real datasets demonstrate that our method consistently outperforms the baselines.