EgoExoMoCap: Distributed Ego-Exo Human Motion Capture

AI summary

Overview

Research area: Computer vision / human motion capture, specifically monocular and multi-device full-body pose and motion estimation from head-mounted devices (HMDs).

Technical level: Advanced. The paper assumes familiarity with SMPL body models, 6D rotation representations, vision transformers, 2D keypoint detectors, SLAM tracking, and pose-estimation metrics (MPJPE, MPJVE, jitter).

Scope: A distributed ego-exocentric motion capture framework (EgoExoMoCap) that fuses continuous head (and optional wrist) tracking from one person's smart glasses with intermittent camera observations from another person's smart glasses to reconstruct full-body motion in the wild.

What This Paper Is About

Full-body motion capture outside the lab has previously been tackled in two separate ways: egocentric tracking, where a device estimates the pose of the person wearing it, and exocentric tracking, where a camera estimates the pose of somebody else in the scene. EgoExoMoCap argues these two paradigms have "largely been explored in isolation" and asks what happens if several people wearing camera-equipped glasses (such as Aria glasses) act simultaneously as both motion subjects and mobile observers of each other. The goal is a lightweight, portable system that requires nothing more than two or more people each wearing a pair of smart glasses, yet still produces robust global-space body motion in the presence of occlusions and out-of-view moments.

Key Contributions

1. A lightweight, portable in-the-wild motion capture solution that relies only on a set of people wearing HMDs such as Aria glasses, in contrast to bulky multi-camera rigs or obtrusive motion-capture suits.
2. A multi-modal framework combining heterogeneous ego- and exocentric signals — continuous head trajectories plus intermittent image features — that works with as few as two subjects and scales naturally to multi-subject, multi-observer setups.
3. A wearer-conditioned ray representation for exo-ego fusion, in which 2D exocentric keypoints are lifted with the observer pose, scaled by the observer–wearer head distance, and canonicalized into the wearer's head-local frame.
4. A learned visibility gating mechanism driven by DINOv3 image context, producing per-joint confidence scores that modulate the exocentric rays so the network falls back on egocentric tracking when the exocentric signal is unreliable.
5. Evaluation on indoor and outdoor sequences from two in-the-wild datasets, Nymeria and EgoHumans, reporting state-of-the-art performance in real-world scenarios.

Main Findings

• Best accuracy on Nymeria under both tracking setups. In the three-point setup EgoExoMoCap reaches 5.72 cm MPJPE (2.72 cm upper body, 10.05 cm lower body), 12.77 cm/s MPJVE and 2.16 jitter; in the one-point setup it reaches 8.28 cm MPJPE (6.10 upper, 11.44 lower), 17.23 cm/s MPJVE and 2.47 jitter. The strongest baseline, PromptHMR+EgoPoser, reports 6.47 cm MPJPE (three-point) and 9.03 cm (one-point).
• The only metric where a baseline wins is jitter. RPM achieves the best jitter on Nymeria (1.29 in three-point, 1.15 in one-point), which the authors hypothesize comes from its PCAF module balancing smoothness against a potentially minor adherence to input signals.
• Egocentric-only methods fail on invisible body parts. They return plausible motions (often with low jitter) but "can hardly reconstruct lower body movement in a faithful way," e.g. kneeling or sitting poses.
• Exocentric-only methods struggle with occlusion and camera motion. PromptHMR, even when fed ground-truth wearer root position and rotation, reports 13.48 cm MPJPE (three-point and one-point alike on Nymeria), and its finetuned variant 10.65 cm.
• Naive ego-exo fusion degrades smoothness. The custom PromptHMR+EgoPoser baseline improves MPJPE over either input alone but reports higher MPJVE and jitter than the proposed early fusion, which the authors attribute to the estimate relying too heavily on unreliable exocentric input.
• The EgoHumans cross-dataset results confirm the trend. EgoExoMoCap reports 7.62 cm MPJPE (three-point) and 11.54 cm (one-point), versus 8.62 cm and 13.27 cm for PromptHMR+EgoPoser.
• Multiple observers help. On the EgoHumans multi-observer subset, the multi-observer variant improves to 7.11 cm MPJPE (7.11 / 5.89 upper / 8.87 lower, 37.31 MPJVE, 2.58 jitter) in three-point setup versus 8.80 cm for EgoExo-single-observer, and to 10.48 cm in one-point versus 13.70 cm. Figure 5 describes single-observer MPJPEs of roughly 9 to 11 cm, dropping to 6.43 cm once views are aggregated.
• Triangulated-keypoint exo tokens are worse than learned fusion. EgoExo-triangulation reports 8.49 cm MPJPE (three-point) and 13.34 cm (one-point), because it does not adequately account for per-view confidence.
• DINO-based scores beat detector confidence. Removing learned gating raises MPJPE from 5.72 cm to 6.30 cm; substituting ViT-provided confidences gives 6.29 cm but remains worse than learned gating "especially for lower-body accuracy," and score masking gives 6.32 cm.
• Egocentric signal matters most. Removing the ego inputs raises MPJPE to 11.45 cm; removing exocentric images gives 7.53 cm; replacing EgoNet region proposals with a YOLO detector gives 6.43 cm.
• Ray design choices are validated. Skipping depth scaling gives 6.26 cm, keeping rays in world space gives 6.99 cm, keeping them in exo space gives 6.14 cm, and omitting lifting entirely gives 6.26 cm.
• Low sensitivity to the coarse ego prior. Perturbing EgoNet joint positions with Gaussian noise of σ = 1/2/5/10 cm increases final full-body MPJPE by only 0.005/0.02/0.14/0.54 cm respectively.

Methodology in Plain English

The setup is deliberately minimal: one wearer whose motion is being reconstructed and one observer who happens to be nearby and looking at the wearer. Both wear glasses with cameras and inertial sensors. The system receives, per timestep, the observer's RGB image plus both people's head position and orientation, and optionally the wearer's left and right wrist positions and orientations.

The pipeline works in stages. First, the wearer's own head (and optional wrist) tracking is normalized and encoded into a 60D egocentric feature vector, which an ego-only temporal network called EgoNet turns into a rough SMPL body pose. Because generic bounding-box detectors struggle under heavy occlusion, this coarse pose is projected into the observer's image to define the region where the wearer actually is.

Inside that region, ViTPose detects 13 body keypoints. Rather than feeding raw 2D coordinates to the network (which would entangle the observer's viewpoint and camera parameters), each keypoint is unprojected into a 3D ray using the observer's intrinsics and extrinsics. The ray direction is normalized so it is invariant to the observer turning their head, then scaled by the Euclidean distance between the two people's heads to recover depth, anchored at the observer's head to produce a proxy 3D endpoint, and finally expressed in the wearer's head-local coordinate frame. This makes the representation independent of where in the world the wearer is and which way they face.

Because 2D detections lie under occlusion, a DINOv3 CLS token (768-dimensional) extracted from the same cropped region is mapped by a small two-layer MLP (768 → 512 → K) and a sigmoid into per-joint confidence scores in [0,1]. These scores element-wise scale the rays; when the wearer is poorly observable the gate suppresses the exocentric evidence automatically. The authors note the learned gate can recognize cases such as the wearer being behind furniture even when the 2D detector still returns high-confidence but wrong detections.

The gated rays form an "Exo Token" and the egocentric signal plus EgoNet pose form an "Ego Token." A Spatial Transformer fuses them per frame, but only the ego token's output is retained, enforcing an egocentric inductive bias so the network still works when exo evidence is fully suppressed. The sequence of fused features is then processed by a Temporal Transformer over a window of 96 frames with bidirectional self-attention, and two MLP heads decode root orientation (6D) and 21 body joint rotations (126 values, 6D each). Root translation is not predicted; it is recovered analytically by subtracting the head-to-root offset obtained through forward kinematics from the known HMD head position.

Training is two-stage: EgoNet is trained first on egocentric signal alone, then frozen while the full fusion network trains end to end. The DINOv3 backbone stays frozen and only the gating MLP is trained. The objective is a weighted sum of three L1 losses — root orientation (λ = 0.02), body joint rotations (λ = 1.0), and 3D joint positions from SMPL forward kinematics (λ = 1.0). The model does not predict SMPL shape parameters; shapes are supplied as input, either subject-specific or the SMPL mean identity.

Why This Matters

Impact on research. The paper reframes exocentric and egocentric sensing as complementary rather than competing, showing that mutual observation among multiple HMD wearers is a practical substitute for studio multi-camera rigs. It also contributes a geometric representation (wearer-conditioned rays with depth scaling and ego-space canonicalization) and an image-context-based gating scheme that other ray-based pipelines could adopt.

Real-world applications:

• Scalable collection of real-world whole-body motion and interaction data for embodied AI and robotics, without a studio.
• VR/AR avatars and telepresence, where several headset users see each other and want full-body representation.
• Sports and outdoor activity capture, as illustrated by the tennis, fencing, basketball and badminton sequences in EgoHumans.
• Human-computer interaction and interactive agents that need to understand coordinated multi-person behavior.

Industry relevance. The work comes from Meta Reality Labs with ETH Zürich, and it is built around Aria glasses, a real hardware platform. It points toward motion capture as a feature of consumer smart glasses rather than a specialist service, which matters for anyone building VR/AR social platforms, robotics data pipelines, or wearable sensing products.

Future Directions

• Real-time and online tracking. The authors explicitly state the current work targets in-the-wild ground-truth motion acquisition rather than online, real-time tracking, and call real-time applications an exciting direction for when hardware and tooling can support it.
• Shape estimation. The method assumes subject shape is provided when computing joint positions; the authors suggest it could instead be estimated from images or sensor-based calibration.
• Failure under person-on-person occlusion and long out-of-view intervals. The authors observe failures when the wearer is largely occluded by another subject, which can confuse the DINO-based visibility scores, and note that long out-of-view intervals or persistently unreliable exo signals can lower accuracy, especially for the lower body.
• Physical plausibility and scene context. Better modeling of foot-ground contact and body-self penetration, together with stronger incorporation of scene context, is suggested to improve motion realism.

Target Audience

Researchers and engineers in computer vision, VR/AR, and embodied AI who work on human pose and shape estimation, multi-person capture, or wearable sensing. It is most useful to readers already comfortable with SMPL-based motion representations, transformer architectures, and standard pose benchmarks, and of direct practical interest to product teams evaluating smart glasses (particularly Aria) as a motion capture platform.

Authors’ abstract

Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data, which is crucial for applications in embodied AI and VR/AR. Existing approaches focus on either egocentric body tracking, estimating the motion of the subject wearing the device, or exocentric tracking, capturing the movements of people in the wearer's surroundings. So far, these two paradigms have largely been explored in isolation. In this paper, we propose a novel distributed framework that jointly leverages ego- and exocentric multi-modal signals for human motion estimation from HMDs. Unlike traditional motion capture systems requiring bulky multi-camera setups or obtrusive mocap suits, our approach, EgoExoMoCap, is as simple as two (or more) people, each wearing a pair of smart glasses. The method leverages head (plus potentially wrist) tracking signals for accurate estimation of global motion in the 3D world and combines context-aware image features based on DINOv3 to achieve robustness in the presence of noise and occlusions. Extensive experiments on two in-the-wild datasets show that our approach can robustly reconstruct motion even in challenging scenarios.