EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

AI summary

Overview

Research area: Computer vision — egocentric (first-person) 3D reconstruction, spanning hand–object interaction, SLAM-based camera tracking, monocular depth, and 6-DoF object pose estimation.

Technical level: Advanced. The paper assumes familiarity with MANO hand models, SLAM, pointmap-based reconstruction (DUSt3R/VGGT family), and standard pose-error metrics (MPJPE, ADD/ADD-S, nATE, AbsRel).

Scope: A full-stack system — capture rig, automatic annotation engine, a 50-hour dataset, a benchmark, and a single feed-forward model — that reconstructs camera, scene depth, hands, and manipulated objects together in one shared metric world frame from monocular egocentric video.

What This Paper Is About

Egocentric video captures what a person sees and does, but recovering the 3D story behind it is fragmented: one method estimates camera motion, another estimates hands, another estimates object pose, and their outputs disagree on scale and coordinate frame. This paper argues that these components are tightly coupled and should be recovered jointly, then builds the missing ingredient — large-scale, geometrically aligned supervision — by automatically annotating 50 hours of task-driven recordings. The result is a single model that, given RGB video and first-frame object masks, outputs camera trajectory, dense metric depth, both hands, and rigid object poses in a common world coordinate system in one forward pass.

Key Contributions

1. EgoX-Engine, an automatic annotation pipeline that couples geometrically verified object tracking (pre-scanned meshes, initial pose estimation, FoundationPose tracking, depth-agreement verification) with SLAM-based world alignment, producing consistent supervision for camera, scene, hands, and objects in a session-fixed metric frame.
2. EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion spanning workspace manipulation, cross-room interaction, and outdoor loco-manipulation, with 101 rigid objects, 270 task scripts, and second-level task captions — described in the paper's dataset table as the only compared dataset covering all three capture settings with 3D hand, 6-DoF, mesh, depth, mask, and caption annotations.
3. EgoX-Bench, a held-out, quality-controlled one-hour subset spanning the three activity settings for joint evaluation of camera, scene, hand, and object reconstruction on identical recordings.
4. EgoX, a unified feed-forward model that jointly predicts camera intrinsics and poses, dense depth, two-hand MANO states, object origins and orientations, and a single positive metric scale per sequence, trained with a geometrically coupled loss that backpropagates through the predicted camera transformations.

Main Findings

• World-space hands: EgoX achieves the lowest PA-, WA-, and W-MPJPE on all four benchmarks. On HOT3D, W-MPJPE drops from 56.30 mm (HaWoR) to 38.89 mm; on EgoX-Bench it drops from 328.81 mm (WiLoR + SLAM) to 113.24 mm. PA-MPJPE on HOT3D is 3.70 mm.
• Object 6-DoF pose: Best ADD, ADD-S, and AUC-S₅₀ on all four benchmarks. AUC-S₅₀ on EgoX-Bench is 76.59 versus 59.80 for GigaPose + MegaPose, even though that pipeline additionally uses metric CAD assets and per-frame masks. On HOT3D, ADD is 28.00 mm and ADD-S is 14.03 mm.
• Headline relative gains: Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0% (camera nATE on HOI4D; hand W-MPJPE and depth AbsRel on EgoX-Bench; AUC-S₅₀ on HOT3D).
• Depth: Lowest AbsRel and highest δ₁.₂₅ on all four benchmarks using its own predicted metric scale, while VGGT-Ω uses a scale fitted from DA3 Nested depth per sequence. On EgoX-Bench, AbsRel is 0.0460 versus 0.0806 for VGGT-Ω and 0.3972 for ReViV; δ₁.₂₅ is 97.37 versus 91.57.
• Camera motion is the partial exception: EgoX beats ReViV on nATE and RRE on all four benchmarks and leads on HOI4D (nATE 0.0066 vs 0.0212 for VGGT-Ω; RRE 0.78° vs 1.50° for VGGT). VGGT-Ω leads on HOT3D and EgoX-Bench, and DA3 Nested leads on H2O — the paper attributes VGGT-Ω's camera advantage to its large-scale geometric training.
• Scale prediction matters beyond rescaling: Replacing ScaleHead with oracle scale still worsens HOT3D W-MPJPE from 38.89 to 40.67 mm and AUC-S₅₀ from 73.71 to 60.87.
• Explicit depth prediction helps entities: Removing DepthHead raises W-MPJPE to 67.53 mm and lowers AUC-S₅₀ to 41.09.
• Efficiency: EgoX processes 150 frames in 14.60 s (10.28 FPS) on one NVIDIA A800, versus 25.37 s for HaWoR, 166.54 s for Any6D, and 134.03 s for GigaPose + MegaPose.

Methodology in Plain English

Capture. Recordings use a head-mounted rig (HOMIE Gen2) with a calibrated front stereo pair for metric depth and four fisheye cameras for SLAM camera tracking. Each recording gets a fixed "session frame" from SLAM, and everything — scene, hands, objects — is expressed in that frame.

Object references. Before recording, target objects are scanned inside a dome of 31 synchronized cameras; segmented views produce metric meshes (SAM 3 masks plus MV-SAM3D), with an AprilTag board providing real-world scale.

Annotation. For each interaction, the engine seeds an object's pose from a first-frame mask plus stereo depth, tracks it through the interaction window with FoundationPose, then keeps only tracked sequences where the placed model agrees with observed depth. Surviving tracks are transformed through SLAM poses into the session frame. Hands are recovered by triangulating 3D keypoints across views and fitting MANO with reprojection, depth, and temporal terms — hand shape shared per recording, pose/position/orientation per frame.

Modeling. DINOv3 ViT-L/16 patch features feed a 24-block Transformer (hidden size 1024; 19 blocks exchange all tokens across views, 5 operate on compact register and entity tokens). Dedicated camera, left-hand, right-hand, and object tokens carry identity across views, with first-frame mask descriptors associating target objects. All outputs are lifted into a first-frame world frame using predicted intrinsics and extrinsics, then a shared ScaleHead predicts one positive scale factor applied consistently to scene points and to camera, wrist, and object positions.

Training signal. Four loss terms — local (camera and view-local entities), dense (depth and scene points), coupled (hand–object geometry composed through the predicted camera), and scale — with each term applied only where a given dataset supplies the needed labels. This lets partially annotated public datasets contribute. Public data (HOT3D, HOI4D, H2O, HO-Cap, H2O-3D, DexYCB) is cleaned and harmonized into a common coordinate and scale convention; the full pool is about 280 hours including real and synthetic sources.

Why This Matters

Research impact. The paper argues that the bottleneck in egocentric 3D reconstruction is aligned supervision, not model capacity, and it supplies both the annotation machinery and a benchmark where all four sub-problems are scored on the same clips. The coupled-loss idea — supervising hand and object geometry through the model's own predicted cameras — gives camera estimation a gradient signal from interaction geometry rather than only from trajectory labels.

Real-world applications:

• Robot learning from demonstration: world-frame hand and object trajectories from ordinary human activity are directly usable as manipulation training data for embodied agents.
• AR/VR assistance and authoring: placing virtual content that respects real scene geometry, real hands, and real object poses without a calibrated studio.
• Skill training and assessment: objective, metric-space measurement of how someone manipulates tools across a workspace or between rooms.
• Markerless motion capture in the wild: head-mounted capture for hands and objects in settings where dome or mocap installations are impractical, including outdoors.

Industry relevance. The pipeline deliberately reduces reliance on calibrated multi-camera or motion-capture infrastructure, which is the main cost driver in scaling interaction data. A single feed-forward model at 10.28 FPS on one A800, replacing staged hand and object pipelines that run at 0.90–1.12 FPS, is a practical argument for deployment in data-engine and teleoperation settings.

Future Directions

• Object mesh prediction. The current formulation assumes rigid objects and does not predict their meshes; the authors list mesh prediction as future work.
• Multiview inputs. Also explicitly named as future work — the current model is monocular, and adding views could improve the depth and camera accuracy where geometry-only baselines currently lead.
• Articulated objects. The paper notes related work (ArtHOI) handles articulated objects while EgoX models rigid ones, leaving doors, tools with joints, and containers as an open extension.
• Closing the camera gap. Since VGGT-Ω leads camera accuracy on HOT3D and EgoX-Bench and DA3 Nested leads on H2O, a clear open question is whether more diverse geometric training data or longer temporal context can make the joint model uniformly best rather than best on HOI4D only.

Target Audience

Researchers and engineers working on egocentric vision, hand–object interaction, and embodied AI who need either (a) a jointly annotated dataset and benchmark for world-frame interaction reconstruction, or (b) a reference architecture for combining camera, depth, hand, and object estimation in one model. It is also relevant to data-collection teams building large-scale manipulation corpora, since the annotation engine is the paper's core practical claim. Readers without background in MANO, SLAM, and pose-error metrics should expect a steep learning curve.

Authors’ abstract

Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.