EgoHMP: Achieving Precise Human Motion Prediction in 3D Scenes via Egocentric Cues

AI summary

Overview

Research area: Computer vision and motion synthesis, specifically human motion prediction (HMP) in 3D scenes conditioned on egocentric video, head-motion sequences, and scene point clouds.

Technical level: Advanced. The paper assumes familiarity with diffusion models, contrastive/cross-modal representation learning, PointNet++-style point-cloud encoders, SMPL-X body parameterization, and GCN-based mesh refinement.

Scope (one sentence): EgoHMP replaces gaze signals with egocentric views plus head motion as a proxy for interaction intent, then uses the resulting intent map to drive a contact-aware diffusion decoder that predicts 10 future frames of physically plausible full-body motion in 3D scenes.

What This Paper Is About

Predicting where a person will move next inside a 3D room requires guessing what they intend to interact with. Prior work often uses eye gaze as that intent signal, but gaze is noisy, drifts, is ambiguous (different targets can produce similar gaze points), and needs specialized eye-tracking hardware.

This paper argues that the egocentric camera view plus the head-motion trajectory already encodes interaction intent more robustly, and it builds a full pipeline that turns those cues into an interaction probability map over the scene point cloud, then into a predicted trajectory and body pose.

Key Contributions

1. A new intent proxy for HMP in 3D scenes. The paper demonstrates that pairing egocentric views with head-motion sequences serves as a robust substitute for gaze when inferring human interaction intent, removing the dependence on eye-tracking hardware.

2. A vision-motion alignment encoder. Visual features (from a pre-trained DINOv2 backbone) and head-motion features are aligned through a combined objective that matches their local feature-distribution disparities (KL-divergence terms) and their global batch-wise semantics (a symmetric cross-entropy term over a temperature-scaled cosine similarity matrix).

3. An egocentric intent predictor with modality-aware modulation. Learnable modulation tokens are applied to each modality via Hadamard product and act as a gradient interception mechanism, letting the network down-weight unreliable cues (e.g. a wildly shifting camera view) and distill a single global intent token, which is then used to query scene points and produce an interaction probability map.

4. A contact-aware motion decoder. A global trajectory is predicted first and used as a self-prompt to relieve the optimization conflict of regressing global locomotion and local articulation in one diffusion model; a multi-stage denoiser then refines poses using explicit human-contact probabilities over 10,475 SMPL-X vertices, projected into a 22-joint kinematic space, before a GCN-based decoder synthesizes the final mesh.

Main Findings

• State-of-the-art on GIMO and GTA-IM. On GIMO, EgoHMP reaches Traj-Path 481 mm, Traj-End 545 mm, MPJPE-Path 114.9 mm, MPJPE-End 138.0 mm, TDE@1 161.3 mm, and 0.81% penetration. The strongest gaze-based baseline, GAP3DS, reports 575 / 623 / 141.2 / 171.4 / 210.5 / 0.95% on the same metrics.

• Large margin on GTA-IM. EgoHMP reports 522 / 623 / 129.6 / 177.2 with TDE@1 182.5 mm and 0.91% penetration, versus GAP3DS at 610 / 799 / 155.0 / 210.1 / 235.4 / 1.12%.

• Gaze-based methods beat non-gaze methods, but neither beats EgoHMP. Non-gaze baselines degrade sharply: Auxformer reaches 688 / 893 / 173.5 / 205.4 with 620.5 mm TDE@1 and 6.65% penetration on GIMO; MDP reaches 612 / 679 / 157.1 / 190.4 with 512.4 mm TDE@1.

• Scene awareness is decisive on Ego-Exo4D. Without 3D scene conditioning, EgoHMP (Base) reports MPJPE-Path 213.5 mm and MPJPE-End 276.2 mm — not the best in that column, where EgoAllo achieves 207.4 / 268.8. Under the scene-aware setting, EgoHMP (Full) reaches 128.6 / 157.0, compared with EgoAllo at 144.7 / 178.4, EgoCast at 153.6 / 193.7, UniEgoMotion at 158.5 / 192.5, and EgoEgo at 263.8 / 335.4.

• The egocentric view is the most important input in ablation. Removing the view signal (✗ F_V) on GIMO degrades MPJPE-Path to 168.9 mm and MPJPE-End to 216.8 mm; substituting gaze for the view yields 145.9 / 171.7 — better than no view, but worse than the full model's 114.9 / 138.0.

• Alignment loss and modulation both matter. Dropping the vision-motion alignment loss (✗ L_A) gives 132.6 / 161.4 mm; dropping the modality-aware modulation tokens gives 130.4 / 150.6 mm.

• Self-prompting and local scene context drive motion quality. Removing the self-prompted trajectory mechanism (✗ S_pm) gives Traj-Path 612 mm and MPJPE-Path 150.4 mm; removing the local scene conditioner (✗ F_local) gives 135.3 / 167.2 mm.

• Contact modeling reduces artifacts. Removing the multi-stage denoiser (✗ D_ms) gives 134.7 / 165.8 mm; removing the joint adaptive reconstruction loss gives 122.8 / 153.5 mm; removing the slide loss gives 115.4 / 140.9 mm. Figure 3(b) visualizes the refinement, showing floating artifacts in the initial stage corrected once human contact is learned.

• Robustness degrades gracefully under perturbation. On GIMO, clean input gives 481 / 545 / 114.9 / 138.0 with 0.81% penetration; Gaussian head-pose noise at σ=0.10 gives 505 / 572 / 128.6 / 155.7 with 0.83%; a 10 cm / 5° calibration offset gives 524 / 594 / 139.3 / 170.4 with 0.88%; dropping 50% of scene points gives 533 / 621 / 145.4 / 174.0 with 0.91%. Human Contact Distance rises from 43.5 mm (clean) to 56.2 mm when 50% of points are dropped.

• Gradient-flow analysis supports the modulation design. Figure 3(a) tracks visual and motion modality gradients over 70 sampled training epochs: under a stable view, the visual branch carries high gradient flow; under an erratic view cluttered with irrelevant objects, the modulation mechanism dampens visual gradients and shifts optimization toward head motion.

Methodology in Plain English

The setup is deliberately simple to state: the model sees 6 frames of egocentric video, 6 frames of head translation and orientation, and an RGB point cloud of the room, and must output the next 10 frames of body motion. Each body pose is represented as a root translation, a root orientation, and a 32-dimensional body embedding.

The pipeline has three stages.

First, make vision and motion speak the same language. Video frames pass through a pre-trained DINOv2 encoder, get temporal positional encoding, and go through a Transformer; head motion goes through a lighter Transformer. A dedicated loss pulls the two feature spaces together by matching how each modality's feature distributions vary across time, plus a contrastive term that pulls matching vision-motion pairs together and pushes mismatched pairs apart within a batch.

Second, decide what the person is about to touch. Scene points are encoded with PointNet++. Learnable per-modality scaling tokens act as a filter on backpropagated gradients, so the network can turn down the volume on whichever cue looks untrustworthy. Everything is fed into a Transformer alongside a special learnable "intent token," which ends up summarizing the goal. Scene points then attend to that intent, and each point receives a score combining a learned per-point MLP score and a similarity score against the global intent vector. Since the benchmarks have no intent annotations, supervision comes from a pseudo-label built from two priors: proximity to the ground-truth contact point (σ₁ = 0.3), and proximity to future head positions (σ₂ = 1.0), weighted by a squared temporal term so head position matters more as the interaction moment approaches.

Third, generate the motion. A transformer decoder first predicts a global trajectory using historical head motion, repeated intermediate states, the highest-probability contact point from the intent map, and distance-weighted scene features. That trajectory becomes a prompt so the diffusion model can concentrate on local pose. A cylindrical mask isolates scene points near the trajectory, and a small set of latent queries distills them into a compact local context. The denoiser works in two stages: a coarse pose prediction capturing the overall motion manifold, then a contact-aware refinement that predicts contact probabilities over 10,475 SMPL-X vertices, projects them into 22 joints using SMPL-X skinning weights, and injects them through Adaptive Layer Normalization to produce a residual correction. Training additionally penalizes joint errors more heavily where contact probability is high and penalizes inter-frame joint velocity at contact joints, which targets sliding. A GCN decoder with VPoser and SMPL-X produces the final mesh.

Why This Matters

For research, the paper reframes intent inference for motion prediction: instead of treating intent as an eye-tracking problem, it treats it as a cross-modal alignment problem between what a camera sees and how the head moves. It also names a concrete failure mode in diffusion-based motion generation — a single model jointly regressing global locomotion and local joint articulation — and offers a cheap structural fix (trajectory as self-prompt) rather than a new architecture. The explicit contact representation gives a measurable handle on floating, penetration, and foot sliding, all reported numerically.

Real-world applications:

• AR/VR and telepresence. Anticipating a user's next movement and interaction target lets virtual content pre-load or reflow before the user acts; the paper situates this work in the AR/VR line of research.
• Human-computer interaction and gestural interfaces. Predicting intent from head motion and a first-person camera avoids requiring eye trackers, which lowers hardware cost for consumer headsets.
• Embodied AI and robot motion planning. A robot with a head-mounted camera could infer where a nearby person is heading and what they will reach for, using the same egocentric-plus-point-cloud inputs.
• Character animation and simulation. Contact-aware generation reduces the cleanup animators normally do by hand on floating feet and scene penetration; the reported penetration rates (0.81% on GIMO, 0.91% on GTA-IM) quantify that gain.

Industry relevance: the method's inputs — headset camera, head pose, and a scanned room mesh — map directly onto what consumer XR hardware already provides, and the absence of a gaze dependency removes a hardware and calibration burden that has limited deployment of gaze-based predictors.

Future Directions

1. Extending beyond locomotion-scale interaction. The contact head predicts over 10,475 SMPL-X vertices, but the refinement is projected onto 22 body joints. Whether this can support fine-grained hand and finger manipulation of specific objects is not demonstrated in the reported results.

2. Closing the intent-annotation gap. Intent supervision here comes from a hand-crafted pseudo-label with fixed deviations (σ₁ = 0.3, σ₂ = 1.0) derived from ground-truth contact position and future head positions. The paper notes that existing benchmarks lack explicit intent annotations; how sensitive results are to that label design, and whether real intent labels would help, is left open.

3. Generalization beyond the evaluated scenes. Evaluation covers GIMO (14 scenes), GTA-IM (10 indoor scenes, 50 unique characters, 22 walking styles), and Ego-Exo4D. Behavior in outdoor, crowded, or multi-person environments is not reported.

4. Longer horizons and real-time deployment. The task is fixed at 6 observed frames and 10 predicted frames at 2 fps for GIMO. Whether the pipeline (DINOv2, PointNet++, a diffusion denoiser, and a GCN decoder) runs at interactive rates, and how error compounds over longer horizons, is not reported.

Target Audience

Researchers and graduate students working on human motion prediction, motion generation with diffusion models, and egocentric vision. It is also relevant to XR/HCI engineers evaluating whether gaze hardware is necessary for intent-aware interfaces, and to graphics practitioners interested in contact-aware pose refinement for reducing penetration and foot sliding. Readers should be comfortable with cross-modal contrastive learning and diffusion-based generative models; the paper is not an introductory treatment.

Authors’ abstract

The complex interactive dependencies between human behavior and 3D scenes make accurately predicting human intent a major challenge for human motion prediction (HMP). While recent studies use human gaze to infer intent, gaze signals suffer from ambiguity, drift, and a reliance on eye-tracking hardware. Research in cognitive science indicates that egocentric views inherently encode potential interactions, while head motion serves as a crucial precursor in motion planning.…