DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction

AI summary

Overview

Research area: Computer vision — monocular 3D human motion reconstruction, generative diffusion models, and world-space (global) trajectory estimation from unconstrained video.

Technical level: Advanced. The paper assumes familiarity with diffusion models, diffusion transformers (DiT), SMPL/SMPLX parametric body models, camera intrinsics/extrinsics, and standard motion-capture evaluation metrics (MPJPE, PVE, RTE).

Scope: The paper introduces a two-stage diffusion framework that splits world-space human motion reconstruction into a camera-space estimation model and a world-space refinement model, evaluated on EMDB, RICH, and Egobody.

Affiliation and venue: The work is by Yufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, and Michael Zollhöfer, from Meta Reality Labs, the University of Pennsylvania, and Carnegie Mellon University. Posted to arXiv as 2603.03265v1 [cs.CV] on 03 Mar 2026 under a CC BY 4.0 license. Project page: yufu-wang.github.io/duomo/

What This Paper Is About

Recovering where a person moves in the real 3D world — not just how their body bends — from ordinary handheld video is hard, because the system must separate the person's motion from the camera's motion while tolerating blurry frames, occlusions, and people walking out of frame. Existing methods sit on either side of a trade-off: end-to-end models that predict world-space motion directly generalize poorly outside the studio data they were trained on, while "lift-then-fix" pipelines are robust to varied motion but produce physically implausible, drifting results. DuoMo's goal is to get both generalizability and global consistency by training two separate diffusion priors and connecting them with a known geometric step.

Key Contributions

1. DuoMo, a two-stage diffusion method that decouples human motion reconstruction into a camera-space estimation stage and a world-space refinement stage, rather than forcing one network to learn the whole 2D-to-world relationship.

2. A world-space motion model trained in per-video coordinate systems. Instead of aligning motion to a fixed canonical studio space (with a fixed "up" axis and flat ground at zero height), DuoMo defines world coordinates relative to each video's starting camera pose, avoiding an error-prone alignment step that fails on terrain like hills or stairs.

3. Mesh-vertex generation without parametric body models. The diffusion models generate the motion of mesh vertices directly rather than the low-dimensional parameters of a model like SMPL, using a 595-vertex LOD6 sparse mesh from MHR.

4. Test-time guided sampling with a 2D reprojection guidance term and a displacement guidance term for long occlusions, implemented as x0-guidance during DDIM sampling, plus a learned iterative converter from the sparse mesh to SMPLX for comparison with prior work.

Main Findings

• World-space accuracy improvements: DuoMo's W-MPJPE is 16% lower on EMDB and 30% lower on RICH than the second best method. Concretely, DuoMo w/ height reaches W-MPJPE 167.1 on EMDB and 80.4 on RICH, versus the best comparison method GENMO at 202.1 and 118.6 respectively.

• Camera-space state of the art: In camera-space reconstruction (Table 1), DuoMo obtains MPJPE 67.1 / PVE 78.2 on EMDB and MPJPE 51.4 / PVE 58.9 on RICH; with height conditioning these improve to MPJPE 59.5 / PVE 70.4 on EMDB and MPJPE 48.0 / PVE 55.2 on RICH.

• Height conditioning matters: Conditioning on ground-truth subject height during inference yields a 10% improvement on both MPJPE and PVE, resolving monocular scale ambiguity that Figure 3 illustrates.

• Robustness to occlusion: On the egocentric Egobody subset, DuoMo reaches W-MPJPE-Occ 193.1 / RTE-Occ 2.6 on occluded segments, versus 688.1 / 10.4 for the camera-model-plus-lifting baseline and 384.1 / 6.1 for GVHMR. Adding guidance improves these to 175.4 / 1.7.

• Dual priors beat either alone: In the ablation on EMDB, the camera-space model plus lifting gives WA-MPJPE 67.0 / W-MPJPE 180.2 but high jitter (32.6) and foot skating (9.2); a one-stage world-space model gives smoother motion (jitter 9.1, foot skating 4.8) but poor accuracy (W-MPJPE 445.1). DuoMo combines both: WA-MPJPE 66.0, W-MPJPE 167.1, jitter 8.7, foot skating 3.7.

• Sparse mesh is competitive with SMPLX parameters: Generating sparse mesh vertices (World-Model-Mesh: WA-MPJPE 65.7, W-MPJPE 164.8, RTE 1.1) slightly outperforms a world-space model outputting SMPLX parameters (70.1, 182.5, 1.3) on EMDB.

• Graceful degradation under camera noise: The paper models camera estimation error using a noise profile derived from comparing TRAM's camera estimation against iPhone-tracked "ground truth" on EMDB, applying accumulating per-frame Gaussian noise on the Lie algebra (rotation) and Euclidean coordinates (translation). The camera-model-plus-lifting baseline degrades rapidly — its foot-skating score explodes — while DuoMo's W-MPJPE degrades gracefully and foot skating stays minimal.

• Inference speed: On an H200 GPU, a 20-second, 30 FPS video takes 2s, 3s, 30s, and 1.5s for extracting keypoints, dense keypoints, image features, and diffusion respectively.

• SMPLX conversion accuracy: The learned sparse-mesh-to-SMPLX iterative network achieves MPJPE error under 5mm when tested on 3DPW.

Methodology in Plain English

The system starts by treating a video as evidence about the person from the camera's point of view, then separately reasons about where the person went in the world. That split is the central design choice.

Stage one — camera-space diffusion. For each frame, the method extracts two kinds of features: dense 2D keypoints corresponding one-to-one with the 595 mesh vertices, and image features from a frozen encoder. Rather than feeding raw 2D keypoint coordinates, it converts them into ray directions using the camera intrinsics and encodes those with positional embeddings; invisible keypoints get a learnable null embedding. These per-frame features condition a diffusion transformer (DiT) that generates the person's mesh pose in each frame's own camera coordinates. RoPE positional embeddings and windowed attention let the model process videos with thousands of frames without chunking. An optional height input, encoded and added to the diffusion time-step embedding, is used when available.

The lift. The camera-space output is transformed into world coordinates using estimated camera poses, where world coordinates are defined by the first frame's camera. This makes every error source — camera pose error, depth ambiguity, and the first model's own imperfections — show up as noise in the resulting proposal.

Stage two — world-space diffusion. A second DiT of the same architecture is conditioned on that noisy proposal and generates globally consistent world-space motion. Because world root positions are unbounded and grow over long sequences, the model predicts root velocity instead of absolute position, and the mesh is recovered by summing velocities. During training, conditioning tokens are randomly replaced with a learnable mask token to simulate the subject being unobservable, which lets the model hallucinate plausible motion through occlusions.

Test-time guidance. Two guidance terms correct known failure modes of velocity-based output: a 2D reprojection loss keeps the world-space motion aligned with the original video (reducing temporal drift), and a displacement loss — applied during occlusions longer than 2 seconds — forces integrated velocities to land where the person actually reappears. Both are applied as gradients on the predicted clean sample during DDIM sampling, which avoids backpropagating through the diffusion network.

Training. The camera-space model uses L1 losses on mesh vertices, root position, and derived joints. The world-space model uses L1 on vertices and root velocity, plus a contact loss applied only on frames where the foot touches the ground, which suppresses foot skating during training instead of as a post-hoc fix. Both models use the same DiT with 8 self-attention layers, d_model = 512, 8 attention heads, and a 2048-dim feedforward hidden size, trained with AdamW for one million steps at a learning rate of 10^-4, effective batch size 256, and sequence length T = 120. Data augmentation perturbs and masks keypoints at both the point and body-part level, and 50% of training samples use the maximum diffusion corruption level (k = 1000).

Why This Matters

Impact on research. DuoMo reframes world-space human reconstruction as a factorization problem rather than a single-network learning problem. It shows that injecting a known geometric operation (camera lifting) between two learned priors outperforms both direct end-to-end prediction and pure lifting, and it opens a path to modeling motion in other object categories directly from surface supervision without a parametric model. The finding that generating mesh vertices beats generating SMPLX parameters on reconstruction accuracy is a notable data point for the field.

Real-world applications

• AR/VR and egocentric capture: The Egobody experiments use video from a moving HoloLens, which maps directly onto headset-based reconstruction of a user's body and trajectory in a room or street.
• Fitness, sports, and coaching: Accurate world-space trajectory and foot contact are exactly what is needed to measure distance covered, gait quality, and foot skating in training footage shot on a phone.
• Character animation and film/games: Recovering a subject's global path from ordinary video reduces the need for mocap stages and hand-authored root trajectories.
• Robotics and imitation learning: Global human trajectories with globally consistent motion can serve as demonstration data for teaching robots to move through real environments.

Industry relevance. The work is authored by Meta Reality Labs alongside academic collaborators, and the target use case — reconstructing human motion from shaky, occluded, moving-camera video such as headset feeds — is central to AR/VR products. The reported inference cost (roughly 6.5 seconds of computation for a 20-second video on an H200, dominated by 30s of image feature extraction) also frames the practical deployment question of whether such a pipeline can run in an interactive setting.

Future Directions

1. Reducing camera pose sensitivity and inference cost. The noise experiments identify camera translation error as a major driver of baseline failure, and the timing breakdown shows image feature extraction dominating runtime; better camera estimation and cheaper feature extraction are natural next targets. The paper states that limitations are discussed in the supplementary material, which is not included in the provided content.

2. Extending the architecture beyond the human body. The authors suggest that because the model generates mesh vertices directly instead of parametric model parameters, the same architecture could learn motion models for other categories from surface supervision. Which categories and datasets this would require is not reported.

3. Closing the occlusion gap. Even DuoMo's best occluded-segment result (W-MPJPE-Occ 175.4, RTE-Occ 1.7) is much larger than its visible-segment error (90.4, 1.0), and the paper explicitly notes that generative plausibility alone "is not sufficient for accurate reconstruction under occlusion." Stronger conditioning on reappearance position is an open problem.

4. Generalizing height conditioning. Height conditioning provides the largest single accuracy gain reported (10% on MPJPE and PVE), but it assumes the subject's height is known. Robustly inferring or jointly estimating metric scale is an unresolved question the paper raises but does not answer.

Target Audience

This paper is most useful to researchers and graduate students working on 3D human pose and shape estimation, video-based motion capture, and diffusion-based generative models for geometry. It will also interest applied engineers building AR/VR body tracking, markerless motion capture, or character animation pipelines, particularly those who need world-space trajectories rather than view-relative pose. Readers without a background in diffusion sampling, camera geometry, or parametric body models will find the method sections difficult, though the two-stage framing and the ablation tables are accessible at a conceptual level.

Authors’ abstract

We present DuoMo, a generative method that recovers human motion in world-space coordinates from unconstrained videos with noisy or incomplete observations. Reconstructing such motion requires solving a fundamental trade-off: generalizing from diverse and noisy video inputs while maintaining global motion consistency. Our approach addresses this problem by factorizing motion learning into two diffusion models. The camera-space model first estimates motion from videos in camera coordinates. The world-space model then lifts this initial estimate into world coordinates and refines it to be globally consistent. Together, the two models can reconstruct motion across diverse scenes and trajectories, even from highly noisy or incomplete observations. Moreover, our formulation is general, generating the motion of mesh vertices directly and bypassing parametric models. DuoMo achieves state-of-the-art performance. On EMDB, our method obtains a 16% reduction in world-space reconstruction error while maintaining low foot skating. On RICH, it obtains a 30% reduction in world-space error. Project page: https://yufu-wang.github.io/duomo/