OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching

AI summary

Overview

• Research area: Computer vision / 3D human motion capture — recovering full-body SMPL motion from sparse, heterogeneous wearable and vision signals.
• Technical level: Intermediate. The paper assumes familiarity with SMPL body models, diffusion/flow-matching generative models, and standard motion-capture error metrics (MPJPE, PA-MPJPE, jerk, foot skating), but the core ideas are explainable without deep math.
• Scope: One paper proposing a single flow-matching framework, OmniPoser, that ingests arbitrary subsets of 3D keypoints, IMU rotations, and head-mounted 6-DoF poses and outputs full-body SMPL motion in real time.

What This Paper Is About

Capturing full-body motion "in the wild" is hard because real sensors are noisy, frequently occluded, and prone to dropout, and each hardware setup (sparse IMUs, head-mounted displays, monocular keypoints) produces a different kind of partial observation. Existing methods are typically built for one fixed sensor configuration and must be retrained for any new one, and the diffusion-based ones are too slow for real-time use. OmniPoser's goal is to recover SMPL motion from any combination of these heterogeneous inputs using a single model that runs at over 700 FPS.

Key Contributions

1. A universal Conditional Flow Matching framework for motion recovery that accepts heterogeneous sparse observations — 3D keypoint positions, joint rotations, and combinations thereof — as input, rather than being tied to one sensor layout.
2. A cross-modal masking mechanism plus a Masked Conditioning Encoder (MCE) that replaces unobserved joints with learnable per-joint embeddings instead of zeros, so the network can distinguish which joints are missing rather than treating all absent joints identically.
3. A decoupled dual-stream generative backbone — a temporal convolutional UNet for the low-dimensional global trajectory and a cross-attention Diffusion Transformer (DiT) with Rotary Position Embedding for high-dimensional local body pose — trained with physics-grounded auxiliary losses, requiring only a single Euler ODE step at inference.
4. Demonstration of competitive performance against specialist state-of-the-art methods and real-time inference across diverse configurations and in-the-wild datasets, including Nymeria, RICH, AIST++, TotalCapture, AMASS, 3DPW and Ego-Exo4D.

Main Findings

• Upper-body keypoints (Setting 1, AMASS): OmniPoser achieves 32.1 mm MPJPE, versus 34.8 mm for RoHM and 88.0 mm for HuMoR, and the lowest skating ratio at 0.031.
• In-the-wild upper-body (Nymeria): OmniPoser records 21.1 mm MPJPE against RoHM's 59.3 mm — the paper describes this as roughly a 64 percent reduction — with lower acceleration error.
• Random joint masking (up to 15 of 22 joints masked, AMASS): Ours-S reaches 35.7 mm MPJPE versus RoHM's 38.2 mm and HuMoR's 80.5 mm, with contact accuracy 0.97 and skating 0.018. Ours-U is reported at 49.3 mm.
• Head-mounted 6-DoF only (Setting 2, AMASS): Ours-S at 83.2 mm MPJPE and Ours-U at 86.2 mm, compared with EgoAllo's 119.7 mm (about 30 percent better) and EgoEgo's 167.4 mm (about 50 percent better). EgoAllo is better on head translation (6.2 mm versus OmniPoser's 17.1 mm), which the authors frame as a trade-off for flexibility.
• Cross-domain (RICH): Ours-S improves MPJPE by 25 percent to 131.6 mm; all methods degrade in contact accuracy (Ours-S at 0.71) due to floor-height ambiguity in RICH unseen during AMASS training.
• Egocentric wild (Nymeria): OmniPoser reduces MPJPE by roughly 60 percent over EgoAllo (40.7 mm versus 103.0 mm) and reaches 0.94 contact accuracy versus EgoAllo's 0.63.
• Six IMUs plus keypoints (Setting 3, AIST++): Best PA-MPJPE (19.4 mm for Ours-S, 21.4 mm for Ours-U) and MPJPE (26.0 mm / 29.3 mm). Ours-S's jerk of 0.29 km/s³ is described as 5x lower than SIP (1.50) and 8x lower than RobustCap (2.49).
• TotalCapture: Ours-S outperforms RobustCap3D in MPJPE (36.2 mm versus 42.3 mm), while SIP is better on global translation (5.42 mm versus 15.9 mm) and hip/shoulder rotation (10.6° versus 13.3°) because it uses stereo-derived metric depth and physical pelvis IMU readings.
• Nymeria (Setting 3): OmniPoser reports 16.6 mm MPJPE and 0.048 m/s³ jerk.
• Ablation on the MCE: Replacing learnable per-joint embeddings with zero-padding worsens MPJPE from 32.1 mm to 88.1 mm, a degradation of 56.0 mm, and raises acceleration error to 4.0 and skating to 0.053.
• Sampling diversity: Joint-position standard deviation over 100 samples grows as inputs get sparser and as ODE steps increase — 58.0 mm for head 6D, 19.4 mm for random ≤15 masked, 18.9 mm for upper-body XYZ, 15.0 mm for 6IMU rotation+XYZ, and 1.7 mm for full joints at 1 ODE step; 92.0 / 35.3 / 31.4 / 14.1 / 1.7 at 30 steps. Performance does not meaningfully improve with more steps, so single-step is used for benchmarking. Users can trade speed for diversity: 5 steps at 250 FPS, 30 steps at 80 FPS.
• Speed: Over 700 FPS, with the single Euler step giving more than a 16x speedup over the 100+ diffusion steps used by baselines such as RoHM.
• Benchmark count discrepancy: The abstract and conclusion state "three settings and seven benchmarks," while the introduction states quantitative evaluation on five benchmarks. Setting 4 (3DPW and Ego-Exo4D) is presented as qualitative visualization only.

Methodology in Plain English

The central insight is that no matter which sensor you use, its readings are only a partial view of the same underlying body state. So the authors define one canonical motion vector per frame containing root translation, root rotation, 3D joint positions, 6D rotations in world and root frames, 10-dimensional body shape, and binary foot-contact labels; this vector is both the input (partially filled) and the output (fully filled).

To handle different sensors, they define a mask scheme that says which joints and which global features are visible for a given configuration — from full optical mocap (all 22 joints, rotations visible), to 5-point VR tracking (head, hands, feet, positions only), to head-only egocentric estimation. Masks can even vary over time to mimic sensor dropout.

Naively zeroing out the missing entries destroys the identity of the missing joints. So the Masked Conditioning Encoder keeps a bank of learnable embedding vectors, one per joint, and substitutes the embedding wherever a value is missing. Four residual temporal convolution blocks with Group Normalization and Mish activations then smooth the conditioning signal and interpolate short occlusion gaps before it reaches the generative model.

Generation uses Conditional Flow Matching: an optimal-transport affine path interpolates between Gaussian noise and the ground-truth motion, and a denoising network is trained with a squared L2 loss to predict the clean target. At inference a single Euler step from t=0 to t=1 maps noise onto the motion manifold.

Because global trajectory is low-degree-of-freedom but needs long-horizon drift-free coherence, while local pose is high-degree-of-freedom but mainly needs biomechanical plausibility, the denoising network is split into two independently trained streams sharing the same conditioning interface: a convolutional UNet for trajectory and a DiT for pose, shape, and foot contact. During training the local stream sees the ground-truth trajectory; at inference the global stream's prediction is fed into the local stream. Losses include the flow matching objective plus representation, joint position, joint velocity, smoothness, foot-contact, and foot-skating terms, and analogous root position/velocity/smoothness terms for the global stream.

Training mask samples mix predefined sensor layouts with data-driven visibility patterns: per-joint visibility probabilities estimated from the large-scale Ego-Exo4D dataset using its COCO pose annotations predicted by MMPose. Training uses AdamW (β1=0.9, β2=0.999, weight decay 0.1, learning rate 2e-4), warmup-cosine scheduling with 4K-step warmup, 480K steps, batch size 256, gradient clipping at norm 1, clips of T=33 frames at 30 fps, on a single NVIDIA A40 (48 GB), converging in about 3 days. Two variants are evaluated: Ours-S (masks drawn from the evaluation setting) and Ours-U (masks drawn from the general configurations), alongside constructed unified baselines RoHM-U and RoHM++-U.

Why This Matters

• Research impact: The paper reframes sparse-sensor motion capture as a single masked-completion problem rather than a family of per-configuration specialist models, and shows that flow matching with one ODE step can match or beat 100+ step diffusion baselines — a meaningful efficiency argument for generative motion models generally.
• Real-world applications:
  • VR/AR and mixed-reality avatars driven only by a headset and two controllers (5-point head/hands/feet tracking), without a full IMU suit.
  • Egocentric wearable capture — reconstructing full-body motion from a head-mounted camera or headset pose for lifelogging, sports analysis, or first-person recording.
  • Sparse IMU suits for clinical gait analysis, rehabilitation monitoring, or sports biomechanics where only a few sensors are practical.
  • Robust motion capture on consumer hardware where signals drop out or drift, since the masking is explicitly designed to tolerate intermittent sensor loss.
• Industry relevance: The over-700-FPS inference on a single GPU, plus reported operating points of 250 FPS at 5 steps and 80 FPS at 30 steps, points toward deployment on consumer-grade devices where a single model must serve many hardware SKUs instead of one model per product.

Future Directions

• Multi-person and contact-rich scenes: The current model handles a single person and ignores body-object and body-body contact forces; extending the masking abstraction to multi-person interaction and object manipulation is named as an active direction.
• Correcting cascaded error: The pipeline feeds the global stream's trajectory into the local stream with no correction mechanism, so trajectory errors propagate into pose recovery; iterative refinement or feedback between the streams is suggested.
• Finer body models: OmniPoser operates on the SMPL 22-joint skeleton without hand or face articulation (SMPL-X); the modular feature-group design is described as readily extensible as training data becomes available.
• Scaling to Nymeria's full annotations: The paper notes that Nymeria contains 260M frames with dense body MVNX annotations, but practical resource constraints in preprocessing and large-scale training mean this is left for future work.

Target Audience

Researchers and graduate students working on 3D human pose and motion estimation, generative motion models (diffusion and flow matching), and wearable or egocentric sensing; engineers building motion-capture or avatar systems for VR/AR, robotics, and sports or clinical applications who need one model to serve multiple sensor configurations; and practitioners comparing generative motion backbones on the accuracy-versus-inference-speed trade-off.

Authors’ abstract

Recovering 3D human motion from sparse wearable sensors is challenging due to the inherent noise, occlusion, and instability of real-world tracking signals. Existing approaches are restricted to fixed sensor configurations and, when built on diffusion models, are computationally prohibitive for real-time deployment. We propose OmniPoser, a universal framework that recovers full-body SMPL motion from a flexible combination of heterogeneous inputs, such as 3D keypoints, sparse IMU rotations, head-mounted 6-DoF poses, or any subset thereof, within a single model. OmniPoser introduces a cross-modal masking mechanism paired with a Masked Conditioning Encoder, enabling one architecture to incorporate diverse sensor configurations through a masked design. A decoupled dual-stream generative backbone generates complete motion via Conditional Flow Matching, requiring only a single ODE integration step at inference. Across three settings and seven benchmarks, including in-the-wild Nymeria and Ego-Exo4D datasets, OmniPoser performs on par with state-of-the-art methods while running at over 700 FPS. The project page is available at https://jamesbridgewater51.github.io/omniposer-eccv/.