InstantHMR : Real-Time Human Mesh Recovery on Your Phone

AI summary

Overview

Research area: Computer vision — single-image 3D human mesh recovery, with a focus on efficient on-device (mobile NPU) inference.

Technical level: Advanced. The paper assumes familiarity with parametric body models (MHR, SMPL), mesh regression, and benchmark metrics such as PA-MPJPE, plus mobile deployment tooling.

Scope: A family of open-source MHR-based single-image mesh regressors that run in real time on phone hardware while staying competitive in accuracy with larger state-of-the-art models, together with a unified re-evaluation of prior methods.

What This Paper Is About

Recovering a full 3D human body mesh from a single photo is useful for many applications, but the best-performing models are too heavy to run on a phone, and the fast ones lose accuracy. The authors build InstantHMR, a set of open-source models based on the Momentum Human Rig (MHR) parametric body model, trained only on publicly available annotations from the SAM 3D Body dataset. The goal is to show that mobile-speed inference does not require giving up much accuracy, and to provide fair, reproducible comparisons against existing methods.

Key Contributions

1. An open-source family of mobile-ready regressors. InstantHMR regresses a 6,890-vertex MHR body mesh from a single image and is designed specifically for mobile inference; code and models are released publicly.

2. Real-time latency on consumer hardware. The method recovers a mesh in 5.5 ms per person on a Samsung Galaxy S23 NPU and 0.98 ms on an NVIDIA RTX 4070 GPU, which the authors state surpasses every method they evaluated.

3. Competitive accuracy at that speed. Reported PA-MPJPE is 40.0 mm on 3DPW, 48.8 mm on EMDB, and 41.7 mm on RICH; on mobile it is 12x faster than the only mobile-compatible method that slightly exceeds it in accuracy.

4. A unified evaluation and reproduction study. Eleven state-of-the-art methods are re-evaluated under a single protocol aligned with the closest MHR-based competitors, including Meta's official MHR-to-SMPL conversion, and latency is measured across eight targets ranging from desktop TensorRT to two phone NPUs.

Main Findings

• Speed advantage: InstantHMR is reported as faster than all methods the authors evaluated, at sub-millisecond GPU latency and single-digit-millisecond NPU latency per person.

• Accuracy versus mobile competitors: Only one mobile-compatible method is reported to be slightly more accurate, and InstantHMR is 12x faster than it on mobile — framing the trade-off as strongly favorable.

• Accuracy versus the broader field: The reported PA-MPJPE figures (40.0 mm on 3DPW, 48.8 mm on EMDB, 41.7 mm on RICH) are presented as competitive, though the abstract gives no per-method breakdown of how these compare numerically to each baseline.

• Reproduction gap in prior work: By re-evaluating 11 methods under one unified protocol, the paper suggests that published comparisons across the field are not directly comparable and provides reproduction results as a corrected reference point.

• Hardware breadth: Latency is characterized across eight deployment targets, from desktop TensorRT to two phone NPUs, indicating the method is not tuned to a single device.

• Data constraint: Training uses only publicly available annotations from SAM 3D Body, which the authors present as a deliberate choice; the abstract does not state dataset size, training duration, or augmentation details.

Methodology in Plain English

The approach is direct regression: a single image goes in, and the network outputs the parameters of a parametric body model (MHR), which are then turned into a full 6,890-vertex mesh. Restricting training to public annotations from SAM 3D Body keeps the pipeline reproducible and avoids reliance on proprietary data. The engineering effort then goes into making that regressor small and fast enough to execute on phone NPUs and on desktop GPUs through TensorRT, and into measuring per-person latency consistently across devices.

The abstract does not describe the network architecture, parameter counts, quantization or pruning strategy, or training schedule; those details are outside what it reports. Similarly, the unified evaluation protocol is described only as being aligned with the closest MHR-based competitors, including Meta's official MHR-to-SMPL conversion, without listing the protocol's individual settings.

Why This Matters

Research impact: The paper pairs a fast model with a re-implementation of 11 baselines under one protocol. Reproduction studies of this kind are useful because cross-paper accuracy and speed numbers are frequently measured on different hardware, different preprocessing, and different body models — an issue that is especially acute when comparing MHR-based and SMPL-based outputs.

Real-world applications (as identified by the authors):

• Robotics — human pose and shape sensing for interaction and imitation.
• Biomechanics — markerless body-shape and joint estimation from ordinary images.
• AR/VR — full-body avatars driven from a single camera frame.
• Exergames — movement tracking in fitness games without dedicated motion-capture hardware.

Industry relevance: Millisecond-scale, per-person inference on a phone NPU makes camera-based body capture viable inside ordinary mobile applications, where the alternative is offloading to a server. The choice of an openly released model, open code, and public-only training data lowers the barrier for commercial and academic reuse, and the latency table across eight targets gives practitioners a concrete basis for hardware selection.

Future Directions

• Closing the remaining accuracy gap: One mobile-compatible method is still slightly more accurate; determining whether that gap can be removed without losing the 12x speed advantage is a natural next step.

• Beyond the three benchmarks: The abstract reports accuracy only on 3DPW, EMDB, and RICH. Behavior in crowded scenes, unusual body shapes, heavy occlusion, and unusual clothing remains uncharacterized.

• Temporal and video extensions: The work is framed around single-image recovery; extending the same efficiency to video, where frame-to-frame consistency matters, is an open engineering question.

• Generalizing the evaluation protocol: The unified re-evaluation covers 11 methods; broadening such protocols — and standardized mobile latency measurement — across the field would make speed and accuracy claims comparable by default.

Target Audience

Researchers and engineers working on 3D human pose and shape estimation, especially those deploying models to edge devices; mobile and AR/VR developers who need real-time full-body capture on phones; and benchmarking-focused readers interested in the reproduction of 11 state-of-the-art methods under a unified protocol. Readers new to parametric body models will find the paper assumes background knowledge of MHR/SMPL conventions and PA-MPJPE evaluation.

Authors’ abstract

Real-time 3D human mesh recovery from a single image on mobile devices is a challenging problem with many applications, including robotics, biomechanics, AR/VR, and exergames. Current state-of-the-art models are either too slow to run efficiently on mobile devices or sacrifice accuracy for speed. We present InstantHMR, a family of open-source single-image regressors suited for mobile inference, based on the Momentum Human Rig (MHR) parametric model and trained exclusively on…