Flow Matching for Probabilistic Monocular 3D Human Pose Estimation

AI summary

Overview

• Research area: Computer vision — probabilistic monocular 3D human pose estimation, specifically 2D-to-3D pose lifting with generative models.
• Technical level: Intermediate. The paper assumes familiarity with pose estimation benchmarks and error metrics, and with generative modelling concepts such as normalizing flows, diffusion, and ordinary differential equation solvers.
• Scope: The paper introduces FMPose, a flow-matching model with optimal transport paths and graph-convolutional conditioning, and benchmarks it against prior probabilistic single-frame lifting methods on Human3.6M, MPI-INF-3DHP and 3DPW.

What This Paper Is About

Recovering a 3D human pose from a single camera image is ambiguous: many different 3D poses project to the same 2D image because depth along the optical axis is unknown. Methods that predict one "best" 3D pose therefore tend to be confidently wrong, especially when joints are occluded or the pose is unusual.

The paper's goal is to predict a distribution of plausible 3D poses from 2D cues instead of a single answer, and to do so with a generative framework that is more stable and faster than the diffusion models currently used for this task. The authors propose FMPose, which learns a continuous normalizing flow from a simple Gaussian source distribution to the distribution of plausible 3D poses using flow matching with optimal transport, conditioned on 2D information aggregated by a graph convolutional network.

Key Contributions

1. The authors state they are the first to use a continuous flow model trained with optimal transport for probabilistic 3D human pose estimation. FMPose learns a straight optimal-transport path from a Gaussian source to the 3D pose distribution via an ordinary differential equation, in contrast to the stochastic denoising path of diffusion-based competitors.
2. They propose a 2D-to-3D lifting condition built with a graph convolutional network whose adjacency matrix is initialized to zeros and learned from scratch during training, rather than fixed to the human skeleton. The condition is extracted from the top-k arguments of HRNet 2D heatmaps instead of only the maximum argument.
3. They report new state-of-the-art results for single-frame multi-hypothesis 3D pose estimation on Human3.6M and MPI-INF-3DHP, plus competitive results on 3DPW, and describe a processing-time advantage over the diffusion-based DiffPose at equal accuracy.
4. They release a code implementation in a public GitHub repository.

Main Findings

• Human3.6M standard protocol (every 64th test frame): FMPose top-k with 200 hypotheses reaches 41.7 ± 0.3 mm MPJPE and 30.6 ± 0.1 mm P-MPJPE, compared with DiffPose at 44.2 ± 0.2 mm and 32.1 ± 0.1 mm. The authors describe this as a 2.5 mm (5.6%) improvement in average MPJPE and 1.5 mm (4.6%) in P-MPJPE over DiffPose. GFPose reports lower numbers (35.8 mm and 30.6 mm) but the authors gray it out as incomparable because it predicts in pixel space and requires ground-truth root translation, perspective ratio and camera intrinsics, and they state that adapting its code to the standard protocol causes a complete collapse in performance.
• Single hypothesis comparison: With a deterministic initial sample at zero (H = 1), FMPose top-k gets 58.9 ± 0.4 mm MPJPE and 41.5 ± 0.1 mm P-MPJPE, versus DiffPose at 64.5 mm and 45.2 mm — a reported reduction of 5.6 mm (8.7%) in MPJPE and 3.7 mm (8.1%) in P-MPJPE.
• Highly ambiguous Human3.6M poses: FMPose random reaches 58.9 ± 0.1 mm MPJPE, 46.7 ± 0.2 mm P-MPJPE, 96.6 ± 0.1 PCK and 194.7 ± 0.9 CPS, versus DiffPose at 66.5 ± 1.4, 48.5 ± 0.2, 94.3 ± 0.1 and 194.2 ± 2.2. The paper reports this as 7.6 mm (11.4%) better MPJPE, 1.8 mm (3.7%) better P-MPJPE, 2.4% better PCK, and the same CPS. Ambiguity here means at least one fitted Gaussian has a width greater than 5 pixels.
• Trade-off between hypothesis-selection strategies: Random sampling widens coverage of pose uncertainty and wins on the ambiguous subset, while top-k selection is more deterministic and wins on more certain poses. The authors adopt top-k as the main configuration for the remaining datasets.
• MPI-INF-3DHP generalization (no extra training or fine-tuning): FMPose achieves PCK of 87.9 ± 0.3 indoor with green screen, 83.3 ± 0.2 indoor without green screen, 84.1 ± 0.2 outdoor and 85.3 ± 0.3 overall, outperforming the closest comparable competitor DiffPose (87.4, 82.5, 83.3, 84.6) on all four setups. GFPose's higher numbers (88.4, 87.1, 84.3, 86.9) are again flagged as non-comparable.
• 3DPW: FMPose trained from scratch reaches 63.9 ± 0.3 mm MPJPE and 42.3 ± 0.2 mm P-MPJPE; the version fine-tuned from the Human3.6M model reaches 56.2 ± 0.2 mm and 35.4 ± 0.1 mm. The authors report an 8.4% MPJPE and 15.1% P-MPJPE improvement over the closest competitor, MultiHMR (61.4 mm, 41.7 mm), and note that the mesh-based baselines listed in the table are for reference only and not directly comparable to the joint-based FMPose.
• Conditioning is essential: Removing the lifting condition entirely degrades results drastically (122.2 ± 0.2 mm MPJPE on Human3.6M and 169.2 ± 0.1 mm on ambiguous poses), which the authors describe as making 3D reconstruction from the Gaussian input impossible.
• Top-k beats max-argument: Using only the maximum heatmap argument instead of top-k costs 2.1 mm (4.8%) in MPJPE and 2.4 mm (3.9%) in ambiguous MPJPE relative to the full model.
• GCN conditioning beats transformer conditioning: Replacing the GCN with a transformer-based conditioning of the same parameter count (4.5M) raises MPJPE by 1.3 mm (3.0%) and ambiguous MPJPE by 1.5 mm (2.5%); the transformer variant also takes 75 ms versus 36 ms.
• GCN beats fully connected layers, and learned adjacency beats skeleton adjacency: Removing the GCN increases MPJPE by 2.3 mm (5.2%); fixing the adjacency matrix to human skeleton connections rather than learning it from zero initialization increases MPJPE by 0.8 mm (1.9%).
• Solver choice: RK2 is selected as the main solver, giving 41.7 ± 0.2 mm MPJPE and 36 ms (28 FPS) for 200 hypotheses, which the authors describe as the best accuracy-versus-time trade-off among RK1 (20 ms, 43.1 ± 0.2 mm), RK3 (55 ms, 41.6 ± 0.2 mm) and RK4 (71 ms, 42.3 ± 0.1 mm).
• Number of ODE steps: Accuracy improves with more steps and plateaus; the best trade-off is reported at 25 steps (36.1 ms, 41.7 ± 0.2 mm MPJPE on Human3.6M, 59.8 ± 0.3 mm on ambiguous poses), versus 5 steps (7.6 ms, 44.2 ± 0.2 mm) and 40 steps (56.7 ms, 41.8 ± 0.2 mm).
• Implementation scale: The full model has 4.5M training parameters and requires 36 ms to produce 200 hypotheses. Training uses about one hour and 86 MB of memory per iteration on an NVIDIA A40. Results are averaged over five random seeds (40 to 44).

Methodology in Plain English

FMPose separates the problem into two stages.

First, an off-the-shelf 2D pose detector (HRNet) produces a heatmap for each human joint, expressing where that joint is likely to be. Rather than keeping only the single most likely position per joint, the method takes the top-k positions ranked by confidence, with k = 48 chosen by ablation, and forms an input tensor of shape J × 2k.

Second, these 2D cues are turned into a "condition" vector using a graph convolutional network in which each node is a human joint. The adjacency matrix that describes how joints influence one another is initialized to all zeros and learned during training, instead of being fixed to anatomical skeleton links. The resulting 64-dimensional condition vector tells the generative model what 2D pose it must explain.

The generative part is a continuous normalizing flow. Training draws a random time t between 0 and 1, draws a Gaussian sample as the starting point, and uses a straight interpolation between that sample and the ground-truth 3D pose to define the velocity the network should predict. Because the path is a straight line under optimal transport, the target velocity is simply the difference between the destination and the source, so training reduces to a mean squared error regression — no expensive simulation of a differential equation during training.

At inference, the model samples a Gaussian starting point and integrates the learned velocity field numerically using an RK2 Runge–Kutta solver over 25 steps, producing one 3D pose per sample. Drawing many samples gives many hypotheses, whose spread represents uncertainty. The whole pipeline is trained end-to-end on Human3.6M and evaluated on Human3.6M, MPI-INF-3DHP (transferred without fine-tuning) and 3DPW, using MPJPE, P-MPJPE, PCK at 150 mm, and Correct Poses Score, taking the minimum error over the hypothesis set.

Why This Matters

• For research: The paper argues that replacing the stochastic denoising path of diffusion with a straight optimal-transport flow path yields both better accuracy and lower computational cost for 2D-to-3D lifting. If it holds up, it shifts the default generative backbone for this task away from diffusion and toward flow matching, and it revives conditional normalizing flows in a form that is practical to train.
• Real-world applications:
  • Autonomous driving and robotics, where systems listed by the authors need to reason about human motion and its uncertainty.
  • Public safety and surveillance, using in-the-wild single-image footage rather than costly video pipelines.
  • Markerless motion capture and sports or clinical movement analysis from ordinary cameras.
  • Human–computer interaction and animation, where multiple plausible 3D poses can be offered rather than one possibly wrong estimate.
• Industry relevance: The reported 36 ms for 200 hypotheses at the full model's 4.5M parameters, and a faster 20 ms RK1 configuration, point toward deployment on modest hardware. The authors also emphasize that probabilistic outputs let downstream models account for a range of possible poses rather than only the maximum-likelihood estimate. The paper does not report latency for the HRNet 2D detector, since it follows the same setup as prior work.

Future Directions

• Extending to video: FMPose is a single-frame method. The paper contrasts it with temporal approaches that use sequences and are limited by costly video processing; whether flow matching can exploit temporal information without sacrificing speed is left open.
• Closing the gap on 3DPW against multi-frame methods: FMPose is described as competitive with methods such as WHAM and ExtPose, but not better; improving single-frame accuracy on in-the-wild data is an open problem.
• Better handling of the sampling trade-off: Top-k and random sampling win on different subsets (certain versus highly ambiguous poses). A principled way to combine or adaptively select between them is not proposed.
• Comparing against the reported GFPose numbers: The authors exclude GFPose as incomparable under the standard protocol. A common evaluation protocol that makes all probabilistic methods comparable would be a useful follow-up.
• Choosing the number of hypotheses and solver settings per task: The paper shows accuracy and cost vary with solver order and step count; adaptively allocating computation per input is not explored.

Target Audience

Researchers and graduate students working on 3D human pose estimation, generative modelling, or both, will benefit most, since the paper sits at the intersection of flow matching and 2D-to-3D lifting. Practitioners building pose-estimation systems that need uncertainty estimates or real-time multi-hypothesis output will also find the benchmark tables and the timing comparisons useful. Readers should already be comfortable with common pose estimation metrics and with the difference between diffusion models and continuous normalizing flows to follow the methodological argument.

Authors’ abstract

Recovering 3D human poses from a monocular camera view is a highly ill-posed problem due to the depth ambiguity. Earlier studies on 3D human pose lifting from 2D often contain incorrect-yet-overconfident 3D estimations. To mitigate the problem, emerging probabilistic approaches treat the 3D estimations as a distribution, taking into account the uncertainty measurement of the poses. Falling in a similar category, we proposed FMPose, a probabilistic 3D human pose estimation method based on the flow matching generative approach. Conditioned on the 2D cues, the flow matching scheme learns the optimal transport from a simple source distribution to the plausible 3D human pose distribution via continuous normalizing flows. The 2D lifting condition is modeled via graph convolutional networks, leveraging the learnable connections between human body joints as the graph structure for feature aggregation. While trade-offs between processing time and precision exist, already in the equal-accuracy comparison, FMPose exhibits significantly faster processing time than the diffusion model, and also offers another faster and more accurate configuration. Experimental results show major improvements of our FMPose over current state-of-the-art methods on two common benchmarks for 3D human pose estimation, namely Human3.6M, MPI-INF-3DHP. Additionally, FMPose shows competitive performance on the more challenging 3DPW dataset. The code implementation is available at https://github.com/cuongle1206/FMPose