QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture

AI summary

Overview

Research area: Computer vision / 3D human motion capture, specifically online kinematics-based 3D human pose estimation from video.

Technical level: Advanced. The paper builds on quaternion algebra, Lie-group constrained integration on the unit sphere, state-space models, and meta-PD control, none of which are explained from first principles.

Scope: The paper proposes QuaMo, an online method that replaces Euler angles with quaternions in a kinematics-based state-space model of human motion, and evaluates it on Human3.6M, Fit3D, SportsPose, and a subset of AIST.

What This Paper Is About

Monocular 3D human motion capture from video loses depth information, so frame-by-frame pose estimators produce poses that look fine in isolation but jitter, skate, or look unnatural when viewed as a trajectory over time. Kinematics-based approaches fix this by modeling how a pose transitions to the next one using physics-style state-space models, but nearly all of them represent joint rotations with Euler angles, which are discontinuous and unstable. The paper's goal is to build a kinematics model whose joint orientation state is a quaternion, so the motion transitions are continuous, and to do so in an online setting where no future frames or trajectory refinement are available.

Key Contributions

1. A quaternion differential equation (QDE) for human kinematics. The joint rotation state is a quaternion and its velocity is given by the standard quaternion differential equation, with the angular velocity driving the Hamilton product. This inherently avoids the singularities and discontinuities that the paper attributes to Euler angles.

2. A second-order acceleration enhancement in the meta-PD controller. An extra term, scaled by a learned factor κ_A, is computed from the second-order quaternion difference between the last three reference poses. It amplifies the control signal during fast pose changes and dampens it as the motion approaches the target pose.

3. Exact integration under the quaternion unit-sphere constraint. Rather than approximating the quaternion update with Euler integration (first-order Runge–Kutta), the next pose is obtained as the Hamilton product of a rotation quaternion with the current pose, giving an exact solution on the S³ manifold.

4. An online system with no future-frame access. The model takes the current state and a single reference pose from an off-the-shelf estimator (TRACE or HMR2.0), making it applicable to real-time use.

Main Findings

• Best results come from HMR2.0 as the controller target. On Human3.6M, QuaMo_HMR2.0 reaches MPJPE 46.7 ± 0.04, P-MPJPE 30.6 ± 0.03, Accel 5.3 ± 0.04, G-MPJPE 88.8 ± 0.21, GRE 78.5 ± 0.33, G-Accel 6.8 ± 0.07, and foot skating 4.3 ± 0.04.

• Large gains over the input estimator. With TRACE as the reference on Human3.6M, QuaMo_TRACE records MPJPE 51.3 ± 0.11, P-MPJPE 37.5 ± 0.05, Accel 5.7 ± 0.03, G-MPJPE 116.2 ± 1.04, GRE 101.4 ± 1.37, G-Accel 7.8 ± 0.06, and FS 6.6 ± 1.78. Relative to raw TRACE, the paper reports improvements of 8.6% in MPJPE, 5.1% in P-MPJPE, and 18.7% in G-MPJPE.

• Improvements over the raw prior are larger still. Relative to HMR2.0 alone, QuaMo improves Accel by 41.8%, G-Accel by 59.5%, and foot skating by 62.6%.

• It beats its closest online competitor. Against OSDCap, using the same TRACE targets, the paper reports gains of 6.3% on MPJPE, 32.1% on Accel, and 12.5% on G-MPJPE. The paper attributes OSDCap's remaining implausibility to its Kalman-filter re-introduction of noisy TRACE inputs.

• Best-on-jitter among online kinematics methods on Human3.6M. The paper states QuaMo consistently records better Accel, G-Accel, and FS than other online kinematics-based approaches, and better MPJPE, P-MPJPE, G-MPJPE, and GRE when using HMR2.0.

• Caveat on "state of the art." The Human3.6M table lists keypoint-based lifter methods with lower MPJPE (PoseMamba 37.1, Mambapose 36.5, PoseAnchor 40.3, KTPFormer 40.1), but the paper states these are shown for reference only and cannot be compared to template-based methods; the table's bold highlighting is within the kinematics category.

• Results hold on more diverse data. On Fit3D, QuaMo_TRACE achieves MPJPE 50.3 ± 0.13, P-MPJPE 35.6 ± 0.04, Accel 3.8 ± 0.01, G-MPJPE 68.8 ± 0.21, GRE 45.2 ± 0.15, G-Accel 5.6 ± 0.03, FS 16.3 ± 0.27. On SportsPose it achieves 71.4 ± 0.30, 48.7 ± 0.21, 5.3 ± 0.19, 112.2 ± 0.75, 82.3 ± 0.36, 13.7 ± 0.13, and 24.1 ± 0.91 respectively, outperforming OSDCap on every metric. On the AIST subset, QuaMo_HMR2.0 achieves MPJPE 89.1, P-MPJPE 60.0, Accel 14.7, G-MPJPE 144.1, GRE 108.7, G-Accel 14.9, and FS 13.0, outperforming the offline method DiffPhy on both pose accuracy and jitter.

• Ablation: quaternions beat other rotation representations. On a Human3.6M subset (camera 60457274), a PD-only baseline with TRACE targets scores MPJPE 74.4 for Euler XYZ, 71.2 for Euler ZXY, 60.6 for axis-angle, and 53.8 for quaternion. The paper notes the Euler variants show reasonably low MPJPE but much larger Accel (13.7 for Euler XYZ, 7.6 for Euler ZXY) because the model must constantly compensate for angle discontinuities.

• Ablation: each component adds accuracy. Adding the data-driven bias term lowers MPJPE to 53.1 ± 0.05, adding the S³-constrained integration lowers it to 52.0 ± 0.07, and adding the acceleration enhancement reaches 51.3 ± 0.08 with P-MPJPE 37.4 and G-MPJPE 114.7. Accel, however, is 5.9 ± 0.02 in the final row versus 5.2 ± 0.02 in the row without the enhancement; the paper states the lowest MPJPE with a reasonable Accel is most desired.

• Foot skating is competitive but not best. TrajOpt reports FS 4.0% versus QuaMo's 4.3%, which the paper explains by TrajOpt's offline trajectory optimization with global refinement.

Methodology in Plain English

The system treats the human body as a physical system that evolves over time. At each moment it tracks two things per joint: the current orientation, stored as a quaternion (a four-number representation of a 3D rotation), and the current angular velocity.

Two equations run in parallel. The angular velocity is updated by a learned controller, similar in spirit to a proportional-derivative (PD) controller in robotics: a proportional term pulls the joint toward the reference pose supplied by an off-the-shelf pose estimator, and a derivative term damps the motion so it does not overshoot or jitter. A neural network ("ControlNet") predicts the controller gains, a bias term, and the scale of the new acceleration term from the current state and the reference pose. The novel acceleration term looks at the last three reference poses, measures how fast the reference is changing, and adds a second-order correction that pushes harder during sudden movements and eases off when the target is reached.

The updated angular velocity then drives the orientation forward. Instead of adding a numerically approximated derivative, the method computes the rotation this angular velocity would produce over the time step and multiplies it into the current quaternion via the Hamilton product, so the orientation stays exactly on the unit sphere at all times. Root translation is handled separately with the same PD-and-Euler-integration scheme.

The resulting pose drives an SMPL body model, with a learnable shape parameter, to produce a mesh, and a linear regressor extracts 17 keypoints. Training combines a frame-wise L1 loss on root-aligned keypoints and root translation, a global consistency loss on second-order finite differences (to suppress jitter and drifting), and a small regularization that keeps body shape near average. Training runs for 35 epochs with batch size 64, an initial learning rate of 5e-4 decayed by a factor of 10 at epochs 20 and 30, a first phase of 5 epochs with frame-wise updates at learning rate 1e-4 and the global loss disabled, λ = 0.01 for the shape loss, Δt = 0.04 (25 Hz), sequences of 100 frames, and results reported over five random seeds (0–4).

Why This Matters

Impact on research. The paper argues that a foundational representation choice — Euler angles — has been an under-examined source of instability in kinematics-based motion capture, and shows empirically that quaternions plus an exact unit-sphere integration give better accuracy and smoother motion in an online setting. It also provides a direct, same-target comparison against OSDCap and against offline methods such as DiffPhy, which is the type of controlled comparison that clarifies where the gains actually come from.

Real-world applications (as identified or implied by the paper):

• Autonomous driving, where human motion prediction feeds into safety and interaction systems.
• Biomechanics and human movement analysis.
• Real-time motion capture and animation pipelines where no future frames are available.
• Sports analysis, since the method is evaluated on the SportsPose dataset of phone-captured sport videos.

Industry relevance. Because QuaMo is online and consumes only the current state plus one reference pose, it fits latency-sensitive deployments — streaming capture, live sports, AR/VR avatars, and robotics — where offline trajectory optimization is not an option. Its reliance on existing pose estimators (TRACE, HMR2.0) means it can be dropped on top of an existing perception stack rather than replacing it.

Future Directions

• Improving jitter without sacrificing accuracy. The final ablation row has the best MPJPE but a slightly worse Accel (5.9) than the configuration without acceleration enhancement (5.2), which suggests tuning the enhancement's scale and interaction with the damping term is unresolved.
• Replacing pseudo ground truth on AIST. The AIST evaluation relies on pseudo ground truth from 3D triangulation and on HMR2.0 rather than the original HUND input, because HUND's implementation is not publicly available.
• Preserving foot-ground contact explicitly. QuaMo's FS of 4.3% is close to but above TrajOpt's 4.0%, and the offline advantage suggests contact modeling or global refinement could close the gap.
• Extending beyond the 24-joint SMPL body. The paper models 24 body joints with the root at the first entry; hands, face, and finer articulation are not addressed here.

Target Audience

Researchers and graduate students working on 3D human pose and motion capture, physics-based or kinematics-based motion reconstruction, and anyone implementing online motion estimation on top of an off-the-shelf pose estimator. Readers need comfort with quaternion algebra, state-space models, and standard pose-evaluation metrics; the paper also suits engineers evaluating whether to swap Euler-angle joint states for quaternion states in an existing temporal model.

Authors’ abstract

Vision-based 3D human motion capture from videos remains a challenge in computer vision. Traditional 3D pose estimation approaches often ignore the temporal consistency between frames, causing implausible and jittery motion. The emerging field of kinematics-based 3D motion capture addresses these issues by estimating the temporal transitioning between poses instead. A major drawback in current kinematics approaches is their reliance on Euler angles. Despite their simplicity, Euler angles suffer from discontinuity that leads to unstable motion reconstructions, especially in online settings where trajectory refinement is unavailable. Contrarily, quaternions have no discontinuity and can produce continuous transitions between poses. In this paper, we propose QuaMo, a novel Quaternion Motions method using quaternion differential equations (QDE) for human kinematics capture. We utilize the state-space model, an effective system for describing real-time kinematics estimations, with quaternion state and the QDE describing quaternion velocity. The corresponding angular acceleration is computed from a meta-PD controller with a novel acceleration enhancement that adaptively regulates the control signals as the human quickly changes to a new pose. Unlike previous work, our QDE is solved under the quaternion unit-sphere constraint that results in more accurate estimations. Experimental results show that our novel formulation of the QDE with acceleration enhancement accurately estimates 3D human kinematics with no discontinuity and minimal implausibilities. QuaMo outperforms comparable state-of-the-art methods on multiple datasets, namely Human3.6M, Fit3D, SportsPose and AIST. The code is available at https://github.com/cuongle1206/QuaMo