SynthGait-19K: A Physically Grounded Synthetic Video Dataset for Gait Parameter Estimation

AI summary

Overview

Research area: Computer vision for health — synthetic video dataset generation and monocular gait parameter estimation.

Technical level: Advanced. The paper assumes familiarity with human mesh recovery (SMPL), video diffusion models, Vision Transformers, and biomechanical gait analysis.

Scope: The paper introduces SynthGait-19K, a synthetic walking-video dataset built from real motion capture, a generation pipeline called Gait2Vid, and a benchmark (including a direct RGB model, GaitXFormer) for estimating six clinical gait parameters from monocular video.

What This Paper Is About

Estimating clinically meaningful gait parameters from ordinary video is attractive for scalable mobility assessment, but training and evaluating such models requires synchronized video and motion-capture or force-plate data, which is expensive, small, and usually captured in one controlled environment with restricted camera viewpoints. This makes it hard to know whether a method generalizes, or which factor (viewpoint, appearance, scene) causes failures. The paper's goal is to build a large, physically grounded synthetic dataset whose motion comes from real people, so that camera and visual conditions can be varied independently of the underlying walking motion, and to use it to train and benchmark gait-estimation approaches.

Key Contributions

1. SynthGait-19K, a synthetic gait-video dataset of 19,272 RGB walking videos derived from 6,427 MoCap sequences across 437 subjects, with paired SMPL motion and annotations for six gait parameters (cadence, walking speed, step length, step width, stooped posture, arm swing). Source data span five public MoCap datasets and 671 minutes of walking, covering 231 healthy or asymptomatic participants and 206 participants from clinical populations. (The paper reports this figure as both 19,272 and 19,273 in different places, and the appendix states "the final 19,273 videos from 6,427 motions were retained.")

2. Gait2Vid, a generation pipeline that unifies heterogeneous MoCap recordings through SMPL, renders depth videos from controllable virtual cameras over a planar ground plane, and uses depth-conditioned video diffusion (Wan2.1-14B-VACE) with text prompts drawn from 200 curated indoor and outdoor scene templates.

3. Validation analyses: kinematic fidelity of generated videos measured against Sapiens2 pose estimates, and independent force-platform validation of the UnderPressure-derived heel-strike annotations used to compute gait labels.

4. A common benchmark spanning direct RGB, pose-based, biomechanical, and human-mesh-recovery approaches, plus GaitXFormer, a V-JEPA2-initialized Video ViT with six learnable gait-query tokens that predicts gait parameters directly from video without an intermediate pose or mesh representation.

Main Findings

• Kinematic fidelity of generated video is comparable to or better than real video: On paired real and VACE-generated GPJATK videos, BBox-NMPJPE dropped from 3.41% to 2.32% (delta -1.09, 95% CI [-1.19, -0.98]), lower-body NMPJPE from 2.90% to 2.20% (-0.70, [-0.79, -0.61]), and knee-angle MAE from 8.71° to 4.86° (-3.86, [-4.11, -3.62]). Lower-body velocity error was nearly unchanged (0.74 to 0.76, +0.02, [-0.01, +0.04]). The authors attribute the lower error to cleaner visual conditions and reduced clothing ambiguity, and note that fitted SMPL is not independent marker-level ground truth, so this measures motion consistency rather than absolute biomechanical accuracy.

• Gait event annotations agree with force plates: Across 6,092 force-platform-observed heel strikes, UnderPressure heel strikes had a mean absolute timing error of 2.31 30-FPS SMPL frames (approximately 77 ms), with 82.3% within 3 frames and 91.7% within 5 frames. The supplement adds that of 3,245 evaluated trials, 3,053 contained at least one force-platform contact, with median absolute error 2 frames and RMSE 3.045 frames.

• Synthetic supervision transfers to real video across two architectures: On the preferred-view GPJATK benchmark, GaitXFormer reached a Fisher-averaged correlation of 0.84, and STT trained on SynthGait-19K (STT†) reached 0.82 — the released STT model, trained on side-view videos from a cerebral-palsy cohort, reached only 0.20 for cadence and 0.67 for walking speed. STT† achieved the strongest step-length result among the evaluated methods (0.73).

• Off-the-shelf HMR and biomechanical methods vary strongly by parameter: Among HMR methods, WHAM scored 0.70, PromptHMR 0.67, CameraHMR 0.59, and FastHMR 0.54 in Fisher-averaged correlation. OpenCap Monocular performed strongly for walking speed (0.87), WHAM for step width (0.69), and FastHMR for arm swing (0.93).

• Spatial gait parameters are more sensitive to synthetic-to-real shift: GaitXFormer's Fisher-averaged correlation fell from 0.87 on GPJATK-VACE to 0.84 on real GPJATK, with cadence, walking speed, and arm swing nearly unchanged, while step length and step width dropped more. WHAM showed a different pattern: overall correlation was similar across generated and real GPJATK (0.68 vs. 0.70), but step width improved from 0.56 to 0.69 on real video while stooped posture fell from 0.79 to 0.64.

• Improved HMR reconstruction does not necessarily improve gait estimation: Adapting WHAM with a SynthGait-19K-trained output adapter reduced body-pose error by 7.43% (10.54° to 9.76°), pelvis MPJPE by 2.78% (46.93 mm to 45.63 mm), and PA-MPJPE by 2.53% (33.28 mm to 32.44 mm). The Fisher-averaged gait correlation moved only from 0.7045 to 0.7080 (Δr = 0.0035, participant-bootstrap 95% CI [-0.0068, 0.0166]); walking speed and step length improved while step width and arm swing decreased.

• GaitXFormer has the best accuracy–runtime trade-off: It processes a 5-second walking clip in 0.27 s on a single RTX3090, versus 1.21–139.10 s for the evaluated baselines including required preprocessing. STT requires only 0.32 s but additionally relies on OpenPose preprocessing.

• Viewpoint matters, and differently per parameter: On multi-view GPJATK, GaitXFormer's cadence correlation was stable across views (0.95 side, 0.94 front, 0.94 back, 0.92 oblique). Stooped posture was far better from the side (0.75) than front (0.26) or back (0.24). Step width was best from the back (0.70) and front (0.63) and weaker from the side (0.50) and oblique (0.48). Walking speed and step length were highest from oblique views (0.91 and 0.70).

• More synthetic data helps: Training GaitXFormer on progressively larger subsets of SynthGait-19K raised the Fisher-averaged correlation from 0.79 at 25% of the training data to 0.80 at 50%, 0.83 at 75%, and 0.84 at 100%, with clearest gains for cadence, step length, and step width.

• Transfer to an independent Parkinson's-disease dataset (PD4T): Under leave-one-subject-out evaluation, a frozen GaitXFormer encoder improved UPDRS classification accuracy from 71.3% to 76.0% and macro F1 from 69.1% to 73.3% relative to a pretrained V-JEPA2 encoder. Predicted gait quantities tracked clinical severity: walking speed ρ = -0.83 (95% CI [-0.88, -0.76]), step length ρ = -0.83 ([-0.88, -0.78]), arm swing ρ = -0.75 ([-0.84, -0.61]), and stooped posture ρ = 0.30 ([0.14, 0.47]), with within-patient directional agreement of 97.9%, 97.9%, 91.5%, and 80.9% respectively, and standardized response means of 1.37, 1.39, 1.24, and 0.53.

Methodology in Plain English

The researchers started from five existing motion-capture datasets that use different marker layouts and joint conventions. They converted every recording into a single common body model (SMPL) so the motions could be combined and rendered consistently, and split subjects 80/20 into training and validation so that all videos of a person stay in one partition.

For each walking sequence, they rendered depth videos from virtual cameras in front/back, side (sagittal), and randomly rotated oblique configurations. Camera pitch was sampled between -5° and 45° for all configurations, and oblique camera yaw was sampled uniformly over 0° to 360°. A synthetic flat ground plane was rendered beneath the walker; the authors show that without this geometry, the video model introduced spurious camera pitch and scale changes across frames, so the static ground implicitly stabilizes the virtual camera. The depth video plus a text prompt describing the subject and environment was fed to Wan2.1-14B-VACE, a depth-conditioned video diffusion model, with TeaCache used to accelerate generation.

Gait labels were then computed directly from the fitted SMPL motion rather than from the generated pixels, so each synthetic video is paired with labels for its conditioning motion. UnderPressure detects heel strikes, and from those plus 3D hip, foot, and neck trajectories and leg length, the pipeline derives walking speed (pelvis forward displacement between first and last heel strike over elapsed time), cadence (steps per minute), step length and step width (forward and mediolateral foot displacement between successive heel strikes, averaged), stooped posture (neck–pelvis forward offset normalized by leg length), and arm swing (normalized forward range of wrist motion, averaged over both wrists).

For generation compute, the campaign used Wan2.1-VACE-14B at 480×832 resolution, 81 frames, 50 denoising steps, and TeaCache with threshold 0.3 on NVIDIA L40S GPUs, split into 48 independent shards running on up to 48 GPUs and requiring approximately 3,131 allocated L40S-hours. The authors note this campaign produced a larger candidate pool than the final retained dataset.

Evaluation used GPJATK, a real dataset with 152 walking sequences from 32 subjects and 608 RGB videos (152 side, 76 front, 76 back, 304 oblique). The main benchmark uses a "preferred-view" protocol: walking speed, step length, stooped posture, and arm swing are evaluated from the side; step width from front and back; cadence across all views. The primary metric is Pearson correlation (r), with the overall score computed by averaging the six parameter-wise correlations using a Fisher z-transformation; mean absolute error in native units is reported in the supplement.

Why This Matters

This work addresses a structural bottleneck in video-based gait analysis: real paired video-and-motion data is scarce, expensive, and difficult to share because gait is identifiable biometric information. By generating large-scale synthetic video whose motion comes from real recordings, the paper makes it possible to vary camera viewpoint, appearance, and scene while holding the underlying walking motion fixed — an experiment that is essentially impossible to run with real capture. The paper's negative result, that better human mesh reconstruction does not automatically yield better gait estimates, is a useful caution for a field that often equates reconstruction quality with downstream usefulness. The paper also shows that a pose-based architecture trained on the dataset improves substantially, so the value is not tied to one model design.

Real-world applications:

• Clinical mobility assessment: Estimating walking speed, cadence, and step length from a single camera could support gait evaluation without a gait laboratory, relevant for the Parkinson's disease and rehabilitation contexts cited in the paper.
• Fall-risk screening in older adults: Spatial gait measures such as step width and stooped posture are associated with fall risk, and the paper's viewpoint analysis indicates which camera placements support which measures.
• Remote and longitudinal monitoring: A model processing a 5-second clip in 0.27 s could enable frequent, low-cost monitoring outside the clinic, where gait patterns often differ from laboratory walking.
• Clinical severity inference from video: The PD4T results show that features learned under gait supervision transfer to UPDRS severity classification, suggesting a route from raw walking video to clinical endpoints.

Industry relevance: the work is directly relevant to digital health and remote patient monitoring, clinical-trial endpoint measurement, elder-care and rehabilitation technology, and camera-based consumer or ambient sensing products, where the cost of collecting paired motion-capture ground truth is the main barrier to deploying gait estimation.

Future Directions

• Broaden real-world visual coverage: The authors list incomplete coverage of severe occlusion, assistive devices, and broader clothing and clinical diversity as limitations, all of which would need to be represented in generation prompts or capture protocols.
• Extend the temporal window: The fixed 5-second window excludes longer-horizon phenomena such as freezing of gait, so longer clips or event-level modeling are natural next steps.
• Generalize evaluation beyond GPJATK and PD4T: The paper's evaluation centers on GPJATK with PD4T validation, and the authors state broader cohorts would better characterize generalization.
• Close the spatial-parameter domain gap and improve HMR-to-gait coupling: Step length and step width are the parameters most affected by synthetic-to-real shift, and the WHAM adapter experiment shows reconstruction gains did not translate uniformly to gait; both point to open problems in bridging visual domain shift and in designing intermediate representations for gait rather than for reconstruction.

Target Audience

Researchers and practitioners in computer vision for healthcare, particularly those working on human motion reconstruction, video-based clinical measurement, and synthetic data generation. It is also relevant to biomechanics and clinical gait researchers interested in camera-based alternatives to instrumented walkways, and to engineers building remote monitoring or mobility-assessment products who need to know which gait parameters are reliable from which camera viewpoints and how much synthetic supervision buys in real-world accuracy. Readers without background in SMPL, human mesh recovery, or video diffusion will find the paper technically demanding.

Authors’ abstract

Accurate estimation of clinically meaningful gait parameters from monocular video is important for scalable mobility assessment, yet progress is limited by the small scale, restricted viewpoints, and limited visual diversity of existing datasets. We introduce SynthGait-19k, a physically grounded synthetic video dataset containing 19,272 walking videos derived from 6,427 MoCap sequences across 437 subjects, with paired SMPL motion and annotations for six gait parameters. To construct the dataset, we develop Gait2Vid, which unifies heterogeneous MoCap recordings through SMPL and synthesizes diverse RGB walking videos under controllable viewpoints and scene appearances. We assess the generated videos for consistency with their conditioning gait kinematics and validate extracted gait events against force-platform measurements. Using SynthGait-19K, we benchmark direct RGB, pose-based, biomechanical, and human-mesh-recovery approaches and analyze viewpoint, training-data scale, and synthetic-to-real domain shift. We also introduce GaitXFormer as a direct RGB reference model for estimating gait parameters. Synthetic supervision transfers effectively to real videos across both GaitXFormer and a pose-based architecture, demonstrating utility across different representations. We further find that spatial gait parameters are more sensitive to visual domain shift and that improved HMR reconstruction alone does not necessarily translate to improved downstream gait estimation.