Deep Learning Pose Estimation for Multi-Label Recognition of Combined Hyperkinetic Movement Disorders

AI summary

Overview

Research area: Computer vision applied to clinical neurology — markerless deep learning pose estimation combined with classical supervised machine learning for movement disorder phenotyping.

Technical level: Intermediate. The pose-estimation step uses a modern deep learning architecture (YOLOv8), but the downstream classification operates on engineered tabular kinematic features rather than deep temporal models, so no specialised sequence-modelling background is required.

Scope: A single-centre, exploratory proof-of-concept study of 25 participants that tests whether smartphone video, YOLOv8 pose estimation, and interpretable kinematic features can support patient-level multi-label classification of eight co-occurring hyperkinetic movement disorders.

What This Paper Is About

Clinicians currently identify hyperkinetic movement disorders such as dystonia, tremor, and chorea by direct observation and rating scales, which suffer from inter-rater variability and cannot easily capture several co-occurring abnormal movements at once. Existing machine learning approaches mostly perform binary classification of a single disorder on short, task-specific video segments, using shallow representations such as raw joint coordinates or velocity. This paper asks whether routine outpatient video processed through deep learning pose estimation plus clinically interpretable kinematic descriptors can produce a simultaneous, patient-level multi-label phenotype profile across eight hyperkinetic movement disorders.

Key Contributions

1. A hybrid pose-based pipeline that couples deep learning markerless pose estimation (YOLOv8, 17 keypoints per frame) with feature-engineered kinematic descriptors and conventional supervised classifiers, explicitly framed by the authors as feature engineering–based supervised multi-label classification rather than true sequential deep learning.
2. A clinically interpretable feature set of 374 derived features per 10-second window (22 descriptors across 17 landmarks), organised into kinematic families (posture/bias, excursions, variability, rhythmicity, directionality, irregularity/complexity) and anatomical regions (cranial, upper limb, lower limb).
3. A patient-level multi-label evaluation protocol using p90 percentile pooling of window probabilities, label-specific control-aware thresholds tuned on training patients only, and a hierarchy of four reporting strategies including prespecified nested cross-validation with per-label model selection as the primary analysis.
4. A decision-level interpretability analysis using permutation feature importance computed on held-out outer test folds only, preserving the full inference pipeline (window scores, p90 pooling, and label-specific thresholds).

Main Findings

• Primary nested cross-validation result: With prespecified nested cross-validation and per-label model selection within training folds, macro-AUPRC was 0.717 ± 0.030, macro-AUROC was 0.767 ± 0.069, Hamming accuracy was 0.764 ± 0.014, and patient–label agreement was 153/200 (76.5%).
• Best single pipeline by discrimination: StandardScaler + multilayer perceptron reached macro-AUPRC 0.821 ± 0.019 and macro-AUROC 0.830 ± 0.029.
• Best single pipeline by Hamming accuracy: MinMaxScaler + support-vector machine reached 0.764 ± 0.041.
• Exploratory upper bound: Post-hoc per-label selection of the best-performing pipeline produced patient–label agreement of 172/200 (86.0%), a 9.5-percentage-point gap above the nested cross-validation estimate that the authors attribute to model-selection optimism.
• High specificity after aggregation: Window-level predictions aggregated within each subject yielded subject-level specificity of 1.00 for all symptoms and all raters, meaning healthy controls were rarely labelled symptomatic at the subject level.
• Phenotype-dependent window-level performance: Dystonia (F1_1 = 0.84 vs 0.89; patient sensitivity = 0.92 vs 1.00), chorea (F1_1 = 0.81 vs 0.96), tremor (F1_1 ≈ 0.82–0.83, ΔF1 ≈ 0.01; patient identification ≈ 0.89–0.92), and tics (patient sensitivity = 1.00 in both annotation sets, though F1_1 = 0.75 ± 0.25 for Rater 1) were the most reliably detected. Athetosis (F1_1 = 0.59 vs 0.49; patient sensitivity 0.80 vs 0.75), stereotypies (F1_1 = 0.54 ± 0.26; patient sensitivity = 0.75), myoclonus (F1_1 = 0.68 vs 0.36; patient sensitivity 0.90 vs 0.43), and ballismus (F1_1 = 0.14 vs 0.23; patient sensitivity 1.00 vs 0.50) were harder.
• Inter-rater agreement tracked difficulty: Window-level Cohen's κ was 0.600 for chorea, 0.366 for myoclonus, 0.259 for athetosis, 0.220 for ballismus, 0.116 for tics, and 0.000 for stereotypies.
• Per-label error decomposition (post-hoc upper bound, 200 patient–label decisions): Tics showed TP=3 and TN=22 with no discordant decisions; dystonia had TP=20, FN=1, FP=1; myoclonus and chorea had no false negatives with remaining discordances dominated by over-calling; tremor had the largest discordance burden (FN=4, FP=4); athetosis showed balanced under- and over-calling (FN=2, FP=3); ballismus maintained FP=0 with FN=2 under post-hoc selection and FN=3 under nested cross-validation.
• Low-prevalence labels are unstable: Ballismus (3 positive patients), stereotypies (7), and tics (3) carry wide confidence intervals. Under nested cross-validation, ballismus sensitivity was 0.00 (95% Wilson CI 0.00–0.56) and tics sensitivity was 0.67 (95% CI 0.21–0.94); under post-hoc per-label best, the same labels reached 0.33 (95% CI 0.06–0.79) and 1.00 (95% CI 0.44–1.00).
• Condition robustness: Dystonia and tremor remained highly discriminable across rest, posture maintenance, and action in several model families, with multiple pipelines achieving near-ceiling performance; the authors note the condition-specific subsets were smaller and treat this as supportive evidence only.
• Focused four-class analysis: Dystonia (prevalence 0.84) showed near-ceiling sensitivity with minimal false alarms, whereas chorea (6/25 positive participants) and athetosis (9/25) remained more challenging, with residual chorea–athetosis confusion consistent with known phenomenological overlap.

Methodology in Plain English

The researchers recorded video during routine outpatient visits at a movement disorders clinic (Montpellier Beau Soleil Clinic, France) using smartphone cameras at a minimum of 30 frames per second and 1920×1080 resolution, with standardised camera positioning and distance. Participants performed predefined motor tasks including resting, posture-holding, finger tapping, arm flexion/extension, target reaching, and writing.

Two physicians with movement disorder expertise independently rated the presence or absence of each of eight hyperkinetic phenotypes — dystonia, tremor, myoclonus, chorea, athetosis, tics, ballismus, and stereotypies — then met to resolve discrepancies into a final consensus label. Each phenotype was evaluated across three conditions: rest, posture maintenance, and action. Clinicians also marked uncertainty ('2'), and windows with uncertain annotations were discarded.

Video was processed frame by frame with YOLOv8 (yolov8x-pose-p6.pt) to extract 17 keypoints per frame. From each landmark's 2D trajectory, a scalar displacement signal was computed as the Euclidean magnitude relative to the image origin. Signals were cut into fixed 10-second windows aligned to the expert annotations. Within each window, the pipeline computed 19 primary descriptors (distributional summaries such as mean, median, standard deviation, variance, interquartile range, range, minimum and maximum; FFT peak frequency and amplitude excluding the DC component; Higuchi fractal dimension; and permutation entropy) plus 3 rolling-mean descriptors, giving 22 features per landmark, or 374 features per window across the 17 landmarks.

Classification used conventional supervised models: gradient-boosted decision trees (XGBoost, LightGBM), random forests, logistic regression, support-vector machines with an RBF kernel, k-nearest neighbours, and a multilayer perceptron. Preprocessing used feature scaling (StandardScaler, MinMaxScaler, RobustScaler, PowerTransformer) followed by univariate feature selection with mutual information (SelectKBest). When the relative difference between class sizes exceeded 20%, random undersampling of the majority class was applied within each training fold, and Platt scaling was used to calibrate probability estimates where needed.

Each of the eight phenotypes was modelled as an independent binary task at the window level. Window probabilities were pooled within each patient using 90th percentile aggregation to obtain one probability per patient and label. Decision thresholds were tuned per label on training patients only, using a policy that minimises a balanced error criterion under explicit constraints on false positives among control subjects. All windows from a given participant were kept in the same cross-validation fold, and folds were balanced by patient-level phenotype profiles using MultilabelStratifiedKFold with a fixed random seed. Feature importance was measured by permutation on held-out outer test folds only, defined as the increase in patient-level error (FP+FN)/N. The full pipeline was implemented in Python 3.10.17, and the code and results are available at https://github.com/xaviervasques/CODY.

Why This Matters

The paper targets an unmet need the authors identify explicitly: no available composite assessment tool enables simultaneous identification and monitoring of movement disorders across diseases that associate multiple hyperkinetic phenotypes, in both children and adults. By working from routine smartphone video rather than specialised motion-capture equipment, the approach is designed to reflect real-world clinic conditions and to be compatible with remote assessment. The authors position these results as proof-of-concept only, requiring external, multicentre, prospective validation before any clinical or trial use.

Real-world applications (as framed in the paper):

• Supporting diagnosis and objective phenotyping of complex, co-occurring hyperkinetic movement disorders in neurology outpatient clinics.
• Disease monitoring over time, including in paediatric and adult populations across the lifespan.
• Remote assessment, using smartphone video in place of in-person rating scales.
• Therapeutic decision-making and the development of targeted, symptom-specific interventions, as well as use in clinical trials.

Industry relevance: The paper does not report industry partnerships, commercialisation plans, or regulatory submissions. Its relevance to industry is structural: the pipeline depends only on smartphone video and open, reproducible Python tooling, which lowers the barrier to deployment, telemedicine integration, and trial endpoint development. The authors state explicitly that no clinical claim should be made without external, multicentre, prospective validation.

Future Directions

• Larger, multicentre cohorts with adjudicated labels. The authors state that these are needed before any clinical claim of discrimination — particularly chorea versus athetosis — can be made.
• Phenotype-specific clinical features on top of the generic descriptors. The authors anticipate complementing the generic set with spatiotemporal gait parameters, tremor-band power and centre frequency for tremor, jerk-spike statistics for myoclonus, segment-level torsion for dystonia, and finger-tap kinematics for action chorea, most naturally within hierarchical pipelines combining a generic stage with phenotype-dedicated downstream modules.
• Moving toward a fully end-to-end system. The current framework is semi-automated and annotation-guided: pose estimation and feature engineering are automatic, but the 10-second window boundaries and per-window labels come from expert clinicians. A fully end-to-end version would require automatic detection and segmentation of motor-examination tasks within long routine recordings.
• Addressing rare and low-agreement phenotypes. Ballismus, stereotypies, and tics showed the widest confidence intervals and the lowest window-level inter-rater agreement, raising the question of whether better annotation protocols, longer windows, or improved 2D pose resolution for fine distal and subtle facial signals could stabilise these labels.

Target Audience

Movement disorders neurologists and clinical researchers interested in objective phenotyping; machine learning and computer vision researchers working on clinical video analysis and human pose estimation; clinical trial methodologists looking for scalable digital endpoints; and biomedical engineers developing remote monitoring or telemedicine tools. The paper is also useful for readers studying the practical limits of small-cohort, single-centre machine learning studies, since it reports how much apparent performance depends on model-selection strategy.

Authors’ abstract

Hyperkinetic movement disorders (HMDs) such as dystonia, tremor, chorea, myoclonus, and tics are disabling motor manifestations across childhood and adulthood. Their fluctuating, intermittent, and frequently co-occurring expressions hinder clinical recognition and longitudinal monitoring, which remain largely subjective and vulnerable to inter-rater variability. Objective and scalable methods to distinguish overlapping HMD phenotypes from routine clinical videos are still lacking. Here, we developed a pose-based machine-learning framework that converts standard outpatient videos into anatomically meaningful keypoint time series and computes kinematic descriptors spanning statistical, temporal, spectral, and higher-order irregularity-complexity features.