Benchmarking Video Foundation Models for Remote Parkinson's Disease Screening

AI summary

Overview

Research area: computer vision and clinical machine learning, specifically the benchmark evaluation of video foundation models (VFMs) for remote Parkinson's disease (PD) screening from webcam-recorded videos of standardized clinical tasks.

Technical level: Intermediate. Readers need some familiarity with self-supervised video representation learning, frozen-backbone linear probing, and screening metrics (AUC, sensitivity, specificity), but no deep clinical background.

Scope: A large-scale, systematic comparison of seven pretrained video foundation models across 16 MDS-UPDRS-inspired tasks using a 32,847-video dataset from 1,888 participants.

What This Paper Is About

Remote screening for Parkinson's disease could make neurological assessment accessible to people who cannot reach a specialist, but existing video-based approaches rely on handcrafted features built to mimic specific clinical scales. The authors ask whether modern video foundation models — large pretrained transformers that learn general visual and motion representations — can distinguish PD from non-PD participants without any task-specific fine-tuning, and, critically, which model architecture suits which clinical task.

Key Contributions

1. A new webcam-recorded video dataset assembled from eight independent clinical and non-clinical studies (2017–2025), totaling 32,847 videos from 1,888 participants, 727 of whom have PD, spanning 16 standardized clinical tasks grouped into four clinical domains.
2. A systematic benchmark of seven state-of-the-art VFMs — VideoPrism, V-JEPA2, V-JEPA2 (SSv2), ViViT, VideoMAE, VideoMAEv2, and TimeSformer — using frozen embeddings with a task-specific linear classification head.
3. A task-model saliency analysis showing that the best architecture depends on the physiological domain being assessed, rather than one model dominating everywhere.
4. Ablation experiments on multi-view aggregation and oversampling (RandomOverSampler), plus public release of code and anonymized structured data.

Main Findings

• Overall performance range: Across tasks, the best models achieved AUCs of 76.4–85.3% and accuracies of 71.5–80.6%.
• High specificity, lower sensitivity: Specificity reached up to 90.3%, but sensitivity ranged only from 43.2% to 57.3%, which the authors attribute to non-optimal decision thresholds. They argue this supports screening for ruling out healthy individuals more than confirming disease.
• Upper-limb tasks are most salient: Flip Palm and Open Fist produced the highest AUCs (85.3 ± 0.2% and 84.3 ± 0.1%). Flip Palm, with V-JEPA2-SSv2, produced the best result overall: AUC 85.3 ± 0.2%, accuracy 80.6 ± 0.3%, sensitivity 56.2 ± 0.9%, specificity 90.3 ± 0.5%, PPV 69.7 ± 0.9%, NPV 83.9 ± 0.2%.
• Lower-saliency tasks: Fine-motor Finger Tapping and the vocal Tongue Twister task showed AUCs below 78 (Finger Tapping best result was 76.4 ± 0.3% with TimeSformer; Tongue Twister best was 77.1 ± 0.2% with VideoPrism).
• VideoPrism generalizes broadly: It ranked first on 10 of 16 tasks, with strongest advantages in Visual Speech Kinematics, Facial Expressivity, and Oculomotor/Cervical/Cognitive Control (e.g., Eye Gaze 81.0 ± 0.1%, Pangram Utterance 81.3 ± 0.2%, Reverse Count 80.0 ± 0.1%, Open Fist 84.3 ± 0.1%).
• V-JEPA2 dominates gross motor tasks: The V-JEPA2 family was most effective for upper-limb motor kinematics (Extend Arm 83.2 ± 0.1%, Nose Touch 83.0 ± 0.1%, Facial Expression Surprise 80.2 ± 0.1%), and V-JEPA2-SSv2 consistently outperformed base V-JEPA2, suggesting a benefit from motion-centric SSv2 fine-tuning.
• TimeSformer stays competitive on rhythmic motion: TimeSformer was the best model for Finger Tapping despite its simpler divided space-time attention.
• Multi-view training did not help: Single-view without oversampling achieved a mean accuracy of 74.97% (std 3.12%) and mean AUC of 80.54% (std 2.42%) across the 16 tasks; multi-view training did not significantly outperform (p > 0.70 for both accuracy and AUC).
• Oversampling slightly hurt: It non-significantly degraded mean accuracy to 74.51% (std 2.99%) and mean AUC to 79.83% (std 2.82%), suggesting frozen VFM embeddings already handle the existing class imbalance.

Methodology in Plain English

Participants performed 16 standardized motor, speech, facial, and cognitive tasks at home or under clinical supervision, guided by instructional videos on a web-based platform. The tasks map to four domains: upper-limb motor kinematics (finger tapping, flip palm, open fist, extending arms, nose touching), visual speech kinematics (pangram, tongue twisters, sustained phonation of 'a', 'e', 'o' — analyzed video-only, without audio), facial expressivity (smile, disgust, surprise), and oculomotor/cervical/cognitive control (eye gaze, head pose, reverse counting).

Each pretrained model was frozen — its weights were never retrained — and used purely as a feature extractor. Videos were resized and sampled at 15 FPS following each model's own requirements. A small neural network with one hidden layer (ReLU activation, random dropout) plus a linear head was trained on those embeddings with binary cross-entropy loss to predict PD versus non-PD, with the hidden layer size treated as a tunable hyperparameter.

Data were split 60% training, 20% validation, 20% test by unique participant, so test evaluation was always on unseen subjects. Hyperparameters (learning rate, hidden nodes, dropout) were tuned with Weights & Biases using equal compute time per task and model, and the final model per task was chosen by peak validation AUC. Results are reported as means with 95% confidence intervals across 30 random seeds where feasible. All experiments ran locally on a 32-core AMD Ryzen Threadripper PRO 5975WX workstation with dual NVIDIA RTX A6000 GPUs (48 GB vRAM each) and 256 GB RAM, keeping patient videos on-premises.

PD status came from self-report for 445 participants and clinical confirmation for 282. The cohort includes 991 female participants, 1,366 White, 68 Asian, 77 Black, and 27 from other groups, while 350 did not disclose race; Hoehn & Yahr clinical stage was available for 158 participants.

Why This Matters

Research impact: This establishes a reproducible baseline showing that frozen, off-the-shelf video foundation models already capture clinically meaningful PD signs without any fine-tuning, and it replaces the assumption of a single "best" model with evidence that architecture choice should match the clinical domain. It also exposes a calibration gap between high AUC and low sensitivity that future screening work cannot ignore.

Real-world applications:

• At-home pre-screening tools that flag people who should seek specialist evaluation, using only a smartphone or webcam.
• Triage support in regions with limited neurology access, where geographic and financial barriers currently delay diagnosis.
• Modular monitoring pipelines that combine several task-specific models (e.g., V-JEPA variants for limb tasks plus VideoPrism for facial and speech tasks) rather than one general model.
• Objective, video-based measurement of motor signs that could complement specialist MDS-UPDRS ratings, which require trained raters.

Industry relevance: The finding that a frozen backbone plus a lightweight trained head performs competitively points to low-cost deployment, since no per-task retraining of large video models is needed. Privacy constraints also favor the open-weight, locally deployable models evaluated here over cloud-dependent alternatives.

Future Directions

• Explore task-specific fine-tuning and parameter-efficient methods such as LoRA, which the frozen protocol deliberately excluded, to find the achievable performance ceiling.
• Improve calibration (for example, post-hoc scaling of predicted probabilities) so decision thresholds yield sensitivity that supports actual screening use.
• Combine multiple tasks and modalities, including audio for speech tasks, since the study used video-only input for communication tasks.
• Develop more complex multi-view aggregation architectures (the ablations suggest simple aggregation is insufficient) and evaluate newer VFMs as they appear.
• Address generalizability by testing on more racially and ethnically diverse cohorts, since the current cohort is predominantly white.

Target Audience

Clinical machine learning and computer vision researchers working on video representation learning or medical video analysis; neurologists and movement disorder specialists interested in remote monitoring and digital biomarkers; digital health engineers and product teams building remote screening or tele-neurology tools; and benchmark-focused methodologists who care about rigorous, leakage-free evaluation design and model selection across heterogeneous tasks.

Authors’ abstract

Video-based assessments offer a scalable pathway for remote Parkinson's disease (PD) screening. While traditional approaches rely on handcrafted features mimicking clinical scales, recent advances in video foundation models (VFMs) enable representation learning without task-specific customization. However, the comparative effectiveness of different VFM architectures across diverse clinical tasks remains poorly understood. We present a large-scale systematic study using a novel video dataset from 1,888 participants (727 with PD), comprising 32,847 videos across 16 standardized clinical tasks. We evaluate seven state-of-the-art VFMs -- including VideoPrism, V-JEPA, ViViT, and VideoMAE -- to determine their robustness in clinical screening. By evaluating frozen embeddings with a linear classification head, we demonstrate that task saliency is highly model-dependent: VideoPrism excels in capturing visual speech kinematics (no audio) and facial expressivity, while V-JEPA proves superior for upper-limb motor tasks. Notably, TimeSformer remains highly competitive for rhythmic tasks like finger tapping. Our experiments yield AUCs of 76.4 - 85.3% and accuracies of 71.5 - 80.6%. While high specificity (up to 90.3%) suggests strong potential for ruling out healthy individuals, the lower sensitivity (43.2 - 57.3%) highlights the need for task-aware calibration and integration of multiple tasks and modalities. Overall, this work establishes a rigorous baseline for VFM-based PD screening and provides a roadmap for selecting suitable tasks and architectures in remote neurological monitoring. Code and anonymized structured data are publicly available: https://anonymous.4open.science/r/parkinson\_video\_benchmarking-A2C5