MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching

AI summary

Overview

Research area: Computer vision for action quality assessment (AQA) and skilled activity understanding, combined with multimodal physiological sensing, biomechanics, knowledge-graph representation, and AI-driven fitness coaching.

Technical level: Intermediate to Advanced. The dataset and coaching framing are accessible, but the modeling paradigm (compositional reasoning over structured ontologies, cross-modal video-to-EMG prediction) assumes familiarity with standard AQA architectures, video-language models, and sEMG signal processing.

Scope: The paper introduces a biomechanically grounded multimodal ecosystem — a dataset, a fitness knowledge graph, a compositional reasoning model called CUBIST, and three benchmark tasks — for evaluating and coaching weight-loaded strength exercises.

What This Paper Is About

Existing action quality assessment methods look at movement from the outside only, using RGB video and pose data, and treat an action as one undifferentiated blob to be scored. This misses what is happening inside the body: which muscles are firing, whether force is balanced, and whether coordination is breaking down. The authors argue this is why visually "correct-looking" lifts can still cause injury, and why current systems cannot say which specific step of a movement went wrong or how to fix it.

The goal is to pair external motion with internal muscle activity, organize expert knowledge into a structured, decomposable representation of each exercise, and then build a model that reasons over that structure to produce fine-grained error attribution and corrective feedback rather than a single opaque score.

Key Contributions

1. MyoMechanix, a multimodal weight-loaded exercise dataset. Described as the first biomechanically grounded multimodal ecosystem for weight-loaded actions and the largest multimodal AQA benchmark to date: 7,512 samples of 20 actions from 38 subjects across three ability levels (novice, amateur, expert), totaling more than 40 hours of recordings. It contains synchronized five-view RGB video, 3D pose, sEMG, and additional physiological signals (heart and breathing signals), with expert annotation.

2. The Fitness Knowledge Graph (FKG). An ontology that organizes expert annotations into structured relationships linking actions, phases (e.g., preparation, concentric, eccentric), ordered action key steps (AKs), error types (ETs), and corrective feedback messages (FBs). It supports a compositional, penalty-based scoring protocol in which errors are weighted by their biomechanical impact.

3. CUBIST (Compositional Ontological Reasoning Engine). A new modeling paradigm inspired by decomposition–analysis–recomposition: it splits an action into structured components, analyzes each with multimodal inputs and specialized modules, then recomposes the results into a coherent assessment with error attribution and feedback.

4. Three benchmarks built on the ecosystem: MyoMechanix-AQA for structured action quality assessment, MyoMechanix-VideoQA for language-grounded diagnosis and feedback, and MyoMechanix-Video2EMG, a new cross-modal task inferring muscle activation patterns from video.

Main Findings

• Multimodal structured sensing helps. Combining physiological signals with structured representations improved performance, interpretability, and error attribution over existing state-of-the-art baselines. CUBIST achieved state-of-the-art results.

• VideoQA improves vision-language models. Training on MyoMechanix-VideoQA enhanced language-grounded, fine-grained action understanding in vision-language models, supporting query-conditioned reasoning about posture, phases, key steps, errors, muscle involvement, causality, and corrections.

• Video2EMG is promising. Results on inferring muscle activation from video suggest a cheap video-based alternative to expensive, specialized EMG sensors.

• The benchmark remains hard. The authors state explicitly that MyoMechanix is challenging with considerable room for improvement, motivating further research.

• Dataset comparison. Against FitAQA (5,512 samples, 30 exercises, 1 view, novice/amateur, web-sourced) and EgoExo-Fitness (6,131 samples, 12 actions, 6 views, novice/amateur), MyoMechanix has 7,512 samples, 20 exercises, 5 views, and covers novice, amateur, and expert levels with motion-capture sourcing. It is also the only one of the compared fitness datasets listed with equipment-based weight loading and a "high" injury risk rating.

• Annotation depth. The paper claims MyoMechanix is the first AQA benchmark to simultaneously provide final macroscopic scores, microscopic execution parameters, and interpretable intermediate reasoning-step annotations.

• Note on quantitative results. The provided paper content is truncated before the experimental results tables, so specific numeric scores for CUBIST and the baselines are not reported here.

Methodology in Plain English

The team did not scrape videos from the web. They collected everything in-house using high-precision motion capture, five RGB cameras (including ordinary phone cameras), a 16-channel sEMG system sampling at 2 kHz, and physiological sensors, with strict hardware-level synchronization between the EMG and motion-capture systems so that muscle signals line up precisely in time with movement.

They selected 20 weight-loaded compound exercises using four principles: covering both upper- and lower-body muscle groups, involving complex and injury-prone joints such as shoulders, knees, and hips, including free-weight movements that demand stabilization, and prioritizing widely practiced exercises.

Multiple fitness experts then annotated every sample. Instead of assigning a subjective overall score, the experts marked specific execution errors at the level of individual action key steps. These annotations were organized into the Fitness Knowledge Graph, drawing on authoritative training standards, which turns each exercise into a structured procedure. Errors are weighted by biomechanical severity, so a final score can be computed compositionally and traced back to the errors that produced it.

CUBIST then exploits this structure: it decomposes an action into its graph-defined components, analyzes each component using the available modalities, and recomposes the per-component analyses into a final assessment plus explanations. For Video2EMG, the model is trained to map video frames to the continuous sEMG signals measured at the same moments, using real muscle activation as supervision rather than predicting discrete muscle names.

Why This Matters

Research impact. The work argues for a shift from purely visual pattern recognition toward biomechanically grounded, multimodal, compositional reasoning. It introduces physiological signals into AQA for the first time, contributes structured process-level annotations that most AQA datasets lack, and defines a new cross-modal task (Video2EMG). The authors position this as a foundation for next-generation Physical AI and for representation learning research, with relevance to fitness, rehabilitation, healthcare, and machine learning.

Real-world applications:

• AI fitness coaching that identifies the specific step where a lift went wrong and prescribes a correction.
• Rehabilitation monitoring, where movement quality is tracked without specialized hardware.
• Motor function assessment and remote neuromuscular analysis.
• Clinically informed evaluation of movement disorders.
• Smart healthcare systems built on low-cost, video-based sensing.

Industry relevance. The Video2EMG result points toward software substitutes for expensive EMG hardware, which matters for consumer wearables, fitness apps, tele-rehabilitation platforms, and any product that wants physiological insight from a phone camera. The ecosystem's phone-camera-compatible video and expert-validated feedback also make it directly applicable to scalable coaching services.

Future Directions

• Closing the performance gap. The authors state that MyoMechanix remains challenging and explicitly call for further research in physiologically grounded and compositional action understanding; improving CUBIST and baselines on the hardest cases is the obvious next step.

• Extending Video2EMG. Validating video-based surrogate sensing more broadly, and outside the fitness setting, would test whether it generalizes to rehabilitation and clinical neuromuscular assessment.

• Broadening domain coverage. The dataset covers 20 weight-loaded compound exercises with 38 subjects across three ability levels; expanding the exercise catalog, load ranges, and populations (for example clinical or rehabilitation groups) is a natural extension.

• Deploying the feedback loop. Turning VideoQA-style diagnostic reasoning into interactive, query-conditioned coaching systems and general-purpose foundation models for fine-grained human action understanding remains an open problem the authors frame as a goal rather than a solved result.

Target Audience

Researchers working on action quality assessment, skilled activity understanding, and multimodal video understanding; computer vision scientists interested in structured, knowledge-graph-based reasoning and vision-language models; biomechanics and sports science researchers who need quantitative, physiologically grounded movement analysis; and fitness technology or digital health practitioners building coaching, rehabilitation, or remote assessment products.

Authors’ abstract

Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/