MotorSense: A Video-EMG Dataset of Motoric Representations for Action Understanding

AI summary

Overview

Research area: Computer vision / multimodal embodied learning — specifically egocentric video paired with surface electromyography (EMG) for hand action understanding.

Technical level: Advanced. The paper combines a large-scale wearable-sensor dataset, CLIP-style contrastive pre-training, 3D hand/contact reconstruction, and LoRA-conditioned video diffusion.

Scope (one sentence): The paper introduces MotorSense, a 100-hour dataset of synchronized egocentric video, bimanual EMG and IMU, and shows that aligning video representations to muscle activity ("sensorimotor alignment") beats visual-only self-supervised pre-training on action recognition, 3D hand reconstruction, 3D contact reconstruction, and EMG-conditioned video generation.

What This Paper Is About

Computer vision models learn hand actions only from what they look like on camera; they never see the muscle signals that actually produced the movement. MotorSense closes that gap by recording 100 hours of unscripted kitchen and household manipulation with wrist-worn EMG bands synchronized to egocentric video, so that the "cause" of an action (muscle activity) is captured alongside its "effect" (pixels). The goal is to use EMG as a free, dense supervisory signal during pre-training, then throw the EMG away and deploy a purely visual model that has internalized fine-grained hand motor structure.

Key Contributions

1. The MotorSense dataset. The first large-scale paired egocentric video and surface EMG dataset: 100 hours of unscripted hand–object manipulation with synchronized video, bimanual EMG, IMU, and audio, collected with non-obstructive wrist-worn bands rather than sensorized gloves or multi-camera rigs.
2. Sensorimotor alignment as pre-training. A CLIP-style contrastive objective jointly trains a video encoder and an EMG encoder (plus a separate IMU tower) in a shared embedding space; EMG is used only during pre-training and discarded at inference, leaving a transferable vision backbone.
3. Transfer to three action-centric vision tasks. The aligned backbone is evaluated on action recognition, 3D hand reconstruction, and 3D hand contact reconstruction, outperforming every baseline across all tasks and datasets evaluated.
4. Motor-code conditioning for video diffusion. A VQ-VAE learns discrete "motor codes" from raw EMG + IMU without manual labels, which are fed through a HyperNetwork-predicted LoRA to control an image-to-video diffusion model, producing finer hand–object interaction fidelity than no conditioning, language conditioning, or video-history conditioning.

Main Findings

• Large frozen-backbone gains in action recognition. With the backbone frozen, sensorimotor alignment reaches 59.85 verb / 29.79 noun / 21.29 action top-1 on EPIC-Kitchens-100, 33.57 on Something-Something v2, 36.11 keystep on Ego-Exo4D, and 35.02 on MotorSense — margins of +15.86, +5.06, +7.50, +4.40, +8.29, and +14.71 respectively over the best baseline. Baselines were K400-MAE initialization and SimCLR, MoCo, and BYOL trained on the same MotorSense recordings with RGB-only objectives, which isolates the contribution of EMG from that of extra in-domain video.
• Gains persist end-to-end but shrink. Fine-tuned fully, the method scores 74.09 / 52.25 / 43.81 on EPIC-Kitchens-100, 68.76 on SSv2, 51.47 on Ego-Exo4D, and 47.51 on MotorSense (deltas of +3.67, +0.09, +1.96, +1.19, +1.99, +4.68).
• Better 3D hand reconstruction. Using a hand-only ArcticNet-SF with a ResNet-50 backbone, MotorSense pre-training reduces MPJPE from 24.29 to 21.71 mm and A-MPJPE from 38.77 to 35.97 mm on ARCTIC; from 24.28 to 19.65 mm and 51.20 to 44.04 mm on H2O; and from 30.85 to 28.39 mm and 54.58 to 48.85 mm on HOT3D.
• Better contact prediction. On the contact-annotated MotorSense subset, the method reaches 55.20 ± 0.62 IoU and 97.19 ± 0.15 AUROC, beating the strongest baseline (PixPro, 50.59 ± 0.66 and 93.36 ± 0.25) by +4.61 and +3.83. Image-level self-supervised methods (DenseCL, PeCLR, PixPro) trained on MotorSense frames gave at most marginal gains over ImageNet.
• The embedding space organizes by action without labels. A t-SNE plot shows the usual "modality gap" between video and EMG clusters, but cross-modal nearest neighbors retrieve the same hand action from a different recording, object, scene, and participant. Within the EMG cluster, actions like mix salad and scrape bowl form compact regions and verbs group together (e.g. the cut family) despite never being trained on those labels.
• Motor codes give the best generative control. With a frozen CogVideoX-5B-I2V-style backbone, motor-code conditioning achieves FVD 304, LPIPS 0.27, PSNR 22.5 using 50M trainable parameters, versus video-history conditioning at 373 / 0.30 / 22.0 with 250M parameters, full supervised fine-tuning at 772 / 0.35 / 17.3 (5.57B parameters), zero-shot at 926, and language conditioning at 935. Language conditioning "adds little," confirming that text under-specifies fine-grained manipulation.

Methodology in Plain English

Recording. Two MindRove armbands sit just above each wrist, each with 8 dry stainless-steel electrodes arranged around the forearm, sampling at 500 Hz with 24-bit resolution and streaming over Wi-Fi; each band also carries a 6-axis IMU sampled at 50 Hz. Video comes from Meta Project Aria Gen 1 and Gen 2 glasses or a head-mounted smartphone, with video and EMG placed on a single clock so each frame matches the muscle activity of both arms.

Preprocessing. EMG is high-pass filtered (4th-order zero-phase Butterworth at 10 Hz) to remove drift and z-scored per channel per recording. Windows with gaps, saturation, or artifacts are dropped, removing about 4% of candidates.

Pre-training. A vision tower encodes a short video clip and pools spatially only (no temporal pooling — stated as a crucial design choice). An EMG tower uses strided 1D convolutions, residual depthwise-convolution blocks, and an LSTM, with 1 second of context padded around the video window; IMU gets a tower of the same design. Embeddings are L2-normalized and compared by cosine similarity with a learned temperature, and a symmetric InfoNCE loss averages the EMG and IMU contrastive terms. Crucially, each batch samples windows from the same recording, because otherwise the vision encoder can cheat by identifying recording sessions rather than learning action features.

Transfer. The EMG tower is discarded; the vision backbone gets a task-specific head and is fine-tuned (or kept frozen) for action classification or 3D regression.

Generation. A VQ-VAE (embedding dim 256, codebook of 64 entries, code rate 64 codes per second, commitment weight 0.25, 1.5 s windows, separate models per hand) converts EMG + IMU into discrete motor-code sequences. A HyperNetwork maps these codes to low-rank adapters inserted into a frozen video diffusion backbone, so only a motor-code embedding network, the HyperNetwork, and the adapters are trained.

Why This Matters

Impact on research. The paper reframes manual labeling as a bottleneck that an embodied sensor can bypass: EMG is dense, occlusion-free, and free of labeler bias, and it encodes forces that cameras cannot see. It also shows that a signal available only at training time can still shape a deployable vision model, which is a cheap recipe for injecting physical grounding into video backbones.

Real-world applications:

• VR/AR input. Wrist-worn EMG bands are already emerging as control interfaces; motor codes let a user's intended hand motion drive generated or rendered video.
• Robot understanding of human collaborators. A robot could infer intent and applied force from a human partner's muscle activity, not just their visible pose.
• Assistive and rehabilitative technology. Motor-grounded representations could model hand action across diverse bodies and motor abilities.
• High-stakes training. The paper points to surgical training, where virtual practice could ease reliance on scarce resources such as cadavers, plus personalized CGI.

Industry relevance. Consumer-grade EMG hardware already exists, the pre-training pipeline needs no manual annotation, and the generative component is deliberately lightweight (a HyperNetwork plus LoRA over a frozen backbone), making the approach attractive to companies building wearables, egocentric assistants, or content-generation tools.

Future Directions

• Higher-density sensing. The authors flag their 8-channel consumer band as a limitation, noting commercial devices with 16 channels and research systems with upwards of 128 channels would provide a richer signal as such hardware becomes accessible.
• Scaling data and participants. Action segments are annotated on 85 recordings (5,209 segments, 57 verbs, 180 nouns) and contact on 57 smartphone recordings (4,700 intervals); the Aria portion covers only 8 attributed participants, so broader demographics and environments are open.
• Richer generative control. Extending motor-code conditioning beyond the single-reference-frame image-to-video setup — and beyond the LightLoRA design borrowed from Video2LoRA — to longer horizons and more dexterous manipulation is a natural next step.
• Sensorimotor alignment under occlusion and in the wild. The paper hypothesizes the largest benefit where hands are occluded by manipulated objects; systematically testing that hypothesis and transferring to non-kitchen, in-the-wild domains remains open.

Target Audience

Researchers and graduate students in computer vision, egocentric/action recognition, and multimodal representation learning; HCI and wearable-sensing researchers interested in EMG as an input modality; robotics researchers modeling human manipulation; and practitioners in VR/AR or generative video who want physically grounded control signals. Readers should be comfortable with contrastive learning, VQ-VAEs, and diffusion models, though the motivation and results are stated clearly enough for a motivated newcomer to follow.

Authors’ abstract

Computer vision models learn hand action only from its visual consequences, without access to the underlying embodied motor signals. We address this absence with MotorSense, the first large-scale paired egocentric video and surface electromyography (EMG) dataset. MotorSense pairs bimanual EMG bands with egocentric video and yields synchronized video, bimanual EMG and IMU, and audio across 100 hours of unscripted manipulation. EMG offers a powerful scalable alternative to manual annotation: it captures finger movement, hand configuration, and the forces exerted between hand and environment as a dense signal free of occlusion and labeler bias, whereas manual annotation is costly, subjective, and unreliable under occlusion. We use MotorSense to learn sensorimotor alignment, producing an organization of action that is shared across the visual and motoric spaces, through cross-modal pre-training between video and EMG. The resultant vision models are then transferred to downstream tasks that benefit from sensorimotor alignment: in this case, action recognition, 3D hand reconstruction, and 3D contact reconstruction. We further demonstrate a generative application in which EMG motor signals are used to condition a video diffusion model, yielding videos of fine-grained hand manipulation. Across four action-centric computer vision tasks and eight popular video datasets, we demonstrate that sensorimotor alignment provides a stronger foundation for visual representation learning than conventional self-supervised learning methods over video, outperforming every baseline across all tasks and datasets evaluated.