SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

AI summary

Overview

Research area: Computer vision and multimodal large language models (MLLMs), specifically egocentric (first-person) video understanding and human body-motion perception.

Technical level: Intermediate. The paper is a benchmark-and-evaluation study, so the core ideas are accessible, but familiarity with vision-language models, video question answering, and 3D body pose helps.

Scope: SOMA is a 1,632-question, 12-task benchmark built from Nymeria and Ego-Exo4D first-person video that tests whether MLLMs can perceive and reason about the wearer's own body, its motion, and how actions unfold in time.

What This Paper Is About

Current evaluations of MLLMs mostly test recognition of objects, scenes and coarse actions, and mostly from a third-person view, so it is unclear whether models actually understand the moving human body. This is especially hard egocentrically: the wearer's body is only partly visible, motion is spread across frames and often blurred, and cues must be inferred indirectly. The paper builds a benchmark that isolates this capability, measures 37 models and five humans against it, diagnoses how models fail, and tests whether adding an explicit body-motion signal to a model helps.

Key Contributions

1. A body-centric egocentric benchmark. SOMA contains 1,632 human-verified multiple-choice questions across 12 tasks grouped into three capability axes: Body Configuration (Hand Use, Bimanual Coordination, Body Posture, Concurrent Body-Part State), Motion Dynamics (Repetition Counting, Movement Direction, Opposite Motion Discrimination), and Temporal Reasoning (Adjacent Action Identification, Chronological Ordering, Missing Event Inference, Next Action Prediction, Repeated Event Reasoning). It is drawn from 397 unique videos producing 1,611 clips, with an average clip length of 35.6 seconds.

2. A multi-stage, leakage-controlled construction pipeline. Questions are generated from Nymeria action annotations and motion narrations using a Gemini 2.5 Flash selector plus generator, yielding 3,928 candidates; a blind-language filter using four models (Gemini 2.5 Flash, GPT-4o, Qwen2.5-72B, InternVL3-78B) removed 45.9% of them, leaving 2,127 for human review, where a reviewer checked each item against the video and a synchronized 3D body-pose rendering.

3. A large-scale evaluation, including a human baseline. 37 closed-source, open-source and specialized egocentric MLLMs were evaluated, alongside five compensated human evaluators who independently answered all 1,632 questions over more than 450 hours of work.

4. A training dataset and multimodal baseline. SOMA-Train provides 28,596 question-answer pairs over 110.2 hours of aligned egocentric video and body motion, kept disjoint from the evaluation set at the participant and scene level; two backbones (Qwen3-VL-8B and InternVL3.5-8B) were extended with a motion pathway and fine-tuned in motion-only, video-only and video+motion settings.

Main Findings

• A large but uneven human–model gap. The best model, Qwen3.6-27B, reaches 50.0% zero-shot accuracy and the best closed-source model, Gemini 3.5 Flash, reaches 47.0%, against 68.5% mean human accuracy and a 25.0% random-chance floor.

• The gap is task-dependent. Relative to the best model per task, humans lead most on Repetition Counting (45.3 points), Repeated Event Reasoning (25.4) and Bimanual Coordination (25.3), but the gap reverses on Concurrent Body-Part State (−1.1) and Next Action Prediction (−10.0), where the answer is not visible in the clip and must be inferred by both humans and models.

• Frame-level recognition beats temporal integration. Best-model accuracy is highest where evidence is concentrated in a few frames, such as Hand Use (74.6%) and Opposite Motion Discrimination (71.4%), while on Chronological Ordering and Repeated Event Reasoning most models score below 40%.

• Egocentric specialization does not help. The three specialized egocentric models do not outperform general-purpose open-source models of similar size, which the authors attribute to training data focused on actions, objects and intentions rather than whole-body posture and motion.

• Scaling helps but does not close the gap. Qwen3-VL gains 13.1 points from 2B to 32B, Qwen3.5 gains 17.8 points from 0.8B to 27B, and InternVL3.5 improves by 8.1 points from 1B to 38B but is not monotonic.

• More frames saturate quickly. Across nine models tested with 1 to 64 uniformly sampled frames, accuracy rises 5.4 points on average from 1 to 16 frames, while frames beyond 16 add at most 0.5; the 1-to-16 gain ranges from 1.2 points (VideoLLaMA3-7B) to 10.3 points (Qwen3.5-9B).

• Test-time reasoning is not a fix. Enabling "thinking" helps only five models by at least one point, all from the Qwen3.5–3.8 family, with gains of 1.0 to 10.2 points; the other seven show negligible change or decline.

• A right-hand bias. Averaged over the 37 models, accuracy is 78.8% for "right hand" but 44.3% for "left hand" on visually unambiguous questions, whereas humans score 92.7% (left) and 89.8% (right); the authors link this to right-handedness being roughly 90% prevalent in the population and thus over-represented in training corpora.

• Systematic under-counting. 24 of the 37 models lean toward counting too few repetitions, consistent with repetitions falling between sampled frames, though the asymmetry is strongly model-dependent.

• Errors concentrate on the wearer's own body. In a 200-question stratified analysis of questions both Qwen3.8-27B and Gemma-4-31B answered incorrectly, four error types were identified – Visual Grounding, Motion and Body-State Perception, Temporal Localization, and Prior-Driven Inference – with motion and body-state perception dominating on the Body Configuration and Motion Dynamics axes.

• Body-motion input gives a real but modest boost. For Qwen3-VL-8B, average accuracy moves from 35.39 zero-shot to 41.49 with video fine-tuning and 42.44 with video+motion; for InternVL3.5-8B, from 35.20 to 41.42 and 41.93. Motion-only fine-tuning stays above chance (28.21 and 29.54) but low, and on Motion Dynamics motion helps Qwen3-VL-8B while slightly hurting InternVL3.5-8B; existing motion-only models evaluated zero-shot remain near chance.

Methodology in Plain English

The authors start from two existing egocentric video corpora that already contain fine-grained action annotations and, for Nymeria, natural-language descriptions of posture, hand motion and leg motion. For each of the 12 tasks, they first use an LLM to select a task-relevant stretch of a video from those annotations, then a second LLM turns that stretch into a four-option multiple-choice question with one correct answer and three task-specific distractors. Repetition Counting is the exception: exact counts could not be extracted reliably from narrations, so those questions were written and counted manually.

Because a language model can often guess an answer from the question text alone, the authors run a blind filter in which four LLMs answer each question with no video access, and discard any question that at least two of them get right. A human then reviews every surviving question with the video and a synchronized 3D body-pose rendering, and can accept, edit or delete it.

For evaluation, all models are run zero-shot with temperature 0 and 16 uniformly sampled frames per clip using the lmms-eval framework, with ablations over frame count (1 to 64) and test-time reasoning. Five paid evaluators answer the same questions without prior familiarity. Finally, two open-weight models are given an extra motion pathway – 3D body motion encoded with a frozen VQ-VAE, projected into the language model's embedding space as motion tokens – and fine-tuned on SOMA-Train to see whether explicit motion supervision closes part of the gap.

Why This Matters

Impact on research. SOMA separates "understanding the body" from "recognizing the scene the body is in," a distinction existing egocentric benchmarks do not make. Its finding that egocentric-specialized models offer no advantage on body motion, and that adding temporal frames or reasoning tokens yields little, identifies a specific missing capability rather than a generic scaling problem. The release of SOMA-Train and of a video+motion baseline gives the field a concrete starting point.

Real-world applications:

• Augmented-reality and wearable assistants that need to know what the wearer's hands and body are doing to offer timely help.
• Robot learning from first-person demonstrations, where the demonstrator's body state matters as much as the objects being manipulated.
• Coaching and rehabilitation, including counting repetitions and judging posture during exercise or physiotherapy.
• Assistive technologies for older adults or people with limited mobility, where detecting posture and falls matters.

Industry relevance. Companies building smart glasses, embodied agents or video-capable assistants need reliable first-person body perception; the paper's diagnosis of right-hand bias and under-counting points to concrete failure modes that would surface in deployed products. The finding that motion signals are complementary but insufficient with vanilla baselines also flags where engineering effort is likely needed.

Future Directions

• Better multimodal fusion. The video+motion gain is modest and inconsistent across backbones and axes; stronger architectures or training curricula for integrating proprioception-like signals remain open, and the authors note world models as a possible route.

• Fixing inherited priors. The right-hand bias and the under-counting tendency suggest that debiasing training data, or training models to override priors when visual evidence is weak, is a concrete target.

• Temporal integration beyond frame sampling. Because accuracy saturates by 16 frames while temporal tasks remain the weakest, methods that explicitly model motion across time, rather than sampling more frames, appear necessary.

• Egocentric models trained on bodies, not just actions. The lack of advantage for specialized egocentric models raises the question of what supervision would make them sensitive to posture, limb use and movement direction.

Target Audience

Researchers and engineers working on multimodal LLMs, video question answering and egocentric vision; teams building AR, wearable or robotics systems that depend on first-person human motion understanding; and benchmark designers interested in the blind-language filtering and human-verification pipeline used here. It is also a useful read for anyone assessing whether current video models genuinely understand people or merely the scenes around them.

Authors’ abstract

Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone (68.5% accuracy), yet clearly outperform the strongest MLLM (Qwen3.6-27B), which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. Full benchmark, training data, and code will be released at: https://cnai.epfl.ch/soma