Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis

AI summary

Overview

• Research area: Computer Vision / Multimodal Large Language Models (MLLMs) for embodied agents; benchmark design and failure diagnosis.
• Technical level: Advanced. The paper assumes familiarity with multimodal benchmarks, vision-language prompting, and embodied agent pipelines, though its central argument (task success hides which capability failed) is stated in accessible terms.
• Scope: This paper introduces BEAR, a skill-level diagnostic benchmark of 14 atomic embodied skills (4,469 interleaved image–video–text samples), uses it to diagnose 20 MLLMs, and turns the diagnosis into a tool-augmented agent, BEAR-Agent.

What This Paper Is About

Existing embodied benchmarks mostly score an agent on whether a whole task succeeded, so when a model fails there is no way to tell which underlying capability broke down. BEAR instead breaks embodied tasks into 14 atomic skills across 6 categories, from low-level perception to high-level planning, so failures can be attributed to specific capabilities. Using this, the authors diagnose 20 MLLMs and then build BEAR-Agent to address the two bottlenecks they find.

Key Contributions

1. BEAR, a diagnostic benchmark. Described as the first systematic embodied diagnostic benchmark, with 4,469 interleaved image–video–text samples spanning 6 categories and 14 atomic skills, curated from 13 distinct data sources. It includes a sixth Long-horizon category of 35 episodes collected from AI2-THOR, decomposed into skill-oriented steps.
2. A hierarchical skill-level diagnosis and cross-skill failure attribution framework. It supports horizontal diagnosis (which skills act as bottlenecks inside long-horizon tasks), vertical diagnosis (fine-grained failure analysis of all 14 atomic skills), and cross-skill aggregation to find shared bottlenecks.
3. Diagnostic findings that prior benchmarks did not expose. Perceptual capabilities are the major bottleneck behind reasoning failures, and unstable spatiotemporal modeling is a recurring cross-skill limitation that Chain-of-Thought prompting and test-time compute scaling do not fix.
4. BEAR-Agent, an actionable tool-augmented fix. A multimodal conversable agent that supplies visual and spatial tools (trajectory visualization, GroundingDINO grounding, 3D scene-graph construction) and improves performance in both offline BEAR evaluation and online simulation / real-robot execution.

Main Findings

• MLLMs score low overall. Most evaluated models achieve only 20–40% on BEAR, and even GPT-5, the strongest model, reaches just 52% (52.17 in the results table), far below human performance of 89.40%.
• Proprietary models lead open-source ones. Proprietary models average 39.2%, outperforming open-source models by 13.4%. GPT-5 leads at 52.2%, exceeding the best open-source model InternVL-3 by 18.3%, though recent open-source models begin to surpass GPT and Claude variants.
• CoT and test-time scaling give limited help. Chain-of-Thought prompting produces inconsistent gains across models, with most improvements negligible and typically under 10%; test-time compute scaling shows similarly limited effects.
• Perception, not reasoning logic, is the dominant failure source. Across skill-level failure roadmaps, errors categorized as perceptual account for 54.8% versus 22.4% for reasoning. In reasoning-oriented skills, models often decide wrongly because the visual information they rely on is misperceived or poorly grounded.
• Failures propagate from grounding to reasoning. Examples include localization failures in Pointing and Bounding Box, arrow-color confusion in Trajectory Reasoning, confusing a target journal with a poster in Spatial Reasoning, and action-understanding errors in Task Planning (an attention map from InternVL3-8B shows the model attending to a stirring motion but misreading the action).
• Unstable spatiotemporal modeling is the second bottleneck. In planning, models frequently omit prior steps or historical context (32.0%); in spatial reasoning, the absence of a stable reference frame leads to incorrect left–right judgments and distorted directional understanding (30.3%).
• Long-horizon failures concentrate in perception and spatial skills. Analysis of GPT-4o failures during long-horizon execution indicates perceptual skills (Pointing, Bounding Box) and spatial reasoning skills account for 88% of failures.
• BEAR-Agent improves offline performance. It yields an average gain of 9.12% for GPT-5, a 17.5% relative improvement, and improves overall performance across all categories from low-level pointing to long-horizon reasoning; largest gains appear in Pointing, Bounding Box, and Trajectory Reasoning.
• BEAR-Agent improves online execution. Integrated with the MOKA baseline in simulation, it produces an average 20.17% improvement in task performance; in real-robot experiments it gives a 23.4% improvement over InternVL3-8B.
• Benchmark composition. Of the 4,469 questions: 2,886 (64.6%) contain only one image, 995 (22.2%) only one video, and 588 (13.2%) interleaved data; 2,563 (57.4%) are multiple-choice and 1,906 (42.6%) free-form; 4,169 (93.3%) are newly generated. There are 2,079 unique images and 918 unique videos, with a maximum of 82 question words, maximum 15.9 choice words, average 20 question words, and average 3.7 choice words.
• Human baseline. Human performance is reported from 5 volunteers on BEAR-mini, a subset with 40 samples per skill, reaching an average of 89.40% versus random-choice baselines around 25% per category.

Methodology in Plain English

The authors start by observing that embodied tasks can be decomposed into small, capability-oriented steps. From analyses of large-scale household activity datasets such as BEHAVIOR-1K and ALFRED, plus insights from human cognition, they define five core categories — Task Planning, Spatial Reasoning, Bounding Box, Pointing, and Trajectory Reasoning — split into 14 atomic skills. A sixth category, Long-horizon, uses 35 AI2-THOR episodes where every step maps back to one of the atomic skills, letting the authors check whether a task-level failure can be traced to specific steps.

Data is curated from 13 sources spanning real images, videos, and simulation episodes (for example OpenImages for Pointing and Open-X-Embodiment for Trajectory Reasoning). A multi-stage pipeline combines automated semantic filtering via GPT-o3 with at least three rounds of human verification by trained annotators, plus strict ethical filtering. The authors control question distribution, distractor design (including "none of the above" options), correct-answer position balance, and difficulty calibration (for instance, removing ground-truth masks that are too small or too large in Pointing). Only validation and test sets are used, to reduce data contamination.

For evaluation, they follow VLMEvalKit's default protocol, processing inputs either in a merged setting (multiple frames combined) or a sequential setting (frames processed individually). Success rate is the metric for Pointing, Spatial Reasoning, Task Planning, and Long-horizon; for Long-horizon, an episode counts as successful only if all steps are answered correctly; Bounding Box uses average Intersection over Union (IoU). Diagnosis then proceeds in three layers: horizontal (which skills bottleneck long-horizon tasks), vertical (fine-grained failure roadmaps for all 14 skills), and cross-skill attribution (aggregating error types across skills).

Finally, the two diagnosed bottlenecks motivate BEAR-Agent: a conversable agent that hands the MLLM tools implemented as Python functions, including trajectory visualization, foundation models such as GroundingDINO, Set-of-Mask, and DepthAnything, and a semantic scene graph with a notebook for temporal events. The model writes code to call tools, the agent executes it and returns results, and the conversation terminates when the model signals its final answer. The agent is tested offline on BEAR with GPT-5 and InternVL3-14B against one-shot, few-shot, and chain-of-thought baselines, and online in Maniskill simulation (three sets of table-top manipulation tasks, each with four language instructions, 20 rollouts per variation) and with a real UR5 robot with a custom 3D-printed fingertip (10 instructions, 30 rollouts).

Why This Matters

Impact on research. The paper argues that task-level binary success signals conflate many capabilities and cannot guide improvement. By providing skill-level attribution and cross-skill failure patterns, it offers a way to direct effort at upstream grounding rather than at reasoning prompts, and it backs the diagnosis with an agent that produces measurable gains.

Real-world applications.

• Robot manipulation, where object naming and spatial placement instructions depend on accurate grounding (demonstrated with a UR5 arm).
• Household-service agents, the setting reflected in the BEHAVIOR-1K and ALFRED activity data motivating the skill taxonomy.
• Simulation-based training and evaluation pipelines for embodied agents, as shown with Maniskill.
• Multimodal evaluation tooling more broadly, since the diagnostic method applies to any model that must ground language in images or video.

Industry relevance. The finding that tool augmentation lifts GPT-5 by a 17.5% relative gain, and that proprietary models average 39.2% versus 13.4% lower for open-source models, is directly relevant to teams deciding whether to invest in better model backbones or in surrounding perception infrastructure. The reported gains in simulation (20.17% over the MOKA baseline) and on real hardware (23.4% over InternVL3-8B) suggest that modest perception scaffolding may matter as much as model scale.

Future Directions

• Extending BEAR-Agent beyond its current limited scope by designing targeted training data for supervised fine-tuning and reinforcement learning, as the authors state in their limitations.
• Building methods specifically for unstable spatiotemporal modeling, since the paper reports that CoT prompting and test-time compute scaling do not effectively address it and that additional methods should be designed.
• Improving perceptual grounding as a holistic lever, given that perceptual errors account for 54.8% of limitations and appear to propagate into reasoning failures across skills.
• Broadening the diagnosis to more models and environments, since the benchmark evaluates 20 MLLMs; the paper notes that only 15 models are shown in the main table due to space limits.

Target Audience

Researchers and engineers working on embodied AI, multimodal large language models, and robot learning who need to know why a model fails rather than only that it failed. It is also useful for benchmark designers interested in skill-level decomposition and failure attribution, and for practitioners evaluating whether perception tooling or model choice is the better investment for embodied deployments.

Authors’ abstract

Understanding the capability bottlenecks of embodied multimodal large language models (MLLMs) is crucial for improving embodied agents. However, existing embodied benchmarks mainly focus on task-level evaluation and fail to provide actionable insights into the underlying causes of model failures. To address this limitation, we introduce BEAR, a benchmark that decomposes embodied tasks into 14 atomic skills for fine-grained skill-level evaluation. BEAR comprises 4,469 interleaved image-video-text samples spanning 14 skills across 6 categories, ranging from low-level perception to high-level planning. We evaluate 20 MLLMs on BEAR under a hierarchical skill-level diagnosis framework and uncover two key findings: (1) perceptual capabilities are major bottlenecks behind reasoning failures, and (2) current models suffer from unstable spatiotemporal modeling that remains largely unexposed in prior benchmarks. Motivated by these findings, we further propose BEAR-Agent, a multimodal conversational agent that augments MLLMs with visual and spatial reasoning tools. BEAR-Agent substantially improves performance across embodied skills, achieving a relative improvement of 17.5% on GPT-5 over the base model on BEAR, while also outperforming strong baselines in both simulation and real-world robotic experiments. Project page: https://bear-official66.github.io/