Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes

AI summary

Overview

Research area: Multimodal AI evaluation — spatial reasoning and embodied perception in vision-language models (VLMs), with direct relevance to Vision-Language-Action (VLA) robotics.

Technical level: Intermediate. The benchmark design and evaluation protocol are accessible to anyone familiar with multimodal models; the correlation analysis and CoT-augmentation experiments assume some background in VLM evaluation.

Scope: The paper introduces MV-RoboBench, a 1,708-item multiple-choice benchmark that measures whether VLMs can fuse synchronized multi-camera views into reliable spatial and action-oriented reasoning for robotic manipulation.

What This Paper Is About

Robots increasingly carry multiple cameras to compensate for occlusion, depth ambiguity, and narrow fields of view, but existing VLM benchmarks almost exclusively test single-image spatial reasoning. The authors ask whether state-of-the-art VLMs can actually exploit complementary multi-view inputs to reason about 3D scenes and make manipulation decisions. To answer this, they build a human-annotated benchmark grounded in real robotic demonstrations and evaluate a broad set of open-source, proprietary, and reasoning-optimized models against human performance.

Key Contributions

1. The first multi-view spatial reasoning benchmark for robotic manipulation. MV-RoboBench contains 1,708 manually curated five-choice QA items drawn from 980 real manipulation episodes in the AgiWorld and BridgeV2 datasets, covering both single-arm and dual-arm settings.

2. A two-axis task taxonomy. The benchmark splits into four spatial understanding subtasks (cross-view matching, distance judgement, viewpoint identification, 3D spatial consistency) and four robotic execution subtasks (action planning, step execution, trajectory selection, affordance recognition), enabling analysis of how perception relates to action.

3. A large-scale zero-shot evaluation across 30+ models with CoT-style augmentations. The authors test blind LLMs, proprietary multimodal models, proprietary reasoning models, open-source VLMs (Gemma-3, InternVL3, Qwen2.5-VL), and open-source MoE models (Llama-4), plus three augmentation strategies: textual scene descriptions, synthesized novel views, and depth priors.

4. Two correlation analyses with practical implications. An internal axis links spatial to robotic accuracy within the benchmark, and an external axis tests whether performance on a general single-view spatial benchmark (OmniSpatial) transfers to multi-view embodied reasoning.

5. Open release of data, templates, and a standardized evaluation protocol to support future spatially grounded VLM and VLA development.

Main Findings

• A large gap to human performance remains. The best model, GPT-5, reaches 56.41% average accuracy; Gemini-2.5-pro scores 49.52%, o4-mini 46.47%, and the best open-source model (Qwen2.5-VL-72B) only 24.29%. Human participants average 91.04%.

• Reasoning-optimized architectures dominate. Explicit reasoning models vastly outperform perception-oriented multimodal models. On the hardest subtask, 3D Spatial Consistency, most non-reasoning models sit at or below random-choice accuracy (19.07%), while reasoning models score 49–82%.

• 3D spatial consistency is the primary failure mode. Maintaining consistent relative object positions across viewpoints appears to require explicit reasoning rather than pattern matching; it is the single strongest discriminator between model families.

• Multi-view input is genuinely necessary. A single-view ablation shows significant drops — for example, roughly a 19% decline in Distance Judgement for GPT-5 — confirming that the benchmark tests integration rather than redundant perception.

• CoT-style augmentations help inconsistently. Depth priors gave the largest single gain for GPT-4.1 (+3.25 points) and a modest gain for Qwen2.5-VL-7B, while CoT prompting helped Gemma-3-12B substantially (+3.70). Synthetic novel views consistently hurt performance, and generic prompt-level augmentation was neutral or negative for already-optimized proprietary models.

• Spatial and robotic reasoning correlate positively, but only for capable models. Proprietary and reasoning-optimized systems show a monotonic trend where better spatial accuracy accompanies better execution accuracy; most open-source VLMs cluster near random on both axes.

• Single-view spatial skill does not transfer. Models that score well on the OmniSpatial benchmark often remain near random on MV-RoboBench, indicating that viewpoint integration, occlusion resolution, and cross-view fusion impose demands that existing benchmarks do not exercise.

Methodology in Plain English

The researchers started from existing robotic datasets that already contain synchronized footage from multiple cameras. They filtered image pairs for temporal separation, scene diversity, and clarity, using GPT-4.1 only as a triage aid to check whether a pair could support one of eight defined task types — never to generate questions or answers. Trained human annotators then wrote five-choice questions from task-specific templates, deliberately keeping distractors plausible but unambiguous. A separate human-in-the-loop review stage discarded or corrected flawed items and rebalanced answer distributions to remove positional bias.

For evaluation, every task was framed as a multiple-choice question with a single correct answer, and all models were run zero-shot under one unified prompt so that no model-specific prompt engineering could tilt the comparison. Accuracy was the sole metric. Five computer science participants completed the same items to establish a human reference point.

Beyond the baseline sweep, the authors tested three ways of giving models extra scaffolding: a textual chain-of-thought prompt, an automatically generated holistic scene description, an intermediate view synthesized by the VGGT reconstruction model, and per-pixel depth maps from MoGe-2 rendered as color-coded images. They also cross-referenced model scores against OmniSpatial, a well-known single-view spatial benchmark, to test transfer.

Why This Matters

Research impact. The paper redirects attention from single-image spatial QA to the multi-view, action-grounded setting that real robots actually operate in. Its two negative results — that generic CoT scaffolding is unreliable and that single-view benchmark gains do not transfer — argue that multi-view spatial reasoning needs dedicated architectures and training signals rather than prompt tricks or more perception data.

Real-world applications:

• Warehouse and logistics robots that use head-mounted and wrist cameras together to pick items from cluttered bins where any single view is partially occluded.
• Home service and assistive robots that must disambiguate objects on tabletops from multiple viewpoints before grasping.
• Surgical and teleoperated robotics, where multi-camera rigs provide complementary depth cues and where an incorrect spatial judgement has direct safety consequences.
• Autonomous mobile manipulators and humanoids, where cross-view consistency is a prerequisite for reliable navigation-plus-manipulation pipelines.

Industry relevance. VLA models are increasingly the backbone of commercial robotics stacks, and they inherit their perception from VLMs. MV-RoboBench gives teams a concrete diagnostic for whether a candidate backbone can fuse camera arrays, and it isolates specific bottlenecks — viewpoint transformation, occlusion, frame-of-reference errors — that vendors can target during fine-tuning. The finding that depth priors help only sufficiently capable backbones also gives practical guidance on where to invest.

Future Directions

• Architectures with explicit geometric priors. The authors argue scaling perception alone is insufficient and call for designs that encode cross-view consistency directly rather than hoping it emerges from attention over concatenated images.
• Training pipelines that align perception with action grounding. The observed spatial-execution correlation suggests joint objectives could raise both, but only for models with enough capacity to exploit them.
• Larger, more diverse multi-camera datasets. The benchmark draws from two existing datasets; broader coverage of sensor configurations, baselines, and environments would test whether the failure modes generalize.
• Better multi-view synthesis and depth integration. Current novel-view synthesis methods failed badly under narrow baselines and gripper-centric viewpoints, leaving open how to supply useful visual scaffolding without adding noise.

Target Audience

Robotics and embodied-AI researchers building VLA systems; multimodal model developers who want to know where their models break down on 3D and cross-view tasks; benchmark and evaluation researchers interested in human-in-the-loop dataset construction and transfer analysis; and industry engineers selecting or fine-tuning vision-language backbones for multi-camera robotic platforms.

Authors’ abstract

Vision-language models (VLMs) are essential to Embodied AI, enabling robots to perceive, reason, and act in complex environments. They also serve as the foundation for the recent Vision-Language-Action (VLA) models. Yet most evaluations of VLMs focus on single-view settings, leaving their ability to integrate multi-view information underexplored. At the same time, multi-camera setups are increasingly standard in robotic platforms, as they provide complementary perspectives to mitigate occlusion and depth ambiguity. Whether VLMs can effectively leverage such multi-view inputs for robotic reasoning therefore remains an open question. To bridge this gap, we introduce MV-RoboBench, a benchmark specifically designed to evaluate the multi-view spatial reasoning capabilities of VLMs in robotic manipulation. MV-RoboBench consists of 1.7k manually curated QA items across eight subtasks, divided into two primary categories: spatial understanding and robotic execution. We evaluate a diverse set of existing VLMs, including both open-source and closed-source models, along with enhanced versions incorporating CoT-inspired techniques. The results show that state-of-the-art models remain far below human performance, underscoring the substantial challenges VLMs face in multi-view robotic perception. Additionally, our analysis uncovers two key findings: (i) spatial intelligence and robotic task execution are positively correlated in multi-view robotic scenarios; and (ii) strong performance on existing general-purpose single-view spatial understanding benchmarks does not reliably translate to success in the robotic spatial tasks assessed by our benchmark. We release MV-RoboBench as an open resource to foster progress in spatially grounded VLMs and VLAs, providing not only data but also a standardized evaluation protocol for multi-view embodied reasoning.