SpatiaLab: Can Vision-Language Models Perform Spatial Reasoning in the Wild?

AI summary

Overview

Research area: Computer vision and multimodal AI — specifically spatial reasoning in vision-language models (VLMs).

Technical level: Intermediate. The concepts are accessible, but familiarity with VLMs, multimodal benchmarks, and evaluation paradigms (MCQ vs. open-ended) helps.

Scope: Introduces SpatiaLab, a 1,400-item real-world visual question-answering benchmark that systematically measures how well 25+ state-of-the-art VLMs reason about spatial relationships compared to humans.

What This Paper Is About

Existing benchmarks for spatial reasoning in AI rely heavily on synthetic, puzzle-like, or LLM-generated images that simplify the visual world and reduce task complexity, creating the illusion that models have nearly solved spatial reasoning. This paper argues that such benchmarks mask real failures, and it introduces SpatiaLab — a benchmark of 1,400 questions built on natural, cluttered, real-world photographs covering 30 distinct spatial task types. The goal is to measure how VLMs actually perform when confronted with the visual noise, occlusion, depth layering, and multi-step relational structure of real scenes.

Key Contributions

1. A new benchmark (SpatiaLab): 1,400 visual QA pairs organized into six top-level categories (Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, 3D Geometry) and 30 subcategories, with ≥25 questions per subcategory and ≥200 per category. Supports both multiple-choice (SpatiaLab-MCQ) and open-ended (SpatiaLab-Open) evaluation.

2. Large-scale model evaluation: Systematic testing of 25+ VLMs spanning proprietary, open-source, reasoning-focused, and spatial-specialist categories, benchmarked against human annotators (87.57% MCQ, 64.93% open-ended).

3. Diagnostic error analysis: Identification of systematic failure modes — occlusion inference, scale consistency, multi-step navigational planning, and 3D geometry — traced to specific architectural and training limitations rather than random noise.

4. Intervention study: Evaluation of four improvement strategies — supervised fine-tuning (SFT), chain-of-thought (CoT) prompting, CoT with self-reflection, and a multi-agent architecture (SpatioXolver) — with an honest accounting of when each helps and when it backfires.

Main Findings

• Large human–model gap persists. Top MCQ performance reaches 54.93% (InternVL3.5-72B) against a human baseline of 87.57%; top open-ended performance is 40.93% (GPT-5-mini) against 64.93% for humans.

• Open-ended evaluation exposes a systematic drop. Across 25 models, the average MCQ→open-ended gap is 23.0% (σ = 5.5%), with spatial navigation showing the largest deficit. This suggests MCQ scores alone overestimate practical spatial competence.

• Scale alone does not determine performance. Llama-3.2-11B scores only 30.50% on MCQ, while reasoning-tuned models like o4-mini (53.21%) and GPT-5-mini (54.29%) rival much larger open-source models. Instruction-tuning and reasoning capability matter more than parameter count.

• Spatial-specialist models underperform expectations. SpaceOm, SpaceThinker, and SpaceQwen score only in the low-to-mid 40s on MCQ and roughly 10–13% on open-ended, indicating that task specialization does not automatically generalize.

• Category-wise strengths are uneven. Orientation and 3D Geometry tend to be easier for strong models, while Spatial Navigation, Depth & Occlusion, and Size & Scale are consistently harder. Failures spike when multiple cues (e.g., depth ordering plus relative size) must be combined.

• Chain-of-thought prompting provides little benefit. CoT often reduces accuracy on this benchmark, helping only with orientation tasks and amplifying flawed perceptual priors elsewhere — the opposite of its effect in textual reasoning.

• SFT boosts MCQ but hurts open-ended. Fine-tuning Qwen2.5-VL-3B on 40% of SpatiaLab raised MCQ accuracy by ~11 points overall but yielded only ~1 point gain on open-ended tasks, with regressions in Relative Positioning and Size & Scale — a sign of overfitting to answer distributions.

• Multi-agent reasoning (SpatioXolver) is selective. It improved orientation dramatically (+36% open-ended) but degraded depth & occlusion (−24%) and spatial navigation (−12%), showing that step-by-step reasoning cannot compensate for weak perceptual grounding.

• Failure modes are structural, not random. Recurring errors include spatial mislocalization in cluttered scenes, reliance on object-size priors over image cues, occlusion/ordering mistakes with thin or partially hidden structures, attribute confusion (perceptual vs. functional), and fluent but visually ungrounded open-ended rationalizations.

Methodology in Plain English

The authors built SpatiaLab by collecting images through three channels: automated web crawling, targeted online retrieval of specific scene types, and manual snapshots taken in varied indoor and outdoor settings. Images were profiled along six meta-dimensions (lighting, texture complexity, edge complexity, dominant spatial relation, material type, and gravity constraints) to ensure coverage of real-world variability. On average, images contained 21.48 objects, 11.88 partially visible objects, 3.23 depth layers, and reference chains of ~2 links — deliberately cluttered scenes that punish shortcut strategies.

Annotation proceeded in three phases: annotator training, QA generation balancing perceptual grounding with higher-order inference, and dual-format encoding (4-option MCQ and open-ended). Quality control used a three-tier review pipeline ending in a gold-standard round, producing 1,400 validated items.

Evaluation used direct prompting for MCQ and an LLM judge (Gemini-2.5-Flash) for open-ended answers, validated against human annotators with a Cohen's kappa of 0.738 (substantial agreement). To probe whether spatial reasoning can be improved, the authors then tested four strategies — SFT on a stratified 40% split, CoT prompting, CoT with self-reflection, and the multi-agent SpatioXolver pipeline.

Why This Matters

Impact on research. SpatiaLab exposes a measurement gap: model performance on synthetic spatial benchmarks has been misleadingly optimistic. The paper reframes spatial reasoning evaluation around real-world clutter, depth layering, and multi-cue fusion, and provides a publicly available benchmark to guide reproducible comparison. Its finding that CoT and SFT can hurt open-ended performance is a cautionary signal for the broader multimodal reasoning community.

Real-world applications:

• Robotics and embodied agents — reliable occlusion inference, depth ordering, and spatial navigation are prerequisites for manipulation and safe movement in cluttered environments.
• Autonomous driving — depth perception and relative positioning under partial visibility directly affect safety-critical decisions.
• Augmented and virtual reality — spatial consistency across viewpoints and scale-consistent object placement depend on the same capabilities this benchmark probes.
• Geospatial analytics and accessibility tools — describing and reasoning about scenes from photographs (e.g., for visually impaired users) requires grounded spatial language, not plausible-sounding description.

Industry relevance. Any company deploying multimodal models in physical-world or image-heavy contexts (robotics, AR/VR, automotive, industrial inspection, real-estate or medical image analysis) needs to know where these models fail. SpatiaLab provides that diagnostic signal, and its finding that specialized models underperform generalists informs build-vs-buy decisions and training data strategy.

Future Directions

• Geometry-aware training and supervision. The paper explicitly calls for unified geometric encodings and physics-aware training data, since current models lack stable internal representations of orientation, physics, and reference frames.

• Better open-ended evaluation and distractor design. Negative MCQ-vs-open gaps in some categories indicate that MCQ distractors can misrepresent true competence; the authors recommend auditing distractors and reporting per-subtask diagnostics alongside open-ended results.

• Rethinking fine-tuning objectives. SFT's "teach to the test" pattern and the risk of catastrophic forgetting of linguistic priors suggest the need for objectives that reward grounded spatial reasoning rather than discrete-choice alignment.

• Stronger perceptual grounding for agentic and reasoning pipelines. Since multi-step reasoning amplifies rather than corrects weak perceptual priors, future work must connect reasoning modules to object-centric binding, multi-scale feature retention, and verification pipelines.

Target Audience

This paper is most valuable for:

• Multimodal AI researchers studying VLMs, visual question answering, or spatial cognition in machines.
• Benchmark designers interested in how evaluation format (MCQ vs. open-ended) shapes conclusions about model capability.
• Practitioners deploying VLMs in robotics, autonomous systems, AR/VR, or any domain where spatial errors have real consequences.
• Cognitive scientists and HCI researchers comparing machine spatial reasoning to human spatial cognition, since the paper explicitly anchors its taxonomy in psychological faculties and includes human baselines.
• Graduate students entering multimodal evaluation research — the benchmark construction, error taxonomy, and intervention study offer a clear template for rigorous evaluation work.

Authors’ abstract

Spatial reasoning is a fundamental aspect of human cognition, yet it remains a major challenge for contemporary vision-language models (VLMs). Prior work largely relied on synthetic or LLM-generated environments with limited task designs and puzzle-like setups, failing to capture the real-world complexity, visual noise, and diverse spatial relationships that VLMs encounter. To address this, we introduce SpatiaLab, a comprehensive benchmark for evaluating VLMs' spatial reasoning in realistic, unconstrained contexts. SpatiaLab comprises 1,400 visual question-answer pairs across six major categories: Relative Positioning, Depth & Occlusion, Orientation, Size & Scale, Spatial Navigation, and 3D Geometry, each with five subcategories, yielding 30 distinct task types. Each subcategory contains at least 25 questions, and each main category includes at least 200 questions, supporting both multiple-choice and open-ended evaluation. Experiments across diverse state-of-the-art VLMs, including open- and closed-source models, reasoning-focused, and specialized spatial reasoning models, reveal a substantial gap in spatial reasoning capabilities compared with humans. In the multiple-choice setup, InternVL3.5-72B achieves 54.93% accuracy versus 87.57% for humans. In the open-ended setting, all models show a performance drop of around 10-25%, with GPT-5-mini scoring highest at 40.93% versus 64.93% for humans. These results highlight key limitations in handling complex spatial relationships, depth perception, navigation, and 3D geometry. By providing a diverse, real-world evaluation framework, SpatiaLab exposes critical challenges and opportunities for advancing VLMs' spatial reasoning, offering a benchmark to guide future research toward robust, human-aligned spatial understanding. SpatiaLab is available at: https://spatialab-reasoning.github.io/.