Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models

AI summary

Overview

• Research area: Computer vision and multimodal AI — specifically the evaluation of spatial reasoning in Vision-Language Models (VLMs).
• Technical level: Intermediate. The paper is readable without deep mathematics, but it assumes familiarity with VLMs, benchmark evaluation (accuracy, confidence intervals), and fine-tuning techniques such as LoRA and reinforcement learning.
• Scope (one sentence): The paper introduces Spatial-DISE, a cognitively grounded benchmark and dataset for measuring spatial reasoning in VLMs across a 2x2 taxonomy (Intrinsic/Extrinsic × Static/Dynamic), and uses it to evaluate 32 state-of-the-art models against a human baseline.

Paper identifier: arXiv:2510.13394v4 [cs.CV]. Authors are affiliated with the School of Computer Science & Informatics, University of Liverpool.

What This Paper Is About

Current benchmarks for testing spatial reasoning in Vision-Language Models mostly measure what a model "sees" in a fixed scene, and they lack a systematic cognitive framework and sufficient scale, especially for dynamic tasks that require mental transformation. Spatial-DISE addresses this by organizing all spatial tasks into four quadrants (Intrinsic-Static, Intrinsic-Dynamic, Extrinsic-Static, Extrinsic-Dynamic) and by generating a large, verifiable set of spatial reasoning questions using a combination of real-world data collection and synthetic generation in Blender. The goal is to provide a balanced, large-scale, reproducible benchmark that reveals precisely where VLM spatial cognition breaks down.

Key Contributions

1. A cognitively grounded taxonomy. A unified 2x2 framework (Intrinsic vs. Extrinsic, Static vs. Dynamic) that can classify any spatial reasoning task, rather than being organized around isolated task types.
2. A scalable and verifiable data generation pipeline. An automated Blender-based pipeline that programmatically generates complex 3D spatial reasoning tasks, with reproducibility ensured through seeded randomization (each item begins with a question_id hashed into a random seed) and systematic distractor generation strategies.
3. A unified and verifiable cognitive benchmark and dataset. Spatial-DISE Bench (559 evaluation VQA pairs across 10 tasks and all four quadrants) plus Spatial-DISE-12K (over 12,000 verified 3D spatial reasoning VQA pairs across five 3D tasks), intended both for evaluation and for future model training.
4. An empirical map of VLM cognitive spatial reasoning limits. Evaluation of 32 state-of-the-art VLMs, with error taxonomy analysis and fine-tuning experiments, defining where current capabilities plateau and how far they remain from human performance.

Main Findings

• Spatial reasoning is a universal weakness across current VLMs. Across all 32 tested models, the average accuracy was 28.4%, only marginally above random guessing (25%) and far below the human baseline (76.8%). The best overall model, Doubao1.5-VL-thinking, reached 42.0%. Proprietary models averaged 31.9% overall; open-source models averaged 26.2%.

• Multi-step mental transformations break models most severely. The Fold and Punch task, which requires simulating a fold, a punch, and then an unfold, produced only 30.8% accuracy for the top model (Doubao-1.5-thinking), with an average across models of 25.4% — essentially random chance. The authors interpret this as a deficit in "spatial working memory" that prevents models from tracking an object through a sequence of changes.

• Static understanding is not a prerequisite for dynamic reasoning in these models. Several top models performed better on dynamic tasks than static ones. Gemini2.0-Flash scored higher on dynamic tasks (38.3%) than static tasks (23.6%), and Doubao-1.5-thinking surpassed the human baseline on Extrinsic-Dynamic questions. The authors argue models are learning fragmented strategies rather than building dynamic reasoning on a stable model of the static world.

• One model exceeded humans on Extrinsic-Dynamic tasks by computing rather than simulating. Doubao-1.5-thinking outperformed the human baseline on Extrinsic-Dynamic questions (61.9% versus the human 61.1% on that quadrant). The authors attribute this to the model converting the task into an algorithmic comparison of geometric features (edges, angles, connection points). Humans found these tasks hardest to simulate: 3D Combination had the longest mean human response time (59.2s) of all tasks.

• Post-training and spatial specialization yield only limited gains. Reasoning-enhanced and spatial-specialized models improved over their base models but remained far from human performance. For example, SpaceThinker (SFT) reached 32.6% overall, and SpaceOM (SFT) reached 25.9%.

• Fine-tuning on Spatial-DISE-12K substantially improves in-domain performance. Qwen2.5-VL-7B improved from 26.1% to 47.0%, and SpaceOm from 25.9% to 41.3%. The largest single jump was Qwen2.5-VL-7B on Intrinsic-Static, from 16.1% to 51.7%. Even after fine-tuning, the best model (Qwen2.5-VL-7B-sft at 47.0%) remained well below the 76.8% human baseline.

• Training transfers are quadrant-specific and asymmetric. Fine-tuning Qwen2.5-VL-7B on individual DISE quadrants produced a strong diagonal pattern: training on a quadrant mainly improved that quadrant, while off-diagonal transfer was minimal or negative. One example of asymmetry: a mild positive transfer of +3.9 pp from 3D Extrinsic-Dynamic to Intrinsic-Static, contrasting with strongly negative transfer from Intrinsic-Static to Extrinsic-Dynamic. Training on Extrinsic-Dynamic 3D tasks produced broadly positive transfer to 2D tasks, whereas Intrinsic-Static or Extrinsic-Static 3D training produced narrow or negative transfer.

• Out-of-domain gains are consistent but selective, without catastrophic forgetting. On external benchmarks, SpaceOm improved from 27.91% to 34.28% on OmniSpatial and from 27.22% to 32.6% on SPACE; Qwen2.5-VL-7B improved from 21.8% to 34.0% on OmniSpatial and from 28.7% to 32.2% on SPACE. CVBench changed only marginally (75.9% to 77.4% for Qwen2.5-VL-7B), which the authors cite as evidence that targeted spatial training does not cause catastrophic forgetting.

• Errors are dominated by reasoning, not perception. Of 200 analyzed incorrect responses (50 each from GeminiFlash2-0, Qwen2.5-VL-3B, Doubao-1.5-thinking, and SpaceThinker), reasoning errors accounted for 72.5%, perceptual errors 17.5%, and comprehension errors 10%. Within the 145 reasoning errors, Failure in Rule Application was the largest (44.8%), followed by Failure in Mental Simulation (40.0%) and Failure in Holistic-Local Processing (15.2%).

• The perceptual confounder check found no parsing bottleneck. Running Qwen2.5-VL-7B with the standard merged image layout versus separate images for question and options produced nearly identical accuracy: 26.1% versus 24.9%. If layout parsing were the main difficulty, separating the images should have helped.

Methodology in Plain English

The researchers first defined a taxonomy from cognitive science: one axis separates intrinsic information (parts inside a single object and how they are arranged) from extrinsic information (relationships between objects), and the other axis separates static tasks (nothing moves) from dynamic tasks (something must be mentally transformed). Crossing these gives four quadrants, and the team designed 10 tasks to fill them — for example 2D/3D Shape Finding (Intrinsic-Static), 2D/3D Rotation and Folding & Fold&Punch (Intrinsic-Dynamic), 3D Projection (Extrinsic-Static), and 2D/3D Combination (Extrinsic-Dynamic).

Data came from a three-stage curation pipeline. First, they collected existing validated spatial reasoning problems from academic psychometric tests and professional aptitude assessments, yielding an initial corpus of 1180 VQA pairs used as a conceptual foundation and template library. Second, they built an automated Blender pipeline that turns those concepts into new problems: each item starts from a unique question id that is hashed into a random seed, then core 3D assets are generated, question and answer images are rendered from chosen camera angles, plausible "near-miss" distractors are produced through strategies such as geometric variations, pattern/orientation errors, incorrect views, and component replacement, and everything is rendered under controlled lighting, materials, and camera parameters. Third, synthetic instances passed a human quality-control protocol checking solution uniqueness, accuracy and clarity, and redundancy elimination; failing items were removed. This produced the 559-pair evaluation bench and the 12,000+ pair training set.

Evaluation used multiple-choice accuracy with exact match via VLMEvalKit, with Deepseek-R1 used to parse answers from malformed model outputs. The human baseline was established with 54 participants aged 15 to 55 using a matrix-sampling design that collected 1,679 valid responses, so each question was answered by roughly three people; results were averaged and cross-validated with Item Response Theory. Error analysis combined an automated LLM judge (Doubao-1.6-thinking) with human verification on 200 incorrect responses. Fine-tuning experiments used LoRA applied to all linear layers on the Spatial-DISE-12K training split.

Why This Matters

For research, Spatial-DISE supplies both a diagnostic framework and a recomputable data-generation method. The taxonomy lets researchers state which kind of spatial competence a model lacks rather than reporting a single aggregate score, and the seeded generation pipeline gives the community a way to produce more dynamic reasoning data instead of being limited by small hand-built datasets. The finding that training gains are quadrant-specific challenges the assumption that spatial reasoning is one transferable skill.

Real-world applications the paper motivates:

• Robotics: manipulation and assembly require reasoning about how parts move and reorient, which maps directly onto the dynamic quadrants where models perform near chance.
• Autonomous navigation: understanding changing spatial relationships between objects and the agent over time is an Extrinsic-Dynamic problem the benchmark explicitly tests.
• Augmented reality: overlaying content correctly depends on projection and viewpoint reasoning, the Extrinsic-Static and Extrinsic-Dynamic tasks.
• Any deployment requiring multi-step physical reasoning: the paper's Fold and Punch result indicates that systems needing to simulate a sequence of state changes cannot yet rely on current VLMs.

For industry, the paper's fine-tuning results are directly actionable: a targeted spatial training set improved a 7B open-source model by 20.9 percentage points on the benchmark and also improved external benchmarks, without degrading general visual performance. That is evidence that spatial competence can be added in a targeted, measurable way — but also that it does not generalize automatically, so production teams should evaluate on the specific spatial quadrant their application depends on.

Future Directions

• Closing the simulation-to-reality gap by transferring abstract cognitive concepts learned from synthetic 3D scenes into real-world settings; the paper notes that Spatial-DISE-12K consists entirely of synthetic 3D scenes.
• Moving from isolated puzzles to interactive tasks such as navigation and robot manipulation, so evaluation reflects the sequential, embodied settings where spatial reasoning is actually used.
• Adopting process-oriented assessment that requires textual justifications or action plans, to distinguish genuine mental simulation from fragile heuristic pattern matching.
• Improving the error-analysis methodology. The authors state their hybrid LLM-plus-human pipeline may bias human annotators toward the LLM's proposed label, and that analyzing only 200 errors from four models means the error table should be read as qualitative trends rather than precise estimates.
• Determining whether robust, human-like spatial schemas can be learned at all, given that even the best fine-tuned model (Qwen2.5-VL-7B-sft at 47.0%) remains markedly below the human baseline of 76.8%.

Target Audience

This paper is most useful to VLM and multimodal researchers who build or evaluate models with spatial or embodied reasoning requirements, benchmark designers interested in cognitively grounded taxonomies and synthetic data generation, and applied researchers in robotics, augmented reality, and autonomous navigation who need to know how much spatial reasoning they can currently expect from a VLM. Engineers deciding whether to fine-tune an open-source VLM for spatial tasks will also find the fine-tuning and transfer results directly relevant.

Authors’ abstract

Spatial reasoning ability is crucial for Vision Language Models (VLMs) to support real-world applications in diverse domains including robotics, augmented reality, and autonomous navigation. Unfortunately, existing benchmarks are inadequate in assessing spatial reasoning ability, especially the \emph{intrinsic-dynamic} spatial reasoning which is a fundamental aspect of human spatial cognition. In this paper, we propose a unified benchmark, \textbf{Spatial-DISE}, based on a cognitively grounded taxonomy that categorizes tasks into four fundamental quadrants: \textbf{I}ntrinsic-\textbf{S}tatic, Intrinsic-\textbf{D}ynamic, \textbf{E}xtrinsic-Static, and Extrinsic-Dynamic spatial reasoning. Moreover, to address the issue of data scarcity, we develop a scalable and automated pipeline to generate diverse and verifiable spatial reasoning questions, resulting in a new \textbf{Spatial-DISE} dataset that includes Spatial-DISE Bench (559 evaluation VQA pairs) and Spatial-DISE-12K (12K+ training VQA pairs). Our comprehensive evaluation across 28 state-of-the-art VLMs reveals that, current VLMs have a large and consistent gap to human competence, especially on multi-step multi-view spatial reasoning. Spatial-DISE offers a robust framework, valuable dataset, and clear direction for future research toward human-like spatial intelligence. Benchmark, dataset, and code will be publicly released.