Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically spatial intelligence in vision-language models (VLMs), evaluated through a visual question answering (VQA) benchmark.

Technical level: Intermediate. The paper is readable without deep 3D-geometry background, but it assumes familiarity with VLM evaluation, multiple-choice scoring, and ranking-based metrics.

Scope: The paper introduces SSI-Bench, a human-curated benchmark of 1,000 ranking questions built from real-world 3D structures, and uses it to measure how well 31 VLMs reason about geometry, topology, and physical feasibility compared to humans.

What This Paper Is About

Most spatial reasoning benchmarks for VLMs use everyday, largely unconstrained scenes — indoor navigation or object arrangements — where one image can support several plausible 3D interpretations (an object may look small because it is small or because it is far away). The authors argue this makes many 3D relations underdetermined from a single image. They build a benchmark from complex real-world structures (roofs, stairs, bridges, towers, frames, railings) where geometric regularities, topological connectivity, and physical feasibility narrow the space of plausible 3D configurations, so that ranking questions over hidden 3D properties have determinate answers.

Key Contributions

1. SSI-Bench, a human-curated VQA benchmark for Structure-Centric Spatial Reasoning (SCSR) in constraint-governed spaces, composed of 1,000 multiple-choice ranking questions over geometric and topological criteria, each with 3 or 4 candidates.
2. A formalization of SCSR in which a structural scene is represented as nodes, members, geometric degrees of freedom, and discrete attributes, with admissible states restricted to a feasible set defined by equality and inequality constraints.
3. A systematic evaluation of 31 VLMs (10 proprietary models from four families and 21 open-source models from six families) plus human and random baselines, revealing a large gap to human performance.
4. An error analysis of four dominant failure modes that points to limitations in structural grounding and globally coherent 3D reasoning, rather than only visual recognition.

Main Findings

• Large human-model gap: The best open-source model reaches 22.2% and the strongest closed-source model reaches 33.6% average Taskwise Accuracy, while humans score 91.6%. The random-ranking baseline is 12.85%.
• Proprietary models lead: Gemini-3-Flash is the top overall model at 33.60%, followed by Gemini-3-Pro at 29.50% and GPT-5.2 at 29.10%. The best open-source result is GLM-4.6V at 22.20%, with Qwen3-VL-235B-A22B at 21.90%.
• Progress over generations is incremental: Within Gemini, accuracy rises from Gemini-2.5-Flash (22.30%) and Gemini-2.5-Pro (26.10%) to Gemini-3-Pro (29.50%) and Gemini-3-Flash (33.60%). In the GPT line, GPT-4.1 (22.40%) and GPT-4o (22.60%) rise to GPT-5 mini (25.90%) and GPT-5.2 (29.10%). GLM shows smaller movement (GLM-4.5V 21.40% to GLM-4.6V 22.20%). The authors state that scaling alone does not close the gap.
• Several models sit near chance: The random baseline is 4.17% for four-candidate tasks and 16.67% for three-candidate tasks. The lowest reported average is InternVL3.5-2B at 11.10%, which also scores 0.00% on the topological Multi-View sub-task.
• Task-level difficulty varies: Humans score highest on Area (98.00%) and Ground Height (94.29%), and lowest on the topological Multi-View sub-task (83.33%). Among model cells, the highest reported is GPT-5 mini at 43.69% on Relative Distance.
• Thinking helps only modestly: Gemini-3-Pro improves from 27.1% (low thinking) to 29.5% (high thinking), and Qwen3-VL-30B-A3B improves from 20.6% (Instruct) to 22.5% (Thinking).
• Token usage is a weak proxy for reasoning quality: Accuracy is non-monotonic with thinking-token usage, peaking at moderate usage and dropping at very high usage. The decline is more pronounced for Qwen3-VL-30B-A3B; Gemini-3-Pro is comparatively stable.
• Thinking gains are task-dependent: Improvements appear on some single-view geometric and topological criteria such as Ground Angle and Hop Distance, but effects are mixed or worse on Multi-View and Volume, which depend on globally consistent 3D reconstruction.
• Four error modes identified: From a manual inspection of 100 randomly sampled questions using Gemini-3-Pro's reasoning traces: member-extent errors (mistaking a visible fragment for the whole member), object-recognition errors (misidentifying components or misjudging coarse orientation), computational and comparison-logic errors (optimizing the wrong quantity, such as 2D projected area instead of 3D convex-hull volume), and 3D spatial-logic errors (weak depth reasoning, failed cross-view correspondence, incorrect relational composition).

Methodology in Plain English

The authors built the benchmark entirely by hand. Ten researchers with backgrounds in AI and structural engineering spent over 400 hours reviewing roughly 20,000 structure-related images from royalty-free sources (Unsplash, Pexels, Pixabay), plus some of their own photography for the multi-view subset, and retained over 2,000 candidates covering structure forms such as space frames, steel towers, cable-stayed bridges, timber trusses, reinforcement frameworks, and pipeline systems. Images were compressed so the longer side is at most 1920 pixels.

They then defined ten task categories across Geometric and Topological families — Ground Height, Ground Angle, Dimension, Relative Distance, Area, Volume, Multi-View (geometric), Hop Distance, Cycle Length, and Multi-View (topological). Ground Height, Ground Angle, and Dimension use four members; the remaining tasks use three groups. Annotators used Label Studio to record an ascending order under each criterion, mark ties, and draw highlighting polygons that respect occlusion. Questions were generated by permuting candidates into answer options with a separate annotated image per option, and each question was independently reviewed, with disagreements escalated to a third reviewer. Candidate sets were deliberately chosen so the ordering cannot be recovered from 2D pixel layouts alone.

Evaluation ran all 31 models at temperature 0, with images resized so the longer side is at most 512 pixels, using a unified prompt template per question type and requiring a parsable Python list encoding a permutation. Two metrics are reported: Taskwise Accuracy (exact match on the full permutation) and Pairwise Accuracy (pairwise ordering consistency). Six independent evaluators with basic structural knowledge, not involved in annotation, answered the full benchmark as the human baseline.

Why This Matters

Research impact. The paper argues that existing spatial benchmarks reward 2D shortcuts because their scenes are weakly constrained, and that structurally constrained settings demand genuine constraint-consistent 3D inference. It positions SSI-Bench as a complement to scene-centric benchmarks and as a diagnostic tool that separates spatial reasoning from object semantics and layout priors.

Real-world applications:

• Robotics and embodied AI, where agents must infer component relations and support paths in built environments.
• Infrastructure inspection, where assessing a bridge, truss, or tower requires judging relative height, angle, dimensions, or connectivity from photographs.
• Construction and structural engineering workflows that rely on interpreting real-world photographs rather than blueprints.
• Accessibility and assistive systems that describe spatial relations in complex scenes to users.

Industry relevance. The paper reports that current open-weight models remain less reliable than proprietary ones at inferring latent 3D state and applying structural constraints, and that scaling within families yields uneven, modest gains. That points to structural grounding and constraint-consistent reasoning as unsolved capability gaps for anyone deploying VLMs on physical-world tasks. The authors also note dual-use considerations and state that SSI-Bench is an offline diagnostic benchmark with no action policies, control interfaces, or deployment data.

Future Directions

• Enhancing fine-grained structural perception through captions or annotations describing component locations, extents, connectivity, and relations.
• Strengthening spatial reasoning with large-scale chain-of-thought data designed for structure-centric tasks.
• Moving beyond scaling toward structure-aware training signals and better coverage of constrained 3D structural configurations, since larger or newer variants within the same family do not reliably improve.
• Extending the annotation and question-generation stages to support semi-automatic data scaling by expanding candidate annotations and sampling new ranking questions.

Target Audience

Researchers and engineers working on vision-language models, multimodal reasoning, and spatial intelligence benchmarks; teams building embodied AI or robotics systems that must reason about physical structures; and interdisciplinary researchers in AI and structural engineering. Readers looking for a compact, well-defined evaluation of whether current models can recover 3D structure from real photographs will find the paper's benchmark design, leaderboard, and failure taxonomy most useful.

Authors’ abstract

Spatial intelligence is crucial for vision--language models (VLMs), yet many scene-centric benchmarks evaluate unconstrained environments where a single image may admit multiple plausible 3D interpretations. We introduce SSI-Bench, a VQA benchmark for Structure-Centric Spatial Reasoning (SCSR) in constraint-governed spaces. Built from complex real-world 3D structures, it uses structural constraints from geometry, topology, and physical feasibility to make component relations more determinate from visual evidence. The benchmark contains 1,000 ranking questions spanning geometric and topological reasoning, where correct ordering requires resolving all candidate-wise 3D relations, imposing stronger demands on spatial understanding. It is created through a fully human-centered pipeline with over 400 researcher-hours of image curation, component annotation, and question design. Evaluating 31 VLMs reveals a large gap to humans: the best open-source model achieves 22.2% accuracy and the strongest closed-source model reaches 33.6%, while humans score 91.6%. Further results show that chain-of-thought reasoning brings only marginal gains, and error analysis reveals fundamental limitations in current models' spatial understanding within constraint-governed spaces. Project page: https://ssi-bench.github.io.