SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs

AI summary

Overview

Research area: Computer vision and multimodal machine learning — specifically the evaluation of spatial reasoning in thinking-enabled vision-language models (VLMs).

Technical level: Intermediate. The paper assumes familiarity with VQA benchmarks, scene graphs, chain-of-thought reasoning, and supervised fine-tuning (SFT/LoRA), but the central idea — auditing the reasoning process rather than the final answer — is explained in accessible terms.

Scope: The paper introduces SpatialChain, a dataset and evaluation protocol that pairs spatially-oriented GQA questions with scene-graph-grounded reasoning chains, and uses it to audit whether nine thinking-enabled VLMs reach correct answers through faithful spatial reasoning or through linguistic shortcuts.

What This Paper Is About

Thinking-enabled VLMs keep posting higher scores on spatial benchmarks, but accuracy alone cannot tell whether a correct answer came from genuine spatial inference or from memorized language patterns such as "cup on table." The authors build SpatialChain, a dataset of spatially-oriented GQA questions bundled with reasoning chains derived from structured scene graphs, plus a two-axis evaluation that scores both the final answer and the quality of the reasoning trace. The goal is to make an otherwise invisible failure mode — answering correctly for the wrong reasons — measurable.

Key Contributions

1. SpatialChain dataset: 28,350 training and 899 test examples that pair spatially-oriented GQA questions with scene-graph-grounded reasoning chains, retained only when the generated answer matches the symbolic ground truth. Each chain is stored in a <think>...</think><answer>...</answer> format compatible with thinking-mode VLMs.

2. Two-axis evaluation protocol: objective chain-overlap metrics (ROUGE-1, ROUGE-L, token-level F1, entity overlap, length ratio) combined with a scene-graph-aware LLM judge that scores reasoning faithfulness and completeness independently of the final answer.

3. Audit of nine thinking-enabled VLMs: the protocol shows that chain quality significantly predicts answer correctness for seven of nine models, and that four models achieving ≥79% VQA accuracy exhibit shortcut rates above 39% — correct answers whose reasoning the judge marks as unfaithful.

4. Supervised fine-tuning evidence: SFT on SpatialChain improves Qwen3-VL-8B by +6.2 pp in-domain and reduces its shortcut rate to 22%, the lowest of all evaluated models, while revealing a stylistic specialization effect on external benchmarks that motivates replay-augmented training.

Main Findings

• Accuracy hides unfaithful reasoning. Four of nine models reach ≥79% VQA accuracy while showing shortcut rates above 39%. Claude Sonnet 4.6 (79.9% accuracy, 39% shortcut) and InternVL3.5-4B (81.3% accuracy, 58% shortcut) are the clearest cases: competitive benchmark scores mask reasoning the judge flags as largely ungrounded.

• Chain quality predicts correctness for most, but not all, models. Seven of nine models show a significant positive correlation between chain quality and correctness (p < 0.05): all Qwen3 variants, GLM-4.1-9B, InternVL3.5-4B, and GPT-5.4. The strongest effect is in the Qwen3 base models (ΔR-1 ≈ +0.067).

• Two exceptions reveal distinct failure modes. Claude Sonnet 4.6's non-significance (p = 0.07, Δ = +0.007) comes from extremely short output (length ratio = 0.36) — too little reasoning content for quality to vary. InternVL3.5-8B generates the longest chains in the evaluation (length ratio = 7.07) yet shows zero correlation with correctness (Δ = +0.001, p = 0.46), i.e. verbose output that resembles reasoning but is decoupled from the spatial judgment.

• Accuracy and chain overlap correlate only moderately. Across all nine models, Pearson r = +0.69 (p = 0.04) and Spearman ρ = +0.64 (p = 0.06); the association collapses when the two fine-tuned variants are removed (r = +0.45, p = 0.32).

• Fine-tuning produces the most faithful chains. Qwen3-VL-8B FT leads with 85.03% accuracy and the lowest shortcut rate (0.222), followed by Qwen3-VL-4B FT at 82.23% accuracy and a 0.194 shortcut rate. Both FT variants lead on judge faithfulness (≈0.63) and completeness (≈0.70).

• Fine-tuning trades off on external benchmarks. SFT on Qwen3-VL-4B improves in-domain accuracy (0.784 → 0.822, +3.8 pp) but degrades ERQA (0.453 → 0.380, −7.3 pp) and EmbSpatial-Bench (0.791 → 0.757, −3.4 pp). After fine-tuning, 100% of outputs adopt a fixed "Looking at…" opener (vs. 0% before) and average chain length collapses to 15% of its pre-SFT value on ERQA — a stylistic specialization rather than loss of spatial knowledge.

• Yes/No asymmetry signals language priors. The polarity gap Δpol = Yes-acc − No-acc separates models that confirm spatial descriptions regardless of visual evidence from models whose reasoning admits mismatch. GLM-4.1-9B is notably balanced (80.39% Yes, 82.98% No), while GPT-5.4 shows the largest gap (79.68% Yes, 59.27% No).

• The faithfulness signal is robust. A second judge from a different provider (GPT-5.6) preserves the ranking by shortcut rate (ρ = 0.88; faithfulness scores ρ = 0.74, pass/fail κ = 0.84). Shortcut rates are identical for τ ∈ [0.5, 0.6]. On 198 human-annotated items, judge–human agreement on faithfulness (ρ = 0.49) matches human–human agreement (ρ = 0.44).

• Pass rate and shortcut rate are not contradictory. GLM-4.1-9B achieves the third-highest pass rate (80.9%) alongside a 45% shortcut rate, because the pass criterion averages faithfulness and completeness while many individual chains fall below the faithfulness cutoff.

• Strongest zero-shot systems are close. GLM-4.1-9B and InternVL3.5-4B tie at 81.34% accuracy, with Claude Sonnet 4.6 at 79.89%. GPT-5.4 is weakest overall at 72.18%, trailing the 4B Qwen3 base by roughly 6 pp.

Methodology in Plain English

The researchers started from GQA, a visual question answering dataset built on 113,018 Visual Genome images with structured scene graphs that encode objects, attributes, and pairwise spatial relations. They kept only questions that genuinely require spatial reasoning, in two stages:

1. Type filtering: 17 relation-centric GQA question types were retained, yielding 135,061 candidates from 49,503 unique images in the balanced training split. For the test split, candidates were ranked by the length of their GQA semanticStr field and only the 1,000 longest were kept, biasing the test set toward compositional, multi-step queries.
2. LLM verification: GPT-5-nano acted as a binary classifier to remove questions answerable by object recognition alone (e.g., "Is the dog sitting?"). This left 86,773 training candidates and 900 test candidates.

Because extended-thinking generation is expensive, 30,000 training candidates were randomly sampled for chain generation. Reasoning chains were produced by Claude Haiku 4.5 with extended thinking (budget 3,000 tokens, max output 4,096 tokens) via the Anthropic Message Batches API, with the scene graph supplied as internal context but a system prompt forbidding any explicit reference to scene-graph fields. A chain was retained only if the model's visible answer matched the GQA ground truth; generation errors removed a few more, giving 28,350 training and 899 test examples.

Evaluation runs on two axes. Downstream accuracy is binary Yes/No exact match after lower-casing and stripping articles, with accuracy also broken out by answer polarity. Reasoning quality is measured first with surface-overlap metrics (ROUGE-1, ROUGE-L, token-level F1, entity overlap, length ratio) computed on all 899 test examples and stratified by correctness, and second with a rubric-based LLM judge (Claude Sonnet 4.6) that receives the question, ground-truth answer, scene graph, candidate answer, and candidate reasoning, and returns three scores in [0,1]: answer correctness, reasoning faithfulness, and reasoning completeness.

Nine systems were evaluated: base and fine-tuned Qwen3-VL-4B/8B, InternVL3.5-4B/8B, GLM-4.1-9B (open-weight, served with vLLM on a single A100 80 GB GPU at τ = 0.6, top-p = 0.95, top-k = 20, max 16,384 tokens), and Claude Sonnet 4.6 and GPT-5.4 (proprietary APIs). Fine-tuning used LoRA (r = 16, α = 32, RSLoRA) on all linear layers for 2 epochs with an effective batch size of 12 on a single A100 80 GB GPU. The judge was validated against 198 human annotations and against a second judge from a different provider.

Why This Matters

Impact on research. Answer-only benchmarks cannot distinguish genuine spatial inference from memorized co-occurrence patterns. SpatialChain supplies a verifiable reference — chains derived from symbolic scene graphs and retained only when consistent with the ground truth — making it possible to ask whether a model reasoned about the correct spatial relations, not merely whether it looked at the right region. It also provides structured spatial supervision usable for reinforcement learning, where outcome and process rewards can be computed automatically from the verifiable answer and scene-graph reference.

Real-world applications:

• Robotics and embodied agents: object-relation understanding is a stated prerequisite for manipulation and navigation, and SpatialChain offers structured spatial supervision for Vision-Language-Action models.
• Safety and reliability auditing: shortcut rates quantify how often a deployed VLM reaches correct conclusions through ungrounded reasoning, which matters where a right answer for the wrong reason can fail unpredictably on new inputs.
• Accessibility and assistive systems: spatial description tools (e.g., describing a scene to a visually impaired user) depend on relational accuracy, not just correct object labels.
• Model selection and procurement: the pass/shortcut decomposition gives buyers a way to compare models that look identical on standard accuracy tables.

Industry relevance. The finding that four models achieve ≥79% accuracy with shortcut rates above 39% directly affects how vendors report benchmark performance and how practitioners decide when a VLM is trustworthy enough to deploy in spatial tasks.

Future Directions

• Per-step verification: the current filter verifies a chain's conclusion, not each intermediate step. Verifying every step against scene-graph node IDs is identified as the natural next step, since the generator can hallucinate a locally unsupported step that still ends in the right answer.

• Breaking the generator–judge family overlap: the faithfulness judge is Claude Sonnet 4.6, from the same family as the reference chain generator (Claude Haiku 4.5), and one evaluated system shares that family. References generated by a non-Claude model remain untested.

• Moving beyond binary Yes/No: the task uses binary answers with a 58% yes-rate in training, which enables the polarity-gap analysis but caps the accuracy axis and is amplified by SFT. Extending beyond binary VQA is stated as the focus of future versions.

• Replay-augmented training and metric relations: replay augmentation is proposed to mitigate the stylistic specialization effect on external benchmarks, and extending scene-graph relations from symbolic (left/right, in front/behind) to metric (exact distances, 3D depth) would broaden the benchmark's coverage of geometric understanding.

Target Audience

Researchers and engineers working on vision-language models, multimodal reasoning, and VQA benchmarking; practitioners evaluating whether a thinking-enabled VLM's spatial outputs can be trusted in robotics, embodied AI, or accessibility settings; and anyone studying chain-of-thought faithfulness, LLM-as-judge methodology, or the gap between benchmark accuracy and genuine reasoning capability.

Authors’ abstract

Thinking-enabled vision-language models (VLMs) report ever-higher accuracy on spatial benchmarks, yet final-answer scores cannot reveal whether a correct prediction reflects faithful spatial reasoning or a linguistic shortcut. We introduce SpatialChain, a dataset of 28,350 training and 899 test examples pairing spatially-oriented GQA questions with scene-graph-grounded reasoning chains, retained only when the generated answer matches the symbolic ground truth, and a two-axis evaluation combining objective chain-overlap metrics with a scene-graph-aware LLM judge that scores faithfulness and completeness independently of the final answer. Applied to nine thinking-enabled VLMs, the protocol surfaces three findings invisible to standard accuracy: (i) four of nine models achieve $\geq$79% VQA accuracy while exhibiting shortcut rates above 39%, i.e., correct answers whose reasoning the judge marks as unfaithful; (ii) chain quality significantly predicts answer correctness for seven of nine models, but the two exceptions (Claude Sonnet 4.6, InternVL3.5-8B) reveal qualitatively distinct failure modes, terse output vs. verbose-decorative reasoning, that benchmark accuracy alone conflates; (iii) SFT on SpatialChain improves Qwen3-VL-8B by +6.2 pp in-domain and reduces its shortcut rate to 22%, while a stylistic specialization effect on external benchmarks motivates replay-augmented training as mitigation. The faithfulness judge is validated against 198 human-annotated items, where judge-human agreement matches human-human agreement, and against a second judge from a different provider, which preserves the model ranking ($ρ$ = 0.88). Data, generation scripts, and evaluation code are released at https://github.com/spatialchain/SpatialChainBenchmark.