SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

AI summary

Overview

Research area: Computer vision and vision-language models, specifically the evaluation of spatial reasoning benchmarks for embodied and multimodal AI.

Technical level: Advanced (assumes familiarity with vision-language models, 3D scene data such as RGB-D scans, supervised fine-tuning, and reinforcement learning from verifiable rewards).

Scope: This paper introduces and validates SpaceCast-Bench, a 3,862-question benchmark built from 182 real-world indoor scenes that tests whether vision-language models can predict spatial relations in scene states they never observe.

What This Paper Is About

Existing spatial benchmarks mostly measure spatial perception: reading off relations that are already visible in the input image or video. The authors argue that real spatial intelligence requires predictive spatial reasoning, meaning the ability to build a scene model from partial observations, mentally apply an intervention, and answer questions about the unseen result. SpaceCast-Bench was built to measure that capability directly, and the paper also tests 21 models on it and studies whether fine-tuning on programmatically generated data improves the ability.

Key Contributions

1. A new benchmark for predictive spatial reasoning. SpaceCast-Bench contains 3,862 questions from 182 scenes across 16 task types, organized into three capability levels (L1 Static Perception, L2 Local Prediction, L3 Global Prediction) that progressively require scene understanding, spatial state updating, and inference over unobserved outcomes.

2. A geometry-grounded construction pipeline. The authors build depth-verified "view chains" from posed RGB-D scans, apply controlled local or global transformations in 3D, and derive labels programmatically while keeping the resulting state hidden from the model. The pipeline also generates a training pool of 12,249 unique questions from 54 disjoint scenes.

3. A large-scale controlled evaluation of 21 models. The evaluation covers 9 proprietary, 6 general-purpose open-weight, and 6 spatially specialized models, plus controlled analyses of chain-of-thought prompting, bridge views, and external 3D or world-model evidence.

4. Evidence that benchmark-aligned training transfers. Fine-tuning Qwen3-VL-4B lifts its overall SpaceCast-Bench score from 34.0% to 65.7% with Staged SFT, and improves the macro average across six out-of-domain benchmarks.

Main Findings

• A large human–model gap. Human performance is 87.2% overall, while the strongest model, Gemini 3.5 Flash, reaches only 58.0%. The random baseline is 25.9%.

• Model ranking is counterintuitive. Gemini 3 Flash follows at 56.7% and Qwen3.7-Plus at 55.3%; the weakest proprietary model, Claude Sonnet 4.6, scores 36.0%. The best general-purpose open-weight model, Qwen3.5-27B, reaches 39.3%, which the authors note is 18.7 pp below the top proprietary model and 47.9 pp below human level.

• Spatially specialized models score lowest. This group ranges from 25.7% to 29.1%, below every proprietary and general-purpose open-weight model and near the 25.9% random baseline. The authors interpret this as evidence that current spatial fine-tuning does not transfer to spatial state updating.

• Capability levels separate model families. Proprietary and open-weight models drop from L1 to L2, with the largest declines for Gemini 3.5 Flash (−21.8 pp), Qwen3.7-Plus (−20.6 pp), and Qwen3.6-Flash (−18.3 pp). Strong proprietary models then recover at L3 (Gemini 3.5 Flash: L2 45.3%, L3 69.4%; Qwen3.7-Plus: L2 43.2%, L3 66.6%), while spatially specialized models fall further (SenseNova-SI-1.2: L2 30.5%, L3 16.1%; Cambrian-S-7B: L2 28.8%, L3 17.1%).

• Attachment is easiest; direction and distance are hardest. Attachment accuracy is highest among the selected models (Qwen3.7-Plus 80.4%, GPT-5.6-Luna 74.2%, Qwen3.5-27B 68.0%), while spatially specialized models score disproportionately low on it (SenseNova-SI-1.2 13.4%, SpatialLadder 9.3%). Occlusion ranks second for most models.

• World-centric direction is the strongest reference frame. Qwen3.7-Plus reaches 58.4% and GPT-5.6-Luna 49.9% on world-centric direction, with camera-centric competitive (53.9% and 45.7%). Object-centric reasoning is consistently weakest; Qwen3-VL-4B shows the largest camera–object gap (39.3% vs. 16.2%). Spatially specialized models show no clear frame preference, clustering in the 22–32% range.

• Chain-of-thought prompting does not uniformly help. Its effect varies sharply by model family and capability level. It improves global tasks for Qwen3.7-Plus but harms its static and local performance, while Qwen3-VL-4B shows the opposite pattern.

• Bridge views matter. Removing bridge views lowers directional accuracy in all three reference frames, with the largest drop in world-centric direction (39.9% to 34.6%, −5.3 pp), versus −1.9 pp for camera-centric and −1.7 pp for object-centric.

• Explicit 3D evidence is more reliable than generated outcomes. Textual 3D metadata yields gains of +30.7 pp and +10.9 pp for proprietary and open-weight models respectively, and BEV images also help both groups. Spatially specialized models show little benefit and decline with 3D metadata (−3.3 pp). Generated outcome images and videos produce much smaller and less consistent changes; in a manual audit of 600 local-intervention questions, 80.3% of incorrect responses under Qwen-Image-Edit and 38.3% under GPT-image-2 were associated with generation failures such as artifacts, missing objects, deformation, and scale or perspective mismatch.

• Fine-tuning helps, and staged training transfers better. On SpaceCast-Bench, Mixed SFT reaches 64.8% overall and Staged SFT reaches 65.7%, both far above the 34.0% base model. GRPO reaches 37.7%. Staged SFT raises Local performance from 51.3% to 54.0%, while Mixed SFT is slightly stronger on Global (73.1% vs. 70.8%). Across six out-of-domain benchmarks, Staged SFT achieves the strongest macro average (38.1%), ahead of Mixed SFT (37.2%) and the base model (33.5%), with the largest gains on DSI-Bench (+10.2 pp), SPARTQA (+6.0 pp), CLEVRER (+5.8 pp), and VSI-Bench (+4.7 pp); MindCube remains slightly below the base model. GRPO shows a different transfer profile with substantial gains on MindCube and DSI-Bench but slight declines on SPARTQA and CLEVRER.

• Dataset composition. ScanNet contributes 718 questions from 134 scenes and ScanNet++ contributes 3,144 questions from 48 scenes. By level, 1,368 questions are L1, 1,709 are L2, and 785 are L3. Among L2 and L3 questions, 2,217 change the queried relation under the transformation and 277 preserve it.

• Quality control results. Ten annotators each reviewed questions, with at least two independent reviews per question. Before adjudication, single-answer questions achieved 98.9% pairwise agreement and attachment-chain questions achieved 91.8% exact-set agreement.

Methodology in Plain English

The benchmark is built from 134 ScanNet and 48 ScanNet++ scenes, which supply calibrated RGB images, depth maps, reconstructed 3D geometry, and instance-level semantic labels. From each object's mesh the pipeline derives its surface geometry, 3D center, and bounding box, and it builds a directed "attachment graph" where an edge means displacing the parent also displaces the child, inferred from surface contact, support overlap, and semantic compatibility.

Questions are conditioned on a specific set of objects and answered from a sequence of views. For cross-view questions, the generator picks two "anchor" views that are mutually disjoint — each anchor shows its assigned object group but not the other group's objects. It then connects the two object groups with a straight 3D corridor, samples that corridor roughly every 0.09 m, and keeps camera frames that see the corridor with depth-consistent coverage. A consecutive pair of views is a valid transition only if it overlaps the current coverage frontier by at least 10% of the corridor and advances toward the opposite anchor. No bridge view may re-expose both queried groups. Thirteen of the 16 task types use this construction; the other three need only a single anchor view. Up to six bridge views are retained, and all views show only the pre-transformation scene.

Labels come from a spatial relation engine that computes answers directly from 3D geometry. Direction projects object centers onto the floor plane and quantizes the relative vector into eight 45-degree sectors plus above and below, expressed in camera-centric, object-centric, or world-centric frames. Distance uses sampled mesh points to approximate shortest surface-to-surface separation and maps it into four categorical bins. Visibility and occlusion use mesh ray casting from the calibrated camera center, distinguishing not occluded, occluded, and not visible. Attachment queries the transitive closure of the attachment graph and requires exact-set identification of every object displaced when a root object moves.

For predictive tasks, the pipeline applies the transformation — translating, orbiting, or removing an object, or rotating the whole layout — reruns the engine on the resulting state, and keeps the result state hidden from the model. Ambiguous cases, labels near decision boundaries, and invalid motion endpoints are excluded. Automatic filtering also removes insufficiently grounded questions and near-duplicates, and answer positions are balanced before the benchmark is frozen.

Evaluation uses a chain-of-thought prompt and deterministic decoding. The primary metric is the unweighted macro-average of per-task accuracy across all 16 task types, with separate Static, Local, and Global averages. For training experiments, the authors train Qwen3-VL-4B with rank-32 LoRA adapters on all language-model linear layers and the multimodal aligner while freezing the vision encoder (learning rates 1e-4 and 1e-5), and with rank-16 adapters for GRPO (learning rates 1e-5 and 5e-6). The training pool covers all 16 task types from scenes disjoint from the evaluation set.

Why This Matters

Impact on research. The paper argues that predictive spatial reasoning is a distinct, largely unsolved capability that current spatial benchmarks do not measure, and that spatial fine-tuning as practiced today does not transfer to it. It provides both a diagnostic instrument and a programmatic data-generation pipeline, and it shows that training on that generated data improves in-domain and out-of-domain performance, which gives the research community a reproducible path to study and improve the capability.

Real-world applications:

• Robot manipulation and multi-step planning. The motivating example is a robot retrieving an envelope beneath a book with a cup resting on top, which must predict that sliding the book displaces the cup and decide to move the cup first.

• Embodied agents in indoor environments. The benchmark is built entirely from indoor scans, matching household and office settings where agents must reason about unseen outcomes of actions.

• Spatial world models. The paper positions predictive spatial reasoning as precisely what world models must learn: representing an environment so that action consequences can be anticipated even when never observed.

• Evaluation of multimodal assistants. The reference-frame and relation-type breakdowns allow targeted diagnosis of whether a system fails on camera-centric, object-centric, or world-centric reasoning, or on attachment, occlusion, distance, or direction.

Industry relevance. The benchmark, dataset, and generation pipeline are released publicly (GitHub and HuggingFace), and the training results indicate that benchmark-aligned data can be used as a post-training resource for open-weight vision-language models. The finding that spatially specialized models underperform general-purpose ones is directly relevant to teams deciding how to allocate training effort for spatial capabilities. The paper's ethics statement notes that evaluated models remain unreliable and should not be treated as ready for safety-critical embodied deployment.

Future Directions

• Extending beyond indoor scans. The benchmark covers only indoor environments from ScanNet and ScanNet++ (ScanNet++ alone contributes 3,144 of the 3,862 questions), so outdoor settings and object arrangements or scales absent from those two sources are not tested.

• Modeling physical dynamics. The three intervention types (translation, rotation, removal) update geometry programmatically without simulating deformation, friction, or collision-driven motion, which the authors list as a limitation.

• Improving generated outcome evidence. Because 80.3% of Qwen-Image-Edit failures and 38.3% of GPT-image-2 failures were linked to generation problems, and because errors persist even after successful generation, the authors argue progress requires advances on both generation fidelity and cross-view evidence integration.

• Explaining and fixing the spatial fine-tuning gap. Spatially specialized models remain near random chance and score disproportionately low on attachment, which the authors attribute to attachment relations being underrepresented in existing spatial training data and task design — an open question for future training recipes.

Target Audience

Researchers and engineers working on vision-language models, spatial reasoning, embodied AI, and robot learning who need a diagnostic benchmark for predictive spatial reasoning. It is also useful for practitioners designing post-training data pipelines, and for benchmark designers interested in how to construct verified, programmatically labeled questions from 3D scans. Readers without background in 3D scene representation or multimodal model training will find the paper denser, though the task descriptions and failure analyses are accessible.

Authors’ abstract

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe-transform-infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.