Scaling Spatial Intelligence with Multimodal Foundation Models

AI summary

Overview

Research area: Multimodal computer vision and spatial intelligence — the ability of AI models to understand, reason about, and act within three-dimensional space.

Technical level: Intermediate. The central thesis (better data beats architectural tricks) is easy to grasp, but the paper assumes familiarity with multimodal LLM training, benchmark evaluation protocols, and concepts like chain-of-thought reasoning and overfitting.

Scope: A systematic study of whether scaling spatially-grounded training data can teach existing multimodal foundation models genuine 3D spatial reasoning, resulting in the open-sourced SenseNova-SI model family.

What This Paper Is About

Multimodal foundation models like GPT-5 can describe images fluently yet fail at tasks humans find trivial — judging which object is closer, imagining a scene from another viewpoint, or reasoning about the layout of a room. The authors argue this gap stems not from weak model architectures but from a scarcity of high-quality, balanced spatial training data.

Rather than designing new model components, they assemble eight million spatially grounded question-answer samples organized under a principled taxonomy of spatial abilities, then fine-tune three existing open-source multimodal models on it, measuring whether spatial intelligence improves and whether that improvement is genuine.

Key Contributions

1. SenseNova-SI-8M dataset. A curated collection of roughly 8.5 million QA pairs built under a five-capability taxonomy (Metric Measurement, Spatial Relations, Mental Reconstruction, Perspective-taking, Comprehensive Reasoning). It combines ~4M reorganized open-source samples with ~4.5M newly generated pairs from richly annotated 3D scene datasets, with explicit emphasis on perspective-taking, a capability that prior datasets largely ignored.

2. A family of open-sourced spatial intelligence models. SenseNova-SI is built on three distinct backbones — Qwen3-VL, InternVL3, and Bagel — without altering their original architectures, demonstrating that the approach is portable across different design philosophies (language-first scaling, natively multimodal pretraining, and unified understanding-generation).

3. State-of-the-art spatial benchmark results with preserved general ability. The best variant posts 68.8% on VSI-Bench, 43.3% on MMSI, 85.7% on MindCube, 54.7% on ViewSpatial, 47.7% on SITE, 63.9% on BLINK, 55.5% on 3DSR, and 72.0% on EmbSpatial, while retaining 84.9% on the general MMBench-En benchmark.

4. Rigorous diagnosis of what was actually learned. Through debiased benchmarks, circular answer-ordering tests, and no-image evaluations, the authors show that the gains come from visual grounding rather than language shortcuts — a distinction most prior spatial models fail.

Main Findings

• Data scaling, not architecture, drives spatial ability. The 2B-parameter SenseNova-SI model outperforms existing 7B spatial-intelligence baselines trained on comparable data volumes, and the 8B version surpasses proprietary systems including GPT-5 on several spatial benchmarks. The authors conclude that algorithmic innovation in this area may be premature while data scaling remains underexploited.

• Perspective-taking is the highest-leverage capability. Training on large volumes of viewpoint-transformation data produced the largest gains — MindCube improved by 44.2 points and ViewSpatial by 16.0 points for InternVL3-8B. The 8B model convincingly beats GPT-5 on perspective-taking, where proprietary systems remain notably weak.

• Scaling saturates. Performance gains diminish as data increases, and the authors state plainly that data scaling alone is unlikely to reach human-level spatial intelligence, motivating their decision to release weights so others can build algorithms on top.

• Early signs of emergent generalization. Models trained on a single data source transferred to seemingly unrelated tasks — training on ego-exo viewpoint association improved maze pathfinding and camera-position reasoning. The authors speculate that perspective-taking contains "meta-tasks" whose learning propagates outward. Gains in comprehensive reasoning appeared even though very little such data was included, suggesting capability synergy.

• Extrapolation beyond the training window. Trained on at most 16 video frames per sample, SenseNova-SI generalizes to 32–64 frames at inference, scoring comparably to a model explicitly trained on 64–128 frames while using far fewer input frames.

• Gains are not overfitting or shortcut exploitation. On VSI-Debiased, SenseNova-SI degrades far less than Cambrian-S. On MindCube, the prior open-source state of the art scored 50.7 with no images at all — nearly identical to its score with images — while SenseNova-SI dropped from 85.6 to 52.5 when images were removed, confirming real visual dependence. Under hard circular answer reordering, SenseNova-SI lost 10 points versus nearly 30 for the comparison model.

• Text-based chain-of-thought does not work well for spatial reasoning. Three CoT schemes, including an elaborate world-coordinate cognition map, produced limited gains that could not justify their token overhead. Adding GRPO reinforcement learning on top actually reduced performance. The authors attribute this to long spatial reasoning chains becoming internally inconsistent.

• Direct downstream benefit for robotics. Without any fine-tuning, SenseNova-SI improved success rates on the spatial subset of EmbodiedBench from 10.4% to 16.6% (official prompt) and 20.8% to 33.3% (spatial-oriented prompt) relative to its base model — roughly 60% relative improvement, though still below GPT-4o's 37.5%/45.8%.

• Some perception benchmarks regressed. BLINK and EmbSpatial scores dipped slightly on the 8B models, indicating a trade-off between specialized spatial training and certain low-level perceptual or embodied-recognition tasks.

Methodology in Plain English

The researchers did not modify any model architecture. Instead, they treated data as the intervention.

First, they adopted an existing taxonomy that decomposes spatial intelligence into distinct cognitive capabilities, then audited public datasets against it. They found the field's data heavily skewed toward measuring object sizes and relative positions, while viewpoint transformation and mental 3D reconstruction were severely underrepresented.

To fill those gaps, they mined 3D-annotated scene datasets — scanned indoor rooms, egocentric and exocentric video, tabletop scenes with many similar objects — where camera poses, object poses, and cross-view instance associations are known precisely. From these annotations they programmatically generated QA pairs with known ground truth. To make questions non-trivial, they filtered for recognizable objects, removed degenerate camera angles, and deliberately enforced that image pairs in a set were connected but not trivially similar.

They then fine-tuned three existing foundation models for a single epoch each on the combined 8.5M samples (128 GPUs, batch size 2048, about three days per run, learning rate 5×10⁻⁶), and evaluated on eight spatial benchmarks plus a suite of general multimodal benchmarks to check for catastrophic forgetting.

To separate real capability from memorization, they ran three diagnostic tests: re-evaluating on a debiased benchmark variant stripped of text-answerable questions, re-running with images removed entirely, and systematically permuting multiple-choice answer orders.

Why This Matters

Impact on research. The paper reframes spatial intelligence as a data problem before an architecture problem. It provides the field with an open baseline, a documented data recipe, and — critically — evidence that the community's overfitting-prone benchmarks may have been rewarding language pattern-matching rather than spatial understanding. The negative result on chain-of-thought is arguably as valuable as the positive results: it suggests that text-token reasoning may be the wrong medium for 3D reasoning.

Real-world applications:

• Robotic manipulation. The EmbodiedBench experiments show a spatially aware model can plan more reliable action sequences for a virtual robot arm from natural-language instructions containing words like "left of" and "on top of."
• Embodied agents and navigation. Extrapolation to longer frame sequences supports tasks requiring tracking a scene over time, such as route planning through unfamiliar buildings.
• Augmented and mixed reality. Strong allocentric perspective-taking underlies systems that must translate between a user's first-person view and an object- or world-centric coordinate frame.
• Visual assistants for accessibility. Describing spatial layouts — how far away something is, what is occluded behind what — is a prerequisite for helping blind or low-vision users navigate physical space.

Industry relevance. The paper is authored by SenseTime Research with academic collaborators, and the models are released publicly with code. Companies building vision-language-action systems, autonomous platforms, or AR products now have a free, strong spatial foundation model to fine-tune rather than having to reproduce the expensive 8M-sample training run themselves.

Future Directions

• What comes after data scaling? The authors acknowledge saturation and explicitly position SenseNova-SI as a launchpad for algorithmic innovation. The open question is what that innovation looks like — new architectures, training objectives, or reasoning mechanisms not based on text tokens.

• Multimodal reinforcement learning for spatial reasoning. The paper reports that GRPO on text chains did not help. Whether RL operating directly over visual or 3D representations could succeed remains unexplored and is flagged as an open gap.

• Closing the perception trade-off. Slight regressions on BLINK and EmbSpatial suggest that spatial specialization can cost general perceptual or embodied grounding. Understanding and eliminating that tension is a concrete next step.

• Bridging the human-model gap. Even the best results remain well below human performance on most benchmarks (for example, 68.8% versus 79.2% on VSI-Bench). Identifying which specific cognitive operations remain hardest would guide the next round of data design.

• Robotics without fine-tuning. The EmbodiedBench results are promising but partial, and the gap to GPT-4o persists. Whether modest fine-tuning or better prompting closes it is an open engineering question.

Target Audience

Researchers and engineers working on multimodal foundation models, spatial and 3D reasoning, and embodied AI will get the most from this paper, particularly those deciding whether to invest in architecture design or data curation. It is also valuable to benchmark designers and evaluation researchers, since its debiasing methodology exposes flaws in how the field currently measures spatial ability. Roboticists seeking an off-the-shelf spatially capable perception backbone will find the released models directly useful, though they should read the downstream section carefully to calibrate expectations. Readers without background in multimodal training will find the results meaningful but may need to consult the referenced benchmarks for context.

Authors’ abstract

Despite remarkable progress, multimodal foundation models still exhibit surprising deficiencies in spatial intelligence. In this work, we explore scaling up multimodal foundation models to cultivate spatial intelligence within the SenseNova-SI family, built upon established multimodal foundations including visual understanding models (i.e., Qwen3-VL and InternVL3) and unified understanding and generation models (i.e., Bagel). We take a principled approach to constructing high-performing and robust spatial intelligence by systematically curating SenseNova-SI-8M: eight million diverse data samples under a rigorous taxonomy of spatial capabilities. SenseNova-SI demonstrates unprecedented performance across a broad range of spatial intelligence benchmarks: 68.8% on VSI-Bench, 43.3% on MMSI, 85.7% on MindCube, 54.7% on ViewSpatial, 47.7% on SITE, 63.9% on BLINK, 55.5% on 3DSR, and 72.0% on EmbSpatial, while maintaining strong general multimodal understanding (e.g., 84.9% on MMBench-En). More importantly, we analyze the impact of data scaling, discuss early signs of emergent generalization capabilities enabled by diverse data training, analyze the risk of overfitting and language shortcuts, present a preliminary study on spatial chain-of-thought reasoning, and validate the potential downstream application. All newly trained multimodal foundation models are publicly released.