SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models

AI summary

Overview

Research area: Computer Vision / Vision-Language Models — spatial reasoning and progressive multimodal training.

Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, reinforcement learning for LLMs/VLMs (GRPO), and spatial reasoning benchmarks, but its core argument is conceptual and easy to follow.

Scope: This paper proposes a 26,610-sample multimodal dataset (SpatialLadder-26k) and a three-stage progressive training framework that produces SpatialLadder, a 3B-parameter model that sets state-of-the-art results on spatial reasoning benchmarks.

What This Paper Is About

Vision-Language Models can describe images well but remain weak at spatial reasoning — judging distances, directions, sizes, and relative positions. The authors argue this is not a reasoning failure but a grounding failure: models are asked to reason about space without first being trained to perceive it. Their goal is to build spatial intelligence in the correct order, from perceiving objects, to understanding spatial relationships, to reasoning about them.

Key Contributions

1. SpatialLadder-26k dataset: A multimodal dataset of 26,610 samples spanning four task categories — object localization (5,929), single-image spatial reasoning (5,929), multi-view spatial reasoning (5,752), and video spatial reasoning (9,000) — built with a standardized pipeline from ScanNet 3D scene reconstructions plus 9,000 videos sampled from SR-91k.

2. Three-stage progressive training framework: Stage 1 establishes spatial perception through object localization, Stage 2 develops spatial understanding across seven spatial dimensions and three modalities, and Stage 3 strengthens complex reasoning using GRPO reinforcement learning with verifiable, task-specific rewards.

3. SpatialLadder, a 3B-parameter model: Built on Qwen2.5-VL-3B, it achieves 62.3% overall in-domain accuracy (a 23.4% improvement over the base model), surpassing GPT-4o (41.5%) by 20.8% and Gemini-2.0-Flash (52.2%) by 10.1%.

4. Evidence for the perception–reasoning gap hypothesis: Controlled experiments with 200 spatial orientation tasks show that progressively adding perceptual hints improves Qwen2.5-VL-3B from 36.5% to 41.5% (+5.0% with location hints) to 46.0% (+4.5% with directional hints), motivating the progressive training design.

Main Findings

• In-domain state of the art: SpatialLadder reaches 62.3% overall accuracy, with 45.7% on VSI-Bench (+16.3% over base), 70.2% on SPBench-SI (+29.9%), and 70.9% on SPBench-MV (+34.3%). It matches Spatial-MLLM-4B's VSI-Bench result (47.3%) closely without specialized 3D encoders.

• Strong out-of-domain generalization: 50.8% overall across CV-Bench, SPAR-Bench, and ViewSpatial-Bench — a 7.2% improvement over the base model and above GPT-4o (48.1%). The largest gain is 16.5% on ViewSpatial-Bench person-perspective tasks.

• Progressive training removes dependence on hints: After training, SpatialLadder scores 82.0% with no hints, 82.5% with location hints, and 83.5% with full hints — a 1.5% range, versus the base model's heavy reliance on external scaffolding.

• Stage 2 is the most critical component: Removing Stage 2 (spatial understanding fine-tuning) causes a 9.4% accuracy drop; removing Stage 1 causes 1.8% and removing Stage 3 causes 2.1%. Excluding single-image and multi-view data causes the most severe degradation at 16.4%.

• Semantic entropy follows the intended curriculum: Entropy rises from 1.24 to 1.47 during Stages 1–2 (exploration and foundation building), then falls to 0.66 during Stage 3 reinforcement learning, indicating convergence to focused reasoning.

• Visual attention becomes object-centric: On 400 SPBench-SI samples, SpatialLadder achieves 73.5% accuracy and 37.7% visual attention IoU versus the base model's 32.1% accuracy and 33.8% IoU, with lower attention entropy (0.176 vs. 0.193).

• Chain-of-thought adds a modest but consistent gain: 0.8% improvement, with the model exhibiting self-verification and error-correction behavior.

• Data quality filtering is aggressive: Constraints on scene diversity, object diversity, object identifiability, and a 40% minimum visibility threshold eliminate approximately 90% of initially generated samples.

Methodology in Plain English

The authors first ran a diagnostic: they gave a 3B VLM 200 spatial orientation questions and gradually added visual hints (bounding boxes, then directional arrows). Accuracy climbed steadily, showing the model could reason spatially but wasn't perceiving well enough to do so on its own.

They then built a training curriculum in three rungs. First, the model learns to point at objects — outputting bounding boxes and labels in JSON for spatially referenced objects, using 6k samples. Second, it learns spatial understanding tasks across seven dimensions (relative direction, relative distance, absolute distance, object size, counting, room size, appearance order) spanning single images, eight-view multi-view sets, and 1–4 minute videos at 24 fps. Third, it learns to reason explicitly using GRPO reinforcement learning, where rewards come from two sources: a format reward checking proper reasoning tags, and an accuracy reward that is binary for multiple-choice questions and graduated by relative error for numerical answers.

Data comes from ScanNet 3D reconstructions, converted into 2D images with unified metadata (3D boxes, 2D boxes, depths, visibility ratios, object sizes), with question-answer pairs generated from templates adapted from VSI-Bench and filtered for quality. Training used four NVIDIA A6000 GPUs with 48GB each.

Why This Matters

Impact on research: The paper argues against treating spatial reasoning as a monolithic capability to be trained end-to-end. It shows a 3B model can beat much larger proprietary systems and specialized 3D-encoder architectures on spatial tasks, suggesting that data curation and training order may matter more than model scale or architectural modification. It also releases two new in-domain benchmarks (SPBench-SI with 1,009 questions, SPBench-MV with 319 questions) built from ScanNet validation scenes with zero scene-level overlap with training data.

Real-world applications:

• Robotics navigation, where agents must judge distances and directions in physical space.
• Autonomous driving, requiring reliable spatial judgment about surrounding objects.
• Virtual reality systems, where user-perspective spatial understanding is essential.
• General embodied AI and any interface where a model must answer "where is X relative to Y?"

Industry relevance: A 3B model that outperforms GPT-4o by 20.8% and Gemini-2.0-Flash by 10.1% on in-domain spatial benchmarks is deployable at far lower cost than proprietary frontier models, and the finding that progressive training substitutes for specialized 3D encoders lowers the architectural barrier to entry.

Future Directions

• Scaling the dataset and curriculum order: The authors state that analysis of dataset scaling and progressive training order is provided in their appendices, but the truncated content does not report those results — the paper's conclusion points explicitly to future research directions in its Appendix E, which is not included in the available content.
• Extending modality coverage: Video reasoning currently rests on 9,000 samples sampled from SR-91k; whether the same progressive recipe scales to longer or more complex temporal sequences remains open.
• Understanding the entropy trajectory: Semantic entropy increases before it decreases; whether this exploration-then-convergence pattern generalizes to other capability hierarchies beyond spatial reasoning is untested.
• Robustness of the reward design: Stage 3 uses binary accuracy rewards for multiple-choice and graduated rewards for numerical answers; how sensitive results are to those thresholds is not reported in the available content.

Target Audience

Researchers and engineers working on multimodal models, spatial reasoning, or reinforcement learning for VLMs will get the most from this paper. It is also valuable for practitioners who need strong spatial capabilities in a small (3B) deployable model, and for dataset builders interested in a standardized pipeline for grounded, high-quality spatial question-answer construction. Readers without exposure to GRPO or VLM fine-tuning will need to consult the cited background work.

Authors’ abstract

Spatial reasoning remains a fundamental challenge for Vision-Language Models (VLMs), with current approaches struggling to achieve robust performance despite recent advances. We identify that this limitation stems from a critical gap: existing methods attempt to learn spatial reasoning directly without establishing the hierarchical foundations of perception and understanding. To address this challenge, we present a comprehensive methodology for building spatial intelligence progressively. We introduce SpatialLadder-26k, a multimodal dataset containing 26,610 samples spanning object localization, single image, multi-view, and video spatial reasoning tasks, constructed through a standardized pipeline that ensures systematic coverage across modalities. Building on this dataset, we design a three-stage progressive training framework that (1) establishes spatial perception through object localization, (2) develops spatial understanding through multi-dimensional spatial tasks, and (3) strengthens complex reasoning via reinforcement learning with verifiable rewards. This approach yields SpatialLadder, a 3B-parameter model that achieves state-of-the-art performance on spatial reasoning benchmarks, with 23.4% average improvement over the base model, surpassing GPT-4o by 20.8% and Gemini-2.0-Flash by 10.1%. Notably, SpatialLadder maintains strong generalization with 7.2% improvement on out-of-domain benchmarks, demonstrating that progressive training from perception to reasoning is essential for robust spatial intelligence.