SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem

AI summary

Overview

• Research area: Computer Vision / Vision-Language Models — specifically 3D spatial reasoning in Large Vision-Language Models (LVLMs).
• Technical level: Intermediate. Readers should be comfortable with LVLM training pipelines, supervised fine-tuning, LoRA, and reinforcement learning (GRPO), but the core idea — block-stacking puzzles as training data — is intuitive.
• Scope in one sentence: The paper introduces a 15,000-sample synthetic block-stacking dataset and two training strategies (direct answering and reasoning-based) that improve LVLMs' spatial reasoning on real-world benchmarks without any real-scene annotations.

What This Paper Is About

LVLMs handle 2D image understanding well, but they are weak at mentally reconstructing the 3D structure of a scene from a 2D image — a capability the authors call spatial intelligence. Prior work tries to fix this with real-scene spatial question-answering datasets that need dense geometric and object-level annotations, which are expensive, slow to produce, and noisy because they rely on external perception modules. The authors take a different route inspired by how children develop spatial cognition: train models on synthetic block-stacking problems that are cheap to generate, clean, and scalable.

Key Contributions

1. A new framing of the problem. The paper shifts spatial-intelligence training away from annotation-heavy real-scene supervision toward foundational spatial-skill learning through structured synthetic tasks.
2. SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering three task types — 3D-to-2D projection, viewpoint transformation, and structural combination — each contributing 5k samples, extended with controlled color cues that act as anchors for reasoning.
3. Two training strategies, SpatialBlock-direct (supervised answer-only prediction) and SpatialBlock-reason (LoRA initialization followed by GRPO reinforcement learning with a multi-objective reward on accuracy, format, and length).
4. Empirical evidence that synthetic block data transfers to real scenes. Models trained on only 15K synthetic samples outperform existing spatial-specialist models on real-world benchmarks such as MindCube, MMSI-Bench, SPBench, and MMMU.

Main Findings

• Large in-domain gains. On SB-Bench — the held-out test split of SpatialBlock-15k containing 600 questions — direct models reach 94.8 (3B), 95.0 (7B), 95.7 (4B), and 97.2 (2B), far above their backbones (29.9 for Qwen2.5-VL-3B-Instruct, 20.8 for Qwen2.5-VL-7B-Instruct, 17.0 for Qwen3-VL-4B-Instruct, and 19.0 for InternVL3-2B-Instruct).

• Better than spatial specialists on MindCube. The 3B direct model outperforms the state-of-the-art SpatialLadder by 2.7%, the 7B direct model beats its backbone by 17.6%, and the 4B direct model improves over its backbone by 25.1%, achieving 51.3 — described as the best open-source performance on that benchmark.

• Reasoning models help on multi-image and hard reasoning tasks. On MMSI-Bench, SpatialBlock-3B-reason outperforms SpatialLadder-3B by 3.2% using a simpler pipeline based solely on synthetic block-stacking data, and SpatialBlock-7B-reason outperforms both SpaceR and Spatial-SSRL with only 15K training samples.

• Improved reasoning quality, not just answers. The reasoning model achieves a higher alignment score of 21.1 with ground-truth reasoning steps on MMSI-Bench, compared to 17.8 for the baseline, scored by GPT-5 on a 1 / 0.5 / 0 scale averaged and scaled to 100.

• Proprietary models struggle on this task. On SB-Bench, GPT-5 scores 43.8, Gemini-2.5-flash 36.4, and Claude-Sonnet-4.5 46.3 — far below the trained models despite human participants reaching 95% average accuracy on 30 sampled questions from SB-Bench across 31 participants.

• Color cues matter. Removing visual cues drops MindCube accuracy by 7.9% (direct) and 7.3% (reason), and MMSI-Bench by 1.9% (direct) and 4.8% (reason).

• All three question types are needed. Training on Q1 only, Q2 only, or Q3 only yields SB-Bench scores of 48.0, 51.5, and 40.8 respectively, far below the 90.2 obtained with all three types combined.

• Task design drives the gains. A model trained with the identical recipe on SpatialLadder-26k (using 11.5k multiple-choice questions, 1.3 epochs to match training size) scores 24.7 (direct) and 28.0 (reason) on SB-Bench, but reaches 72.1 and 68.8 on SPBench, its in-domain benchmark.

• LoRA initialization beats cold-start. Without initialization, SpatialBlock-3B-reason drops to 67.8 on SB-Bench. Cold-start with full-parameter tuning on Qwen2.5-VL-7B reaches 96.5 in-domain but only 43.0 on MindCube, and GPT-5-generated cold-start data reaches 75.3 and 41.7.

• Block tasks generalize better than conventional spatial questions. Against a same-sized Synthetic-Real dataset built from relative direction and distance questions, SpatialBlock scores higher out-of-domain (MindCube 49.1 vs 32.8; MMSI-Bench 28.1 vs 25.5; MMMU 46.4 vs 43.6) while Synthetic-Real wins only in-domain (relative direction 43.4 vs 31.4; relative distance 76.9 vs 67.4).

• General visual perception is not degraded. On MMMU, trained models stay in a comparable range to their backbones despite seeing no real-scene images during training.

• Counting and size estimation improve, absolute distance does not. On numerical SPBench tasks, SpatialBlock-3B-direct reaches 72.1 on object counting versus 35.1 for Qwen2.5-VL-3B-Instruct, and 19.9 versus 17.6 on object size, but drops on absolute distance (23.2 vs 30.9).

• Stable across viewing angles. Training uses a fixed 45° rendering angle; both model variants maintain consistent performance when the same problems are re-rendered at 30° and 60°.

Methodology in Plain English

Dataset construction. All 3D structures are built on a 3 × 3 spatial grid where each grid position holds between one and four vertically stacked blocks. Configurations with severe occlusion — where blocks are completely hidden from the viewing perspective — are excluded so the structure stays inferable from the image. Three question types are generated from these structures:

• Q1, 3D-to-2D projection: predict the 2D appearance of a 3D structure from a given viewing direction, requiring depth-aware occlusion reasoning. Distractors are built by removing a visible top block, swapping a block's color with a neighbor in the same column, or a combination of both.
• Q2, viewpoint transformation: identify where a highlighted block ends up after a randomly chosen transformation (90°, 180°, or 270° rotation, or vertical flip). Only blocks still visible after transformation are used as anchors. Distractors include the original structure with a different block highlighted and a structure produced by a different transformation.
• Q3, structural combination: infer the union of two structures whose attachment point is marked by overlapping, transparently colored blocks. Distractors differ in only one of the two input structures.

Color as a functional cue. A second version of each task adds color meaningfully rather than decoratively: colors encode front-to-back depth ordering in Q1, one anchor block keeps a distinct color across a transformation in Q2, and same-colored blocks across two separate images mark the correspondence point in Q3.

Two training strategies. The direct model minimizes standard cross-entropy over the ground-truth answer tokens only. The reasoning model first adapts via LoRA — rank 8 for the 3B and 2B models, rank 16 for the 7B and 4B models — deliberately avoiding full fine-tuning because it degrades inherent chain-of-thought ability, and avoiding teacher-model cold-start because even GPT-5 produces unreliable reasoning on these tasks. It is then trained with GRPO, using a binary reward summing accuracy (final answer matches ground truth), format (a reasoning trace followed by the answer in <answer>/</answer> tags), and length (response length L must satisfy 50 < L < 1024). Training uses EasyR1 defaults, batch size 512, 5 rollouts, maximum prompt length 4096, and 1 epoch over SpatialBlock-15k.

Evaluation. Four backbone models are used — Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen3-VL-4B, and InternVL3-2B. Evaluation covers SB-Bench in-domain plus four out-of-domain benchmarks: MindCube, MMSI-Bench, and SPBench for spatial reasoning, and MMMU for general visual perception. Only multiple-choice questions are used, to match the training answer format. For direct-model training, early stopping picks the iteration just before test accuracy begins to decline, at a point where it exceeded 93%.

Why This Matters

Impact on research. The paper challenges the assumption that spatial intelligence requires dense real-scene 3D annotation, showing that a compact synthetic dataset of a structured cognitive task can transfer to real-world benchmarks. It also reports that state-of-the-art proprietary models fail basic block-stacking reasoning while humans reach 95% accuracy, which reframes spatial reasoning as a foundational gap rather than a scaling problem. The finding that a LoRA-initialized RL pipeline beats cold-start teacher distillation on this task is a practical signal for how reasoning models should be trained when no reliable teacher exists.

Real-world applications:

• Robotics manipulation, where a model must infer the 3D arrangement of objects before grasping or placing them.
• Autonomous driving, where scenes must be understood structurally from 2D camera input.
• Augmented and virtual reality, where a system must reason about how a scene changes under viewpoint shifts.
• Assistive and embodied agents that interpret a user's physical environment and act on spatial relationships.

Industry relevance. The method replaces an expensive annotation pipeline with synthetic data that is cheap and scalable to generate, and it works with 2B to 7B open-source backbones — so spatial capability can be added without proprietary model access or large annotation budgets. Code and data are released at https://github.com/rsoohyun/SpatialBlock.

Future Directions

• Closing the remaining out-of-domain gap. Direct models excel at canonical 90-degree viewpoint transformations, while reasoning models do better on diverse viewpoint changes and multi-image reasoning — the paper does not report a single model that dominates both, leaving open how to combine them.
• Metric spatial reasoning. The model improves counting and size estimation but is less effective at absolute distance, because the training objective prioritizes structural understanding and spatial integration over precise metric regression; extending block tasks to metric supervision is an open question.
• Better cold-start trajectories. Training with traces from Qwen2.5-VL-7B and GPT-5 yields only marginal gains because those traces contain inaccurate block descriptions and reasoning, so producing reliable teacher reasoning for spatial tasks remains unsolved.
• Beyond blocks. The dataset is limited to 3 × 3 grids with one to four stacked blocks per cell and excludes heavily occluded configurations; whether the same cognitive-development framing scales to more complex objects or richer scenes is not reported.

Target Audience

Researchers and practitioners working on vision-language models, multimodal reasoning, and embodied AI, particularly those building spatial-reasoning training pipelines or looking for cheaper alternatives to annotated 3D data. It is also relevant to engineers fine-tuning small open-source LVLMs (2B–7B) who need spatial capability without a large annotation budget, and to cognitive-science-adjacent readers interested in the block-play analogy for machine spatial learning.

Authors’ abstract

Large Vision-Language Models (LVLMs) have achieved strong performance on diverse visual tasks, yet their ability to reconstruct and reason about the 3D structure of the scene depicted in 2D images -- referred to as spatial intelligence -- remains limited. Existing approaches attempt to address this gap by using real-scene spatial question answering datasets that require dense geometric annotations. However, constructing such labels is costly, time-consuming, and often noisy due to reliance on external perception modules. In this work, we propose a novel paradigm inspired by human cognitive development: learning foundational spatial skills through structured block-manipulation tasks. We introduce SpatialBlock-15k, a synthetic dataset of 15,000 block-stacking problems covering 3D-to-2D projection, viewpoint transformation, and structural combination. The dataset further incorporates controlled color modulation as visual cues to encourage anchor-based reasoning in visually complex conditions. Experiments demonstrate that LVLMs trained on our dataset through either direct answering or reasoning-based prediction significantly outperform baselines and generalize to real-world spatial tasks, despite the dataset's synthetic and compact nature. Code and data are available at https://github.com/rsoohyun/SpatialBlock.