Spatial-Interactor: Learning Spatial Reasoning through Interaction with the Observable Physical World

AI summary

Overview

• Research area: Spatial reasoning for vision-language models (VLMs) — specifically how models track how physical scenes change across time and viewpoints, and how they can learn that from interaction data rather than from static question-answering.
• Technical level: Advanced. It assumes familiarity with VLM post-training, supervised fine-tuning, GRPO reinforcement learning, and distillation.
• Scope: The paper proposes Spatial-Interactor, a post-training framework plus a 108K-example dataset (LSI-108K) that teaches VLMs local physical state transitions and their long-horizon integration, and reports gains across four model backbones and multiple spatial benchmarks.

What This Paper Is About

VLMs are good at describing images but still struggle with the physical world: they lose track of spatial state, confuse earlier observations, and miss fine-grained changes when scenes evolve. The paper argues this is because most spatial training uses static questions about object attributes and relations, which gives almost no supervision about how a scene changes from one moment to the next. The goal is to train VLMs on interaction trajectories — a before-observation, an action, and an after-observation — so they learn both to model single changes and to accumulate many changes across a long trajectory.

Key Contributions

1. Spatial-Interactor framework. An interaction-centric post-training method that treats interaction trajectories as direct supervision for physical-world state transitions, covering both local transition modeling and long-horizon integration.
2. LSI-108K dataset. A three-level curriculum of 108K verifiable QA pairs synthesized from large-scale simulated interactions (AI2-THOR, ProcTHOR, HSSD, Replica, SIMS-V) and real-world trajectories (ScanNet, ScanNet++, MultiScan, ARKitScenes, RoomTour3D, and robot data from BridgeData V2). The levels are L1 passive world-state transitions, L2 active self-state transitions, and L3 long-horizon interaction trajectories.
3. Two-stage training recipe. Supervised Fine-Tuning (SFT) on L1 and L2 for local transition modeling, followed by On-Policy Distillation (OPD) inside the GRPO framework, where a privileged teacher with segment-level transition descriptions supervises the student's on-policy chain-of-thought for L3.
4. Diagnostic evidence for the problem. Two experiments (a frame-shuffling test and a local-to-long-horizon comparison) that show current VLMs largely ignore temporal order and fail to integrate consecutive transitions.

Main Findings

• Frame shuffling barely hurts base models. On VSTI-Bench, randomly shuffling the 32 sampled frames of each video reduced Qwen2.5-VL-3B and Qwen2.5-VL-7B accuracy by only 0.9 and 0.8 points respectively, with no sub-task category changing by more than 1.4 points — evidence that these models rely mostly on order-invariant visual cues.
• Long-horizon integration collapses. Performance dropped from 54.7 to 8.6 for Qwen2.5-VL-7B and from 88.7 to 23.9 for GPT-5.5 when moving from the local SAT-Real interaction task to the long-horizon VSTI-Bench camera-displacement task.
• Explicit transition text helps. Supplying GPT-5.5's own descriptions of local state transitions across consecutive segments raised its camera-displacement score from 23.9 to 35.5.
• Large consistent gains over base models. Spatial-Interactor improved Overall by 16.4 to 25.0 points across four backbones (Qwen2.5-VL-3B/7B, Qwen3-VL-4B/8B). Relative Distance rose by 8.3–19.2 points, Route Planning by 2.6–7.7 points, and Camera Displacement by 5.1–28.0 points.
• Best reported scores on three benchmarks. Spatial-Interactor variants achieved the best VSI-Bench, MindCube, and VSTI-Bench averages, with the 8B model reaching the best Overall score of 65.9. SpatialLadder remained strongest on SPBench-MV.
• Generalization to unseen benchmarks. On MMSI, ViewSpatial, SAT-Real, and SAT-Syn — all absent from the training mixture — Overall improved by 4.3–10.6 points. Most of the gain appeared after SFT, and OPD added a further 1.0–1.4 points over the SFT checkpoint for every backbone.
• Interaction data adds value beyond public spatial QA. With Qwen2.5-VL-7B, external-data SFT reached 56.0 Overall; adding the interaction-derived L1 and L2 data raised it to 58.4 and improved every benchmark average by 1.4–3.9 points.
• OPD beats matched GRPO. From the same Full SFT checkpoint, standard GRPO reached 59.2 Overall while OPD reached 60.1, with OPD exceeding GRPO by 5.1 points on Route Planning and 3.1 points on Camera Displacement.
• Curriculum order matters. Training sequentially on L1 → L2 was the only configuration that improved all four held-out benchmarks over Base, raising the average from 44.1 to 49.8.
• Training dynamics improve together. Across the four backbones, mean on-policy task reward increased by 6.8–8.9 points between the first and final 100 OPD steps, rollout diversity declined but stayed nonzero, and divergence between the plain and privileged branches fell by 27–56%.
• Temporal order matters more than trace length. On all 839 Camera Displacement questions, an ordered four-segment trace raised a frozen SFT checkpoint's score from 32.6 to 35.4; shuffling the same descriptions lowered it to 33.8, and eight ordered segments yielded 35.3.
• Trained models actually use frame order. Spatial-Interactor-3B and -7B dropped by 6.6 and 7.0 points under frame shuffling, versus only 0.9 and 0.8 points for their Base counterparts, with the clearest changes on Camera Displacement and Camera Motion.
• Closed-loop interaction improves. On WalkerBench Standard-100 with Qwen2.5-VL-7B, overall success rose from 7.0 to 14.0 and navigation from 5.0 to 10.0, while mean actions fell from 17.71 to 6.52. On eight ESI-Bench categories (30 matched questions each), average success rose from 32.9 to 41.2 and mean rounds among solved episodes fell from 12.34 to 10.55.

Methodology in Plain English

The core idea is that an interaction naturally pairs a before-picture, an action, and an after-picture. That pair of pictures plus the action tells the model exactly what changed, and the change can be checked against simulator or sensor metadata.

The authors build data in three levels. L1 covers changes in the world while the camera stays roughly still — objects moving, attributes changing, things being occluded or becoming visible, and single- or multi-step operations. L2 covers changes caused by the observer — camera translation, rotation, and elevation — plus the cross-view spatial reasoning needed to keep the scene straight. L3 covers whole camera trajectories, asking about path length, endpoint displacement, trajectory shape, turning points, revisited places, and reverse paths.

Targets are computed from state, pose, and trajectory metadata and turned into question-answer pairs by deterministic templates. Negligible transitions, discontinuous or redundant trajectories, and ambiguous targets are removed, and a frozen vision-language verifier rejects questions that cannot be answered from the visuals without altering their ground truth — so the labels stay verifiable.

Training happens in two stages. First, SFT on 82,596 L1–L2 examples from LSI-108K plus 80K public spatial QA samples. Second, OPD on 10,712 L3 RoomTour examples and 10,783 long-horizon VSTI-Bench examples. In OPD, each long video is split into four contiguous intervals from 32 uniformly sampled frames, and a frozen VLM annotator writes question-independent descriptions of the environment, the visible change, and coarse camera motion for each interval. These descriptions form a privileged trace seen only by the teacher. The teacher is a stop-gradient snapshot of the policy before the update — not an exponential-moving-average model — and it never generates its own answer. Instead, both student and teacher score the same student-generated reasoning prefix, with the teacher additionally seeing the trace. The training objective adds a process term (teacher-to-student forward KL on the teacher's top-k non-special-token support, applied only at reasoning positions) to the verifiable answer reward and a reference-policy KL, with the process weight gradually decayed. A separate frozen SFT reference policy provides the conventional regularizer and never sees the trace. Both stages update the language model and multimodal projector while keeping the visual encoder frozen, and training and evaluation records are kept disjoint at the scene and video levels.

Why This Matters

The paper reframes spatial training: instead of asking "what is where?", it asks "what changed, and why?" This directly addresses the failure modes that keep VLMs out of real dynamic environments, and the diagnostic experiments suggest the fix should be process-level supervision rather than more static QA. Because the gains appear across four backbones of different sizes and generations, and transfer to benchmarks never seen in training, the recipe looks like a general post-training add-on rather than a benchmark-specific trick. The increased sensitivity to frame order in trained models is a behavioral signal that they are relying on temporal evidence rather than shortcuts.

Real-world applications:

• Visual navigation and object search in homes and offices, where an agent must remember how a room changed as it moved through it.
• Robotic manipulation, using action-conditioned scene changes to predict the spatial consequence of a grasp or placement.
• AR and 3D scene understanding, where moving the camera changes the observation and the model must preserve scene identity across views.
• Interactive assistants and embodied agents that must make repeated decisions as their own actions alter what they see next.

Industry relevance: the method builds on existing open VLM backbones and reuses simulation and real trajectory datasets rather than requiring new hardware. Its reliance on verifiable, automatically generated labels makes the data pipeline cheap to scale, which matters for teams that cannot afford large human annotation efforts.

Future Directions

• Adaptive trajectory segmentation. The conclusion names this explicitly; the current design splits videos into four contiguous intervals, and the trace experiment showed that eight ordered segments (35.3) performed about the same as four (35.4), leaving open how segmentation should be chosen.
• More diverse real interactions. Also named in the conclusion: extending beyond the current simulation and real-trajectory sources.
• How to extend the privileged-trace idea beyond spatial reasoning. OPD added 1.0–1.4 points on benchmarks outside the long-horizon training tasks, which raises the question of whether process supervision from privileged descriptions transfers to other domains.
• Isolating why OPD helps long-horizon integration. The paper shows the process weight is decayed and that the privileged-branch divergence falls 27–56%, but does not report an ablation of the trace contents themselves, the annotator model, or the decay schedule.

Target Audience

Researchers and engineers working on vision-language models, multimodal post-training, reinforcement learning from verifiable rewards, and embodied or navigation agents. It is most useful to readers already comfortable with SFT and GRPO-style policy optimization who want a concrete recipe for converting interaction trajectories into training signal. Readers looking for an introductory treatment of spatial reasoning benchmarks will find the diagnostic experiments and benchmark tables accessible, but the method sections assume prior exposure to distillation and policy-gradient training.

Authors’ abstract

Spatial reasoning is essential for vision-language models (VLMs) to understand and act in the physical world. Reasoning in dynamic environments requires VLMs to perceive local state transitions caused by object motion and viewpoint changes and integrate them over long trajectories to maintain an updated spatial state, yet existing VLMs remain limited in both capabilities. Current spatial training primarily focuses on static questions about object attributes and spatial relations, providing limited direct supervision for state transitions; in contrast, interaction trajectories naturally connect a preceding observation, an action, and a subsequent observation, offering direct supervision for local state transitions, while complete trajectories reveal dependencies among consecutive transitions. We therefore introduce Spatial-Interactor, a framework that trains VLMs to model physical-world state transitions through interaction, organizing this learning process into a three-level curriculum covering L1 passive world-state transitions, L2 active self-state transitions, and L3 long-horizon interaction trajectories. Accordingly, we construct the Learning from Spatial Interaction dataset (LSI-108K) from simulated and real interaction trajectories, with tasks aligned with the objective of each level. Our two-stage training strategy applies Supervised Fine-Tuning (SFT) to L1 and L2 for local transition modeling, and On-Policy Distillation (OPD) then uses privileged self-distillation: a teacher branch given segment-level transition descriptions supervises the student's on-policy CoT, helping the student learn to integrate consecutive transitions over L3 long trajectories. Experiments across multiple VLMs and spatial benchmarks show consistent gains in local transition modeling and long-horizon integration.