SpatialASM: Internalizing Active Mentalization for Spatial Reasoning

AI summary

Overview

Research area: Computer vision and multimodal learning — specifically spatial reasoning in Large Vision-Language Models (LVLMs), with a focus on post-training data synthesis and reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with vision-language model post-training (SFT, GRPO), novel-view synthesis, and 3D reconstruction metrics.

Scope: The paper proposes SpatialASM, a post-training framework that uses an allocentric novel-view generator as a training-time simulator to internalize "Active Spatial Mentalization" — the joint ability to plan useful viewpoints and mentally imagine the scene from them — into LVLMs.

What This Paper Is About

Current LVLMs handle static spatial perception reasonably well but fail on problems that require reasoning about a 3D scene from viewpoints the input images do not provide, such as cross-view correspondence and viewpoint transformation. The authors argue that what is missing is Active Spatial Mentalization (ASM): the ability to proactively decide which alternative viewpoints would reveal useful evidence and to mentally construct what the scene looks like from those viewpoints.

The goal is to teach this plan-imagine-reason behavior to LVLMs without attaching an external world model at inference time, and without relying on scarce human-written spatial chain-of-thought data.

Key Contributions

1. A formulation of Active Spatial Mentalization (ASM) as two coupled skills — view planning (selecting task-relevant viewpoints given the question and observations) and view imagination (mentally constructing the layout expected from those viewpoints) — and the claim that its absence is a key bottleneck for LVLM spatial reasoning.

2. The SpatialASM framework, built on an allocentric novel-view generator (NVG) used only at training time, plus a Geometry Perturbation (GeoP) metric that acts as a model-based proxy for whether a rendered view is geometrically consistent with the input scene.

3. Two complementary forms of supervision derived from answer-only spatial QA data: explicit ASM trajectories (view plan, imagined-view captions, reasoning, answer) for supervised fine-tuning, and automatically verifiable perspective-taking multiple-choice tasks for reinforcement learning with GRPO.

4. An empirical demonstration across four LVLM backbones (Qwen3-VL-4B, Qwen3-VL-8B, Qwen3-VL-30B-A3B, Qwen3.5-9B) and seven spatial benchmarks that SpatialASM improves average spatial accuracy by +1.2 to +3.5 points while keeping general-VQA performance within 0.6 points of the baselines.

Main Findings

• Consistent gains across four backbones: Average accuracy over the seven spatial benchmarks rises from 43.8 to 47.3 (+3.5) for Qwen3-VL-4B, 46.2 to 49.1 (+2.9) for Qwen3-VL-8B, 44.7 to 47.2 (+2.5) for Qwen3-VL-30B-A3B, and 51.1 to 52.3 (+1.2) for Qwen3.5-9B.

• Largest gains on the hardest tasks: The biggest per-benchmark jumps appear on tasks requiring spatial mental modeling and viewpoint transformation — +8.7 on Spatial457-6d (Qwen3-VL-8B, 34.8 to 43.5) and +5.2 on MindCube (Qwen3-VL-4B, 33.9 to 39.1). Modest gains appear on perception-centric benchmarks such as 3DSR and EmbSpa.

• General visual ability is preserved: Changes in average general-VQA accuracy stay within 0.6 points on all four backbones (63.7 to 63.1 for 4B, 64.4 to 65.0 for 8B, 69.4 to 68.9 for 30B-A3B, 78.1 to 78.5 for 9B). The paper contrasts this with HoliSpatial, which it reports degrades by 1.3% on average on the general-VQA benchmarks after SFT.

• External ASM helps only sometimes; internalization is more reliable: On MindCube subtasks, adding external ASM to baseline Qwen3-VL-8B improves Among (33.0 to 42.0) but degrades Around (52.0 to 45.5). SpatialASM-8B improves under vanilla reasoning on both (39.5 Among, 58.0 Around) with roughly comparable inference time (18.6 s vs 12.6 s on Among; 7.2 s vs 7.5 s on Around), and combining it with external ASM gives the best results (45.0 Among, 61.5 Around).

• Allocentric framing matters more than raw geometric fidelity: SpatialASM-NVG gives the highest downstream accuracy in the novel-view generator comparison (55.5 / 70.5 with Qwen3.5-35BA3B; 68.0 / 80.5 with Kimi-k2.5 on Among / Around) even though SVC achieves a lower GeoP (0.087 and 0.072 versus 0.118 and 0.099 for SpatialASM-NVG).

• GeoP filtering helps both training stages: On Qwen3-VL-4B, SFT with filtering reaches an average of 49.9 versus 48.4 without filtering and 46.5 for the baseline; RL with filtering reaches 48.8 versus 48.2 without. SFT and RL are complementary — SFT drives MindCube (+4.8) and ViewSpa (+3.0) gains, while RL is stronger on EmbSpa, and the combination reaches the best average (50.6).

• Existing LVLMs can plan only when explicitly prompted: In the preliminary study on 200 questions from each of the MindCube Among and Around subtasks, planning-based external ASM generally beats vanilla reasoning and LVLM-based planning matches or beats random planning — but Qwen3-VL-8B drops under both random and LVLM-based planning, indicating the capability is not autonomously invoked.

Methodology in Plain English

The authors use a world-model-like image generator as a training-time scaffold rather than an inference-time tool.

First, they fine-tune an image-editing model (FLUX.2-klein-9B) into SpatialASM-NVG, which renders a scene from a requested viewpoint. The viewpoint is specified allocentrically — a single tuple (anchor image index, pixel location on that image, viewing direction) — so a target view can be requested in one step instead of a sequence of camera-motion commands.

Second, they check whether each rendered view is trustworthy using GeoP: reconstruct a 3D point cloud from the original images, reconstruct again with the rendered view added, and measure the Chamfer distance between the two point clouds. A high value means the rendered view forced the reconstruction to change geometry, which signals inconsistency.

Third, they synthesize reasoning trajectories. An LVLM is prompted to plan helpful viewpoints; the NVG renders them; the LVLM re-answers using those renders. A trajectory is kept only if the model answered wrongly from the original images, answered correctly after seeing the rendered views, and every rendered view passed the GeoP threshold. Then the rendered images are replaced by text captions describing object layout, so the training target teaches the model to imagine the view internally rather than consume an image. The generator is never called at inference.

Fourth, for reinforcement learning they build automatically checkable multiple-choice questions from GeoP-filtered candidate views: one asks the model to locate on the anchor image where a given rendered view was captured; the reverse asks the model to pick which of four candidate views matches a described location and direction.

Training is two-stage — supervised fine-tuning on SpatialASM-66k for 10 epochs at learning rate 5e-6, then GRPO with global batch size 128, learning rate 1e-6, for 300 steps. Qwen3-VL-4B and Qwen3-VL-8B are fully fine-tuned; Qwen3-VL-30B-A3B and Qwen3.5-9B use LoRA with rank 32.

Data comes from DL3DV (10k scenes) for the generator training set SpatialASM-edit140k, from Holi-spatial, VST-500k and Mindcube-train for the 128k reasoning trajectories (SpatialASM-128k, with the HoliSpatial-only SpatialASM-66k subset used for tuning), and from HoliSpatial and DL3DV 2D observations for SpatialASM-RL-58k.

Why This Matters

Impact on research: The paper offers a general recipe for converting answer-only spatial QA datasets into chain-of-thought supervision without human annotation, addressing a widely noted scarcity of spatial CoT data. It also reframes the training-time versus inference-time world-model debate by showing what should be distilled (question-conditioned view planning) rather than only how views change under motion, and it shows that geometric fidelity alone does not predict downstream usefulness of generated views.

Real-world applications:

• Robotic manipulation and embodied agents that must reason about occluded or unobserved regions before acting.
• Autonomous driving, where decisions depend on what would be visible from another position or heading.
• AR/VR and 3D content tools that need to answer questions about a scene from arbitrary user viewpoints.
• Assistive and instructional systems that must explain spatial layouts (assembly, navigation) from photos taken elsewhere.

Industry relevance: Because the approach requires no architectural modification and no external simulator at inference time, it is a portable post-training recipe that can be layered onto existing LVLM checkpoints — attractive for teams that cannot afford multi-step tool rollouts at serving time, though the paper does not report serving cost beyond the per-subtask timings in Table 2.

Future Directions

• Extending ASM beyond static scenes: the authors state they plan to extend the approach to dynamic real-world scenes, embodied interaction, and longer-horizon 3D reasoning.
• Internalizing other tool-assisted reasoning: the conclusion explicitly names internalizing other forms of tool-assisted reasoning as future work.
• Combining internalized ASM with test-time simulators: the paper's own results show internalized and external ASM are complementary (highest accuracy when combined), leaving open how to schedule or gate external calls.
• Open questions the paper raises but does not resolve: how to set the GeoP thresholds (the specific values of the SFT and RL thresholds are not reported), how many viewpoints K to plan, and why GeoP does not correlate perfectly with downstream accuracy — the authors attribute the latter to the allocentric interface but do not test this directly.

Target Audience

Researchers and engineers working on vision-language models who are interested in spatial reasoning, post-training data synthesis, and reinforcement learning for multimodal models. It is most useful to readers already comfortable with SFT/GRPO pipelines and novel-view synthesis evaluation; readers new to LVLM post-training will find the high-level idea accessible but the implementation details demanding. Practitioners building embodied, robotics, or AR/VR systems on top of open LVLM checkpoints are a secondary audience.

Authors’ abstract

When solving complex spatial reasoning tasks, ordinary humans naturally revisit the scenes from task-relevant viewpoints and mentally reconstruct the spatial layouts from those viewpoints. But can Large Vision-Language Models (LVLMs) proactively think in the same way? We term this ability Active Spatial Mentalization (ASM), an intrinsic component of visual intelligence that remains substantially underdeveloped in current LVLMs. To bridge this gap, we introduce SpatialASM, a scalable framework to internalize ASM into LVLMs through training-time spatial simulation. SpatialASM employs an allocentric novel-view generator to ground hypothetical viewpoints in visual evidence and produces two complementary forms of supervision: i) explicit ASM trajectories that teach LVLMs when and how to adaptively invoke view planning and imagination during spatial reasoning, and ii) automatically-constructed perspective-taking tasks that further strengthens viewpoint-conditioned scene imagination. On seven benchmarks spanning relatively perception-oriented spatial understanding to more challenging tasks requiring multi-image, cross-view and viewpoint-transformation reasoning, SpatialASM yields performance gain on LVLMs of four LVLM backbones in spatial reasoning while retaining their original general vision capabilities. The experimental results demonstrate that our approach improves average accuracy across the seven spatial benchmarks. And the synthesis of ASM trajectories provides a scalable pathway to alleviate the scarcity of spatial Chain-of-Thought (CoT) supervision for eliciting active viewpoint exploration and mental scene simulation.