S-Space: Exploring Spatial Workspace in Multimodal Language Models

AI summary

Overview

Research area: Mechanistic interpretability of multimodal large language models (MLLMs), applied to spatial reasoning — computer vision and vision-language research.

Technical level: Intermediate. The paper assumes some familiarity with linear representation hypotheses, logit contrasts, and activation-level causal interventions, but its core idea (a three-axis "coordinate system" inside the model) is explained geometrically and is accessible to a motivated non-specialist.

Scope: The paper identifies and characterizes a three-dimensional linear subspace — S-Space — inside MLLMs where object-name token representations encode continuous horizontal, vertical, and distance coordinates, and then uses it both to causally test how spatial information is used and to diagnose why spatial reasoning fails.

What This Paper Is About

Multimodal language models can answer spatial questions about images, but it is unclear how spatial information is represented inside them and where exactly their reasoning breaks down. This paper uncovers an internal, three-dimensional "spatial workspace" in which each object's token representation encodes a continuous coordinate along left-right, above-below, and close-far axes, and shows this workspace causally drives the model's spatial answers. The authors then use this workspace as a diagnostic instrument to separate spatial perception from spatial computation, concluding that models often perceive locations accurately but fail at the transformations needed to reason across viewpoints.

Key Contributions

1. Identification of an internal spatial workspace. The authors claim to be the first to systematically identify and characterize a linear spatial subspace in MLLMs in which spatial information is encoded, updated during reasoning, and causally used for responses.

2. A simple gradient-based method for finding S-Space. The three axes are recovered by differentiating answer contrasts (e.g., logit(right) − logit(left)) with respect to object-token activations, taking a role-antisymmetric difference between query and reference object gradients, and averaging over prompt templates, swapped object orders, and examples. A forward pass then reads out continuous coordinates by linear projection.

3. A causal and persistence test of the workspace. Coordinate editing inside S-Space selectively alters spatial judgments while largely preserving non-spatial (color) judgments, and the same fixed axes remain readable under prompts that never ask a spatial question.

4. S-Space as a diagnostic tool for spatial reasoning failures. By reading coordinates independently of the model's answer, the authors decompose errors into perception versus computation, and expose a systematic coupling between cardinal and image-relative reference frames.

Main Findings

• A three-axis, model- and layer-specific but token-position-shared subspace. S-Space is defined per model and per layer, spans the horizontal, vertical, and distance directions, and is shared across all token positions within that layer. The authors search in middle layers, since early layers stay anchored to inputs and later layers align with next-token prediction.

• Accurate coordinate readout. Projecting object-token representations onto S-Space yields coordinates that correlate strongly with ground-truth object positions and with pairwise coordinate differences along all three axes. The benchmark figures report correlations and least-squares fits for Molmo2-ER at layer 21; the specific correlation coefficients are not stated in the provided text.

• Coordinate readout rivals verbal answers and generalizes off-distribution. Answering relative spatial questions purely from S-Space coordinate differences achieves accuracy comparable to the models' direct verbal answers, and transfers to CV-Bench, EmbSpatial, and SpatialTunnel, including benchmarks outside the distribution used to identify the axes.

• Causal, selective use. Editing S-Space coordinates and writing them back into hidden representations substantially alters spatial judgments compared with random editing, with one principled exception: shifting both objects together leaves relative judgments unchanged. The same intervention has little effect on unrelated color judgments.

• Persistence across prompts. Replacing relative-comparison prompts with single-object queries, or even with non-spatial color questions, still yields coordinates that support accurate spatial judgments — evidence that S-Space is an intrinsic organization of the representations rather than an artifact of a prompt template.

• The workspace extends beyond visible objects and physical space. A goalkeeper diving toward an unseen ball places the token "ball" in the direction of the dive; the abstract token "you" projects toward the close end of the distance axis. Holding a delivery-robot image fixed while swapping "moving past you" for "moving toward you" shifts "you" from farther to closer while the robot's coordinate stays nearly unchanged. The concepts "socialist" and "conservative" project toward the left and right ends of the horizontal axis respectively, mirroring political metaphor.

• The main bottleneck is computation, not perception. On SpinBench perspective-taking, explicit transformation of S-Space coordinates outperforms both direct generation and chain-of-thought. Without a premise, Molmo2-ER scores 60.96 directly versus 94.52 with S-Space + rotation (+33.56); Qwen3.5-4B scores 60.96 directly, 62.33 with reasoning (+1.37), and 91.10 with S-Space + rotation (+30.14); Qwen3.6-27B scores 60.96 directly, 83.56 with reasoning (+22.60), and 95.21 with S-Space + rotation (+34.25).

• Even given correct perceptual premises, models still fail. In the premise setting, Molmo2-ER moves from 61.81 to 99.31 (+37.50), Qwen3.5-4B from 60.42 to 72.22 (+11.80) with reasoning and 97.92 (+37.50) with S-Space + rotation, and Qwen3.6-27B from 72.92 to 91.67 (+18.75) with reasoning and 99.31 (+26.39) with S-Space + rotation. Models often fail to transform correct spatial information they already have.

• Chain-of-thought partially, imperfectly transforms S-Space. In a controlled chain-of-thought analysis on Qwen3.6-27B (layer 43), object coordinates become progressively more aligned with the requested viewpoint, shifting from an advantage for rotated readout early to an advantage for direct readout later. The model internally simulates part of the perspective transformation, but not reliably enough to produce the correct verbal answer.

• Reference-frame coupling is visible in S-Space. Reading cardinal-direction tokens from Qwen3.6-27B's chain-of-thought on MMSI-Bench shows "east" aligning with right, "north" with above, and "northeast" reflecting both — partial sharing of features between cardinal and image-relative directions, hypothesized to reflect map-like biases in web-scale pretraining data.

Methodology in Plain English

The authors start from the hypothesis that concepts are encoded as directions in a model's internal activations. To find the direction corresponding to "rightness," they take an image and a question such as "Is the apple to the left or right of the cup?" and compute how the answer signal (the difference between the "right" and "left" output scores) changes as they nudge the apple's internal representation. The gradient points in the direction that would most favor one answer.

Doing this naively would pick up noise from object identity and phrasing, so they apply two corrections. First, they subtract the gradient for the reference object from the gradient for the query object, since a spatial relation acts oppositely on the two. Second, they average this quantity over five prompt templates, over swapped object orders, and over many images. Repeating for the vertical and depth axes yields three unit directions that span S-Space.

Reading out is then trivial: project any object's hidden representation onto those three directions to get a coordinate. To check whether the model actually uses this workspace, they edit the coordinates, write the edited values back into the hidden states, and let generation continue with everything outside the subspace untouched. To diagnose reasoning, they read coordinates from the model, apply an explicit geometric rotation for the requested viewpoint, and compare the resulting answer against what the model says on its own.

Training data for finding the axes came from COCO-2017: 6,000 images for axis identification (one object pair each) and a disjoint set of 1,800 images for layer selection, with 1,000 examples per spatial relation for the axes and 300 per relation for layer selection. Ground-truth horizontal and vertical positions come from annotated bounding-box centers normalized by image dimensions; depth comes from averaging Depth Anything V2-Small inverse-depth predictions within each instance mask. Models studied are Molmo2-ER, Qwen3.5-4B, and Qwen3.6-27B, plus MolmoAct2-Pretrain and MolmoAct2 (VLA variants derived from Molmo2-ER) for examining how representations change with action training. Notably, the same three-way readout achieving 60.96 direct accuracy without a premise is reported for all three main models on SpinBench.

Why This Matters

Impact on research. The paper shifts the discussion of MLLM spatial failures from end-to-end accuracy to internal mechanism. Rather than proposing another benchmark or a larger model, it supplies a measurable internal quantity that can be probed, edited, and tracked across reasoning steps. It also reframes a commonly stated weakness — poor perspective-taking — as a computation deficit over adequate perceptual representations, which points improvement efforts at transformation operators, reasoning procedures, and reference-frame handling rather than at perception alone. The identification of reference-frame coupling in S-Space gives a concrete mechanistic hypothesis for a well-known error class in geographic and map-based reasoning.

Potential real-world applications:

• Robotics and embodied agents — turning reliable perceived coordinates into reliable action decisions, and identifying whether a failure came from sensing or from planning.
• Navigation and assistive systems — diagnosing when a vision-language assistant confuses egocentric "left" with global "east" before deployment.
• Agentic pipelines with dedicated spatial tools — the authors explicitly suggest connecting separate modules for transformation, simulation, and memory instead of relying on one monolithic model.
• Model auditing and safety-adjacent evaluation — the finding that abstract political concepts inherit physical left/right geometry illustrates how pretraining data can imprint biases that are invisible in benchmark scores.

Industry relevance. The gradient-based axis identification is cheap — it needs only relative spatial questions and a few thousand images — and the readout is a single linear projection at inference time, so the diagnostic could plausibly be added to internal model-evaluation suites without retraining. For teams building VLA models, the comparison across Molmo2-ER and its action-trained derivatives offers a way to check whether action training preserves or distorts spatial structure.

Future Directions

• Closing the computation gap. Since explicit rotation over S-Space dramatically outperforms the model's own reasoning even when the correct premise is supplied, the open question is what training signal or architectural component would let the model perform these transformations itself.

• Decoupling reference frames. The observed coupling of "east" with right and "north" with above suggests targeted data or fine-tuning interventions; whether the coupling can be removed without damaging other spatial abilities is untested.

• Extending the analysis to action. MolmoAct2-Pretrain and MolmoAct2 are described as derived from Molmo2-ER through successive stages of action training; a systematic account of how S-Space changes across those stages is a natural follow-up.

• From diagnosis to intervention. The paper frames S-Space as a diagnostic; whether coordinate editing or subspace-level regularization can be used during training to improve spatial reliability, rather than only to probe it, remains open. Relatedly, the case studies suggest limits of language-inherited spatial structure, but how far that structure can be extended by scaling is not reported.

Target Audience

Interpretability researchers studying vision-language models; MLLM and VLA developers who need to know whether spatial failures originate in perception or reasoning; roboticists and embodied-AI engineers building systems on top of multimodal models; and benchmark designers interested in why spatial evaluations produce the errors they do. Readers looking for a new state-of-the-art spatial model will not find one here — the value is diagnostic and mechanistic — but those already familiar with activation probing and causal interventions will find the setup directly reusable.

Authors’ abstract

Multimodal language models (MLLMs) exhibit promising spatial reasoning capabilities, yet the internal mechanisms supporting them remain poorly understood. We uncover S-Space, a linear representation subspace in which object-token representations encode continuous coordinates along horizontal, vertical, and distance axes. Although we identify these axes by differentiating spatial-answer scores with respect to object-token activations, S-Space is intrinsic rather than induced by specific data or prompts, remaining readable under non-spatial questions. We show that S-Space functions as an internal spatial workspace of MLLMs. Perception encodes object locations into it, forming spatial representations that can be directly read out to support accurate spatial judgments. Internal reasoning transforms it for changing viewpoints. Verbal responses selectively access it, evidenced by causal interventions that alter spatial judgments while largely preserving non-spatial responses. Case studies further show that S-Space assigns locations to unseen entities and the observer, and connects physical directions with abstract linguistic meanings, suggesting both the capabilities and limitations of large-scale language training. Finally, we use S-Space as a diagnostic tool for spatial reasoning failures. Direction-word readouts reveal systematic coupling between cardinal and image-relative reference frames. More importantly, a perception-computation decomposition exposes a key bottleneck: MLLMs can form capable spatial representations, yet fall short in the spatial transformations required to reason reliably across viewpoint changes.