Spatial Harness: Grounding Spatial Reasoning in Executable World States with Physics Engines

AI summary

Overview

Research area: Multimodal machine learning — specifically spatial reasoning in vision-language models (VLMs), and tool-augmented / agentic reasoning grounded in physics-engine simulation.

Technical level: Intermediate. The central idea is intuitive, but the paper assumes familiarity with VLMs, camera pose estimation, 3D scene reconstruction, and benchmark evaluation for spatial question answering.

Scope (one sentence): The paper proposes Spatial Harness, a two-part system (a Perceiver and a Reasoner) that converts images plus a spatial question into an explicit, executable world state and answers by running viewpoint changes and measurements in the MuJoCo physics engine, evaluated across four spatial-reasoning benchmarks.

The paper is arXiv:2611.11344v1 [cs.LG], dated 08 Nov 2026, from the TMLR Group at Hong Kong Baptist University and ByteDance Seed.

What This Paper Is About

VLMs answering spatial questions perform two jobs in a single forward pass: figuring out where things are in a scene (perception) and working out what happens when a viewpoint changes (reasoning). Both steps stay hidden inside the model, so a wrong answer could come from a misread scene, a botched mental rotation, or both — and nothing in the output distinguishes these cases. The paper's goal is to pull the scene out of the model into an explicit, executable representation — a world state specifying object and camera positions and headings — so that every spatial consequence (distances, what a turn reveals) is computed by a physics engine rather than imagined by the VLM, letting perception and reasoning be inspected and improved separately.

Key Contributions

1. Executable world states. The authors define a scene representation that is concrete enough for geometric quantities — distance between entities, direction of an entity from a viewpoint, the view from any camera placed in the scene — to be computed directly from it, and they connect such states to a physics engine through an explicit interface.

2. Spatial Harness architecture. A Perceiver fuses question-derived constraints (stated object relations, stated camera motions) with specialized visual estimates (detection, depth, orientation, camera pose) into a shared coordinate frame, and a Reasoner answers by issuing Python calls into a MuJoCo scene for viewpoint control, geometric measurement, and rendering.

3. Controlled evaluation separating perception from reasoning. Using MindCube's annotated cognitive maps ("GT maps") as a fixed scene, the authors compare direct model-internal reasoning against execution on an identical layout, isolating gains that come purely from executing operations rather than from better scene estimation.

4. Cross-benchmark results and ablations. Evaluation on MindCube-Tiny, SAT, MMSI-Bench, and SpatialRGPT-Bench with open- and closed-weight VLMs, plus ablations on state construction (text channel, visual channel, individual perception modules), feedback type (numerical vs. rendered), and estimated-state quality.

Main Findings

• End-to-end gains on MindCube. With Qwen3.5-27B, Spatial Harness raises accuracy from 52.70% (Base) to 71.14%, which is 18.44 points above Base and 7.52 points above the strongest baseline, pySpatial (63.62%). Across all four VLMs tested end-to-end on MindCube it achieves the highest overall accuracy, exceeding pySpatial by 5.40 points on average.

• Other backbones on MindCube (end-to-end). Qwen2.5-VL-72B-Instruct: 41.46% (Base) to 55.33%; Kimi K2.5: 50.67% to 72.67%; GPT-5.4-mini: 44.25% to 59.62%. MindJourney stays within 4.42 points of Base.

• Direct reasoning with a supplied map does not reliably help. With the GT map, Base accuracy drops from 35.59% to 30.35% for Qwen2.5-VL-7B. Newer models use the map better (Base rises by 19.11 points with Qwen3.5-27B and 30.60 points with Kimi K2.5), but execution on the same map still dominates.

• Execution with an identical state is the source of gain. Holding the GT map fixed, Qwen3.5-27B scores 71.81% with Base versus 98.25% with Spatial Harness. With the same maps, improvements over Base (w map) appear on every backbone: +26.16 (Qwen2.5-VL-7B), +37.43 (32B), +42.76 (72B), +26.44 (Qwen3.5-27B), +12.83 (Kimi K2.5), +38.07 (GPT-5.4-mini), +22.22 (GLM-5V-Turbo).

• Results on SAT, MMSI-Bench, SpatialRGPT-Bench. Spatial Harness has the highest average accuracy with both models tested: 61.57 vs. 46.31 (Base) for Qwen2.5-VL-72B-Instruct, and 68.67 vs. 54.47 for Qwen3.5-27B — 15.26 and 14.20 points above the strongest baseline respectively. On SAT it reaches 71.60% and 83.20%; pySpatial falls 13.20 and 13.80 points below Base on SAT. On MMSI-Bench it improves Qwen3.5-27B from 33.40% to 39.50%, but with Qwen2.5-VL-72B it stays at Base level (29.00% vs. 29.60%) because that model often misreads coordinates in the world state and ignores engine feedback. On SpatialRGPT-Bench it reaches 84.11% and 83.31%, roughly 43 and 35 points above the strongest baseline.

• Both perceptual channels matter. Fixing the Reasoner on MindCube with Qwen3.5-27B: letting the VLM generate the map directly scores 41.33% (vs. 71.14% for the full Perceiver); removing the visual channel drops to 52.95%; removing the text channel drops to 68.38%. The visual channel contributes more.

• Camera-motion constraints drive the Rotation category. Dropping camera/view constraints from the text channel lowers accuracy to 62.29%, mainly on Rotation (81.50% to 48.00%). Replacing the relative-pose estimator costs the most in the visual channel (54.10%), mainly on Among and Around; replacing the orientation estimator costs the least (70.57%), since only 36 of the 1,050 questions depend on an object's facing.

• Numerical and rendered feedback are complementary. With the GT map fixed, multi-turn self-review raises accuracy only from 71.81% to 77.81%; numerical feedback alone reaches 95.62%, rendered feedback alone 93.71%, and both together 98.25%.

• Estimated states are structurally sound but incomplete. On all 1,050 MindCube questions with Qwen3.5-27B, 99.52% of estimated maps are valid and they recover 72% of annotated entities after name alignment; reported map-quality scores are 0.9952 (validity), 0.7237 (coverage), 0.7310 (directional similarity), and 0.8980 (facing similarity). The gap to the 98.25% GT-map result suggests better perception would raise accuracy further.

Methodology in Plain English

The system splits spatial question answering into two stages built on the same underlying VLM.

Stage 1 — the Perceiver builds a scene. A text channel parses the question into four things: which objects and viewers are involved, relations stated between objects, statements about camera movement between images, and the type of answer required (direction, distance, facing, camera motion). A visual channel runs specialized models on each image: detection gives direction, depth gives distance, orientation estimation gives facing, and — when there are multiple images — camera pose estimation gives the rotation and translation between viewpoints. The VLM also writes a qualitative description of each image to cover anything the specialized models miss.

Compilation into a shared frame. The first image's camera is placed at the origin as the reference. Other cameras' poses are filled in from a fixed priority order of sources: question-stated viewpoints, camera pose estimation, entities shared with an earlier image, and finally the VLM's description. Each observation is then projected into that shared frame using a rotation by the camera heading. If an object appears in several images, it gets several candidate positions; the highest-confidence one wins and headings are decided by vote. Because these candidates can violate stated relations, a deterministic local search then nudges entities on a grid to best satisfy the relations while staying close to the measurements. The resulting state is stored as JSON containing the layout, the per-view observations, and the stated constraints.

Stage 2 — the Reasoner executes. Each object becomes a labeled cylinder "proxy" at its estimated position and heading in MuJoCo (faithful meshes from images are judged unreliable), and each input image becomes a camera at its estimated pose. The Reasoner writes Python against a tool interface with three capabilities: move/set the camera, measure (current pose, distances to entities, motion between two views), and render the current view with labeled proxies. The camera pose persists between calls, so multi-step movements are executed one call at a time, and the returned measurements and images are appended to the context to guide the next operation. The loop runs for at most 5 turns, and an answer is accepted only after at least one engine call; a routing step skips the engine entirely if the question needs information the state does not describe.

Validation strategy. Running the same VLM as both Perceiver and Reasoner lets the authors run two clean comparisons: end-to-end (everything estimated by the system) and GT-map (scene given to both methods), the latter isolating reasoning quality from perception quality.

Why This Matters

The work reframes a common failure mode in multimodal AI — fluent spatial explanations that are silently wrong — as an interface problem: expose the assumed scene, let a deterministic engine compute the consequences, and both become checkable. This is a template for making reasoning about the physical world auditable rather than merely plausible, and it shows that the gains are not simply from more inference turns (self-review only added 6 points where execution added more than 20).

Real-world applications:

• Embodied agents and robotics: a robot that maintains an explicit, engine-queryable world state can check "what will be in view after I turn" before acting, rather than trusting a language model's imagination.
• Assistive and AR/VR navigation: answering "what is behind me after this turn" or "which object is closer after moving forward" with distances computed from geometry rather than estimated from appearance.
• Visual question answering for accessibility: describing a scene from an arbitrary requested viewpoint with measurements that trace back to an inspectable layout.
• Quality control and safety review in deployed multimodal systems: since the world state is stored as JSON and the queries as code, failures can be attributed to either a bad scene estimate or a bad spatial operation.

Industry relevance: the method is model-agnostic (tested across Qwen, Kimi, GPT, and GLM backbones), uses an open-source physics engine (MuJoCo), and gains most where geometry matters — SpatialRGPT-Bench numerical estimation jumps roughly 43 and 35 points — which is exactly the regime relevant to robotics, simulation, and spatial computing products. Code is released at https://github.com/tmlr-group/Spatial-Harness.

Future Directions

• Adding a time dimension. The authors state the main limitation plainly: the world state describes a static scene at a single moment and cannot represent change over time. Recording entity and camera poses per frame would let the engine instantiate the scene at any moment and measure inter-frame change, supporting video and motion-ordering questions — explicitly planned as future work alongside the video benchmarks such require.
• Closing the perception gap. Estimated states recover 72% of annotated entities (coverage 0.7237) while GT-map execution reaches 98.25%, so improving the Perceiver — or using engine renders to check the state against input views and flag disagreements for correction — is a clear route to further gains. The paper notes this self-checking use of the engine as an unexplored possibility.
• Robustness of the Reasoner's use of feedback. In MMSI-Bench, Qwen2.5-VL-72B did not improve because it misread coordinates in the world state and failed to act on engine feedback; understanding when a backbone can exploit grounded feedback is an open question.
• Generalizing beyond static spatial QA. The authors hope physics engines become a general substrate for grounding multimodal reasoning about the physical world, which raises open questions about extending the interface to dynamics, physical parameter estimation, and interaction, and about how far the "proxy cylinder" abstraction can be pushed before scene fidelity becomes the bottleneck.

Target Audience

Researchers and engineers working on multimodal reasoning, spatial intelligence, and embodied AI who want to understand how to ground VLM reasoning in external, verifiable computation. It is also useful for practitioners building agentic tool-use pipelines who need a concrete design for what state the tools should share, and for benchmark designers interested in the technique of holding the scene fixed to separate perception errors from reasoning errors. Readers need comfort with VLM evaluation and basic 3D geometry; the writing is dense with formulas but the argument is carried by the tables and figures.

Authors’ abstract

Spatial reasoning requires understanding how objects and observers are arranged in a scene and how their relations change with viewpoint or movement. Vision-language models (VLMs) typically perform both steps through model-internal reasoning, forming an implicit scene interpretation from partial views and imagining the consequences of spatial transformations. A misread scene can lead to a wrong answer, while even a correct interpretation does not ensure that imagined…