S-Agent 2.0: Spatial Reasoning with a General Coding Agent

AI summary

Overview

Research area: Spatial reasoning for vision-language models (VLMs), specifically agentic systems that use external 3D perception tools, built on top of general-purpose coding-agent harnesses.

Technical level: Advanced. The paper assumes familiarity with vision-language models, agent loops/tool calling, multi-view geometry, and benchmarks for embodied spatial reasoning.

Scope: The paper proposes S-Agent 2.0, a spatial reasoning agent formed by pairing an unmodified off-the-shelf coding agent (OpenHands, Codex, or Claude Code) with a command-line perception toolkit and a persistent scene memory, and evaluates it on five multi-view and video spatial-reasoning benchmarks.

What This Paper Is About

Agentic VLMs have improved spatial reasoning by calling 3D perception tools while they reason, but on many questions the tools return correct evidence and the model still answers wrong. The authors diagnose this as an unclosed "evidence loop": the model acquires measurements but fails to remember them or to interpret raw numbers into usable spatial statements. S-Agent 2.0's goal is to close that loop at the interface rather than by writing another specialized controller or training the backbone.

Key Contributions

1. A spatial agent built from a stock general coding agent plus a command-line perception toolkit. No custom state machine or hand-designed plan-act-reflect loop is written; perception is added as ordinary shell commands, and the authors show the same method runs under OpenHands, Codex, and Claude Code.

2. A toolkit designed around closing the evidence loop. Each call (i) acquires scene evidence, (ii) writes it into a persistent scene memory keyed by instance name, and (iii) returns a reading already interpreted in plain spatial language with a stated reference frame. The model's own perception (including imagined observers) is registered into the same scene via add_instance and can correct tool errors via drop.

3. Strong empirical results with an untrained 27B backbone. The toolkit adds +13.9 and +20.5 points for Qwen3.8-27B over the same agent without tools on MMSI-Bench and ReVSI, and +11.2 and +9.6 for Gemini-3.8-Flash. S-Agent 2.0 reaches 66.5 / 82.3 (Qwen3.8-27B) and 74.1 / 90.5 (Gemini-3.8-Flash), ahead of the specialized SpatialClaw at 62.6 / 74.5 and 67.0 / 79.7 on the same backbones.

4. An ablation that decomposes the interface. Removing scene memory, the interpreted readout, or the model-perception interface each costs accuracy, with the model-perception interface costing the most.

Main Findings

• Tool gains are large on both backbones, but shrink with capability. On the general coding agent, adding the toolkit moves Qwen3.8-27B from 52.6 to 66.5 on MMSI-Bench and 61.8 to 82.3 on ReVSI; Gemini-3.8-Flash moves from 62.9 to 74.1 and 80.9 to 90.5. Adding tools to the no-tool baseline costs nothing in setup complexity because the harness is unchanged.

• The advantage over the specialized baseline persists on the stronger model. SpatialClaw's ReVSI tool gain drops from +13.2 (Qwen3.8-27B) to +2.7 (Gemini-3.8-Flash), while S-Agent 2.0's stays at +9.6 on the stronger backbone. S-Agent 2.0 scores above SpatialClaw on every benchmark and backbone tested.

• It leads all five benchmarks among agentic baselines. On MMSI-Bench / MindCube / SPAR-Bench / MMSI-Video-Bench / ReVSI with Qwen3.8-27B: S-Agent 2.0 scores 66.5 / 88.3 / 72.4 / 56.4 / 82.3, versus SpatialClaw 62.6 / 83.2 / 70.2 / 53.1 / 74.5, S-Agent 48.3 / 75.5 / 63.0 / 37.9 / 59.5, SpaceTools 43.2 / 58.6 / 55.3 / 39.3 / 64.4, and pySpatial 36.3 / 66.3 / 42.1 / 26.9 / 39.3.

• It is competitive with proprietary frontier models. With the 27B open backbone and no training, it beats every proprietary model on MMSI-Bench except GPT-6 (71.0 vs 66.5). With Gemini-3.8-Flash it posts the highest MMSI-Bench (74.1) and ReVSI (90.5) averages in the tables, ahead of GPT-6 (71.0 and 84.2). GPT-6 retains the lead on MMSI-Bench region-to-region relations, camera motion, and multi-step reasoning.

• Gains concentrate on cross-object geometry. On ReVSI with Qwen3.8-27B, the largest margins over SpatialClaw are relative direction (95.2 vs 77.8, +17.4) and route planning (94.4 vs 72.2, +22.2). The one deficit is absolute distance (−2.8 on Qwen3.8-27B, −1.7 on Gemini-3.8-Flash).

• The harness is swappable. With the same toolkit and workspace, MMSI-Bench scores 66.5 (OpenHands), 67.0 (Codex), and 63.8 (Claude Code); SPAR-Bench 72.4 / 71.8 / 72.8; ReVSI 82.3 / 76.6 / 79.1.

• Every interface component matters, ranked as the diagnosis predicts. Removing the interpreted readout costs the least (−1.3 MMSI, −1.2 ReVSI); removing scene memory costs more (−3.5, −2.5); removing the model-perception interface costs the most (−8.8, −5.9, down to 57.7 and 76.4).

• Residual errors are grounding errors. The failures that remain are at the point where the question enters the loop: simulating an imagined viewpoint incorrectly (registering an observer with a borrowed heading), or measuring the right quantity in the wrong sense (reporting elevation above the floor when the question asks for the object's own height).

• Baseline failures are not acquisition failures. In the authors' analysis of SpatialClaw trajectories, tools usually return correct masks and positions; the model either misreads a correct reading (sign flips, wrong coordinate frame) or never carries the reading into the final answer, falling back on its own look at the images.

Methodology in Plain English

The authors first re-analyze where an existing specialized agent (SpatialClaw) breaks. They conclude that specialized systems invest almost entirely in acquiring evidence — richer perception and geometry libraries — while leaving the model to remember and interpret the numbers. That yields two failure modes: misreading a correct measurement, and dropping a correct measurement before answering.

They then invert the allocation. The agent itself is an off-the-shelf coding agent (OpenHands is the default), which already supplies a shell, a file system, code execution, long-horizon context management, and a tool-calling loop. Perception is added as one executable, python3 /app/perception.py <command> [flags], whose replies are JSON on stdout and whose rendered images (segmentation overlays, top-down maps) are written to disk for the agent to open like any file.

Three design commitments follow from the diagnosis:

1. Two kinds of perception on one geometric footing. Tool perception (recon fusing all frames into a metric, gravity-aligned point cloud; segment grounding text prompts to masks) supplies precision. Model perception (add_instance, drop) lets the VLM register objects it recognizes or observers it imagines, and delete instances the tools got wrong. Both write to the same scene memory, so the question's premise is measured with the same geometry as the scene.

2. Memory kept by the environment, not the context window. The scene memory holds the raw frames, a fused point cloud, and a registry of named instances, each with an xyz position, an extent, and a 6-DoF pose. Readouts are computed in closed form from stored geometry, so later steps need only remember an instance's name rather than transcribe numbers across a growing transcript.

3. Numbers interpreted on arrival. Every reply is distilled into typed numeric fields with units plus a one-sentence conclusion in plain spatial language that names the entities, declares the reference frame (gravity-aligned, camera, or image plane), and states the relation in the vocabulary the answer options use. Full measurements stay on disk for the agent to read if needed.

Evaluation covers five benchmarks (MMSI-Bench, MindCube, SPAR-Bench, ReVSI, MMSI-Video-Bench) with two backbones, Qwen3.8-27B (FP8, served locally) and Gemini-3.8-Flash. The perception backends are frozen off-the-shelf models: Depth Anything 3 for metric reconstruction, π³_mos for motion masks, GeoCalib for gravity, and SAM 3 for promptable segmentation. No backbone is trained.

Why This Matters

Impact on research. The paper argues that the bottleneck in agentic spatial reasoning has shifted from perception to grounding, and that interface design — not controller design or additional training — is a productive place to intervene. The ablation ranking (interpretation < memory < model-perception interface) and the observation that the specialized baseline's advantage collapses on a stronger backbone (ReVSI +13.2 to +2.7) support a general claim: gains that come from a hand-built harness may not survive model improvements, whereas gains that come from closing the evidence loop do (+9.6 retained).

Real-world applications:

• Robotics and embodied agents that must reason about object positions, relations, and motion before acting in physical space.
• AR/VR and 3D content tools where a user asks relational questions about a reconstructed scene or an imagined viewpoint.
• Video analysis and route planning, where evidence is spread across frames and no single image shows the relation asked about.
• Cross-video memory and spatial construction tasks, such as reconstructing a layout from long recordings.

Industry relevance. The method requires no model training and drops into existing production coding-agent harnesses (OpenHands, Codex, Claude Code), which lowers the barrier to adoption: perception capability is delivered as a command-line tool rather than a fine-tuned model or a bespoke controller. That makes it practical to swap perception backends, swap the backbone, or swap the harness without retraining.

Future Directions

• Grounding the question into a measurement. The authors identify this as the remaining gap and name two classes: incorrectly imagined viewpoints (observers registered with a borrowed heading) and quantities read in the wrong sense (elevation vs object height). Improving this is the stated next target.
• Closing the absolute-distance deficit. ReVSI absolute distance is the only question type where S-Agent 2.0 trails SpatialClaw on both backbones (−2.8 and −1.7), so it marks a specific unresolved weakness in metric readout.
• Extending the harness-swap evidence. Three harnesses are tested; whether the interface transfers to further agent loops, or to harnesses with different context-management behavior, is left open.
• Refining the model-perception interface, which is the most costly component to remove (−8.8 MMSI, −5.9 ReVSI). How much of the model's own spatial judgment can be routed through add_instance/drop without introducing the grounding errors identified in the qualitative analysis remains an open design question.

Target Audience

Researchers and engineers working on spatial or embodied reasoning with vision-language models, agentic tool-use system designers, and practitioners building perception pipelines on top of coding-agent harnesses. Readers evaluating whether specialized agent architectures are worth their complexity will find the tool-gain-shrinkage analysis most useful; readers interested in multimodal interface design will find the three-requirement decomposition and the ablation ranking most directly applicable.

Authors’ abstract

Spatial reasoning asks a model to work out where objects are in 3D space, how they relate to one another, and how they move. Agentic VLMs have recently brought large improvements on such tasks by letting the model actively gather evidence with 3D perception tools instead of answering in a single pass. Even so, on many questions the tools return the right evidence and the model still does not arrive at the correct answer: the gap between model and tools, where raw readings become usable conclusions, is left unfilled. We present S-Agent 2.0, a spatial reasoning agent that equips an off-the-shelf general coding agent with a command-line perception toolkit, and we design the interface around one principle: help the VLM close the evidence loop: acquire scene evidence, remember it in a persistent scene memory, and interpret each reading in plain spatial language. This design keeps the full flexibility of a coding agent, and every reading it acts on stays grounded in the tools' geometry. On MMSI-Bench and ReVSI, the toolkit improves a strong open-source model, Qwen3.8-27B, by +13.9 and +20.5 points over the same agent without tools, and a strong closed-source model, Gemini-3.8-Flash, by +11.2 and +9.6 points. With the open backbone S-Agent 2.0 reaches 66.5 on MMSI-Bench and 82.3 on ReVSI, ahead of the specialized system SpatialClaw on the same backbone (62.6 and 74.5), and with Gemini-3.8-Flash it reaches 74.1 and 90.5 against its 67.0 and 79.7. Both tool gains shrink as the backbone grows more capable, but SpatialClaw's shrinks far more on ReVSI, from +13.2 to +2.7, while ours stays at +9.6. S-Agent 2.0 also works across different general coding agent harnesses, such as Codex and Claude Code.