SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

AI summary

Overview

• Research area: Computer Vision / Vision-Language Models / Embodied AI (spatial reasoning, tool use, reinforcement learning).
• Technical level: Advanced.
• Scope (1 sentence): The paper introduces Double Interactive Reinforcement Learning (DIRL) and Toolshed, a two-phase training framework plus serving infrastructure that teaches a 3B vision-language model (SpaceTools) to coordinate many vision and robotic tools for spatial reasoning and real-robot manipulation.

What This Paper Is About

Vision Language Models can describe scenes well but are weak at metrically precise spatial reasoning, which robots need. One fix is to let a VLM call external computer vision tools (depth estimators, segmentation models, pose estimators, grasp predictors), but existing approaches either rely on handcrafted prompts, enforce fixed tool pipelines, or have only been shown with a single tool. This paper's goal is to let a VLM learn how to select, sequence and recover from errors across a large set of tools, without predefined pipelines.

Key Contributions

1. DIRL (Double Interactive Reinforcement Learning): a two-phase training paradigm — a teaching phase (supervised fine-tuning) followed by an exploration phase (interactive RL) — that enables training with a large set of tools, where naive multi-tool RL would otherwise fail in a combinatorial action space.
2. Toolshed: an interactive platform (stated to be open-sourced) that hosts compute-intensive computer vision and robotic tools as on-demand services during training and inference, with per-tool resource/environment isolation, decoupled scaling from the policy's inference loop, and asynchronous parallel workers per tool.
3. SpaceTools: a VLM trained for spatial reasoning via interactive multi-tool use, which reports state-of-the-art results across spatial reasoning benchmarks and controls a real robot through alternating perception and action tool calls.
4. Full interactivity during training: unlike prior work, the model calls tools interactively during training (Table 1 contrasts this against fixed pipelines and precomputed contexts), exposing it to real and stochastic tool outputs.

Main Findings

• Benchmark performance: SpaceTools achieves state-of-the-art results on RoboSpatial-Home, BLINK, RefSpatial, CVBench, and BOP-ASK, surpassing proprietary, open-source, and spatial VLM baselines. Specific margins reported in the text: +7.5% over Gemini-ER 1.5 on RoboSpatial, +24.4% over Claude Sonnet 4.5 on pose estimation, and +8.3% over GPT-5 on grasp prediction.
• Improvement over no-tool baselines: SpaceTools-3B achieves higher accuracy on all tasks than tool-free fine-tuning of the same base model on the same 8k VQA pairs — +12% on RoboSpatial over tool-free SFT and +16% on RoboSpatial over tool-free RL.
• Ablation — IRL-trained teacher: Removing it causes a sharp drop, especially on fine spatial grounding tasks (RefSpatial and RoboSpatial); mean score falls from 52.48 to 41.68.
• Ablation — universal teacher: Removing it also degrades performance, especially pose tasks requiring multi-tool composition; mean falls to 42.86 (pose drops to 8.92 from 34.37).
• Ablation — Stage-2 IRL: Removing the exploration phase lowers the mean to 50.99 from 52.48.
• Comparison to non-interactive training: Tool SFT (mean 39.19) and Tool NIRL (mean 38.06) underperform DIRL (52.48) by +13.4 and +14.4 mean improvement respectively, suggesting interactive RL is key to reasoning over complex tool sequences.
• Real robot manipulation: SpaceTools completes pick-and-place with an 86% success rate using a 7-DOF robot, and reports 86 (6/7) on Pick, 83 (5/6) on Relational Pick, 86 (12/14) on Pick & Place, and 10s Time-to-First-Movement, versus Claude Sonnet 4.5 + Toolshed (86, 50, 79, 30s), GPT-5 + Toolshed (71, 33, 65, 36s), and π0.5 (0 on all three, 1s TTFM).
• Learned tool-selection behavior, not prompt engineering: Qualitative analysis shows the model adapts strategies per task — pointing for spatial compatibility and relationship, depth estimation for relative-depth queries, and multi-tool composition for pose or grasp prediction — and exhibits corrective behaviors such as falling back to self-estimation when a tool fails or switching pointing tools.
• Zero-shot tool augmentation for frontier models: Adding Toolshed improves GPT-5 on RefSpatial (23.10 to 36.10) and pose (9.03 to 15.00), while high-level tasks such as RoboSpatial and BLINK show mixed trends because models overuse tools and misread nuanced outputs.
• Out-of-domain generalization: A model trained only on RoboSpatial reaches 72.3% on that benchmark and still achieves 34.3% on RefSpatial, where other fine-tuning approaches score zero.
• Structural format reward: An additional format score was tried but provided no measurable improvement and was excluded from final training.

Methodology in Plain English

The model, a Qwen2.5-VL-3B-Instruct base, is placed in a loop where it may think inside <think> tags, emit tool calls inside <tool_call> tags, and finish inside <answer> tags, for up to a maximum number of turns. Tools include segmentation, pointing, depth estimation, cuboid fitting, grasp generation, cropping, array indexing, perspective projection, plus robotic tools (image capture, depth capture, grasp, place).

Training happens in two rounds of interactive RL:

1. Teaching phase: The authors build an 8k-trajectory supervised dataset — 2k demonstrations from an "IRL-trained teacher" (a model trained with interactive RL on a single pointing tool, RoboRefer) and 6k from a "universal teacher" (Claude Sonnet 4.5 using the full toolset, keeping only correct trajectories). The base model is then fine-tuned with next-token prediction over assistant turns.
2. Exploration phase: RL resumes with all tools available, letting the model refine tool chaining. The strong initialization from phase 1 prevents exploration collapse in the large multi-tool action space.

The RL algorithm is GRPO: for each input, N rollouts are launched asynchronously under the current policy, and the policy is updated with a KL-regularized GRPO objective against a reference policy. Rewards are task-specific and normalized: binary correctness for multiple choice; Mean IoU for 2D bounding boxes; Normalized Negative Distance to Centroid for pointing (clipped with a binary accuracy term); convex-hull IoU over eight projected 3D corners for pose; and Normalized Negative Coordinate Error for grasp estimation with δ_max = 10, reported as MACE and grasp Success Rate (percentage of grasps with MACE > 40). SFT used LLaMA-Factory, RL integrated Toolshed into the VERL framework, and mock robot tools were used during training.

Why This Matters

• Research impact: It shows that tool coordination for spatial reasoning can be learned through interactive RL rather than baked into architecture or large-scale dataset engineering, and that a large multi-tool action space can be made tractable by progressive training.
• Real-world applications:
  • Robot pick-and-place and grasping in unstructured environments.
  • Relational manipulation (e.g., "pick the object next to the other one") driven by language.
  • Perception modules in embodied agents that need metric 3D understanding (depth, pose, occlusion, fit).
  • Practical infrastructure for serving heavy vision models on demand alongside training or inference workloads.
• Industry relevance: The paper targets the cost and latency bottlenecks of running many compute-heavy models (segmentation, depth, grasp prediction) in an agentic loop, and the released tool-serving platform is aimed at making such pipelines deployable at scale.

Future Directions

• Longer-horizon tasks: Extending from short- and medium-horizon tasks (spatial QA, grasp-and-place) to complex, multi-stage tasks and richer environments such as large-scale robotic simulation, interactive games, or physics-rich virtual worlds.
• Visual tool outputs: The work primarily explores tools returning structured text or variables (e.g., point cloud); reasoning directly over image-level tool outputs could unlock finer-grained behaviors.
• Robustness and recovery: Systematically improving how the model perceives, verifies, and recovers from tool errors or inaccuracies, including modular upgrades of individual tools without breaking overall coordination.
• Training and infrastructure scaling: Alternative RL formulations such as stepwise rewards for large tool action spaces, continual learning of new tools, and infrastructure improvements in scheduling, caching, batching, and asynchronous execution — including real (rather than mock) robot execution during interactive learning — plus lighter-weight tools and memory-optimized deployment.

Target Audience

Researchers and engineers working on vision-language models, embodied AI and robot learning, reinforcement learning for reasoning, and ML systems/infrastructure for serving many models inside an agentic loop. Readers who benefit most are those interested in tool-augmented reasoning, interactive RL training recipes, or transferring spatial understanding to real robotic control.

Authors’ abstract

Vision Language Models (VLMs) demonstrate strong qualitative visual understanding, but struggle with metrically precise spatial reasoning required for embodied applications. The agentic paradigm promises that VLMs can use a wide variety of tools that could augment these capabilities, such as depth estimators, segmentation models, and pose estimators. Yet it remains an open challenge how to realize this vision without solely relying on handcrafted prompting strategies or enforcing fixed, predefined tool pipelines that limit VLMs' ability to discover optimal tool-use patterns. Reinforcement Learning could overcome this gap, but has so far been limited to reasoning with a single visual tool due to the large search space in multi-tool reasoning. We introduce Double Interactive Reinforcement Learning (DIRL), a two-phase training framework where VLMs learn to coordinate multiple tools through interactive exploration and feedback. In the teaching phase, we combine demonstrations from a single tool specialist trained via interactive RL with traces from a frontier model using all tools. In the exploration phase, the model further refines multi-tool coordination through continued RL. Our model, SpaceTools, with tool-augmented spatial reasoning ability, achieves state-of-the-art performance on spatial understanding benchmarks (RoboSpatial-Home, BLINK, BOP-ASK) and demonstrates reliable real-world manipulation using a 7-DOF robot as a tool. DIRL provides substantial improvements over the vanilla SFT (+12% on RoboSpatial) and RL (+16% on RoboSpatial) baselines. Project page: https://spacetools.github.io/.