SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

AI summary

Overview

Research area: Computer vision and embodied AI — specifically spatial intelligence and 3D scene understanding evaluated through vision-language model (VLM) agents.

Technical level: Intermediate. The task setup is intuitive (look around a room, figure out where objects are), but full appreciation of the metrics (3D IoU, one-to-one matching within equivalence groups, omission penalties) requires some familiarity with 3D geometry and agent benchmarking.

Scope in one sentence: This paper introduces SpatialScout, an 80-scene benchmark that tests whether VLM agents can actively move a camera, gather views, and produce a complete, metrically accurate map of object centers in a shared 3D coordinate frame.

What This Paper Is About

Most benchmarks for spatial intelligence in VLMs ask models multiple-choice or numeric questions about images and video, which the authors argue can be answered by guessing, elimination, or language priors rather than genuine 3D understanding. SpatialScout instead gives an agent an object inventory, camera calibration, and camera poses, and requires it to actively explore an indoor scene and output explicit 3D coordinates for every target object. The goal is to measure whether models can build a precise and complete metric map through a sustained closed loop of observation, reasoning, and action.

Key Contributions

1. A task formulation for explicit 3D spatial construction. Instead of question answering, agents must output a set of 3D object-center coordinates in a shared coordinate frame, enabling direct quantitative scoring of the resulting map (localization accuracy plus target coverage).

2. A closed-loop active exploration benchmark. SpatialScout contains 80 indoor scenes with 3,140 target objects (abstract figure; see the note on internal inconsistencies under Main Findings), 30–40 targets per scene, spanning individual rooms and apartments from iTHOR, RoboTHOR, ArchitecTHOR, ReplicaCAD, and ProcTHOR sources, rendered with SAPIEN through ManiSkill3.

3. A shared ten-tool agent interface with separate budgets. Agents control camera position and rotation, capture RGB frames, query intrinsics and poses, use a multi-view triangulation helper, submit/revise positions, and finish. Each episode allows 100 model calls and 100 acquired images, and no tool reveals ground-truth positions or scores.

4. A multi-faceted evaluation protocol. Scoring combines mean center error, coverage, and scale-relative 3D overlap (IoU) at strict thresholds of 0, 0.5, and 0.9, computed both over all targets (omissions count as zero) and over submitted targets only, plus replayable map traces that track how the map evolves over interaction steps.

Main Findings

• Best model still far from a reliable map: Claude Opus 5.5 achieves an all-target mean IoU of 41.71%, with 77.36% of targets above 0% IoU, 47.23% above 0.5, and only 3.25% above 0.9. Among submitted objects only, its mean IoU is 46.69% and 3.64% of targets exceed 0.9.

• Second-best is close on coarse placement: GPT-6 Astra (xhigh) reaches 33.16% all-target mean IoU, 67.79% above 0%, 35.15% above 0.5, and 1.62% above 0.9; among submitted targets, 41.16% mean IoU and 2.09% above 0.9.

• Large spread across the seven models evaluated (GPT-6 Astra, GPT-6 Sol, Claude Opus 5.5, Claude Sonnet 5.5, Kimi-K3, Qwen3.8-Max, DeepSeek-V4.1-Flash). Claude Sonnet 5.5 reaches 32.26% all-target mean IoU; GPT-6 Sol 20.83%; Qwen3.8 Max 6.80%; Kimi-K3 4.22%; DeepSeek V4.1 Flash 3.48%. Only the top four models have any targets above 0.9 IoU.

• Reasoning effort helps modestly: GPT-6 Astra at "medium" effort scores 31.18% all-target mean IoU versus 33.16% at "xhigh".

• Precision collapses at strict thresholds: high-IoU matches are rare even among submitted objects, showing that models can recover coarse object layouts but not precise metric centers relative to object extent.

• Omitting objects is the dominant failure mode: in a 40-trajectory case study (success defined as at least 90% coverage and mean center error at most 0.2 m on matched objects), only six trajectories succeeded; incomplete coverage appeared in 26 of the 34 failures.

• Geometric consistency tools are not enough: 21 of the failures involved at least one substantially inaccurate submitted position despite a low triangulation residual, indicating bad visual correspondences that geometry checks cannot catch.

• Tool-use style separates strong from weak models: Claude Opus 5.5 makes the most tool calls; GPT-6 Astra and Claude Opus 5.5 keep a balanced mix of observing, moving, and mapping, while DeepSeek-V4.1-Flash and Qwen3.8-Max move the camera a lot but observe and map relatively little.

• Triangulation support correlates with accuracy: across 80 shared tasks, 99.33% of GPT-6 Astra's submitted objects had prior successful triangulation versus 63.48% for DeepSeek-V4.1-Flash; 2.69% versus 38.65% of triangulations carried geometric warnings; median center error was 7.08 cm versus 68.17 cm, despite DeepSeek averaging more captures (39.63 vs 32.41).

• Accuracy comes at a price: Claude Opus 5.5 has the highest IoU but the highest cost at 28.85 USD per task; only DeepSeek-V4.1-Flash is cheaper than GPT-6 Sol, and Qwen3.8-Max and Kimi-K3 are both more expensive and lower-scoring on all IoU metrics.

• Reported inconsistencies in the paper text: the abstract states 3,140 target objects, while Section 2.2 states "totaling 3,410 instances" alongside a breakdown of 12 scenes with 35 targets and 68 with 40 (which multiplies to 3,140 with 80 scenes). The listed source-scene counts (32 iTHOR, 12 RoboTHOR, 10 ArchitecTHOR, 6 ReplicaCAD, 30 ProcTHOR) sum to 90 rather than 80. Section 4.1 states GPT-6 Astra's submitted-object mean as 41.76%, while Table 3 reports 41.16%. These figures are reproduced as written in the paper; the paper does not reconcile them.

Methodology in Plain English

Each benchmark episode starts with one RGB image, a list of target objects the agent must find (with identifiers and categories), the camera's intrinsics and pose, and a scene-level search radius. The agent then uses ten tools to control a simulated camera: it can set or translate its position, rotate it, capture new frames at 512×512 (delivered as JPEGs with the longest side at most 384 pixels), check where the camera is, and read calibration.

To localize an object, the agent marks the same visual feature in at least two calibrated frames and calls a triangulation tool, which solves for the 3D point that best fits the viewing rays. The tool rejects degenerate queries (for example, cameras too close together or rays nearly parallel) and warns when parallax is weak, residuals are large, or the point falls outside the public search bound. Crucially, the tool only reports geometric consistency — the agent still has to decide whether the marked pixels really correspond and how a surface point relates to the object's center, which is defined as the midpoint of the object's world-axis-aligned bounding box.

The agent submits or revises center estimates at any time, and the final accepted map is scored when it calls finish_mapping, stops using tools, or exhausts the 100 model calls. Scoring matches predictions to targets one-to-one within "equivalence groups" (repeated instances of the same category), then computes center error, coverage, and 3D IoU by translating each target's reference box to the predicted center. Every submission is recorded as a trace, allowing offline replay of how error and coverage changed over the episode. Failure analysis was done on 40 trajectories from four models, and trajectory inspection on 80 shared tasks used Codex powered by GPT-6 Astra.

Why This Matters

Spatial question answering can be passed by guessing or by exploiting language priors, so it gives a weak signal about whether a model actually understands 3D space. By demanding an explicit, scored coordinate map, SpatialScout makes spatial understanding falsifiable and separates "roughly knows where things are" from "can build a usable metric map." This matters for research because it reframes spatial evaluation as an end-to-end agentic task with planning, memory, cross-view association, and revision.

Real-world applications:

• Home robotics: a robot that must locate all instances of requested objects (cups, chairs, appliances) in an unfamiliar apartment before acting.
• Augmented and virtual reality: automatically placing virtual content or annotations at correct metric positions in a scanned room.
• Inspection and digital twins: autonomous capture of object layouts for facility documentation, inventory, or simulation asset creation.
• Assistive agents: wearable or mobile assistants that answer "where is my medication" by exploring rather than relying on a single view.

Industry relevance: the paper's cost analysis (up to 28.85 USD per task) and tool-use analysis give practitioners concrete guidance that longer tool use and better triangulation discipline drive accuracy, while naive extra camera motion mostly adds cost. The shared tool interface also gives model developers a reproducible way to compare agentic spatial competence.

Future Directions

• Better correspondence and center inference: since 21 of 34 failures involved confident but wrong triangulations, future work must improve cross-view feature association and the reasoning that maps a surface point to an object's bounding-box center, beyond what geometric residual checks can catch.
• Viewpoint selection and stopping policies: 13 failures terminated with insufficient coverage despite at least ten response rounds remaining, suggesting explicit coverage-aware stopping and next-best-view planning as open problems.
• Reducing cost without losing accuracy: only one evaluated model scored below GPT-6 Sol on cost, so the accuracy–cost Pareto frontier (Fig. 7) leaves substantial room for more efficient exploration.
• Memory and map revision over long horizons: the benchmark records map traces and supports replay, inviting work on persistent spatial memory and principled revision of earlier estimates as new evidence arrives.

Target Audience

Researchers and engineers working on vision-language models, embodied AI, 3D perception, and agentic benchmarks; robot perception and AR/VR teams that need agents to build metric maps of object layouts; and benchmark designers interested in replacing multiple-choice spatial QA with explicit coordinate-level evaluation. Readers should be comfortable with basic camera geometry and IoU-style metrics to get the most from the results tables.

Authors’ abstract

Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.