SpatialBrain: Spatial Intelligence Growing from Experience

AI summary

Overview

• Research area: Artificial intelligence — spatial reasoning with vision-language models, agentic tool use, and test-time experience accumulation.
• Technical level: Advanced. The paper assumes familiarity with vision-language models, multi-view 3D reconstruction, agent tool-calling, and memory/skill abstractions, though the core idea is explained conceptually.
• Scope: SpatialBrain is a framework that keeps a vision-language backbone frozen while letting an agent accumulate and reuse experience about when and how to apply scene memory and spatial skills across blocks of tasks.

What This Paper Is About

Spatial questions vary widely: some ask for an attribute visible in a single image, others require a reference frame, a measurement across views, or tracking a route. A good reasoner must decide when direct observation is enough and when to reconstruct, measure, or change reference frames. The paper's goal is to build that decision-making capacity from experience, without retraining the underlying model and without carrying over answers or scene coordinates from earlier tasks.

Key Contributions

1. 3DFusionMemory — fuses object, pose, camera, and route observations from the current scene into a queryable set of spatial relations with supporting views, confidence, and ambiguity, so reasoning and tools refer to the same grounded objects.
2. Spatial Skill Library — represents recurring spatial computations as reusable procedures, each with an explicit applicability condition, typed inputs, a parameterized tool procedure, and returned evidence, separating the operation from the scene-specific inputs.
3. Spatial Cognition — stores condition-specific experience about when and how to use memory and skills, selecting a reasoning route, a memory-access mode (off, answer-checking only, or planner-visible), permitted skills, and verification behavior from an input-derived task signature.
4. A blockwise self-evolution loop — completed trajectories are abstracted into label-free process records and consolidated between task blocks, so later decisions draw on earlier execution without transferring scene coordinates or reference answers and without gradient updates.

Main Findings

• Improvement over direct answers: Using GPT-5.6-sol, SpatialBrain improves on direct answers on four of five benchmarks and ties on the fifth, raising the unweighted mean from 67.5 to 72.6.
• Per-benchmark scores: MMSI 57.1 (direct 53.8), MindCube 87.6 (67.2), OmniSpatial 69.1 (68.3), SpaceDG 57.9 (56.9), SAT 91.3 (91.3).
• Largest gain on MindCube: a 20.4 percentage-point increase; the smaller changes on OmniSpatial and SpaceDG indicate the benefit depends on task setting.
• Mixed category-level changes: regressions are reported on OmniSpatial's egocentric questions and on SpaceDG's distance and counting questions.
• Error reduction on MindCube: errors fall in all three categories, from 344 to 130 overall, a 62.2% reduction.
• MMSI error counts: Direct makes 462 errors and SpatialBrain 429, with 199 corrections and 166 reversals; camera–object errors drop from 53 to 24 and appearance errors from 39 to 25, while multi-step errors rise from 83 to 94.
• Component removal (MindCube, 1,050 items): 87.6 full versus 76.7 without the Skill Library, 75.2 without 3DFusionMemory, 78.8 without Spatial Cognition, and 67.2 for direct answers; gaps are larger on Among and Around than on Rotation.
• Memory coverage varies by benchmark: saved 3D objects occur in 50.6% of MMSI tasks and 37.5% of SpaceDG tasks, versus 21.0%, 29.2%, and 18.0% for MindCube, OmniSpatial, and SAT.
• Experience breadth: across five independent GPT-5.6-sol streams, MMSI reaches 165 task signatures and 24 seed skills, while MindCube reaches 12 and 10; on MMSI, task coverage keeps growing after skill coverage plateaus.
• Experience delivery: in the executable configuration, MMSI forms 18 experience records (six become available) and MindCube five (three available); earlier nonempty experience reaches 231 of 1,000 MMSI tasks and 212 of 1,050 MindCube tasks.
• Earlier retrieval experiment: a GPT-5.5 study with 500 MMSI source tasks produced 81 procedural records across 36 task–path families and 16 capability skeletons; among 500 target tasks, 332 were eligible, 122 retrieved a record (36.7%) and 59 executed a partially matching sequence (17.8%). This experiment did not improve aggregate accuracy or cost.
• Cost is not uniformly lower: among 485 both-correct fixed-policy/SpatialBrain pairs with 480 complete token records, 221 use fewer SpatialBrain tokens and 71 also use fewer non-final top-level calls and less observed time.

Methodology in Plain English

The system acts as an agent on top of a fixed vision-language model. For each task it receives images, a question, and optional answer choices. A controller uses a "task signature" derived only from the inputs — reference frame, relation scope, output form, temporal structure, metric demand, and view count — to decide whether to answer directly or use tools, whether to consult scene memory and how, which skills to permit, and how to verify.

Scene evidence is built fresh for each scene. Perception and geometry tools project detections into a shared 3D frame using depth and camera poses, matching the same object across views and fusing centers through confidence-weighted updates; queries return distances and relations with confidence and supporting views. A library of parameterized skills performs recurring computations such as relative-direction or rotation estimation on the current task's objects.

Tasks are processed in blocks. All tasks in a block read the same committed snapshot of experience and keep separate scene memories, which allows parallel execution. After a block finishes, each completed trajectory is abstracted into a record containing the task signature, resource decisions, tool sequence, process checks, failures, and available costs — but no answers, literal coordinates, or raw task content. These records update the strategic store and the skill library, and the next block can make different resource choices. Resource changes are conservative: route and memory alternatives need sufficient historical support, acceptable process risk, and lower estimated token cost, and at most one learned route or memory factor changes per request. Process feedback is explicitly not treated as a correctness reward.

Why This Matters

• Research impact: the paper reframes spatial reasoning as resource selection rather than answer prediction, and separates scene evidence ("what the current scene supports") from experience ("when and how to use a resource") — a distinction that makes self-evolution inspectable and keeps model weights fixed.
• Real-world applications:
  • Indoor navigation assistants that must decide when a quick look suffices and when to reconstruct a room from multiple photos.
  • Robotics or embodied agents that need reusable procedures for relative directions and route tracking across changing environments.
  • Inspection or documentation workflows where multiple views of a site must be fused into consistent spatial relations.
  • Question answering over photo sets where measurement and reference frames matter.
• Industry relevance: because experience is external and the backbone is unchanged, the approach can be layered onto existing model deployments and updated without retraining; the paper's cost accounting (token use, call counts, observed time) speaks directly to deployment economics.

Future Directions

• Establishing whether self-evolution produces general improvement rather than accumulation and use: the authors state their curves and trajectories do not show this, and call for an initial-frozen control with the same resources and starting experience.
• Separating component effects more cleanly, since removing 3DFusionMemory also removes memory-based answer checks and earlier component runs were not verified as complete removals.
• Improving calibration of perception and geometry errors, which the authors describe as remaining imperfectly calibrated.
• Matched evaluations that measure both answer quality and the full cost of acquiring and using experience, since published baselines differ in protocols, backbones, and serving precision.

Target Audience

Researchers and practitioners working on vision-language agents, spatial reasoning, and tool-augmented reasoning systems, particularly those interested in test-time adaptation and external memory or skill libraries that avoid model retraining. It is also relevant to engineers evaluating the cost trade-offs of agentic tool use, though readers without background in multi-view geometry and agent architectures will find the technical sections dense.

Authors’ abstract

Spatial reasoning requires more than access to geometric tools: a reasoner must decide when direct observation is sufficient and when to reconstruct, measure, or change reference frames. We present SpatialBrain, a framework that retains experience about these decisions while keeping its vision-language backbone fixed. The key distinction is between evidence about a scene and experience about how to reason with that evidence. 3DFusionMemory fuses current-scene observations into queryable spatial relations. A Spatial Skill Library provides reusable tool procedures, and Spatial Cognition associates task conditions with experience about when and how to use memory and skills. Completed trajectories are abstracted into process records and consolidated between task blocks, allowing later decisions to draw on earlier execution without transferring scene coordinates or reference answers. Across five spatial-reasoning benchmarks, SpatialBrain improves on direct answers from the same backbone on four and ties on the fifth. Component comparisons, experience-coverage curves, and traced resource decisions characterize how the framework accumulates and uses these resources.