Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?

AI summary

Overview

Research area: Spatial embodied intelligence and multimodal foundation models — specifically whether large models can actively explore partially observable environments and build internal spatial beliefs.

Technical level: Advanced. The paper assumes familiarity with partially observable decision processes, benchmark design, multimodal (vision-language) agents, and spatial cognition concepts such as egocentric versus allocentric reasoning and cognitive maps.

Scope (one sentence): The paper defines "Theory of Space," builds a text-based and vision-based benchmark for measuring it, and uses direct belief probing to diagnose where state-of-the-art foundation models fail.

What This Paper Is About

Embodied agents usually see only part of their environment, so they must act — move, turn, look — to gather the information they need. Existing benchmarks mostly test models either as passive viewers of complete observations or as task-driven goal solvers, leaving unanswered whether a model can decide what to observe next to build a coherent map of a space.

The paper proposes Theory of Space as a counterpart to Theory of Mind: the ability to construct, revise, and exploit an internal spatial belief from sequential, partial observations. Rather than scoring task completion alone, the work asks a model to externalize its evolving cognitive map so the quality of its underlying spatial model can be measured directly.

Key Contributions

1. A formal definition and framework for Theory of Space. The paper formalizes an agent interacting with a spatial structure S to produce a history h_t = (o_{0:t}, a_{0:t}), and defines belief manipulation through three operations: Construct (approximating the posterior B_t(S) ≈ P(S | h_t)), Revise (updating after an environmental shift S → S'), and Exploit (using a policy π(a_t | B_t) to perform a downstream task, measured by a performance metric J).

2. A benchmark with parallel text and vision worlds. Procedurally generated multi-room indoor layouts on an N × M grid, built with ThreeDWorld and Objaverse assets. Each scene has n objects with 2D coordinates and cardinal orientations (N, S, E, W). The main experiments use three connected 6 × 6 rooms with 4 objects each (total 12 objects), 384 × 384 images, 100 scenes, and three questions per task per scene, yielding 3 × 9 × 100 = 2700 questions per setting.

3. A nine-task exploitation suite plus five assessment dimensions. Tasks split into Route belief (egocentric, path-based: pairwise direction, perspective taking, perspective decision, action-to-view, view-to-action) and Survey belief (allocentric, map-like: allocentric mapping, mental rotation, location-to-view, view-to-location). Assessment dimensions are D1 belief construction efficiency, D2 cognitive map quality, D3 uncertainty map quality, D4 belief revision, and D5 belief exploitation success.

4. Spatial belief probing and the false belief paradigm. The agent is prompted at each step to output a JSON-structured cognitive map, and separately to select which of a set of candidate points lie in unobserved regions. A false belief setting covertly relocates or reorients objects after exploration to test whether the agent overwrites obsolete priors.

Main Findings

• The Active-Passive Gap. Performance drops when models must gather information themselves rather than reason over a supplied history. In the abstract's normalized form, GPT-5.2 falls from 0.57 to 0.46; the introduction reports the same effect as GPT-5.2 57.1 → 46.0 and Gemini-3 Pro 60.5 → 57.3.

• Inefficiency of autonomous exploration. Rule-based proxy agents reach target coverage in approximately 9 steps, while foundation models explore redundantly and require ≥ 14 steps without improving belief accuracy. In the text world, following a concise Scout trajectory, GPT-5.2 and Gemini-3 Pro reach 83.9 and 86.7 respectively, surpassing their own active exploration scores of 72.0 and 81.5.

• A persistent modality gap. Text performance is substantially higher than vision performance for every model. Under passive evaluation, GPT-5.2 averages 90.4 in text versus 57.1 in vision, and Gemini-3 Pro averages 86.5 in text versus 60.5 in vision. Open-source models trail substantially (for example GLM-4.6V at 23.4 text and 16.7 vision).

• Active exploration is the primary bottleneck. Active agents score lower than they do when reasoning over program-generated histories, fail to achieve complete information coverage, and the active-versus-passive gap widens as environment scale grows. Gemini-3 Pro degrades least among the models tested.

• Incomplete coverage and premature termination. GPT-5.2 acquires a substantial amount of information early but its rate of gain slows in later turns, and it often terminates prematurely, leaving uncertainty. None of the models achieves full coverage relative to the proxy agent.

• Perception is an initial bottleneck, and orientation is especially hard. In vision settings, recognizing an object's facing direction is particularly challenging; agents frequently fail to infer orientation and reach near-chance (or worse) facing correctness, consistent with roughly 36% accuracy on perspective-taking tasks.

• Belief instability causes decay. Perception scores show models can capture local spatial detail reasonably well, but this fidelity does not translate into final map correctness: agents often overwrite correctly perceived facts with incorrect predictions in later turns. Self-tracking is not a primary bottleneck — models can often maintain an accurate belief about their own pose.

• Belief Inertia. Using the false belief paradigm, agents fail to overwrite obsolete spatial priors with new sensory evidence. The issue appears in text agents but is notably severe in vision-based models: despite directly observing the new configuration, models persist in their initial, now incorrect coordinates.

• Exploration behavior differs by model. GPT-5.2 explores unsystematically, prioritizing any newly discovered door and leaving rooms partially unexplored, compounded by object omission and path redundancy. Gemini-3 Pro adopts a more methodical "rotate-and-scan" strategy mirroring the Scout proxy.

• Human baselines. Three human subjects were benchmarked across five text and five vision scenes. Humans consistently outperformed foundation models in both domains, particularly in vision (96.4 average in the vision active setting versus 46.0 for the best model), and reached near-perfect accuracy (99.0) when allowed tools such as protractors and compasses.

• Leaderboard specifics (active exploration). Gemini-3 Pro leads every active task and all reasoning metrics, with a vision average of 57.3 at 13.6 average steps; GPT-5.2 achieves the lowest exploration cost in the text world (11.4 steps, 72.0 average). In the multi-room table, active accuracy falls as rooms increase — for GPT-5.2 in vision from 51.5 (2-room, 10.8 steps) to 40.3 (4-room, 23.2 steps).

Methodology in Plain English

The researchers built a controllable "world" generator instead of using fixed datasets. Rooms are laid out procedurally on a grid, objects are placed with coordinates and orientations, and the agent starts at a random position knowing only how many rooms exist and what objects are present.

The agent acts through a small, high-level action set: Goto (move to a visible object), Rotate (turn in place by 90°, 180°, or 270°), Observe (perceive objects in a 90° field of view), and Query (get a visible object's absolute 2D coordinates). Observe costs 1 and Query costs 2, though Query is rarely invoked by any model, so the authors measure efficiency by step count.

Every observation is returned in two parallel forms. The Text World gives symbolic descriptions with discrete bins (for example, "chair is front-left and near"), isolating pure spatial reasoning. The Visual World gives egocentric RGB images rendered in ThreeDWorld with Objaverse assets, so perception must recover the spatial relations. Directions are discretized into eight 45° allocentric bins (N, NE, E, SE, S, SW, W, NW) and five egocentric labels within a 90° FOV; distance uses six bins — same =0, near (0,2], mid (2,4], slightly far (4,8], far (8,16], very far (16,32].

Evaluation runs in two phases. In the Exploration Phase the agent acts on its own for up to 20 steps and decides when to stop. In the Reasoning Phase it answers open-ended questions rather than multiple-choice ones, to reduce knowledge leakage.

To separate exploring from reasoning, the authors also run a passive condition in which the model reasons over logs produced by scripted proxy agents: Scout for visual environments (rotates at each location to guarantee all objects are seen) and Strategist for text environments (belief-driven edge-coverage with AC-3 constraint propagation).

Efficiency is quantified by a normalized information gain, E = 1 − (Σ log₂ max(1, C_i)) / (N log₂ M), where M is the number of possible positions per object under a uniform prior and C_i is the number of positions still consistent with observations. The score runs from 0 (no information gained) to 1 (all objects perfectly localized). Vision environments use node coverage instead, since they lack this symbolic ambiguity.

Belief is measured by probing: the agent externalizes an allocentric cognitive map as JSON, scored on positional accuracy (K/N)·e^(−RMSE/L), directional accuracy, and facing accuracy, combined into an equally weighted (1/3 each) Correctness score at the final turn. Per-turn diagnostics cover Perception, Self-tracking, Local↔Global consistency, and Stability. Uncertainty modeling is scored with F1 over candidate points in observed and unobserved areas, and belief revision is scored with F1 on detecting object identity and transformation type.

Six models were evaluated: GPT-5.2, Gemini-3 Pro, Claude-4.5 Sonnet, GLM-4.6V, Qwen3-VL (235B-A22B-Thinking), and InternVL-3.5 (241B-A28B). Closed-source reasoning models used temperature 1 and a maximum of 32768 tokens; other models used temperature 0. InternVL-3.5 supports at most 10 images and was therefore omitted from the vision-based world setting.

Why This Matters

The paper argues that behavioral success alone — "did the agent find the chair" — cannot reveal the quality of an agent's internal spatial model, and that passive and task-driven benchmarks systematically miss the exploration capability that embodied intelligence depends on. It reframes exploration as a first-class decision-making problem and provides a diagnostic vocabulary (perception, self-tracking, stability, local↔global consistency, uncertainty modeling, belief inertia) for attributing failures to specific representational breakdowns rather than to a single accuracy number.

Real-world applications:

• Home and service robots that must build a map of an unseen house from partial views and update it when furniture moves.
• Warehouse and logistics robots that must decide where to look next efficiently rather than sweeping an entire aisle.
• Search-and-rescue or inspection drones operating in cluttered, partially observable indoor environments where coverage steps are costly.
• AR/VR navigation assistants and spatial computing interfaces that need a persistent, revisable model of a user's surroundings.

Industry relevance: The findings — that models terminate exploration prematurely, explore with high redundancy, struggle with orientation in vision, and resist revising stale priors — point to concrete engineering gaps for anyone building embodied agents on top of foundation models. The proxy-agent comparison also suggests that pairing a model with a lightweight structured exploration policy may currently outperform letting the model plan its own trajectory.

Future Directions

• Multi-agent exploration. The paper explicitly names extending Theory of Space beyond single-agent settings, where coordination and the sharing or alignment of spatial beliefs across agents introduce new challenges.

• Fixing the exploration policy. Since short proxy trajectories produced higher accuracy than autonomous exploration, an open question is how to train or prompt models to plan efficient, coverage-complete exploration rather than unsystematic door-chasing.

• Improving belief stability and orientation perception. The diagnosis of decay — where verified facts are overwritten by later incorrect predictions — and near-chance facing accuracy identify specific targets for better spatial memory and viewpoint reasoning in visual models.

• Overcoming belief inertia. The false belief results show models fail to overwrite obsolete priors even after directly observing a change; how to give models the plasticity to revise cognitive maps in dynamic environments remains unresolved.

Note: the provided paper content is truncated during Section 5.1, so the numeric values behind Table 5 (the cognitive map probing metrics) and the quantitative false belief / belief inertia results are not reported in the text available here.

Target Audience

Researchers and engineers working on embodied AI, multimodal foundation models, robot navigation, and spatial reasoning benchmarks; cognitive scientists interested in computational models of cognitive maps and belief revision; and practitioners building agents that must operate under partial observability, who will find the failure taxonomy and the active-versus-passive comparison directly actionable.

Authors’ abstract

Spatial embodied intelligence requires agents to act to acquire information under partial observability. While multimodal foundation models excel at passive perception, their capacity for active, self-directed exploration remains understudied. We propose Theory of Space, defined as an agent's ability to actively acquire information through self-directed, active exploration and to construct, revise, and exploit a spatial belief from sequential, partial observations. We evaluate this through a benchmark where the goal is curiosity-driven exploration to build an accurate cognitive map. A key innovation is spatial belief probing, which prompts models to reveal their internal spatial representations at each step. Our evaluation of state-of-the-art models reveals several critical bottlenecks. First, we identify an Active-Passive Gap, where performance drops significantly when agents must autonomously gather information. Second, we find high inefficiency, as models explore unsystematically compared to program-based proxies. Through belief probing, we diagnose that while perception is an initial bottleneck, global beliefs suffer from instability that causes spatial knowledge to degrade over time. Finally, using a false belief paradigm, we uncover Belief Inertia, where agents fail to update obsolete priors with new evidence. This issue is present in text-based agents but is particularly severe in vision-based models. Our findings suggest that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.