UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

AI summary

Overview

Research area: Multimodal large language model (MLLM) agents, embodied spatial reasoning, urban navigation, and simulation-based agent evaluation.

Technical level: Advanced. The paper combines a Unity-based 3D geospatial engine, a closed-loop agent protocol, and large-scale benchmarking of frontier MLLMs.

Scope: The paper introduces UrbanGround, a real-scale, physically constrained sandbox built from Hong Kong territory-wide 3D geospatial data, and uses it to measure how far current MLLM agents can convert local urban perception into sustained, goal-directed action.

What This Paper Is About

MLLMs can look at a single street view and describe what they see, but urban agency depends on whether that local evidence stays useful after the agent starts moving and the scene changes. Existing evaluations either use game worlds with game-specific mechanics, bounded indoor spaces, or urban data sampled from fixed viewpoints that do not preserve continuous physical contact with a real city. The authors build UrbanGround, a Unity-based replica of Hong Kong driven by the Hong Kong Lands Department's 3D Visualisation Map and 3D Pedestrian Network, so that every action produces a new observation under real scene geometry, and they use it to test grounding, persistence, and adaptation across three research questions.

Key Contributions

1. A real-scale urban sandbox. UrbanGround streams territory-wide georegistered 3D city tiles into Unity, supplies physical collision, records trajectories in geographic coordinates, and releases the environment on the web and as native builds for macOS, Windows, and Linux, with code and app downloads hosted on GitHub and a project page at https://urbanground.github.io/.
2. A formal closed-loop definition of spatial agency. The paper decomposes spatial agency into three cumulative capabilities: grounding, persistence, and adaptation, and expresses the interaction as a loop over hidden state, model-visible observations, a working spatial state, structured actions, and a transition function.
3. A five-level evaluation ladder. Levels 1 through 5 increase the spatial state that must remain usable across action, from local recognition and orientation through short- and long-range navigation, implicit-instruction exploration, multi-task planning, and dynamic environment interaction, instantiated across urban regions of Hong Kong.
4. A large empirical study of contemporary MLLM agents. The study contains 810 manually verified base instances, evaluates ten models from the GPT, Claude, Gemini, Doubao, GLM, and Kimi families, and reports answer accuracy, navigation success, pedestrian-network adherence, safe progress rate, and pedestrian-collision rate.

Main Findings

• Strong atomic perception, weak directional reasoning. On RQ1, models scored highest on visual recognition, with Gemini-3.6-Flash at 93.8 and Kimi-K3 at 92.5 accuracy. Orientation was markedly worse, with Claude-Opus-5 at 58.3, Kimi-K3 at 55.0, and Gemini-3.1-Pro at 23.3, a level the paper describes as approaching the random-guessing baseline for a four-option question. Active exploration fell between the two, from Claude-Opus-5 at 82.5 down to Gemini-3.1-Pro at 46.3. Overall RQ1 accuracy ranged from 53.2 (Gemini-3.1-Pro) to 79.1 (Claude-Opus-5).

• Generation gaps show up in orientation and exploration, not recognition. GPT-5.2 and Gemini-3.1-Pro remain close to newer models in visual recognition but fall markedly behind on orientation and active exploration, which the authors interpret as recent progress primarily improving preservation of directional spatial evidence.

• Answer correctness does not imply physically valid movement. Pedestrian-network adherence stayed below complete compliance in every local task, with overall values from 73.6 (Gemini-3.1-Pro) to 80.2 (Claude-Opus-5). In the reported example, GPT-5.5 correctly answered "Bank of East Asia" for the bank next to Beijing Tong Ren Tang but crossed the road directly and left the registered pedestrian connection.

• Navigation collapses beyond the immediate scene. Short navigation success reached 75.0 for GPT-5.5, Claude-Opus-4.6, and GLM-5V-Turbo, but long navigation fell to near zero: 0.0 for GPT-5.5, GPT-5.2, GLM-5V-Turbo, Gemini-3.6-Flash, and Gemini-3.1-Pro, and 3.8 for the best model, Kimi-K3. Overall eight-task navigation success ranged from 8.1 to 22.9, with Claude-Opus-4.6 highest at 22.9 and Kimi-K3 at 22.5.

• Instruction-dependent navigation fails even at short distances. Routes requiring the agent to infer the intended destination are comparable in length to short-range navigation, yet performance falls substantially; the best instructional navigation scores were 42.0 (Kimi-K3), 30.0 (Claude-Opus-5), and 28.0 (Claude-Opus-4.6).

• Failure is a horizon effect, not a single task split. A distance-stratified analysis binned ShortNav and InstructionNav episodes into four equal-count bins by initial straight-line start-to-goal distance; success concentrates in the shorter-distance bins and generally declines as the navigation horizon increases.

• Partial progress without completion. More than 50 percent of long-range episodes still end closer to the goal than they begin, and multi-stop agents reach 10 to 20 percent of their required destinations on average while rarely completing the entire plan.

• Interactive navigation magnifies model differences. Models remain relatively close on local QA tasks, but navigation produces a much wider separation; GPT-5.5 and GPT-5.4 diverge sharply on short routes (75.0 versus 15.0).

• Lower visibility hurts question answering, less so navigation. Under matched conditions, dusk and night reduced local QA accuracy for every reported model, for example Claude-Opus-5 from 79.1 (clear) to 66.4 (dusk) and 69.6 (night), and GPT-5.5 from 63.6 to 59.1 and 61.4. Short navigation under the same conditions was not systematically affected, and the paper reports that agents generally performed worse under dusk than at night.

• Failure to revise obsolete plans. In the road-closure task, goal-reaching rate was 0.0 for eight of ten models and 3.3 for Claude-Opus-4.6 and Kimi-K3, while pedestrian-network adherence remained high (93.0 to 96.4) and safe progress rate stayed low (10.0 for Doubao-Seed-2.0-Pro up to 46.7 for Claude-Opus-5). Agents continued producing locally compliant movement after the route became invalid.

• Moving pedestrians are not handled safely. Pedestrian-collision rate was high for every model, ranging from 76.3 (Doubao-Seed-2.0-Pro) to 90.0 (Gemini-3.6-Flash), while pedestrian-network adherence stayed between 90.6 and 98.5.

• Reported scope limits. The provided paper content is truncated mid-sentence in Section 4.4, so results beyond the road-closure and pedestrian tables, and the details of Appendices A through E, are not reported here.

Methodology in Plain English

The researchers started from two open datasets published by the Hong Kong Lands Department: a tile-based textured 3D mesh of the entire territory (3D Visualisation Map) and georeferenced 3D pedestrian line features (3D Pedestrian Network). They load the tiled mesh into Unity at runtime, convert geographic coordinates into a shared simulation frame, and stream tiles according to viewing scale. The same tile geometry is used for both rendering and collision, so the agent cannot walk through buildings.

The controlled agent is a first-person character whose motion is continuous and resolved against that collision geometry. It is not restricted to the pedestrian graph, but its trajectory is later compared against the registered network, which is how the authors measure whether the agent respected walkable routes. The simulation layer also runs a clock controlling sky, sun, shadows and artificial lighting, configurable weather including rain and fog, and animated pedestrians drawn from the open-source Microsoft Rocketbox avatar library and routed over the pedestrian network edges.

The agent layer keeps the simulator and the language model as separate processes communicating through a client-server interface, so different models and agent frameworks can drive the same embodiment. At each turn the model receives a first-person RGB observation, the task instruction, and its text interaction history, and it selects exactly one structured action. The physical action set is move, sprint, look, jump, and open_map; the map action set is map_select, map_pan, map_zoom, map_orbit, and close_map; and terminate ends the episode. The map shows a georeferenced overhead view with the agent's location marked, but does not expose a computed route, remaining distance, shortest path, or a route API.

Tasks are organized as a five-level ladder. Across the study there are 810 manually verified base instances distributed across Hong Kong districts that differ in street pattern, terrain, and vertical pedestrian connections. Every instance was completed by human testers under the same 100-step limit used for the models, and because each movement action lasts at most two seconds, the budget permits up to 200 seconds of commanded motion. Question-answering episodes are scored by agreement with the annotated answer, and single-endpoint navigation succeeds when the final position is within 15 meters of the destination and any evaluator-enforced constraint is satisfied.

Why This Matters

Impact on research. The paper separates three things that are often conflated: recognizing a scene, remembering a spatial frame after the scene changes, and revising behavior when the city itself changes. By holding the geographic world and embodiment fixed and only varying the task level, the study gives a clean argument that local multimodal competence does not automatically compose into city-scale agency.

Real-world applications:

• Pedestrian navigation and wayfinding assistants that must keep a route plan coherent as the user turns corners and as the visible scene stops showing the relevant landmark.
• Last-mile delivery and sidewalk robotics, where a system must stay on walkable routes, avoid pedestrians, and re-plan when a road segment becomes unavailable.
• Accessibility tools for blind or low-vision travelers, where directional grounding and reliable route revision matter more than naming a storefront.
• Urban simulation and planning, since the sandbox can replay the same location under controlled light, weather, closure, and pedestrian conditions.

Industry relevance. The work was produced by a team spanning Shanghai Jiao Tong University, the National University of Singapore, Meituan, The Chinese University of Hong Kong, Shanghai University, and the University of Oxford. Meituan's involvement points directly at delivery-oriented navigation, and the benchmark's emphasis on pedestrian adherence, collision rates, and closure compliance maps onto the safety and reliability requirements of any deployed embodied agent operating in a real city.

Future Directions

• Fixing directional grounding. Orientation was the weakest atomic capability and in some models approached the four-option guessing baseline, so improving heading-relative spatial representations is a prerequisite for everything above it.
• Making errors correctable over long horizons. The reported failure mode is error accumulation without effective correction. Better intermediate goal decomposition, progress verification, and recovery policies are the open problem.
• Route revision and pedestrian-aware motion. Low safe progress rate under road closure and high pedestrian-collision rates across all models indicate that plan revision and local obstacle avoidance need dedicated treatment.
• Broadening beyond one city. UrbanGround is built from Hong Kong data specifically; extending the same georegistered approach to other metropolises and to other dynamic events such as larger-scale closures or crowd changes would test whether the findings generalize.

Target Audience

Researchers working on multimodal agents, embodied AI, and spatial reasoning will get the most from this paper, particularly those building or evaluating navigation benchmarks. It is also relevant to robotics and autonomous delivery engineers who need to know where current models break in real urban geometry, and to practitioners who need a reproducible environment in which to test agent frameworks against a fixed embodiment. The paper assumes familiarity with MLLM agent loops, so beginners will find the framework concept approachable but the experimental results dense.

Authors’ abstract

Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city. We propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data. UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city and explore from a first-person view. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can ground a local scene well enough to answer spatial questions after active observation. Then we ask whether that grounding supports navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. Contemporary MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far current MLLM agents can explore reliably in complex, open-ended urban environments.