Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

AI summary

Overview

Research area: Computer vision / 3D vision-language grounding, specifically agentic multimodal models for city-scale outdoor 3D scene understanding.

Technical level: Advanced. The paper assumes familiarity with 3D visual grounding, bird's-eye-view (BEV) scene representations, vision-language models (VLMs), and reinforcement learning (specifically GRPO).

Scope: The paper proposes Urbex, an agentic framework that reframes city-scale 3D grounding from static candidate ranking to active spatial search with a VLM that calls city-aware tools, trained via reinforcement learning on CityRefer and evaluated zero-shot on STPLS3D-Refer.

What This Paper Is About

Grounding an object in a city-scale 3D scene from a natural-language description (for example, a car described relative to a named building or a distant landmark) is usually solved by precomputing object proposals and then ranking them against the text. That approach depends on proposal quality and gets slow as the number of candidates grows, because it must score every instance. Urbex instead lets a VLM agent actively search the scene: it queries landmarks, zooms into regions, renders angled views, and commits a bounding box when it has enough evidence.

Key Contributions

1. A reformulation of city-scale 3D grounding as active spatial search rather than exhaustive candidate ranking, removing the requirement for precomputed instance proposals.
2. A city-aware interactive tool interface with three tools — Landmark Query, Zoom-in, and Oblique Rendering — exposing a large scene through coordinate-aligned, VLM-operable views.
3. A reinforcement learning objective (GRPO) for the tool-use policy, combining a smooth terminal reward (IoU plus center-distance) with lightweight tool-level shaping rewards, and a step cost that bounds interaction.
4. Empirical validation showing the best Hit@1 and IoU on CityRefer, candidate-free grounding, roughly an order-of-magnitude online inference speedup over the strongest ranking baseline, and zero-shot robustness on STPLS3D-Refer.

Main Findings

• Best accuracy on CityRefer: Urbex reaches 25.8 Hit@1 / 39.2 Hit@3 / 42.3 Hit@5 on the ND split and 23.9 / 36.9 / 40.7 on the NO split in the With-GT setting, the best Hit@1 and Hit@3 on both splits. CityAnchor retains the highest Hit@5 on ND at 45.4.
• Candidate-free operation: In the Without-GT setting, Urbex gets 15.6 2D IoU / 11.0 3D IoU / 20.1 point F1 on ND and 14.6 / 7.7 / 17.8 on NO, while always outputting a free-form BEV box rather than selecting from candidates.
• Large efficiency gain: CityAnchor takes about 32.1 seconds per example versus about 3.2 seconds for Urbex under the same online timing protocol. Tools account for about 0.36 s/query. Building the optional oblique cache takes roughly 2.3 hours across 12 CityRefer scenes, while rendering on demand is estimated at 4.4 s/query (7.3 times faster).
• Stronger than prompted VLMs on the same tools: GPT-5.4 (9.3/13.1 Hit@1 on ND/NO With-GT) and Qwen3.6-plus (6.4/11.5) use the identical Urbex tool environment without task-specific training and fall far below the trained policy.
• Indoor pipelines transfer poorly: EDA nearly fails (0.1 Hit@1 on ND, 0.2 on NO), consistent with information loss from downsampling city-scale point clouds to 40,000 points. InstanceRefer does better (1.6 ND / 4.9 NO) but only because it is evaluated as an easier candidate-ranking problem with provided instances.
• Proposal-based pipeline gap: Training SoftGroup on 15 non-CityRefer SensatUrban scenes and feeding its proposals to CityAnchor yields 4.2/2.4/4.7 BEV IoU/3D IoU/point F1 on NO, versus 14.6/7.7/17.8 for Urbex.
• Zero-shot generalization: On STPLS3D-Refer, Urbex leads every reported accuracy metric with 27.2/32.1/33.4 Hit@1/3/5 and 13.0/10.0/16.2 2D IoU/3D IoU/point F1, compared with 2.1/4.2/6.0 Hit@1/3/5 for CityAnchor and 16.3/17.8/19.1 for GPT-5.4.
• Landmark-free transfer: On 30 manually verified landmark-free re-annotations of STPLS3D-Refer, Urbex achieves 23.3 Hit@1 with no further training.
• Every tool matters: Without tools the agent reaches only 10.5 Hit@3 and 2.1 2D IoU. Adding Zoom-in gives 15.5 and 4.1; adding Landmark Query gives the largest jump to 28.2 and 8.9; adding Oblique Rendering at 7 turns gives 34.8 and 11.0; the 8-turn setting gives 36.9 and 14.6, while 9 turns slightly lowers 2D IoU to 13.2.
• Reward shaping matters: IoU-only reward reaches 22.8 Hit@3 / 6.7 2D IoU; adding distance shaping gives 30.6 / 10.9; IoU plus tool-level shaping gives 28.4 / 9.6; the full reward gives 36.9 / 14.6.
• Policy quality beyond the tool sequence: Under a fixed Landmark Query → Zoom-in → Oblique Rendering → Answer workflow, the base Qwen model reaches 2.7 Hit@1 while RL-trained Urbex reaches 10.0. The same workflow lets GPT-5.4 maintain similar Hit@1 while cutting token use from 16,321 to 7,734.
• Landmark noise tolerance: On CityRefer-NO expressions with named landmarks, injecting 10 m of coordinate noise lowers Hit@1 by only 1.2 points, whereas missing or incorrect landmarks cause drops exceeding ten points.
• Boxes are larger than targets: Matching-rule changes (positive-overlap filtering or CIoU instead of IoU) shift Hit@1 by at most 0.29 points, but additionally requiring BEV IoU ≥ 0.25 drops Hit@1 from 23.53 to 18.71. The predicted-to-target area ratio has median 1.40 and P90 9.92, indicating some boxes include substantial surrounding context.

Methodology in Plain English

Rather than feeding the whole city point cloud to a model or scoring a precomputed list of objects, Urbex treats the scene as an interactive environment the agent can look around in.

The environment starts with a global bird's-eye-view (BEV) image. The agent can call three tools. Landmark Query resolves a named place from the query (a building, a road, a facility) and returns its world position plus its location in a chosen view, giving a coarse anchor. Zoom-in crops a selected region from the original high-resolution BEV (whose long side typically exceeds 10,000 pixels) so small objects become visible. Oblique Rendering renders a candidate region from north, south, east, or west to check shape, height, and placement in 3D. Each view carries a transform back to world coordinates, so a box drawn in any BEV-aligned view can be projected into the scene. Oblique views are supporting evidence, not a coordinate frame for the final answer.

The agent is a VLM policy that sees the current observation and the whole interaction history, and each step emits a reasoning trace plus either a tool call with structured arguments or a final box. Training uses GRPO with no supervised trajectory demonstrations: for each query, the behavior policy samples a group of 8 trajectories, computes group-relative advantages from their cumulative rewards, and updates the clipped objective with a KL penalty. The reward combines IoU with the ground-truth box, an exponential center-distance term (λ_iou = 1.0, λ_dist = 0.3, τ = 80 m), a step cost (λ_step = 0.05), and tool-level shaping that rewards zooms moving toward the target and oblique renders overlapping the target. Shaping signals come from ground-truth annotations but never appear in the agent's text observations, so they do not leak at inference. Training runs for 300 optimization steps with a rollout batch size of 256, group size 8, actor learning rate 1e-6, KL coefficient 0.001, bf16, a frozen vision encoder, and 16 NVIDIA H20 GPUs. The base model is Qwen3-VL-4B-Instruct, and each trajectory allows at most eight assistant turns (up to seven tool calls plus a final answer).

Why This Matters

Impact on research. The paper argues that the dominant candidate-ranking formulation for city-scale 3D grounding is tightly coupled to precomputed proposals and scales poorly with candidate count. By showing that an RL-trained agent with a small tool set beats ranking baselines on accuracy and runs about ten times faster online, it suggests sequential, exploratory localization is a viable alternative for large 3D scenes and connects 3D grounding to the broader agentic VLM literature.

Real-world applications:

• Autonomous driving and urban scene understanding, where a vehicle or fleet may need to locate an object described relative to named streets or landmarks.
• Robotic navigation and delivery in cities, where a robot must find a target from a natural-language instruction without a prebuilt instance database.
• Aerial or drone inspection, where an operator describes an object using a nearby building or road direction.
• City-scale digital twin and mapping services, where free-form queries could be answered over dense urban geometry without exhaustive instance segmentation.

Industry relevance. The system's tool interface, candidate-free inference, fast online timing (about 3.2 s per example, with tools at about 0.36 s/query), and public code release make it more directly deployable than pipelines that require ground-truth or predicted instance proposals. The paper's own proposal-based comparison (4.2/2.4/4.7 BEV IoU/3D IoU/point F1 versus 14.6/7.7/17.8 for Urbex on NO) quantifies the practical cost of relying on proposal generation.

Future Directions

• Box refinement after target selection. Since the median predicted-to-target area ratio is 1.40 and P90 is 9.92, and requiring BEV IoU ≥ 0.25 drops Hit@1 from 23.53 to 18.71, tightening the predicted extent is a clear next step.
• Better local discrimination and region revisiting. The reported failure cases involve a zoom that excludes the target (the agent picks another car in the crop) and a zoom that contains the target but the agent selects vehicles in a different parking row — both require improved region selection and local verification.
• Reducing dependence on named landmarks. Landmark Query produced the largest single tool gain (Hit@3 from 15.5 to 28.2), yet missing or wrong landmarks cost more than ten Hit@1 points, so extending the policy to rely more on visual attributes and local spatial relations is motivated; the 23.3 Hit@1 on landmark-free re-annotations is an early signal.
• Broader cross-scene validation. The paper describes the aggregate STPLS3D-Refer results as suggesting, but not proving, that the policy learns reusable search behaviors, leaving broader multi-dataset transfer an open question.

Target Audience

Researchers and engineers working on 3D vision-language grounding, embodied agents, and multimodal tool-use policies; practitioners building city-scale spatial search or mapping systems; and readers already comfortable with BEV representations, VLM agents, and reinforcement learning, since the paper assumes that background without introducing it.

Authors’ abstract

City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large outdoor scenes. We propose Urbex, an agentic vision-language framework that instead casts grounding as active spatial search. Urbex represents a city-scale 3D scene as an interactive multi-view environment, allowing a VLM agent to query landmarks, zoom into local regions, render oblique views, and commit a grounded bounding box. We optimize this tool-use policy with reinforcement learning on CityRefer, using smooth localization rewards and lightweight shaping rewards for efficient exploration. Experiments show that Urbex achieves the best Hit@1 and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is substantially faster than the strongest city-scale ranking baseline. Zero-shot results on STPLS3D-Refer further indicate improved cross-scene robustness.