Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing

AI summary

Overview

Research area: Multimodal large language models (MLLMs), geospatial/spatial reasoning, Geographic Information Science (GIS), and benchmark design.

Technical level: Intermediate. The paper assumes familiarity with terms like topological relations, GeoJSON, vector geometry, and question-answering benchmarks, but it explains each task format clearly enough for readers new to GIS.

Scope: The paper introduces GeoSpatialBench, a benchmark built from real OpenStreetMap urban data in six cities that tests nine MLLMs on 28 formally defined spatial relations through both question answering and constraint-checked spatial editing, across GeoJSON, map-image, and combined inputs.

What This Paper Is About

Existing evaluations of spatial reasoning in MLLMs rely mostly on question answering over small or synthetic scenes, which shows whether a model can name a spatial relation but not whether it can use that relation to change a scene. The authors argue that answering correctly does not prove a model can construct geometry satisfying explicit spatial constraints. GeoSpatialBench therefore pairs QA and constrained spatial editing over the same set of 28 spatial relations, with answers and edited geometries verified by GIS operations rather than by a learned judge.

Key Contributions

1. GeoSpatialBench, a geospatial relation benchmark covering 28 spatial relations drawn from spatial-database and GIS practice, split into four families (topology, direction, metric, ordering), with 8,400 canonical instances derived from real OpenStreetMap data across six cities (Stockholm, Paris, Hong Kong, Wuhan, Detroit, and Nairobi).
2. Constrained spatial editing as a complement to QA, where models must modify only a designated target feature F₀ and the resulting scene is checked against the requested constraints, allowing a direct test of whether a relation a model recognizes can also be applied.
3. A systematic modality comparison of GeoJSON, rendered map images, and a combined "Together" condition, using matched scenes, questions, and reference answers for QA so that representation effects can be compared directly.
4. A nine-model empirical evaluation with GIS-derived ground truth and deterministic scoring, including a strict Constraint Satisfaction Rate (CSR) for structured editing and a separate raster-tolerant success rate for map-image editing.

Main Findings

• GeoJSON beats map images. Structured GeoJSON consistently outperforms map images for both QA and spatial editing, and the paper reports that error rates by relation family are higher under map images in every family.
• Combining modalities does not reliably help. The Together condition, which supplies the annotated map alongside the same GeoJSON, offers no consistent gain over GeoJSON alone for QA, and its effect on editing varies across models.
• Top models are far ahead. Under GeoJSON, GPT-6 Astra reaches 96.5% QA accuracy and 97.6% editing success, and KIMI K3 follows at 90.7% and 85.1%. In Table 1, GPT-6 Astra records 96.36 open-ended, 96.64 multiple-choice, and 97.60 editing under GeoJSON.
• QA and editing are aligned but not equivalent. MiMo-V2.6-Pro scores 67.3% QA accuracy, above Qwen3.8-27B at 55.5%, yet Qwen3.8-27B performs better on editing (54.4% versus 43.0%), so strong relation understanding does not always translate into constrained editing.
• Modality effects are large for some models. For MiMo-V2.6-Pro, mean QA accuracy falls from 67.3% with GeoJSON to 45.2% with map image, while Together reaches 65.2%; the same model reaches 43.0% CSR with GeoJSON, 51.4% with Together, and a 16.5% raster-tolerant success rate with map image.
• Recognition can saturate before intervention fails. Under GeoJSON multiple-choice QA, Equality, Touching, and Disjointness have error rates of only 1.6%, 7.3%, and 8.4%, but editing error rises to 71.2% for Touching and 47.3% for Disjointness, while Equality stays easy at 3.3%.
• Metric reasoning is the persistent bottleneck. Metric relations are hardest in both modalities; under GeoJSON, open-ended metric error reaches 61.4%, and Minimum Distance, Hausdorff Distance, Total Perimeter, and POI Density all exceed 70% error in open-ended QA. Family-level error rates under GeoJSON are 61.40 (metric) versus 25.61 (topology), 44.68 (direction), and 28.14 (ordering) in open-ended QA.
• Relation-level difficulty varies widely. Averaged across models on multiple-choice QA, Equality (T01) reaches 98.4% accuracy and Touching (T04) 92.7%, while Boundary Coverage (T09) falls to 46.4%; among metric relations POI Density (M18) reaches 37.6% and Hausdorff Distance (M08) 46.8%; among ordering relations Along-Line Order (O01) reaches 89.2% versus 23.7% for Layer Hierarchy (O06), whose required road-layer attribute is omitted from geometry-only GeoJSON QA input.
• Editing reverses some family rankings. Under GeoJSON, spatial editing achieves the highest CSR for topology (65.1%) and direction (63.4%), while ordering falls to 47.3% and metric remains lowest at 42.9%, differing from the QA pattern where ordering is relatively strong.
• City variation is small relative to model variation. For open-ended GeoJSON QA, within-model range across the six cities is 1.00–5.71 percentage points, compared with a 67.03 percentage point gap between the highest- and lowest-performing pooled models. The authors state this does not establish generalization to unseen cities.
• Typical failure modes differ by task. In QA, DeepSeek-V4.1-Flash answers the Touching case correctly with GeoJSON but predicts "intersects" from the map image; in editing, InternVL3-8B moves the target in the wrong direction with GeoJSON, and KIMI K3 produces a 42.8 m distance from the map image when the required range is 8–22 m.
• Constraint violations dominate editing failures, and most failed edits violate at least one specified spatial constraint.

Methodology in Plain English

The authors build scenes from real OpenStreetMap data covering roads, buildings, land-use regions, and city-level points of interest, projecting features into a city-specific metric coordinate system before computing distance and area. Each of the six cities contributes 700 QA and 700 editing instances, giving 8,400 canonical instances; QA and editing draw on separate feature pools, so individual instances are not paired.

For every relation, a canonical scene records the required geometries, roles, source attributes, and construction provenance, using observed geometry where available and controlled constructions where the source data lack enough cases. A geometric oracle evaluates each scene and produces the reference answer. Each scene is serialized both as a GeoJSON FeatureCollection with anonymous feature IDs, and as an annotated map rendered in a square Web Mercator view on a 1,536 × 1,536 pixel canvas, with highlighted features, a legend, detail insets, and attribute cards where needed.

QA instances appear in open-ended and four-option multiple-choice formats, and each item is validated for correct geometry and distinct options. Editing instances are built separately: candidates are rejected if the instruction's constraints are already satisfied in the initial scene and kept only when a reference edit confirms a valid solution exists. Models then edit only the target feature F₀.

Scoring is deterministic. Multiple-choice outputs are parsed as option letters and open-ended outputs are normalized, with tolerances of one meter for distance and perimeter, one degree circular for bearing, and 0.001 for density; unparseable answers count as incorrect. Structured editing is scored with CSR, which requires a parseable, nonempty prediction, an actual change to F₀ (or U=1 for attribute edits), and satisfaction of all task constraints, with one corrective retry that does not reveal whether constraints were met. Map-image editing maps predicted pixel geometry back into task coordinates and accepts either metric constraint satisfaction or pixel geometry within four pixels of the reference under Hausdorff distance, provided required attributes match; this raster-tolerant rate is reported separately from strict CSR. The nine models evaluated are Qwen3.5-9B, Qwen3.8-27B, Qwen3-VL-8B-Instruct, and InternVL3-8B (open-weight), plus GPT-6 Astra, GPT-5.6 Sol, KIMI K3, MiMo-V2.6-Pro, and DeepSeek-V4.1-Flash (API-accessed).

Why This Matters

Impact on research. The paper argues that benchmark accuracy on spatial QA can mask gaps in a model's underlying spatial understanding, and it supplies an automatically verifiable framework where the same relation set is tested through both recognition and construction. It also gives a controlled comparison of structured geometry versus rasterized maps under matched scenes, questions, and answers, which most prior geospatial benchmarks do not provide because they differ in representations, relation sets, and task formulations.

Real-world applications:

• Urban and transportation planning — the motivating example is routing a road that must avoid mountains and rivers, cross a populated district, stay connected, and respect a distance limit.
• Navigation and autonomous driving, which depend on reliable reasoning about direction, distance, and connectivity.
• Geospatial analysis workflows, where models would translate natural-language spatial instructions into valid changes to vector datasets like a city's GeoJSON.
• Embodied interaction and world models, where a system must not only perceive spatial arrangements but act on them under constraints.

Industry relevance. The results indicate that organizations feeding map images to models should expect lower reliability than supplying explicit vector geometry, and that pairing images with geometry yields no consistent benefit. Metric quantities such as distance, perimeter, and density remain unreliable even with explicit coordinates, which matters for any downstream GIS or planning pipeline that needs numerically valid output. The authors also note that map-image editing confounds the input representation with the output interface, so cross-modality editing gaps should be read cautiously.

Future Directions

• Relation-aware spatial representations. The authors point to spatial tokenization approaches such as SPOK as one direction for representing geographic space and spatial relations explicitly, possibly bridging recognition, geometric computation, and intervention.
• Closing the recognition-to-editing gap. Editing stays discriminative even where recognition approaches saturation, so methods that translate a correctly identified relation into a valid geometry remain an open problem.
• Improving quantitative geometry. Metric relations are the most persistent weakness under both inputs, and explicit coordinates reduce ambiguity but do not guarantee reliable computation of continuous quantities.
• Generalization beyond the six cities. The authors state their city-level results do not establish generalization to unseen cities, and they note that difficulty with basic 2D geometric constraints sets a baseline for more complex spatial generation.

Target Audience

Researchers and engineers working on multimodal LLMs, spatial reasoning, GIS, embodied AI, and world models; benchmark designers who need a template for constraint-verified, automatically scored spatial tasks; and practitioners building planning, navigation, or geospatial analysis tools who need to know which input representation and which relation families their models can be trusted on.

Authors’ abstract

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models. However, existing evaluations of multimodal large language models (MLLMs) for spatial reasoning rely predominantly on static question-answering (QA) over small-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing. Grounded in real-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground-truth references and constraints verified via GIS operations. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI.