MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

AI summary

Overview

Research area: Natural language processing / evaluation of large language models on geospatial reasoning, with a focus on multilingual and geographically stratified benchmark construction.

Technical level: Intermediate. The paper assumes familiarity with benchmark design, knowledge graphs, retrieval-augmented generation, and agentic tool use, but its core argument is stated in plain terms.

Scope: The paper introduces MultiGlobeQA, a 46,060-question parallel benchmark that measures whether LLMs can compute over geographic entities — distances, containment, directions, grid encodings — rather than merely recall facts about them.

What This Paper Is About

Geospatial reasoning means computing distances, containment, directions, and other spatial relations over real-world places. LLMs store a lot of geographic knowledge but struggle with the geometric and topological computation that questions about physical space require, and existing benchmarks only partially localize where those failures occur: they tend to be synthetic, small, monolingual, or limited in geographic coverage. The goal of this paper is to build a large, execution-verified, multilingual, and income/density-stratified benchmark that isolates which spatial operations fail, under which conditions, and in which regions and languages.

Key Contributions

1. A large execution-based benchmark. MultiGlobeQA contains 46,060 question-answer pairs instantiated from 65 top-level templates across 14 spatial-function families and 15 answer formats, with ground truth computed by executing queries over three geographic knowledge graphs rather than by LLM judging. It also includes 3,589 false-premise questions (7.8% of the English benchmark) and a multimodal slice of 946 questions (2%).

2. Geography-aware stratified sampling. Entities are sampled from WorldKG, KnowWhereGraph, and OSMH3KG under a 4 × 3 grid of income tier (low, lower-middle, upper-middle, high, from the World Bank FY2026 classification) and population-density tier (low, medium, high, from the WorldPop R2025A 1km raster, binned at the global 33rd and 67th percentiles), covering 201 countries and territories.

3. Multi-parallel multilingual release. The same questions are released in 17 languages (English plus 16 targets) spanning six language families, three resource tiers, and seven scripts, with machine translation followed by human post-editing and LLM-based disagreement resolution.

4. A three-tier evaluation that separates knowledge from computation. Four LLMs are evaluated under parametric (T1), reasoning (T2), and agentic (T3, with KG retrieval, web retrieval, or both) settings, plus oracle conditions that inject gold KG triples to approximate perfect retrieval.

Main Findings

• No model beats the majority baseline without external help. The majority-class baseline scores 28.8 EM. Without retrieval or injected gold triples, Gemini-3-Flash reaches 22.7 EM at T1, and the open-weight models only 2.0 to 4.8.

• Explicit reasoning does not close the gap. Moving from T1 to T2 changes accuracy by +1.9 for Gemini-3-Flash and -1.8 for Qwen3.5-35B. Because gold answers are computed from geometry rather than stated in text, more thinking cannot recover a coordinate the model never memorized.

• Retrieval helps substantially, and KG retrieval beats web search. Qwen3.5-27B rises from 4.4 EM at T1 to 44.3 at T3a. KG retrieval outperforms web search for every model, by 2.3 points (Gemini-3-Flash) to 21.7 points (Qwen3.5-27B). Combining both helps the weakest models (Gemma-3-27B-Instruct +7.6, Qwen3.5-35B +7.9 over T3a) but not Gemini-3-Flash (-1.5) or Qwen3.5-27B (-0.2).

• Agentic retrieval is unstable. EM standard deviation stays below 0.5 at T1 and T2 but reaches 9.6 (Gemma-3-27B-Instruct) and 10.8 (Qwen3.5-35B) at T3a, depending on which entities are resolved first.

• Perfect retrieval does not solve the task. With gold triples and a Python interpreter (T3-o), accuracy caps at 61.6 EM, leaving 38% to 55% of questions wrong. Injecting triples lifts parametric accuracy 2.5× (Gemini-3-Flash) to 8.1× (Qwen3.5-27B) and agentic accuracy 1.4× (Qwen3.5-27B) to 2.0× (Gemini-3-Flash); T3a reaches only 49% to 72% of T3-o accuracy.

• Computation, not access, is the bottleneck. Adding tools to the same gold context (T3-o vs. T1-o) raises accuracy by 20.4 to 46.7 points for the open-weight LLMs but only 3.2 for Gemini-3-Flash, and closes the gap between them (Gemini-3-Flash led Qwen3.5-27B by 21.4 points at T1-o; at T3-o it is 60.2 vs. 61.6).

• Failures concentrate in specific spatial functions. Under gold context, accuracy spreads by 81 points across the 14 families: coordinates reach 94, direction 89, shape 25, and grid indexing 13. Administrative hierarchy is the only family substantially answerable from parametric knowledge (28.6 at T1), and that is driven by Gemini-3-Flash alone (75.2 vs. at most 17.0 for the others), as are topological relations (58.1) and direction (36.6).

• Discrete spatial encodings fail unevenly. Geohash recovers to 54.5 under gold context, while H3 and S2 cell indices reach only 16.2, and their composition is never correct in any condition.

• Errors are large when they occur. Across all models and conditions, normalized error is 0 for 18.4% of questions and 1 for 72.0%, leaving 9.6% with partial credit.

• Abstention separates the models more than accuracy does. At T1, coverage ranges from 7% (Qwen3.5-35B) to 94% (Gemini-3-Flash), yet their accuracy when answering is similar (30.8% vs. 24.3%).

• Low-income regions lag, and gold facts widen the gap. High-income questions are answered more accurately by every model (+1.0 Qwen3.5-35B to +5.8 Gemini-3-Flash), and the margin widens under perfect retrieval (+4.4 Qwen3.5-27B to +9.0 Qwen3.5-35B at T3-o). Oracle contexts themselves carry on average about twice as many gold triples for high-income as for low-income questions, so evidence volume and regional familiarity cannot be separated.

• Models reject false premises more than answerable questions. On the same 13 sub-templates under retrieval, Qwen3.5-27B rejects 85.7% of false premises while refusing only 13.6% of answerable ones; Gemini-3-Flash shows 63.2% vs. 33.9%.

• Accuracy is largely stable across languages. At T1, the best-to-worst spread is at most 2.1 points for every model, and 3.9 to 9.0 for Qwen3.5-27B across tiers. Refusal varies more, up to 11.0 points; the largest retrieval deviations are Georgian and Urdu, whose non-Latin entity names pass through an English tool interface.

• Agentic search is expensive. Agentic retrieval multiplies median tokens per question by 107 to 437. The oracle reaches higher accuracy with 1.6 to 6.3 times fewer tokens, and the Qwen models exhaust the step budget on 44% and 59% of retrieval questions versus 6% and 5% under T3-o.

Methodology in Plain English

The authors build the benchmark in four stages.

First, they define a taxonomy of 65 top-level question templates grouped into 14 spatial-function families (distance, containment, topology, network/path, administrative hierarchy, comparison, centrality, grid indexing, transformation, coordinates, direction, shape, uncertainty, spatio-temporal events). Each template expands into 129 parameterized sub-templates, and each sub-template gets one to five human-verified paraphrases, giving 315 English natural-language variants.

Second, they sample entities from three complementary knowledge graphs — WorldKG (OSM-tagged places), KnowWhereGraph (hazard, health, and administrative data), and OSMH3KG (H3-indexed OSM geometries). Each entity is tagged with a World Bank income tier and a WorldPop density tier, producing a 4 × 3 grid of 12 strata, and questions are generated independently within each cell, drawing up to 30 entity tuples per (template, KG, stratum) pair.

Third, ground truth is obtained by execution rather than post-hoc verification: every question template is paired with a query template that is run over the KG, results are validated against type-specific constraints, degenerate executions are discarded, and duplicates on (question, ground truth) are removed. The resulting English benchmark contains 46,060 verified pairs, with roughly 6.9k grounded in KnowWhereGraph, 24k in OSMH3KG, and 15k in WorldKG.

Fourth, the templates are translated into 16 target languages by an adaptive machine translation system with placeholders marked non-translatable, then post-edited by two proficient speakers of English and the target language, who reach 69.7% agreement; a rule-based pass resolves 88.6% of remaining items and an LLM judge arbitrates the rest. On a sample of four languages, a third annotator agreed with the judge on at least 88.9% of arbitrated questions. Instantiating templates with concrete entities introduces grammatical artifacts, which an ensemble of three LLM post-editors fixes on 16.2% of questions, with at least two models agreeing on 90.1%.

Evaluation uses four models (Qwen3.5-35B, Qwen3.5-27B, Gemma-3-27B-Instruct, Gemini-3-Flash) plus a majority-class baseline, across T1 (parametric), T2 (reasoning), and T3 (agentic, with Python execution and retrieval from KG, web, or both), plus oracle conditions injecting gold triples in structured, raw, or verbalized form. Metrics are exact match, normalized error, coverage, and false-refusal / negative-rejection rates, computed over true-premise questions with refusals counted as incorrect. Results are reported on the English text portion of the small split (5,916 questions, up to 200 per KG/SFID cell), which spans 4,979 true-premise questions, with means and standard deviations over three seeds.

Why This Matters

Impact on research. The paper argues that prior benchmarks localize geospatial failures only partially and that tool-augmented evaluation conflates tool assistance with model reasoning ability. By pairing live retrieval with oracle conditions that inject identical gold triples, MultiGlobeQA separates missing knowledge from missing computation, and its income- and density-stratified design makes geographic bias measurable rather than incidental. Because ground truth is computed by executing queries, correctness does not depend on an LLM judge.

Real-world applications:

• Navigation and routing, where distance, direction, and network/path computations are directly operational.
• Logistics and planning, which depend on proximity and containment queries such as how many facilities lie within a given radius.
• Disaster response, where spatial errors in hazard and event questions carry immediate consequences.
• Location-based services and map products, which increasingly sit behind LLM interfaces that must produce correct grid cells, coordinates, or geometries.

Industry relevance. The results show that retrieval and tool use account for nearly all performance gains, yet accuracy plateaus below 61.6 EM even with gold facts and code execution, and grid indexing stays below 13. For teams deploying LLMs in geospatial pipelines, that indicates the remaining bottleneck is reliable spatial computation — not more parametric geographic knowledge — and that agentic retrieval is expensive (median tokens per question multiplied by 107 to 437) and unstable. The authors caution against reading high benchmark scores as evidence of deployment readiness.

Future Directions

• Improving spatial computation rather than knowledge. The authors conclude that closing the gap requires models that compute spatial relations reliably, in context or through tools, rather than models with more memorized geographic facts.

• Reducing search that never reaches the evidence. Under T3-o the Qwen models exhaust the step budget on 6% and 5% of questions, versus 44% and 59% under T3a, indicating most of the agentic budget goes to exploration. How to make agentic retrieval efficient at reaching evidence remains open.

• Testing larger and frontier models. The authors note their academic budget limited them to three locally served open-weight models and one closed model from the small, fast tier of its family, so they cannot rule out that larger or newer models would close the reported gaps.

• Extending beyond schema-constrained answers. Outputs were constrained to the gold answer type's JSON schema, which isolates spatial reasoning from format compliance but means the benchmark does not test free-form spatial answers or whether a model can select the appropriate representation itself.

• Addressing ground-truth drift and source incompleteness. Ground truth is correct with respect to fixed KG snapshots rather than the world, so incompleteness affects set-valued answers (counts, enumerations, nearest-entity questions), and models retrieving current web information may be scored against outdated gold answers. Re-executing queries against newer snapshots would quantify this drift.

Target Audience

This paper is most useful to researchers building and evaluating LLM benchmarks, especially in geospatial, multilingual, and retrieval/agentic evaluation; to practitioners deploying LLMs in navigation, logistics, mapping, or disaster-response systems who need to know where such models fail; and to NLP researchers studying geographic bias and cross-lingual robustness. Readers looking for a new model architecture will not find one here — the contribution is measurement infrastructure and the diagnostic separation of knowledge from computation.

Authors’ abstract

Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logistics, yet large language models (LLMs) struggle with the required geometric and topological computation despite storing considerable geographic knowledge. Existing benchmarks localize these failures only partially: they are synthetic or smallscale, largely monolingual, and offer limited control over geographic coverage. We introduce MultiGlobeQA, a multilingual benchmark of 46,060 question-answer pairs spanning 14 spatial-function families and 15 answer formats, with execution-based ground truth over three knowledge graphs. It covers 201 countries and territories via income- and density-stratified sampling, with parallel questions in English and 16 additional high- and low-resource languages. Across parametric, reasoning, and agentic settings, LLMs collapse on tasks requiring grid indexing and shape computation, while topological relations and directions fare best. Retrieval and tool use yield considerable gains, yet performance plateaus below two thirds even when gold facts are supplied, indicating that computation, not access to knowledge, is the bottleneck. Models also underperform on low-income regions, a gap that gold facts widen rather than close.