GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks

AI summary

Overview

• Research area: Evaluation of Large Language Models on geographic, spatial, and temporal tasks (geospatial NLP benchmarking).
• Technical level: Intermediate. The paper is a benchmark-and-evaluation study; the tasks and metrics are explained with examples and formulas, but readers need some familiarity with LLM evaluation and standard accuracy/BLEU-style metrics.
• Scope: The paper introduces GeoBenchLLM, a benchmark built from twelve public datasets expanded into seventeen textual subdatasets across eight geo-related tasks at three cognitive levels, and reports baseline results for Qwen3 models (0.6B, 1.7B, 8B, with and without thinking) and GPT-OSS-20B/120B.

What This Paper Is About

Existing evaluations of LLMs on geographic data have mostly been conducted in a homogeneous setting, which limits what can be concluded about how well these models generalize. Prior benchmarks are either small, cover too few major tasks, or are restricted to a single context such as cities. The authors build GeoBenchLLM to probe LLMs across a broad set of geo-related tasks, categorized into Knowledge, Reasoning, and Application cognitive levels, and to measure how model size and reasoning (thinking) affect performance.

Key Contributions

1. A comprehensive geo-related benchmark: Twelve publicly available datasets are collected, transformed, and divided into seventeen purely textual subdatasets covering eight tasks at world scale, totaling 421,041 questions (203,014 train, 26,220 dev, 191,807 test).
2. A three-level task taxonomy: Tasks are organized into Knowledge (Coordinates Prediction, Yes/No questions, Regression, Place prediction), Reasoning (Complex Scenario QA, Spatial Reasoning), and Application (POI Recommendation, Pathfinding).
3. New metrics for open-ended geo answers: Because the benchmark is not limited to MCQ and Yes/No formats, the authors introduce metrics including Coordinates Accuracy, Precision, Recall, P-R Mean/Median, Compliance Ratio, Feasible/Success/Optimal Ratios, Unreachable Accuracy, and Distance, alongside Accuracy, Bleu-1, and Bert-Score.
4. Public release and baselines: The benchmark is released at https://github.com/Rfr2003/GeoBenchLLM, and all metrics are made available in a Hugging Face Collection usable with the evaluate library. Baseline results are provided for Qwen3-0.6B, Qwen3-1.7B, Qwen3-8B (with and without thinking) and GPT-OSS-20B and 120B.

Main Findings

• Larger models lead on knowledge but not everywhere: The larger GPT-OSS models achieve the best results in nine of the seventeen subdatasets (GPT-OSS-120B with six wins and nine runner-up placements; GPT-OSS-20B with three wins), and lead in four of the seven Knowledge-level subdatasets, often by a wide margin.
• Thinking can substitute for size on reasoning tasks: Qwen3-8B with thinking enabled surpasses the largest model on five of the ten subdatasets belonging to the Reasoning and Application levels. On PPNL_multi it reaches 0.62 accuracy (measured as Optimal Ratio) while the largest model attains 0.57.
• The size gap narrows outside the Knowledge level: The gap between the largest models and Qwen3-8B in thinking mode is around 24% at the Knowledge level, 13% at the Reasoning level, and 18% at the Application level.
• Knowledge tasks depend on parameter count: On GeoQuestions1089_coord, GPT-OSS-120B outperforms the best Qwen model by 0.29 in coordinates accuracy (0.72 versus 0.43), which the authors interpret as evidence that without external tools or databases, model size plays a major role in embedding real-world knowledge.
• Thinking almost always helps: The thinking version of a given model nearly always outperforms its non-thinking counterpart, sometimes matching the performance of the next larger model, as shown in the per-cognitive-level mean improvement gains in Figure 1.
• Small models still struggle on hard tasks: Performance on tasks such as SpatialEvalLLM (best 0.37) and PPNL_multi (best 0.62) remains well below the ceiling even for the largest evaluated models, and the smallest Qwen3 models score near zero on several of them.

Methodology in Plain English

The authors started by gathering twelve publicly available datasets from author-provided links, chosen to cover a wide range of geographic tasks. TourismQA could not be regenerated from its original code, so it was retrieved from a later work that used it. Where needed, they applied transformations from prior work to make datasets suitable for LLM evaluation, and they split some datasets into subdatasets so that questions using different answer types (coordinates, real numbers, Yes/No, place names) would not overlap across tasks.

They then sorted all tasks into three cognitive levels inspired by Xu et al.: Knowledge tasks are factoid questions answerable by querying a geographic database; Reasoning tasks require applying that knowledge (multiple-choice scenario questions, and spatial reasoning about distance, topology, and direction); Application tasks build on these skills for real-world uses such as POI recommendation and pathfinding, the latter adding a temporal factor.

Because open-ended questions cannot be graded like multiple choice, the authors designed a set of manually specified metrics. Coordinates Accuracy checks whether a predicted coordinate falls inside a circle of a given radius around the gold coordinate using haversine distance. For place prediction and regression, Precision, Recall, and their mean are computed from a distance matrix between reference and predicted value lists, with the median of the P-R Mean used as the main metric to reduce the effect of outliers. Pathfinding uses feasibility, success, and optimality ratios, plus a compliance ratio for output format and an unreachable accuracy. Closed-answer tasks use accuracy, and two text-generation tasks use Bleu-1 (chosen as the main metric over Bert-Score) and Bert-Score.

Finally, they ran all evaluated models on every subdataset, reporting for each subdataset only the main metric marked in their benchmark description table, and plotted the mean improvement gain from the worst result per cognitive level.

Why This Matters

Impact on research: GeoBenchLLM provides a much larger and broader evaluation surface than prior geo-related benchmarks. Its 421,041 examples compare with 955,463 for CityEval, 82,750 for STBench, 39,378 for GeoBenchmark, 900 for Xu et al., 3,154 for MapQA, and 4,000 for the two geographic bAbI tasks. It also covers all three cognitive levels and all four answer formats (generative, regression, Yes/No, MCQ), whereas GeoBenchmark omits Application tasks, CityEval and STBench are limited to MCQ and to specific contexts, and Xu et al. is too small for comprehensive assessment.

Real-world applications (implied by the tasks evaluated):

• Routing and navigation systems that must produce valid paths between locations, including through unordered intermediate stops.
• Location-based recommendation, such as suggesting points of interest from tourist reviews or predicting a user's next visit from check-in histories.
• Geographic question answering and GIS assistants, including yes/no spatial questions, place prediction from descriptions, and numeric estimates such as distances.
• Educational and exam-style geography assessment, drawing on scenario-based multiple-choice questions of the kind used in Gaokao geography examinations.

Industry relevance: The finding that a smaller model with thinking enabled can match a much larger model on reasoning and application tasks—while larger models retain a substantial advantage on knowledge tasks—gives practitioners a concrete basis for choosing model size versus inference-time reasoning budget when building geographic applications. The released metrics, usable with the evaluate library, lower the cost of reproducing and extending these comparisons.

Future Directions

• Test larger models with thinking enabled: The authors only allowed low thinking budgets to GPT-OSS-20B and 120B and state that using a larger model with thinking would likely yield marked improvements across all subdatasets; this remains untested.
• Close the Knowledge-level gap with retrieval: Since size dominates Knowledge results in the absence of external tools or databases, integrating tools, databases, or retrieval into the benchmark's knowledge tasks is a natural extension.
• Improve the hardest remaining tasks: SpatialEvalLLM and PPNL_multi show low absolute scores even for the best models (0.37 and 0.62 respectively), leaving room for methods targeting spatial reasoning and multi-objective pathfinding.
• Broaden model coverage: The reported baselines are limited to the Qwen3 family and GPT-OSS models; the provided content does not report results for other model families, nor does it include a limitations discussion.

Target Audience

Researchers and engineers working on LLM evaluation, geospatial NLP, and geographic question answering, as well as practitioners building location-based services, recommendation systems, or routing applications who need an evidence-based comparison of model size versus reasoning budget. Readers looking for an entry point into geographic LLM benchmarking will also benefit, because the paper documents dataset provenance, task taxonomy, and evaluation metrics in one place.

Authors’ abstract

In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities. In this paper, we present \benchName, a comprehensive benchmark for probing LLMs on geo-related tasks. We leverage a careful selection of twelve publicly available datasets from diverse geo-related tasks and domains, and evaluate a set of LLMs on geo-spatial and temporal understanding using our benchmark. Our results show that reasoning and size have a strong impact on overall performance. GeoBenchLLM is publicly available at https://github.com/Rfr2003/GeoBenchLLM.