OGD4All: A Framework for Accessible Interaction with Geospatial Open Government Data Based on Large Language Models

AI summary

Overview

Research area: Applied machine learning for civic technology — specifically, LLM-based natural-language access to geospatial Open Government Data (OGD), framed between computational social science and GIS.

Technical level: Intermediate. The paper is readable for a general technical audience, but assumes some familiarity with retrieval-augmented systems, tool-calling/coding agents, vector embeddings, and GIS data formats.

One-sentence scope: The paper presents and quantitatively benchmarks OGD4All, an end-to-end LLM framework that retrieves relevant City-of-Zurich open datasets from metadata, generates and sandbox-executes Python analysis code, and returns verifiable multimodal results, evaluated over 199 questions, 430 datasets, and 11 LLMs.

What This Paper Is About

Citizens who want answers from open government data must typically search non-semantic repositories with poor metadata, manually inspect dozens of datasets, and then possess GIS and programming skills to analyze geospatial files. OGD4All addresses this by letting people ask ordinary-language questions and having an LLM retrieve the right datasets, write and run analysis code in a secure sandbox, and return text, maps, and plots — while explicitly refusing to answer when no supporting data exists, to avoid hallucination.

Key Contributions

1. A semantic, agentic dataset retrieval stage that embeds metadata and queries into a dense vector space using OpenAI's text-embedding-3-large, lets the LLM reformulate a question into up to 3 focused subqueries via tool calling, and uses the "no relevant dataset found" (k = 0) case as a termination criterion that blocks hallucinated answers.
2. An iterative code-generation analysis stage built on the CodeAct paradigm, in which an LLM plans, writes Python, executes it in a persistent sandbox, observes logged output, and self-corrects on errors, ending either when it calls a final_answer tool or when it hits a default maximum of 20 steps.
3. A secure-by-design execution layer using HuggingFace's smolagents Python interpreter (which disables extra imports and submodules, caps elementary operations, and blocks undefined operations), with optional support for remote "microVM" sandboxes on E2B for public deployments.
4. A reproducible 199-question OGD benchmark comprising 169 manually crafted and verified question-answer pairs derived from 70 parameterized templates, plus 30 negative examples with no valid answer, each accompanied by relevant datasets and a ground-truth Python script, with stated metrics for recall, precision, answerability, correctness, latency, token consumption, and API cost.

Main Findings

• Highest analytical correctness: 98.2%. GPT-4.1 outperformed all 10 other tested LLMs on analysis correctness; the abstract reports 98% analytical correctness and 94% recall overall.
• Clear open-weight versus closed-weight gap. Mistral Large, Mistral Codestral, Llama 4 Maverick, and GPT-oss 120B reached only 60.9%–69.2% correctness, attributed to weaker reasoning, instruction following, and Python code generation for geospatial analysis.
• Best retrieval grounding. GPT-4.1 achieved both the highest recall and the highest answerability accuracy, with near-perfect rejection of unanswerable questions. GPT-4o scored 5% higher in precision than GPT-4.1, but its lower recall prevents relevant datasets from being identified, which the authors judge worse for analytical completeness.
• Error recovery without fabricated data. For GPT-4.1, the most common causes of incorrect answers were misunderstandings of insufficient data or user intent, poor geometric approximations, and reasoning errors. No incorrect claims were produced through hallucinated data — errors were traceable through the generated code and reasoning traces.
• Cost-effective option. GPT-4.1-mini reached 91.7% correctness at 15 times lower median API cost per question than GPT-4.1.
• Reasoning models are slower. GPT-5, GPT-o1, and Gemini 2.5 Pro showed significantly higher latencies than most non-reasoning models, with a median of 51.1 s for GPT-5. GPT-oss 120B also showed a high median latency of 21.6 s, attributed to frequent coding errors requiring extra error-recovery iterations.
• A newer model did not improve results. GPT-5, described as OpenAI's most recent frontier model as of 10 September 2025, showed a decline in correctness compared to GPT-4.1, suggesting limited benefits for short analytical tasks.
• Different query strategies. GPT-4.1 and GPT-4.1-mini issued more refined subqueries (an average of 2.25 per question) than other models (1.25–1.6), which the authors interpret as systematic self-verification behavior.
• Expert validation. Six municipal experts from the Zurich city administration, briefed with a 4-minute demonstration video, confirmed the system's high value for citizen engagement; their main concern was the risk of generating incorrect information that could affect public trust. They also flagged a dilemma: the stronger performance of proprietary models such as GPT-4.1 conflicts with public data and governance sovereignty principles.
• Benchmark saturation. Frontier models already reach near-perfect scores on recall, answerability, and correctness, limiting the benchmark's ability to discriminate among future frontier models.

Methodology in Plain English

The framework runs in two stages. In retrieval, metadata for all available datasets (attributes, summaries, publication dates) is stored in a database. When a citizen asks a question, an LLM uses tool calling to issue a nearest-neighbour search against that database, decomposing the question into up to 3 subqueries. Because retrieval is semantic rather than keyword-based, a question about "bicycles in the city" can match a dataset described only as "urban cycling"; the subqueries are instructed to be in German because the prototype's metadata is German. Reformulations are shown to the user. If nothing relevant is found, the interaction ends — the anti-hallucination mechanism. Image inputs are base64-encoded, and PDFs are converted to Markdown with pymupdf4llm. In analysis, the retrieved datasets and their metadata are loaded into a persistent Python notebook inside a sandbox. An LLM repeatedly plans, writes code, executes it, and reads the logged output, correcting itself when errors appear, until it submits results through a final_answer tool or reaches 20 steps. Outputs are interactive maps, plots, and text, and users can ask follow-ups that are routed back to analysis or to another retrieval round.

Evaluation uses the 199-question benchmark. The 169 positive questions have unique, unambiguous answers, and 31% require joining two or more datasets. Questions span 55 datasets from Zurich's open data platform, but retrieval runs against a larger pool of 430 datasets; the six most frequent categories are base maps, administration, population, environment, construction & housing, and mobility. Required operations include SQL-style filtering and aggregation, GIS operations such as geocoding and length/area computation, and topological operations such as intersections and overlays. Correctness is graded by GPT-4o as an LLM-as-a-judge over the 169 non-negative examples, which preliminary experiments found more robust to alternate answer representations than substring or fuzzy matching; recall, precision, and accuracy are averaged over all 199 questions, while latency, token consumption, and API cost are reported as medians because those distributions were non-normal, multimodal, or right-skewed. Retrieval and analysis are assessed separately, with ground-truth datasets supplied to the analysis runs. Eleven models were tested, spanning OpenAI (GPT-4o, GPT-4.1, GPT-4.1-mini, GPT-o1, GPT-oss 120B with 117B/5.1B parameters, GPT-5), Google (Gemini 2.5 Flash, Gemini 2.5 Pro), Mistral AI (Mistral Large 123B, Mistral Codestral 22B), and Meta (Llama 4 Maverick 400B/17B).

Why This Matters

Impact on research. The paper argues that existing LLM-plus-GIS systems operate on curated datasets and emphasize technical performance over reproducibility, transparency, or public-sector applicability, and that none of the surveyed systems builds in rejection mechanisms for unanswerable questions or a benchmark covering retrieval, correctness, and deployment factors such as API cost, latency, and token consumption. OGD4All targets those gaps directly by releasing an open benchmark, traces, generated code, and evaluation scripts.

Real-world applications:

• Citizen-facing municipal data assistants that answer specific local questions instead of forcing navigation of dozens of datasets.
• Journalism and accountability work, where claims must be traced back to a specific dataset, code snippet, and reasoning step.
• Evidence-based political deliberation and local community problem-solving, as described in the paper's framing of widely accessible open-data tools.
• Visual citizen communication through the interactive maps and plots the coding agent produces, which the interviewed experts rated highly valuable.

Industry relevance. The work sits at the intersection of geospatial software (an Esri R&D Center Zurich affiliation) and public-sector AI. It shows that transparency can be engineered as a system property — logged reasoning traces, displayed generated code, cited dataset sources, and sandboxed execution — rather than added afterward. It also surfaces a procurement question for governments: the tested proprietary model, GPT-4.1, performs best, but the authors frame the performance gap against open-weight models as a policy challenge for data and governance sovereignty.

Future Directions

• Increase benchmark difficulty by adding question-answer pairs that require domain expert knowledge and local knowledge, since frontier models already saturate the current benchmark.
• Handle ambiguity. Every benchmark question had a well-defined, unique answer; the system's behavior on ambiguous or open prompts was not evaluated, even though ambiguity is a common issue in geospatial questions.
• Develop multimodal evaluation. Maps and plots are commonly produced and were valued by experts, but only textual outputs were graded, so the quality of visualizations was not quantitatively assessed.
• Build sovereign open models and participatory validation. The authors call for open models that meet governance standards for grounded, verifiable reasoning, and for validation involving domain experts and citizens.

Target Audience

Researchers and practitioners working on LLM agents, retrieval-augmented generation, and trustworthy AI; GIS and geospatial data scientists; civic technologists, municipal data officers, and public administrators exploring conversational access to open data; and policy-oriented readers interested in transparency, auditability, and sovereignty requirements for AI in governance. Cost- and latency-conscious engineering teams will also find the 11-model comparison useful.

Authors’ abstract

We present OGD4All, a transparent, auditable, and reproducible framework based on Large Language Models (LLMs) to enhance citizens' interaction with geospatial Open Government Data (OGD). The system combines semantic data retrieval, agentic reasoning for iterative code generation, and secure sandboxed execution that produces verifiable multimodal outputs. Evaluated on a 199-question benchmark covering both factual and unanswerable questions, across 430 City-of-Zurich datasets and 11 LLMs, OGD4All reaches 98% analytical correctness and 94% recall while reliably rejecting questions unsupported by available data, which minimizes hallucination risks. Statistical robustness tests, as well as expert feedback, show reliability and social relevance. The proposed approach shows how LLMs can provide explainable, multimodal access to public data, advancing trustworthy AI for open governance.