Reasoning with Pixel-level Precision: QVLM Architecture and SQuID Dataset for Quantitative Geospatial Analytics

AI summary

Overview

Research area: Computer vision and vision-language modeling, specifically quantitative spatial reasoning over satellite imagery (geospatial analytics).

Technical level: Advanced. The paper assumes familiarity with vision-language model architectures, patch embeddings, semantic and instance segmentation, and LLM code generation.

Scope: The paper introduces a benchmark (SQuID) and a code-generation architecture (QVLM) that avoid compressing images into embeddings so that counting, measuring, and spatial-relationship questions about satellite imagery can be answered with pixel-level precision.

What This Paper Is About

Vision-Language Models can describe a satellite scene fluently but cannot reliably count buildings or measure areas, because their vision encoders convert images into patch tokens and discard the pixel indexing that counting and measurement require. The authors set out to show that this is an architectural defect rather than a training-data problem, and to build both a benchmark that isolates the defect and an alternative architecture that avoids it by generating code that calls segmentation models and then computes answers directly on pixel-accurate masks.

Key Contributions

1. SQuID (Satellite Quantitative Intelligence Dataset): a benchmark of 2,000 satellite-image question-answer pairs across 587 satellite images, organized into three difficulty tiers (basic quantification, spatial relationships, and complex multi-condition reasoning). Each numerical answer comes with an acceptable range derived from the variability observed across 10 human annotators.

2. QVLM (Quantitative Vision-Language Model): a decoupled architecture in which an LLM generates executable Python code that first calls a segmentation model to obtain pixel-level masks and then performs geometric operations (counting, area, distance, buffering, intersection) directly on those masks, so spatial indexing is never compressed away.

3. Human-validated acceptable ranges: the paper computes Median Absolute Deviation (MAD) over 500 annotations from 50 USGS NAIP images at 1.0m GSD, producing tolerance ranges for percentage, proximity, and count questions rather than requiring a single point-estimate match.

4. Segmentation stack and an evaluation of it: a ConvNeXt-UNet segmentation server combining several public land-cover datasets into 7 land-use classes plus an instance model for building roofs trained on AIRS, and an alternative DINOv3 + Mask2Former backend, with the full system, benchmark, evaluation scripts, and DINOv3 training code released publicly.

Main Findings

• Code generation beats direct encoding on the benchmark: QVLM using GPT-5 as the coder with ConvNeXt segmentation reaches 42.0% accuracy on SQuID, compared with 28.1% for GPT-5 used as a VLM prompted with image-question pairs, a +13.9 point improvement.

• The gain holds even with weak components: gpt-oss-120B with ConvNeXt reaches 32.1% and Llama-3.1-8B with ConvNeXt reaches 29.0%; the weakest QVLM variant still surpasses the best VLM baseline (28.1%). The Qwen3-VL-30B-A3B-Thinking VLM baseline scores 26.14%.

• Segmentation quality matters: swapping ConvNeXt for DINOv3 + Mask2Former while keeping GPT-5 as the coder drops accuracy from 42.0% to 30.8%, indicating stronger segmentation directly improves quantitative reasoning.

• Improvement appears in every difficulty tier: comparing QVLM (GPT-5 + ConvNeXt) with GPT-5 as a VLM, Tier 1 improves by +14.22 points (53.52% vs 39.30%), Tier 2 by +19.97 points (54.06% vs 34.09%), and Tier 3 by +8.01 points (18.84% vs 10.83%).

• Geometric question types show the largest deltas: fragmentation reaches 81.63% for QVLM versus 26.53% for the GPT-5 VLM; connectivity 74.04% versus 37.5%; counting 56.74% versus 36.52%; size measurement 33.73% versus 16.27%; proximity percentage 40.65% versus 19.51%.

• Predictions sit nearer the correct answer, not just inside the tolerance: when acceptable ranges are doubled, QVLM rises from 42.0 to 51.4 (a 9.4 point gain) while the VLM rises only from 28.1 to 33.9 (a 5.8 point gain).

• Human agreement varies sharply by question type: overall Krippendorff's alpha across 500 annotations is 0.79; count questions reach α = 0.959 while proximity questions reach only α = 0.424. Mean MAD is 1.735 for percentage questions (26 questions, 260 annotations) and 2.250 for proximity questions (9 questions, 90 annotations); normalized count MADc is 0.19 from 120 count annotations.

• Compression is the stated root cause: a 1024×1024 image processed in 16×16 patches becomes a 64×64 token grid, reducing 1,048,576 pixels to 4,096 tokens, a 256-fold compression, and the paper cites 40-60% k-nearest-neighbor divergence caused by vision encoders.

• Updated supplementary results differ slightly from the main table: the Q-B run moved from 32.14% to 32.10% and the Q-C run from 30.83% to 36.90% once the runs completed after submission.

Methodology in Plain English

The authors start from an observation: when a VLM looks at a satellite image, it chops the image into small patches and turns each patch into a token. A million pixels become a few thousand tokens. Any record of exactly which pixel was where is gone, so the model cannot count or measure precisely.

Their first move is to build a test that exposes this. SQuID questions are generated automatically from existing remote-sensing datasets that already have segmentation masks (DeepGlobe at 0.5m GSD, EarthVQA at 0.3m GSD, and a solar panel dataset at 0.3m GSD), using contour detection to find objects, converting geometry into metric units, and generating questions whose answers come straight from the mask geometry. To check that the automatic answers match what people would say, they had 10 annotators mark grids and take distance measurements on 50 USGS NAIP images at 1.0m GSD through a custom interface, producing 500 annotations used to set tolerance ranges rather than single correct values.

Their second move is an architecture that sidesteps compression entirely. An LLM reads the question and writes Python code. That code calls a segmentation service, which returns binary masks for the requested classes or objects. All the counting, area computation, distance transforms, and buffering then happen on the raw masks using ordinary geometric operations. The LLM never looks at pixels; it only writes the recipe. Because the pieces are separate, any code-writing LLM and any segmentation model can be swapped in, which is exactly what the ablation studies test.

Why This Matters

Impact on research: The paper argues that quantitative failure in VLMs is architectural, not a matter of scale or better data, because no amount of training can recover information the encoder discarded. It provides a benchmark that isolates quantitative spatial reasoning instead of scene description, and a working demonstration that decoupling language understanding from visual analysis is a viable research direction.

Real-world applications:

• Climate and deforestation monitoring that needs hectare-precise area measurements rather than qualitative descriptions.
• Urban planning, such as assessing solar adoption through panel counts and areas, with the paper including a solar potential calculation at 200W/m² efficiency.
• Disaster response, including flood-risk queries counting buildings within 100m of water and fire-risk queries counting buildings within 50m of forest.
• Precision agriculture and land management, using class-level coverage, fragmentation, and connectivity measures.

Industry relevance: The system is explicitly modular, so organizations can add segmentation models for their own topics without retraining the whole system. The authors note the coding LLM could be much smaller than GPT-5, and that the ConvNeXt models are very small at 180M parameters, making for quick inference and less overall compute. The main reported costs are speed (a two-stage computation) and token usage: the developer prompt requires 1600 tokens versus 700 for the VLM path.

Future Directions

• Open-vocabulary segmentation: QVLM currently relies on closed-vocabulary segmentation models, which fundamentally limits it to the topics those models support.
• End-to-end training: connecting the code generation and segmentation stages into a jointly trained system rather than two independent stages.
• Hybrid routing architectures: designs that route between quantitative code-based reasoning and qualitative direct encoding depending on the question.
• Reducing cost and contamination risk: exploring smaller coding LLMs to offset the two-stage latency and token overhead, and addressing indirect dataset bias, since EarthVQA and DeepGlobe both contribute images to model training and to evaluation, a risk the authors note applies to the VLM baselines as well.

Target Audience

Researchers and engineers working on vision-language models, remote sensing, and geospatial analytics who need models to produce numbers rather than descriptions; benchmark designers interested in how human annotator variability can be turned into acceptable answer ranges; and practitioners building applied systems for environmental monitoring, urban planning, disaster response, or agriculture who are choosing between end-to-end VLMs and modular, code-orchestrated pipelines.

Authors’ abstract

Current Vision-Language Models (VLMs) fail at quantitative spatial reasoning because their architectures destroy pixel-level information required for counting and measurements. Vision encoders compress images through patch embeddings, reducing spatial indexing and losing the precise pixel-level tracking required for accurate counting. We present two contributions to address this fundamental limitation. First, we introduce SQuID (Satellite Quantitative Intelligence Dataset), a benchmark of 2,000 satellite image Question-Answer pairs with both numerical range and categorical answers, designed to evaluate quantitative spatial reasoning. The dataset spans three difficulty tiers with annotations automatically generated from human labels and their learned variability. Second, we propose QVLM (Quantitative Vision-Language Model), a code-generation architecture that maintains pixel precision by decoupling language understanding from visual analysis. Instead of encoding images into embeddings, QVLM generates executable code that first calls a segmentation model to obtain pixel-level masks, then operates directly on these masks, preserving spatial indexing throughout the reasoning process. Our experiments show that QVLM using GPT-5 as coder achieves 42.0% accuracy on SQuID compared to 28.1% for a VLM prompted with image-question pairs. Our work reveals that, for quantitative spatial reasoning, architectural decoupling enables better accuracy on quantitative tasks.