EarthRecord: From Pixels to Land Change with Multimodal Agents over Four Decades of Satellite Imagery

AI summary

Overview

Research area: Evaluation of multimodal AI agents for Earth observation — specifically, whether agents can turn raw, multi-decade, multi-sensor satellite archives into reliable land-change analysis.

Technical level: Advanced. The paper assumes familiarity with remote sensing practice (cloud masking, cross-sensor harmonization, surface reflectance products), agentic sandbox evaluation, and the pass^3 scoring protocol.

Scope (one sentence): EarthRecord is a 56-site, 238-question benchmark built on four decades of raw Landsat, MODIS, VIIRS, and Sentinel-2 data, used to evaluate twelve multimodal agents and to test — via a closed-book control and altered-satellite "instrument pairs" — whether their answers rest on prior knowledge, on the pixels, or on real change on the ground.

What This Paper Is About

Multimodal agents can now open files, write code, and run multi-step data analyses with tools, but nobody had measured whether they can do the actual work of land-change science: screening clouds and gaps, mapping transitions from imagery, and harmonizing measurements across sensors and years. Existing benchmarks either ask questions about curated images or hand agents preprocessed inputs, so the raw multi-sensor archive where real analysis begins is never tested. EarthRecord closes that gap by giving agents the unmodified satellite record itself and then probing, with two controls, what their answers actually rest on.

Key Contributions

1. A benchmark built on the raw satellite record. EarthRecord pairs four decades of unscaled Landsat, MODIS, VIIRS, and Sentinel-2 archives at 56 sites with 238 land-change questions spanning ten analysis tasks (153 computable, 73 recognition, 12 insufficient-evidence). Data selection, cloud masking, scaling, mapping, and cross-sensor harmonization are all left to the agent.

2. Two controls for prior knowledge and sensor shifts. A closed book condition removes the record entirely to measure what regional priors alone can answer, and 22 instrument pairs alter one satellite's data (by up to 20% from 2018 onward in 20 pairs, 5% in one, and Landsat 8→7 substitution in one) while nothing changes on the ground. Eight placebo pairs remove 5% of clear scenes at random to measure unrelated variability.

3. Reference answers drawn from four kinds of evidence. 98 computable questions use method-derived references (method median, with ≥75% method agreement for categorical and date conclusions), 13 take references from global products inside the record, 78 use four withheld global products (Hansen forest loss, JRC surface water, GHSL built-up surface, MCD64A1 burned area), and 37 come from human review of AI-drafted visual interpretations of the imagery.

4. An evaluation of twelve multimodal agents. The paper shows where leading agents succeed end to end, where they fail (mapping from imagery), and that roughly half of their right answers follow an altered sensor rather than the ground.

Main Findings

• Leading agents complete most analyses, but the field is wide. On the 226 computable and recognition questions, Opus 5.5 (85.4%), GPT-6 Sol (84.1%), GPT-6.1 Sol (81.4%), and Sonnet 5.5 (81.0%) lead with overlapping 95% intervals. GPT-6 Luna (18.6) and Qwen 3.7 Flash (19.5) fall below the majority-class baseline of 23.9; the always-no-change baseline scores 13.7.

• Mapping from imagery is the dominant bottleneck. The six leading models answer 92.0% of questions computed directly from record variables but only 61.1% of questions that require mapping from the imagery. Holding model and task fixed, the odds of a right answer on a mapping question are 0.31 times those on a computed one (95% interval 0.14–0.66, p = 0.003). Fifteen of the 18 questions no model answers right require a map.

• Some agents abstain rather than attempt a map. On the 41 recognition questions needing a map, GPT-6.1 Sol, GPT-6 Astra, and GPT-6.1 Terra answer "insufficient evidence" in 46.3%, 58.5%, and 66.7% of runs, against 8.1% for GPT-6 Sol and 8.9% for Sonnet 5.5. Their abstaining runs make a median of two or three tool calls and mostly view no image — yet when they do answer, they are right in 85% to 98% of runs.

• The record moves answers from prior knowledge to the pixels. With the record, all twelve models improve on computable questions, by 4.6 to 60.8 points (e.g., GPT-6.1 Terra +60.8, GPT-6.1 Sol +51.6, Opus 5.5 +25.5). Far more model–question pairs flip from wrong to right (40.9%) than from right to wrong (5.8%).

• Prior knowledge persists on recognition questions. Four models score lower with the record on recognition questions (GPT-6 Astra −4.1, DeepSeek Flash −9.6, Haiku 5.5 −20.5, GPT-6 Luna −5.5). At "famous" areas the closed book scores 55.9 versus 50.0 with the record, and removing the image tool barely changes leading models' recognition answers.

• But not yet to the ground. When one satellite's data are altered, 53.5% of cases that were right on the original turn wrong (74/102 on Terra NDVI, 20/50 on Landsat 8, 13/48 on Terra snow), and 91% of changed answers move in the direction of the alteration. Placebo pairs change answers in only 12% of cases.

• Few agents cross-check sensors. Only 7.5% of cases right on the original are right beyond both the closed book and the no-change reference. Of the wrong runs in cases that move, only 21% report the break at all and 3.5% attribute it to the instrument; the rest read it as clearing, drought, or land-use change.

• Higher scores do not protect against this. The eight models above 75 on computable questions still lose 33% to 59% of their right answers under alteration.

• Confidence is an unreliable error signal for some agents. Reported confidence separates right from wrong conclusions with an AUROC of 0.84 for Sonnet 5.5 but only 0.63 to 0.67 for Gemini 3 Flash, Qwen 3.7 Flash, and GPT-6 Luna. Gemini 3 Flash lowers confidence by 0.03 while its accuracy falls 25.7 points, and never declines an answerable question.

• Failures are often inconsistent rather than impossible. Of 347 wrong runs by the six leading models, 121 (35%) are on questions the same model answers right in another pass. A single analytic judgment can decide a question: on the snow question, none of the 10 runs defining winter as December–February finds the decrease, while all 13 using October–April or longer find it.

• Theme difficulty varies sharply. Fire is the hardest theme (median 50.0 across models); the regional questions are easiest (96.8).

Methodology in Plain English

The authors built a sandbox in which an AI agent is handed a site's satellite archive exactly as the data providers distribute it — unscaled single-band rasters, with quality layers and offsets not applied — plus a README giving the conversions. The agent has Python, a shell, geospatial and classical machine-learning libraries, and an image viewer, on a fresh offline CPU-only container with 4 CPUs and 24 GB of memory, 3,600 seconds of working time, and 300 messages. Because the viewer reads rendered images rather than raw bands, the agent must select bands and apply a stretch before it can see anything.

Each of the 238 questions specifies structured answer fields and a designated conclusion field, and every field accepts "insufficient evidence." Answers are scored against references the agent never sees, using pass^3: a question counts as correct only if the conclusion is right in all three independent passes. Intervals are 95% bootstrap intervals over questions.

Two controls isolate what the answers rest on. The closed book gives the model the question and the site's bounding box but no data, measuring what regional priors alone can produce. The instrument pairs corrupt one satellite's record while the ground stays unchanged, then score against the original reference — if an agent's answer flips, it was following the sensor rather than the land. Each of the twelve models runs 1,518 times (18,216 runs total), and after a pilot run on the first model the references across the whole bank were audited, 11 questions with unstable references were removed, and that model was rescored rather than rerun.

Why This Matters

Impact on research. EarthRecord reframes what an Earth-observation benchmark should measure. Scoring the final answer alone cannot distinguish an agent that computed from pixels from one that recalled a regional stereotype, and it cannot distinguish either from an agent that faithfully tracked a drifting sensor. The instrument-pair design turns a known remote-sensing hazard — apparent Amazon green-up from viewing geometry, vegetation browning from a degrading sensor, tropospheric cooling from orbital decay — into a controlled test. The finding that 53.5% of right answers follow an altered satellite is a direct challenge to the assumption that benchmark accuracy implies scientific validity.

Real-world applications (as motivated by the tasks and products in the paper):

• Deforestation and forest-loss monitoring, where references come from Hansen forest loss and 50 of the 73 recognition questions sit in the forest, fire, and urban themes.
• Surface-water and lake-area tracking, benchmarked against JRC surface water and tested through lake-area trend questions.
• Urban expansion and settlement mapping, using GHSL built-up surface as withheld ground truth.
• Burned-area and fire-scar assessment, the hardest theme in the benchmark (median 50.0), where agents misread harvested fields as burned.

Industry relevance. Operational Earth-observation pipelines, geospatial analytics vendors, and disaster-response teams are already deploying agentic tooling. This paper shows that such systems can execute code over an archive competently while remaining unable to verify a change on the ground — and that some agents decline exactly the questions that need a map, quietly, without lowering their confidence. For anyone deciding whether to put an agent in a monitoring loop, that is the failure mode that matters.

Future Directions

• Give agents pretrained vision models for mapping. The authors note their sandbox provides only general geospatial and classical machine-learning libraries on CPU, and that mapping is where the leading models lose the most. Stronger visual mapping components are the natural first fix.

• Build routine cross-sensor checks into the workflow. Since the alterations were all detectable by comparison with a concurrent satellite, but only 21% of wrong runs even noticed the break and 3.5% blamed the instrument, the paper calls for agents that habitually cross-check instruments — including for subtler drift than the clear-cut breaks tested here.

• Make confidence and abstention trustworthy. Early abstention without viewing an image, and agents that never decline or lower confidence while erring (Gemini 3 Flash), both point to a calibration problem the benchmark now makes measurable.

• Reduce cross-pass inconsistency. With 35% of leading-model wrong runs occurring on questions the same model answers right in another pass, the open question is whether better prompting, self-consistency, or explicit analytic-judgment checks can convert latent capability into reliable output.

Target Audience

Researchers and engineers working on multimodal agents and agentic evaluation; remote-sensing and Earth-observation scientists interested in what current AI can and cannot do with multi-decadal archives; benchmark designers who need templates for controlling prior knowledge and instrument confounds; and technical decision-makers in geospatial analytics, environmental monitoring, and disaster response who are considering deploying agents over satellite data. Readers without remote-sensing background will need to absorb terms such as surface reflectance, cloud masking, and cross-sensor harmonization, but the paper's central argument — answered from memory, answered from pixels, or answered from the ground — is legible without them.

Authors’ abstract

While multimodal agents now carry out complex data analysis with tools, their ability to turn decades of satellite imagery into reliable land-change analysis, crucial for Earth observation science, remains unmeasured. Existing benchmarks fall short: they pose questions on curated images or give agents preprocessed inputs, leaving out the raw multi-sensor archives where such analysis begins. We introduce EarthRecord, a benchmark built on the raw record of 56 sites: four…