SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation

AI summary

Overview

Research area: Geospatial foundation models (GFMs) for Earth observation, specifically uncertainty quantification, out-of-distribution detection, and selective prediction.

Technical level: Intermediate. The framework is conceptually accessible, but assumes some familiarity with uncertainty quantification (aleatoric vs. epistemic), ensemble methods, and embedding-space analysis.

Scope: This paper introduces SHRUG-FM, a model-agnostic framework that fuses three reliability signals—input-space OOD, embedding-space OOD, and task-specific predictive uncertainty—into an interpretable decision tree that tells a model when to abstain from making a prediction.

What This Paper Is About

Geospatial foundation models trained on satellite imagery often fail silently when deployed on regions, seasons, or disaster events underrepresented in their pretraining data. Existing models provide no built-in mechanism to flag when a prediction is unreliable, leading to overconfident outputs in high-stakes scenarios like disaster response. SHRUG-FM addresses this by combining multiple complementary signals to identify likely failures and abstain from them, rather than predicting blindly.

Key Contributions

1. Systematic identification of failure sources. The authors formalize three distinct reliability signals for GFM pipelines: input-level OOD (U_I) from geophysical features, embedding-level OOD (U_E) from the foundation model's latent representations, and task-level uncertainty (U_T) from the downstream decoder.

2. A unified selective prediction framework. SHRUG-FM integrates these signals into a single, model-agnostic abstention mechanism via a shallow decision tree classifier that decides whether to accept or reject each prediction.

3. Empirical validation across three high-stakes tasks. The framework is evaluated on burn scar segmentation, flood mapping, and landslide detection, demonstrating consistent risk reduction on retained samples compared to single-signal baselines such as predictive entropy or mutual information alone.

4. Interpretable, glass-box abstention. By using a shallow decision tree (max depth 3), the framework exposes interpretable thresholds (e.g., specific slope values combined with embedding drift), allowing users to understand why a prediction was rejected, not just that it was.

Main Findings

• Fused signals outperform single-signal baselines across all tasks. On burn scar detection, SHRUG-FM achieves Risk@0.5 of 0.090 ± 0.038 vs. 0.137 for the best single-signal baseline (Mutual Information). On flood mapping, it achieves the lowest AURC of 0.115 ± 0.001. On landslide detection, AURC drops to 0.392 ± 0.035 versus 0.526 for aleatoric uncertainty alone.

• Task uncertainty (Mutual Information) is the most robust universal signal. It was selected in 100% of decision trees across every task and seed, confirming that model disagreement is the most consistent indicator of GFM failure.

• Different failure causes require different signals. Spatial density (input-level) was selected in 100% of flood trees, reflecting geographic coverage gaps. Embedding-space signals (NCDD at 70%, normalized distance at 100%) dominated landslide detection, where events are rare and structurally atypical. For burn scars, epistemic uncertainty was the primary indicator.

• The "Input Paradox." Ablation results show that removing input geophysical features occasionally improves ROC AUC (e.g., 0.84 vs. 0.791 for fire), suggesting these features add noise in low-data regimes. The authors retain them anyway for physical grounding and human interpretability.

• UQ signals are essential for flood detection. Removing uncertainty quantification signals collapses flood ROC AUC from 0.725 to 0.524, a dramatic degradation.

• Discard-performance curves remain stable at higher coverage. Single-signal baselines (Entropy, MI) often level off or degrade with increasing coverage—SHRUG-FM maintains or improves F1 on retained samples.

Methodology in Plain English

The researchers built a system that sits on top of an existing Earth observation model and decides whether to trust each prediction. It uses three complementary "health checks":

1. Input check (is this scene unusual?): For each satellite image, the system extracts geophysical properties like elevation, slope, and land cover from external databases (HydroATLAS, Copernicus DEM). It compares these to the distribution of properties seen during pretraining, computing percentile ranks and a spatial density estimate. If a scene comes from an area with few or no pretraining samples nearby, the system flags it.

2. Embedding check (does the model understand what it sees?): The system clusters embeddings from pretraining data into 64 prototypes using k-means. For a new sample, it computes how far the embedding sits from its nearest cluster center (normalized distance) and how ambiguous the cluster assignment is (Nearest Centroid Distance Deficit, or NCDD). This avoids requiring the full pretraining embedding set—only compact prototypes are stored.

3. Task check (is the model confident about this label?): Using an ensemble of models, the system computes predicted probability, average entropy (aleatoric uncertainty), and mutual information (epistemic uncertainty). These are aggregated over the region of predicted event area to produce image-level scores.

These signals are fed into a shallow decision tree trained as a secondary classifier: it learns to predict whether the downstream model's F1 score will fall below a task-specific threshold (0.6 for burn scars and floods, 0.5 for landslides). The tree outputs an accept/reject decision. Evaluation uses risk–coverage curves, AURC, discard–performance curves, and operational risk at fixed coverage levels.

Why This Matters

This work bridges a critical gap between benchmark performance and real-world reliability of geospatial foundation models. As GFMs become central to environmental monitoring and disaster response, the inability to know when a model is wrong becomes a safety hazard.

Real-world applications:

• Disaster response triage. Flagged predictions can be routed to human analysts while accepted ones go to automated pipelines, prioritizing scarce expert attention where it is most needed.
• Climate monitoring. Long-term environmental shifts (droughts, land-use change) create distribution shifts that reliable abstention mechanisms can surface as diagnostics.
• Agricultural and food security monitoring. Underrepresented geographies and seasons can be flagged before decisions are made on crop assessments or food-aid allocation.
• Insurance and risk assessment. Automated spatial analysis of flood or landslide risk can incorporate reliability flags to avoid acting on overconfident model outputs.

Industry relevance: The framework is model-agnostic, meaning companies deploying any GFM (Clay, Prithvi, SkySense, TerraMind, etc.) can layer SHRUG-FM on top without retraining. The suggested deployment pattern—distributing pretrained embedding prototypes and feature quantiles alongside models—points toward a standardization opportunity for GFM providers.

Future Directions

• Broader generalization testing. The authors evaluate a single foundation model (SSL4EO-S12 ViT/s16 MoCo) and three tasks; generalization across architectures, pretraining paradigms, and domains remains open.
• Lightweight uncertainty alternatives. Ensembles are computationally expensive. Density-predicting models that yield a full predictive distribution in a single forward pass are proposed as a promising substitute.
• Metadata distribution standards. The input and embedding signals require access to pretraining distribution statistics. Distributing precomputed prototypes and quantiles alongside pretrained models would reduce integration friction and enable wider adoption.
• Collaboration with domain experts. Translating combined reliability flags into actionable diagnostics and recalibrating thresholds with expert feedback could yield new scientific insights into EO data itself.

Target Audience

Researchers and practitioners working on geospatial machine learning, remote sensing, and Earth observation who need reliable deployment of foundation models in risk-sensitive settings. Also relevant to ML reliability researchers interested in uncertainty quantification, OOD detection, and selective prediction applied to real-world scientific domains—and to disaster-response and environmental-monitoring organizations evaluating whether to trust automated analyses.

Authors’ abstract

Geospatial foundation models (GFMs) for Earth observation often fail to perform reliably in environments underrepresented during pretraining. We introduce SHRUG-FM, a framework for reliability-aware prediction that enables GFMs to identify and abstain from likely failures. Our approach integrates three complementary signals: geophysical out-of-distribution (OOD) detection in the input space, OOD detection in the embedding space, and task-specific predictive uncertainty. We evaluate SHRUG-FM across three high-stakes rapid-mapping tasks: burn scar segmentation, flood mapping, and landslide detection. Our results show that SHRUG-FM consistently reduces prediction risk on retained samples, outperforming established single-signal baselines like predictive entropy. Crucially, by utilizing a shallow "glass-box" decision tree for signal fusion, SHRUG-FM provides interpretable abstention thresholds. It builds a pathway toward safer and more interpretable deployment of GFMs in climate-sensitive applications, bridging the gap between benchmark performance and real-world reliability.