FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models

AI summary

Overview

• Research area: Computer vision, specifically remote sensing foundation models (RSFMs), benchmark design, and group-robustness / debiasing methods.
• Technical level: Advanced. The paper assumes familiarity with self-supervised pretraining, frozen-backbone linear probing, segmentation decoders, and group-robust optimization (GroupDRO, worst-group metrics, calibration metrics).
• Scope: The paper introduces FairRSFM, a biome-aware benchmark and set of mitigation baselines that measure and reduce how unevenly three remote sensing foundation models perform across ecological regions on four downstream datasets.

What This Paper Is About

Remote sensing foundation models are almost always judged by a single average score, which says nothing about whether a model works equally well in forests, deserts, croplands, or boreal and cryospheric regions. FairRSFM assigns every georeferenced sample a biome label using its coordinates, groups those labels into six broad ecological macro-groups, and then measures performance separately within each group under a strictly frozen-backbone protocol. The paper's goal is to show that strong average accuracy can conceal large biome-dependent gaps, and to test whether three light-weight debiasing strategies — BOLP, DBR, and GroupDRO — can narrow those gaps without retraining the backbone.

Key Contributions

1. A biome-aware benchmark (FairRSFM) for studying ecological group robustness across four classification and segmentation datasets: m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K with Dynamic World label maps.
2. A biome-labeling pipeline that first assigns georeferenced samples to one of 14 terrestrial biome classes using a spatial centroid join with an external ecoregion reference map, then consolidates them into six biome macro-groups reflecting spectral, phenological, hydrological, cryospheric, and surface-property regimes.
3. A controlled evaluation of three RSFMs — Prithvi-EO-2.0, SatMAE, and DOFA — under a unified frozen-backbone protocol over three random seeds, reporting both standard task metrics and biome-aware robustness measures (worst-group score, normalized failure range, ECE, equalized-odds-style disparity, and demographic parity metric).
4. An evaluation of complementary mitigation baselines (DBR, BOLP, and GroupDRO) showing that worst-group improvements are model- and task-dependent and can involve trade-offs with aggregate performance.

Main Findings

• Aggregate scores hide biome gaps. With Prithvi-EO-2.0 under standard empirical risk minimization (ERM), m-EuroSAT reaches 90.98% ± 2.36 overall macro-F1, but the mean worst-group score is only 83.72% ± 0.87.
• Dense prediction shows the same pattern. On m-SA-Crop-Type, Prithvi-EO-2.0 under ERM drops from 27.30% ± 0.15 overall mIoU to 18.47% ± 0.16 mIoU in the Xeric and Mineralogical group (N_g = 124), versus 28.55% ± 0.19 on the Transitional Herbaceous and Scrub group (N_g = 875).
• The weakest group varies by dataset and model. On m-EuroSAT under Prithvi-EO-2.0 ERM, the Aseasonal High-Biomass group is weakest at 84.56% ± 0.31 macro-F1 (N_g = 25) compared with 88.96% ± 3.38 for High-Amplitude Phenological. On m-BigEarthNet under the same model, the Transitional Herbaceous and Scrub group is weakest at 46.04% ± 1.85 F1@opt while forest-dominated groups are stronger.
• BOLP helps the worst group without touching the backbone. BOLP raises Prithvi-EO-2.0 worst-group F1@opt on m-BigEarthNet from 46.12% ± 1.51 to 50.27% ± 0.63, and lifts overall m-BigEarthNet performance from 60.09% ± 0.99 to 62.75% ± 0.16, with no gradient updates to the encoder.
• Gains are not uniform and sometimes trade off. On Prithvi-EO-2.0 m-EuroSAT, BOLP and GroupDRO raise aggregate macro-F1 to 93.42% ± 0.35 and 94.14% ± 0.47 respectively, but both also increase the normalized failure range relative to ERM (ERM NFR 5.96% ± 3.28; BOLP 9.82% ± 0.30; GroupDRO 10.34% ± 0.51). On MMEarth20K, BOLP lowers Prithvi-EO-2.0 overall mIoU from 47.99% ± 0.32 to 43.31% ± 0.85, while GroupDRO raises the worst-group score from 33.75% ± 0.58 to 37.16% ± 0.26.
• Model rankings by worst group differ from rankings by average. SatMAE achieves the highest m-EuroSAT ERM average in the reported tables (93.89% ± 0.13) but its worst-group score is 74.45% ± 0.51; GroupDRO on SatMAE trades some average performance (91.26% ± 1.03) for a much higher worst-group score of 89.71% ± 1.51.
• Biome coverage is uneven. Matched-group coverage is 98.8% for m-EuroSAT, 88.5% for m-BigEarthNet, 99.9% for m-SA-Crop-Type, and 100% for MMEarth20K. m-EuroSAT and m-BigEarthNet contain only three matched macro-groups, m-SA-Crop-Type only two, and MMEarth20K all six; the smallest matched-group test sizes are N_g = 25 and N_g = 14.
• Task-specific baselines are model-dependent. For DOFA, DBR gives the best reported numbers on several datasets — for example 28.20% ± 0.49 overall mIoU and a 21.91% ± 0.60 worst-group score on m-SA-Crop-Type, and a 88.27% ± 1.11 worst-group macro-F1 on m-EuroSAT.

Methodology in Plain English

The authors take four standard remote sensing datasets — three drawn from GEO-Bench (m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type) plus a 20,000-sample MMEarth20K subset with 10 m Dynamic World label maps — and add one extra piece of information to every sample: its biome. Because the datasets publish geocoordinates, the authors compute each patch's centroid and look it up against an external global terrestrial ecoregion map with 14 biome classes. Those 14 raw labels are then folded into six macro-groups that pool sparse biomes while keeping broad ecological distinctions, and the mapping is fixed across every dataset, model, and method. Patches that are unknown, water-only, or unmatched are dropped from group summaries.

Three pretrained encoders are used as-is and frozen: Prithvi-EO-2.0 (300M spatio-temporal ViT, 6 HLS bands), SatMAE (ViT-Large, 304M parameters, 10 Sentinel-2 bands), and DOFA (ViT-Base, 86M parameters, wavelength-conditioned embeddings spanning 0.44 μm to 2.20 μm). Only a small task head is trained on top — a linear or MLP probe for classification, an FCN, conv, or UPerNet-style decoder for segmentation — using AdamW with a base learning rate of 10⁻³ cosine-annealed to 10⁻⁶, weight decay 10⁻⁴, up to 50 epochs with patience 15, 224×224 inputs, batch size 64 for classification and 32 for segmentation, across three seeds.

Three debiasing strategies are compared against plain ERM. Dynamic Biome Reweighting (DBR) starts from inverse-frequency group weights and then, every epoch, reweights groups using per-group validation scores: groups performing below the mean get more weight, groups above the mean get less. Biome-Orthogonal Linear Probing (BOLP) works on the frozen embeddings instead — it computes per-group centroids, takes a thin SVD of the centered group-mean matrix, and projects embeddings onto the orthogonal complement of the top k inter-group directions (with k set to the number of matched groups minus one: 2, 2, 1, and 5 for the four datasets), while recentering each group to a common reference. GroupDRO adapts group weights from training losses in the usual way. None of the methods adds trainable parameters beyond the task head.

Evaluation goes beyond average accuracy. The paper reports macro-F1 for single-label classification, macro-F1 at per-class optimal validation thresholds in the range [0.1, 0.95] for multi-label classification, and mean IoU for segmentation — computed within each macro-group — plus a worst-group score, a normalized failure range, expected calibration error with 10 equal-width bins, an equalized-odds-style disparity computed from Youden's J-index, and a demographic parity metric based on positive prediction rates. The paper explicitly frames the latter two as descriptive summaries under ecological shift, not as claims about protected human attributes.

Why This Matters

• Impact on research: The paper reframes standard RSFM transfer evaluation as a group-robustness problem. It argues that dataset-level averages, which dominate RSFM benchmarking today, cannot answer whether a model is consistent across ecological regions, and it provides a reusable, metadata-only protocol (geocoordinates plus a biome map) that other georeferenced datasets can adopt.
• Real-world applications:
  • Crop mapping and agricultural monitoring: m-SA-Crop-Type results show that Xeric and Mineralogical regions lag badly behind Transitional regions, which matters for any operational crop-type product applied globally.
  • Global land-cover mapping: m-BigEarthNet and MMEarth20K results show underperformance in transitional and xeric regions even when forest-dominated regions score well.
  • Flood and disaster segmentation: The paper lists flood segmentation among the Earth-observation tasks RSFMs are used for; uneven performance across hydrological and cryospheric groups could mean uneven disaster response quality.
  • Model selection and procurement: Because worst-group rankings differ from average rankings (SatMAE versus Prithvi-EO-2.0, and GroupDRO versus ERM), the paper's metrics give practitioners a different basis for choosing a backbone.
• Industry relevance: Organizations deploying geospatial foundation models globally — satellite imagery providers, agricultural technology firms, environmental monitoring agencies, and humanitarian mapping groups — get a way to audit where a model is likely to underperform geographically before deployment. The mitigation methods (DBR, BOLP) add no trainable parameters beyond the task head, so they are cheap to try on top of an existing frozen model.

Future Directions

• Extend the benchmark beyond the four datasets. The paper notes the metadata join is lightweight and that FairRSFM is "readily extensible to other georeferenced remote-sensing datasets," so adding more classification and segmentation tasks is a natural next step.
• Resolve uneven macro-group coverage. m-EuroSAT and m-BigEarthNet contain only three matched macro-groups and m-SA-Crop-Type only two, with the smallest matched-group test sizes at N_g = 25 and N_g = 14; expanding or rebalancing these splits would make range-based summaries more reliable.
• Address the ecotone/multi-biome labeling limitation. The authors assign an entire patch to the biome of its centroid, which they acknowledge can be noisy for ecotone or multi-biome tiles and treat as a consistent grouping rule rather than a pixel-perfect ecological map.
• Find mitigations that help the worst group without hurting the average. The reported trade-offs — for example BOLP lowering MMEarth20K overall mIoU from 47.99% to 43.31% while GroupDRO raises the worst-group score from 33.75% to 37.16% — suggest that a single method that improves both aggregate and worst-group performance across models and tasks is still an open problem.

Target Audience

Researchers and graduate students working on remote sensing foundation models, self-supervised geospatial representation learning, and benchmark or robustness evaluation. It is also relevant to practitioners who fine-tune or deploy frozen Earth-observation backbones for land-cover classification, crop mapping, or dense segmentation, and to fairness and distribution-shift researchers interested in how group-robustness methods from the broader machine learning literature (GroupDRO, representation debiasing, null-space projection) transfer to ecology-defined groups rather than human demographic attributes.

Authors’ abstract

Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecological regions. We introduce FairRSFM, a biome-aware benchmark for evaluating ecological group robustness in RSFMs. FairRSFM maps georeferenced samples from 14 terrestrial biome classes into six ecologically meaningful macro-groups and evaluates models under a unified frozen-backbone evaluation protocol. The benchmark covers four downstream datasets: m-EuroSAT, m-BigEarthNet, m-SA-Crop-Type, and MMEarth20K with Dynamic World label maps. Using Prithvi-EO-2.0, SatMAE, and DOFA across three random seeds, we show that aggregate performance consistently masks biome-dependent disparities across architectures and tasks. For example, Prithvi-EO-2.0 reaches 90.98% overall macro-F1 on m-EuroSAT but a mean worst-group score of only 83.72%, while m-SA-Crop-Type drops from 27.30% overall mIoU to 18.47% in the Xeric and Mineralogical group. We further evaluate Biome-Orthogonal Linear Probing (BOLP), Dynamic Biome Reweighting (DBR), and GroupDRO as complementary mitigation baselines. Their effectiveness is model- and task-dependent; for example, BOLP improves Prithvi-EO-2.0 worst-group F1@opt on m-BigEarthNet from 46.12% to 50.27% without updating the RSFM backbone. FairRSFM provides a reusable protocol for diagnosing and mitigating ecological robustness gaps in remote sensing foundation models. Code and datasets are available at: https://github.com/aminurhossain/FairRSFM.