Towards Consistent Geospatial Representations: Aligning One Encoder to Many Foundation Models

AI summary

Overview

Research area: Geospatial foundation models (GFMs) for Earth observation — representation learning, model alignment, and benchmark evaluation in computer vision.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, self-supervised pretraining, linear/ridge probing, and representation-similarity metrics (CCA, CKA).

Scope: A workshop paper (2nd Workshop on Advances in Representation Learning for Earth Observation, arXiv 2610.28313v1, 20 Oct 2026) that releases frozen embeddings of five GFMs across eleven datasets and introduces a method for aligning a single ViT to all five at once.

What This Paper Is About

Geospatial foundation models are pretrained in very different ways — different self-supervised objectives, sensors, and ground resolutions — and no single one wins across benchmarks. The authors ask whether these models' representations are complementary (each capturing something the others miss), and whether one compact encoder can inherit that complementarity instead of forcing practitioners to pick a single model per task.

Key Contributions

1. Pangaea-Latents: an extension of the Pangaea benchmark that stores frozen embeddings from 5 GFMs (CROMA, DOFA, Prithvi, Scale-MAE, TerraMind-L) over all 11 Pangaea datasets, with baselines reproduced across 3 seeds. Because the embeddings are precomputed, training on top of them needs no forward pass through any GFM and fits on a single GPU.
2. A similarity-and-complementarity analysis: pairwise CCA, CKA, a standardized CKA variant, and a ridge-regression complementarity score computed across the five GFM embedding sets, showing the models are neither redundant nor disjoint.
3. GeoMixture: a framework that aligns one ViT-B/16 student to several frozen GFM teachers per dataset using a multi-representation alignment objective (mREPA), with three variants — Naive (align the backbone directly), Sum (sum expert outputs), and MoE (a learned per-location router over teacher-specific experts).
4. A frozen-encoder evaluation on nine Pangaea datasets comparing the GeoMixture variants against the five individual GFMs and a from-scratch UNet baseline, plus a loss ablation.

Main Findings

• GFMs are partially similar, not redundant: averaged over the 11 datasets with standardized CKA, pairwise similarity runs from 0.26 (Scale-MAE × DOFA, dissimilar) to 0.83 (CROMA × TerraMind-L, highly similar).
• Similarity does not predict complementarity: on DynamicEarthNet, Prithvi × DOFA has similarity 0.17 and complementarity 0.10 (dissimilar but complementary); on Sen1Floods11, pairs with similarity 0.36 and 0.70 reach comparable complementarity (0.12–0.16); on CropTypeMapping, CROMA × TerraMind-L has similarity 0.77 but complementarity of −0.03.
• The complementarity score itself is fragile: it becomes unreliable on high-dimensional latents where ridge regression overfits and is sensitive to regularization (e.g., the CROMA self-pair on MADOS), and it collapses to the majority class on imbalanced datasets (e.g., SpaceNet7). The authors suggest global similarity measures may miss localized latent signals that a linear probe cannot reliably capture.
• GeoMixture ties the best GFM on average rank: GeoMixture-Sum reaches an average rank of 3.89 across the nine datasets, tying TerraMind-L (3.89); GeoMixture-Naive is 4.00 and GeoMixture-MoE is 4.33. All three variants outperform four of the five individual GFMs on average rank (the GFMs span 5.22 for CROMA to 7.33 for Prithvi).
• TerraMind-L keeps the best average mIoU, narrowly: 57.05 versus 56.49 for GeoMixture-Sum, 56.27 for GeoMixture-Naive and 56.07 for GeoMixture-MoE — but only thanks to MADOS; without MADOS all GeoMixture variants surpass it.
• Consistency is the real gain: GeoMixture-Naive is never ranked below sixth on any dataset, whereas TerraMind-L ranks first or second on four datasets but sixth or seventh on four others. GeoMixture ranks first on only one dataset (GeoMixture-Naive on DynamicEarthNet, 40.14 mIoU).
• Alignment costs peak performance where one teacher dominates: GeoMixture trails by −4.0 to −14.5 mIoU relative to the best GFM on MADOS, PASTIS and CropTypeMapping, i.e., where aligning to a consensus is most costly.
• Extra routing capacity does not help: GeoMixture-Sum (no trained gate) is the best variant; learning the gate in GeoMixture-MoE brings no further gain, and variants mostly remain within one standard deviation of each other.
• The loss choice matters: in the ablation, negative cosine similarity is best on MADOS but worst on HLSBurnScars for every variant, while MSE plus cosine gives every variant its best average rank.
• Frozen GFMs do not consistently beat supervised training: a from-scratch UNet wins three datasets and outranks four of the five GFMs, largely through AI4SmallFarms (45.41 mIoU versus at most 27.59 for any frozen encoder), despite having fewer parameters than the UPerNet decoder alone. All GeoMixture variants still exceed the UNet in average mIoU (54.67 for UNet).

Methodology in Plain English

The team first did the bookkeeping that makes the comparison possible. They ran five publicly available geospatial foundation models once over every split of the eleven Pangaea datasets, under each model's own preprocessing and band mapping, and saved the last-layer feature grids to disk. Because all models see the same tiles, a given (sample, tile) index refers to the same piece of ground for every teacher, which makes the saved features directly comparable.

With those features in hand, they measured how alike each pair of teachers is using three similarity metrics, and how useful each pair is together using a complementarity score: fit a ridge regression on embedding A, on embedding B, and on the concatenation of the two, then ask what fraction of the remaining headroom the concatenation recovers. A positive score means the pair carries complementary information.

For the alignment method, a single ViT-B/16 student (768-dimensional tokens, 196 tokens per image, ImageNet initialization, a one-convolution tokenizer) is trained on each dataset's unlabeled training images. Each teacher gets a lightweight one-by-one convolution that projects the student's tokens into that teacher's width (768 or 1024), and the student is trained to match all five frozen teachers simultaneously, with the loss normalized across teachers and masked for any teacher unavailable on a given sample. Only the final block of the student is supervised. The MoE variant inserts one small per-teacher expert per token and a router that softly weights the experts at every spatial location; the Sum variant simply adds the experts.

After alignment, the projections are thrown away. The tokenizer, backbone, experts and router are frozen and evaluated under the standard Pangaea protocol, where a UPerNet decoder (plus a temporal attention module for time series) reads the student's features — so the mixture is part of the evaluated encoder, not just a training trick.

Why This Matters

The paper reframes a practical question for Earth observation: rather than pretraining yet another foundation model, can you cheaply combine what several existing ones already know? Its analysis suggests the answer depends on the dataset, and that the intuitive shortcut — "pick dissimilar models because they must be complementary" — does not hold. The release of Pangaea-Latents lowers the compute barrier for studying this, since a student can be trained without a single forward pass through any GFM.

Real-world applications:

• Disaster response: flood mapping (Sen1Floods11), wildfire burn-scar delineation (HLSBurnScars) and building-damage assessment (xView2), where no single pretrained model is reliable across event types.
• Agricultural monitoring: crop-type mapping (CropTypeMapping, PASTIS) and smallholder field-boundary delineation (AI4SmallFarms) across multi-temporal, multi-sensor inputs.
• Environmental and climate reporting: land-cover and urban-growth mapping (DynamicEarthNet, SpaceNet7, FiveBillionPixels) and above-ground biomass regression (BioMassters).
• Marine pollution monitoring: detection of oil spills, marine debris and algal slicks from Sentinel-2 (MADOS).

Industry relevance: organizations building geospatial analytics pipelines currently choose one foundation model per task, or pay to pretrain their own. A single encoder that is competitive on average rank and never collapses on a task is operationally more attractive than a model with higher peaks and deeper troughs, and embedding-only teachers (like AlphaEarth, mentioned as future work) can be folded in without needing released checkpoints.

Future Directions

• Complete the benchmark: the evaluation covers nine of the eleven Pangaea datasets; FiveBillionPixels and BioMassters (the latter scored by RMSE) are omitted, and finishing them would test the conclusions on the largest segmentation dataset and the only regression task.
• Remove the specialization confound: alignment is currently done per dataset on that dataset's unlabeled training images, which the GFMs never saw. Aligning on a large multimodal corpus instead would reveal whether a single GeoMixture encoder actually transfers across the benchmark, rather than measuring how well a representation can be specialized to target imagery.
• Broaden the teacher set and the loss study: the authors plan to scale to additional GFMs, including AlphaEarth, which releases embeddings but no backbone, and to ablate more alignment losses — the current ablation covers only five settings, and the loss is explicitly described as not a neutral choice.
• Understand what drives complementarity: since global similarity measures miss the localized signals that make two models useful together, a better predictor of when mixing helps — one that does not overfit high-dimensional latents or collapse on imbalanced classes — remains an open question.

Target Audience

Researchers and practitioners in remote sensing and geospatial machine learning who work with foundation models, benchmark evaluation, or knowledge distillation and representation alignment. It is most useful to readers deciding which pretrained geospatial encoder to deploy, to teams interested in combining existing models rather than pretraining new ones, and to students who want a compact, well-instrumented example of multi-teacher representation alignment applied to a standardized Earth observation benchmark.

Authors’ abstract

Geospatial foundation models (GFMs) differ widely in pretraining objective, modality and resolution, and none leads across benchmarks. We ask whether their representations are complementary, and whether a single encoder can inherit that complementarity. We introduce Pangaea-Latents, an extension of the Pangaea benchmark providing frozen embeddings of $5$ GFMs over its $11$ datasets, along with reproduced baselines, and find GFM embeddings to be neither redundant nor disjoint. We then propose GeoMixture, a preliminary framework to align a ViT to several GFMs per dataset through a multi-representation alignment (mREPA) objective. Our encoder ties the best GFM in average rank across nine Pangaea datasets, trading peak performance for cross-task consistency.