MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training

AI summary

Overview

Research area: Geospatial machine learning / Earth observation (EO) computer vision, with intersections in self-supervised pretraining, multimodal representation learning, and test-time adaptation.

Technical level: Intermediate. The benchmark design and empirical findings are accessible to a general machine learning audience; the formalization of the test-time training method (gradient normalization and averaging) is more advanced.

Scope: The paper introduces a global multimodal Earth observation benchmark of five tasks sharing 12 aligned modalities, benchmarks eight pretrained models on it, and proposes a model-agnostic test-time training method that exploits all modalities available at test time as reconstruction targets.

What This Paper Is About

Pretrained geospatial models are supposed to generalize to new tasks and new parts of the world with little labeled data, but existing benchmark datasets make this hard to verify: most have few modalities, most downstream tasks are regional rather than global, and few explicitly test whether a model can transfer to an entirely held-out geographic region. The authors build MMEarth-Bench to evaluate these three problems directly, then address a related inefficiency — the mismatch between the modalities a model was pretrained on and the modalities actually available at inference time, where standard practice discards the extra modalities.

Key Contributions

1. MMEarth-Bench dataset. Five new multimodal environmental monitoring tasks — aboveground biomass, soil nitrogen, soil organic carbon, soil pH, and species occurrence — each with 12 aligned modalities and globally distributed data, providing both random (in-distribution) and geographic test splits.

2. A systematic benchmark. Eight recent pretrained models spanning RGB-only, Sentinel-2-only, and multimodal architectures, plus randomly initialized baselines, are evaluated for low-shot transfer and geographic generalization.

3. TTT-MMR: test-time training with multimodal reconstruction. A model-agnostic method that repurposes all task modalities available at test time as reconstruction auxiliary tasks, using their (per-modality normalized, then averaged) gradients as an adaptation signal for the encoder.

4. TTT-MMR-Geo: geographic batching. A variant that batches test data by geographic proximity via recursive spatial partitioning, trading some regularization for specialization, which proves especially valuable for long-tail samples.

Main Findings

• Multimodal inputs help, pretraining helps more at low shot but not always. Models taking multimodal input generally outperform unimodal models, and especially RGB-only models. However, a randomly initialized multimodal model (ConvNeXtV2A-MM) is generally on par with pretrained multimodal models, ranking 1st when averaging rankings across tasks at 100% training data, and still ranking 4th overall with only 5% training data. Its performance relative to pretrained multimodal models does worsen as training data shrinks.

• MPMAE is competitive despite being Sentinel-2 only. The S2-only MPMAE model shows competitive or superior performance to multimodal models on the biomass and soil organic carbon tasks.

• Some tasks are highly data-hungry. Biomass and soil organic carbon are the most sensitive to limited training data; on soil organic carbon, none of the models achieves a positive R² with only 5% of the training data.

• Domain-specific RGB pretraining gave little benefit here. Comparing DINOv3 Web and DINOv3 Sat, the domain-specific pretraining of DINOv3 Sat had little benefit on these tasks.

• A geographic generalization gap remains. For all tasks except biomass, models perform significantly worse on the geographic (Africa) test split than on the random test split, despite global pretraining. Soil nitrogen and soil organic carbon remain difficult even for multimodal models on the geographic split. The randomly initialized ConvNeXtV2A-MM ranks 4th overall on the geographic split, ahead of Galileo.

• Multimodal finetuning is not always better than unimodal finetuning. While multimodal pretraining tends to improve geographic generalization, finetuning the same multimodally pretrained model with multi- rather than unimodal inputs can harm generalization, with exceptions in the geographic split of the soil tasks — an effect appearing more strongly for TerraMind and Galileo than for a Siamese Copernicus-FM.

• TTT improves every model on every task and both splits. Both TTT-MMR and TTT-MMR-Geo improve over joint training on all tasks and both test splits, with larger gains on the geographic split for biomass and soil organic carbon. One-sided (greater) Wilcoxon tests with Holm-Bonferroni correction confirm significance for each boxplot at p < 0.05.

• Geographic batching is usually best. TTT-MMR-Geo is the best method for all models except SatlasNet, Galileo, and ConvNeXtV2A-MM — these may benefit from the extra regularization of random batching. RGB-only models tend to gain the most from TTT. MPMAE, despite having been pretrained by reconstructing the same modalities, shows no special behavior in its TTT gains.

• Long-tail performance favors geographic batching. As sample frequency decreases, joint training and TTT-MMR increasingly underestimate target values, whereas TTT-MMR-Geo is robust to sample rarity. For species, the three methods are more similar overall, but on the random split TTT-MMR-Geo gives greater gains for rarer species.

Methodology in Plain English

The authors assembled five datasets covering biomass, three soil properties, and species occurrence. For every 128×128-pixel tile — each pixel representing 10 m on the ground, so a tile spans roughly 1.6 km² — they gathered 12 modalities: six pixel-level (Sentinel-2, Sentinel-1, ASTER GDEM, ETH Global Canopy Height, Dynamic World, ESA WorldCover) and six tile-level (ERA5 precipitation and temperature, geolocation, Sentinel-2 date, biome, ecoregion). Eight modalities are continuous and four categorical, all accessed through Google Earth Engine. Each task's tiles are split so that all African tiles form the geographic test set, with the rest of the world randomly divided 70% / 15% / 15% into train, validation, and random test sets; 50% and 5% training subsets are also provided, the smaller nested inside the larger.

They then finetune eight pretrained models plus two randomly initialized ConvNeXtV2A baselines using a linear task decoder, training on 5%, 50%, and 100% of the training tiles, and compare performance (R² for regression, mean average precision for species) on the random and geographic test sets.

The proposed method, TTT-MMR, works in two stages. First, joint training: the encoder is trained with the main task decoder for the task loss and a task modality decoder that reconstructs every task modality from the encoder's embedding. Then, at test time, the reconstruction losses for all modalities serve as an adaptation signal. The gradients of each modality's reconstruction loss with respect to the encoder are normalized separately and then averaged, so each modality contributes equally regardless of scale, and the task modality decoder is frozen to force the encoder itself to adapt. The encoder is updated for a few iterations (batch size 8, learning rate 10⁻², up to 5 iterations) before producing a prediction, then reset to its post-joint-training state for the next batch. The number of iterations to use is chosen by running every batch of the validation set and taking the mean of the best per-batch iteration count. TTT-MMR-Geo differs only in how batches are formed: by recursive spatial partitioning, mimicking a k-d tree, so that each batch is a contiguous geographic region. Experiments ran on an NVIDIA H200 140GB GPU, and shaded regions reflect one standard error across three random seeds.

Why This Matters

The paper shows that the field's central promise — globally pretrained, multimodal EO models that transfer to new tasks and new regions with few labels — is only partially realized today, and it offers a practical, architecture-agnostic way to squeeze more out of models without retraining or changing their inputs. Its insistence on geographic test splits and on a randomly initialized multimodal baseline raises the bar for how geospatial models are evaluated.

Real-world applications grounded in the paper's tasks and framing:

• Planetary health and sustainability monitoring, including tracking progress toward the Sustainable Development Goals.
• Disaster response, where Earth observation data is used to respond to natural disasters, often in regions with sparse reference data.
• Soil and agricultural management, using soil nitrogen, organic carbon, and pH estimates to inform land use.
• Biodiversity and conservation, using the species occurrence task built from terrestrial mammal range data.

Industry relevance: Companies and agencies building geospatial analytics pipelines, remote sensing foundation models, and carbon or land-cover products face exactly the problems this benchmark measures — sparse field labels, geographically biased training data, and sensors that provide more modalities than a pretrained model accepts. A model-agnostic adaptation method that works with any architecture and loss, including transformers without batch normalization, is directly deployable in such pipelines.

Future Directions

• Closing the geographic gap on soil properties. Soil nitrogen and soil organic carbon remain difficult on the geographic split even for multimodal models, and the authors state that all the soil tasks and species occurrence offer an opportunity for globally pretrained models to improve methodological geographic generalization. No solution is demonstrated here.

• Preventing overfitting to the labeled region. The authors hypothesize that finetuning on the world without Africa may erase information pretrained models encoded about Africa, and that additional modalities can cause overfitting to the non-Africa training domain. How to retain or restore that pretrained regional knowledge is left open.

• Explaining why geographic batching helps. Geographic batching is shown empirically to improve long-tail performance and is argued to sharpen gradients, but the paper offers this as an interpretation rather than a mechanistic study; how best to choose batching for arbitrary distributions remains an open question.

• Continuing to broaden benchmarks. The paper notes design choices to facilitate future development of multimodal models — shared modalities across all tasks, climate data included, and novel datasets rather than reused ones — but the current benchmark is limited to five tasks and to single-timestamp data, since it is not designed for time-series modeling or for producing the next best task-specific model.

Target Audience

Researchers and practitioners working at the intersection of machine learning and Earth observation: those developing or evaluating pretrained geospatial models, studying self-supervised and multimodal representation learning, or working on test-time adaptation and domain shift. It is also relevant to applied teams in remote sensing, conservation, agriculture, and climate who need models to work in regions where labeled reference data is scarce. The benchmark design choices and the model-agnostic adaptation method make it useful to readers who care about evaluation methodology as much as about any single model.

Note: the provided paper content is truncated within the Conclusion section, so the final sentences of the paper are not reflected above.

Authors’ abstract

Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data. However, most benchmark datasets have few data modalities and poor global representation, limiting the ability to evaluate multimodal pretrained models at global scales. In order to fill this gap, we introduce MMEarth-Bench, a collection of five new environmental tasks with 12 modalities, globally distributed data, and both random and geographic test splits. We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor. Moreover, a simple randomly initialized multimodal model is competitive given enough labeled data. Although data is abundant, models can currently only make use of the modalities on which they were pretrained. To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTT-MMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which is especially beneficial for long-tail distributions. Our dataset, code, and visualization tool are linked on the project page: lgordon99.github.io/mmearth-bench.