Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration

AI summary

Overview

Research area: Multi-modal machine learning for Earth Observation (EO), specifically multi-modal co-learning where several sensor modalities are available during training but only one is available at inference time (the "all-but-one missing modality" setting).

Technical level: Intermediate. The framework is built from standard components (convolutional encoders, contrastive learning, cross-entropy and squared-error losses), but it combines four loss terms and three disentangled feature spaces, so some familiarity with representation learning is helpful.

Scope: The paper proposes MDiCo (Multi-modal Disentanglement for Co-learning), a task-agnostic framework that improves single-modality EO models by exploiting additional sensor modalities during training, and evaluates it on four EO benchmarks covering binary, multi-class, and multi-label classification plus regression.

What This Paper Is About

Remote sensing platforms cannot guarantee that the same sensors are available at training time and at deployment time, so a model trained on optical plus radar data may have to run on only one of them. Most existing EO co-learning methods are "dedicated": they decide in advance which modality will be missing and design the training around that choice, or they are tied to one downstream task. This paper builds a general framework that transfers knowledge between heterogeneous sensor modalities at the feature level, without committing ahead of time to which modality will be available at inference.

Key Contributions

1. A general multi-modal co-learning framework (MDiCo) that disentangles three feature spaces per modality: shared features (common across modalities), specific features (unique to a modality and task-discriminative), and unused features (unique to a modality but not task-discriminative, and discarded at inference).
2. A non-dedicated training strategy that makes no assumption about which modality is present at inference, in contrast to dedicated training approaches that target a predefined missing modality.
3. A four-term loss design combining a main predictive loss, an auxiliary predictive loss over shared and specific features, a contrastive InfoNCE loss for shared features, and a modality discriminant loss for specific and unused features, summed without weights.
4. A broad empirical assessment against competitors from general machine learning, computer vision, and four EO-specific methods, on four EO benchmarks spanning different sensor combinations, tasks, and single-modality inference scenarios.

Main Findings

• Binary crop classification (CropH-b, F1): MDiCo reaches 73.5 with Sentinel-1 only and 83.3 with Sentinel-2 only, exceeding all listed competitors. The next best on Sentinel-1 are DML and ESensI at 71.7 (the paper's underlined second best), and on Sentinel-2 the multi-modal fusion baseline MMGF reports 82.3 and AnySat reports 82.4.
• Multi-class crop-type classification (CropH-m, F1): MDiCo reaches 58.3 with Sentinel-1 and 74.2 with Sentinel-2. The second-best Sentinel-1 results are DisCoM-KD and ESensI at 55.6; the second-best Sentinel-2 result is AnySat at 73.8, and the multi-modal fusion baseline MMGF reports 73.3.
• Multi-label tree species classification (TSAITS, F1): MDiCo reaches 66.4 with the aerial image and 66.3 with Sentinel-2. AnySat reaches the highest listed Sentinel-2 score at 74.8, and DML is the strongest aerial competitor at 65.2 with FCoM-av at 65.9 (underlined second best); the multi-modal fusion baseline OmniSAT reports 73.3 and MMGF 68.6.
• Regression (LFMC, R²): DeCuR reaches 0.248 (marked best in the Sentinel-1 column) and 0.417 (underlined second best) with Landsat-8, while the multi-modal fusion baseline InputFu reports 0.520. MDiCo's own LFMC values are not present in the available content, which is truncated at this table.
• EO-specific methods tend to outperform generic vision methods on the EO classification benchmarks. The paper attributes this to the heterogeneous nature of EO data, which differs from standard natural-image benchmarks.
• Missing-modality fusion methods generalize poorly. ShaSpec and EmbraceNet produce clearly low results on TSAITS and LFMC; on TSAITS, ShaSpec only predicts a no-label pattern (marked with a dagger). The authors state that fusion methods designed for missing-modality settings are not sufficiently robust or generic for single-modality prediction.
• Consistency claim: the authors report systematic improvement of MDiCo across validation scenarios, positioning it as effective for multi-modal co-learning when all-but-one missing modality scenarios arise at inference.

Methodology in Plain English

Each modality gets its own encoder. One part of the encoder produces a shared representation, meant to capture what the modalities have in common. Another part splits the modality's unique information into a specific representation, which stays in the final prediction, and an unused representation, which carries modality-specific information not useful for the task and is dropped at inference. A per-modality prediction head concatenates the shared and specific representations to produce a prediction, so the same trained model can be deployed with either modality alone.

Training is guided by four losses added together without weights. The main predictive loss is the task loss (cross-entropy for classification, squared error for regression) applied to each modality's prediction. The auxiliary predictive loss applies a single shared prediction head to both the shared and specific features of both modalities, pushing task-relevant information into both spaces. The contrastive loss uses InfoNCE with cosine similarity and a temperature of 0.07 to pull the shared representations of the two modalities from the same sample together and push apart those from different samples. The modality discriminant loss trains classifiers to identify which modality a specific or unused feature came from, which keeps each modality's unique features distinguishable.

Evaluation uses four benchmarks. CropHarvest provides Sentinel-1 radar and Sentinel-2 optical satellite image time series at 10 m resolution for cropland binary classification (69,800 samples) and crop-type multi-class classification (29,642 samples, ten classes). TSAITS provides Sentinel-2 time series at 10 m plus a 0.2 m aerial image for multi-label identification of 15 tree species in Germany, with 38,520 training, 6,810 validation, and 5,044 test samples. LFMC provides Landsat 8 and Sentinel-1 time series at 250 m for vegetation moisture regression, with 2,578 samples.

All methods train with access to all modalities and are evaluated with a single modality. Classification is scored with Weighted F1, regression with R². TSAITS uses its predefined validation and test splits; CropH-b, CropH-m, and LFMC use standard 10-fold cross-validation, with results averaged over 5 runs. Multi-temporal modalities use a TempCNN backbone and the mono-temporal aerial image uses ResNet-50, each followed by a 128-unit linear projection layer with 20% dropout. Training uses Adam with a learning rate of 10⁻³, batch size 128, early stopping with patience 5, and 100 epochs. All prediction heads in MDiCo are single linear layers. Class imbalance is handled with per-class weighted cross-entropy, and the regression target is z-score normalized so the squared error does not dominate the modality loss.

Why This Matters

Research impact. The paper argues that dedicated training—fixing in advance which modality will be missing—is too restrictive for real remote sensing deployments, and that a single framework can serve many tasks and modality combinations. It also contributes a comparative study placing EO-specific co-learning methods against general machine learning and computer vision methods under a shared evaluation protocol.

Real-world applications.

• Crop type and cropland monitoring where optical imagery is blocked by cloud cover and only radar is available.
• Forest and tree-species mapping, where aerial imagery and satellite time series are rarely both obtainable for the same area and period.
• Wildfire risk assessment through live fuel moisture estimation, relevant to the western United States data used here.
• Any operational pipeline affected by sensor outages, such as the Landsat 7 ETM+ SLC-off problem after 2003, the Sentinel-1b satellite that stopped operating at the end of 2021, or the NAIP satellite that operates only in the United States, all cited by the authors.

Industry relevance. Satellite operators, agricultural technology companies, and geospatial analytics providers often train on multi-sensor archives but must serve models on limited or variable sensor availability. A framework that keeps single-modality inference models strong without retraining per scenario reduces engineering overhead and makes deployments resilient to sensor degradation, launch delays, and cost-driven modality constraints.

Future Directions

• Results for MDiCo on the LFMC regression benchmark, and the full competitor comparison there, are not included in the available content, so the regression claim remains to be verified against the published table.
• The paper's ablation studies are announced in Sections 4.5 and 4.6, including a comparison against an adaptive loss weighting scheme instead of the uniform sum, but their outcomes are not reported in the available content.
• Section 4.7 examines alternative encoder backbones beyond TempCNN and ResNet-50; the reported numbers are not in the available content.
• Sections 4.8 and 4.9 analyze the internal representations learned by MDiCo and the evolution of individual losses during training; these analyses are not reported in the available content.
• The framework is presented for a two-modality setup and does not assume a fixed encoder shape; extension to more than two modalities is not described in the available content.

Target Audience

Researchers and practitioners in remote sensing and Earth Observation who work with multi-sensor data and face missing-modality constraints at deployment. It is also useful for machine learning researchers interested in co-learning, contrastive representation learning, and feature disentanglement, and for applied engineers building geospatial pipelines that must tolerate sensor unavailability. Readers need basic familiarity with deep learning training loops and evaluation metrics to follow the method, but the framework itself is described without heavy theory.

Authors’ abstract

Multi-modal co-learning is emerging as an effective paradigm in machine learning, enabling models to collaboratively learn from different modalities to enhance single-modality predictions. Earth Observation (EO) represents a quintessential domain for multi-modal data analysis, wherein diverse remote sensors collect data to sense our planet. This unprecedented volume of data introduces novel challenges. Specifically, the access to the same sensor modalities at both training and inference stages becomes increasingly complex based on real-world constraints affecting remote sensing platforms. In this context, multi-modal co-learning presents a promising strategy to leverage the vast amount of sensor-derived data available at the training stage to improve single-modality models for inference-time deployment. Most current research efforts focus on designing customized solutions for either particular downstream tasks or specific modalities available at the inference stage. To address this, we propose a novel multi-modal co-learning framework capable of generalizing across various tasks without targeting a specific modality for inference. Our approach combines contrastive and modality discriminative learning together to guide single-modality models to structure the internal model manifold into modality-shared and modality-specific information. We evaluate our framework on four EO benchmarks spanning classification and regression tasks across different sensor modalities, where only one of the modalities available during training is accessible at inference time. Our results demonstrate consistent predictive improvements over state-of-the-art approaches from the recent machine learning and computer vision literature, as well as EO-specific methods. The obtained findings validate our framework in the single-modality inference scenarios across a diverse range of EO applications.