RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation

AI summary

Overview

• Research area: Computer vision and remote sensing — multimodal Earth observation foundation models.
• Technical level: Intermediate. The core idea is intuitive, but the architecture relies on Vision Transformer and masked-autoencoder concepts.
• Scope: The paper introduces RAMEN, a single sensor-agnostic transformer encoder that unifies heterogeneous Earth observation data and lets users choose the spatial resolution of the learned representations at inference time.

What This Paper Is About

Earth observation data is wildly inconsistent: aerial photos, multispectral satellite bands, radar polarizations, and elevation maps all differ in the number of channels, the ground sampling distance (how many meters each pixel represents), and how often they are captured. Existing foundation models either demand a fixed input resolution or rely on sensor-specific encoders, so applying them to a new sensor requires retraining parts of the architecture. RAMEN solves this by learning one shared representation across all of these modalities while treating spatial and temporal resolution as adjustable inputs — and, crucially, letting practitioners pick the output resolution to trade accuracy against compute cost.

Key Contributions

1. A resolution-adjustable, sensor-agnostic encoder. RAMEN processes imagery from any sensor configuration — optical, radar, elevation — without retraining, using a single shared transformer encoder with only lightweight modality-type projectors.
2. Inference-time control over spatial resolution. Spatial resolution is treated as a controllable output parameter, not a fixed property. Users select the target ground sampling distance at inference, explicitly trading computational cost against spatial precision.
3. A unified treatment of the three axes of heterogeneity. The architecture jointly addresses spectral (channel wavelengths), spatial (ground sampling distance), and temporal (acquisition dates) variability — a combination no prior Earth observation foundation model handles together.
4. State-of-the-art results on PANGAEA. A single pretrained model outperforms larger competitors across eight downstream segmentation tasks, with an average mIoU of 60.03 and the best average rank of 2.63.

Main Findings

• Beats larger models with a smaller backbone: RAMEN achieves 60.03 average mIoU on the 8-task PANGAEA benchmark, surpassing TerraMindv1-L (59.10) despite using a ViT-Base encoder instead of a Large one. It lands in the top two on 6 of 8 tasks.
• The best resolution is task-dependent: On HLS BurnScars (wildfire segmentation), coarser target resolutions actually improve accuracy because burned regions are large and homogeneous. On MADOS (marine pollutant detection), finer resolutions are essential for resolving small details.
• Favorable compute/performance trade-offs: On BurnScars, RAMEN reaches 85.02 mIoU at 817 GFLOPs, versus TerraMind-L's 82.93 mIoU at 980 GFLOPs. On Pastis, 33.26 mIoU at 359 GFLOPs captures about 80% of peak performance with roughly 7.4x faster inference.
• Unlocks detail-critical tasks: On AI4SmallFarms, RAMEN reaches 38.78 mIoU where all compared foundation models plateau below roughly 30, showing that access to finer resolutions enables qualitatively better results.
• Wavelength encoding matters: Progressively encoding accurate Sentinel-2 band central wavelengths instead of treating bands as generic channels yields a +4.6 mIoU gain on Sen1Floods11.
• Adjustable resampling helps most when scales diverge: The mixture-of-convolutions spatial resampler outperforms naive bilinear interpolation at every target resolution, with gains growing as the target resolution moves further from the native one.
• Multimodal fusion is free: Adding Sentinel-1 radar to Sentinel-2 imagery improves average mIoU from 62.72 to 64.27 without any modality-specific architecture changes.
• Temporal modeling is competitive: RAMEN's built-in temporal attention reaches 38.08 average mIoU on three time-series tasks, ahead of Prithvi (34.95) and SatlasNet (33.59), though optional late fusion pushes this to 45.14 on datasets with complex temporal dynamics.

Methodology in Plain English

The researchers built a single transformer encoder that converts every input into tokens, regardless of sensor type. Each input passes through three stages:

1. Channel conditioning. Every spectral band is described by its central wavelength and encoded as a sinusoidal signal; radar polarizations and elevation products get their own learned embeddings. A small network turns these descriptions into a projection matrix that maps raw channels into a shared latent space. Bands are therefore treated by what they physically measure, not by arbitrary channel indices.
2. Adjustable spatial resampling. Features are interpolated to a user-chosen target ground sampling distance. Because sensors differ by orders of magnitude in scale, a mixture of four small convolution "experts" — weighted by the log-scaled ratio between input and target resolution — corrects the feature statistics after interpolation.
3. Temporal attention. For time series, a lightweight attention encoder aggregates observations across dates, with positional encoding based on the actual acquisition day.

The pretrained model is trained with a masked image modeling objective: random patches of the unified multimodal token sequence are hidden, and the model must reconstruct the missing content at each modality's native spectral, spatial, and temporal resolution. Training samples random datasets, random modality subsets, and random target resolutions each iteration, forcing the model to generalize across scales rather than specialize. After pretraining, the encoder is frozen and only a lightweight segmentation decoder is fine-tuned per downstream task.

Why This Matters

Impact on research. Most Earth observation foundation models handle one axis of heterogeneity at a time — spectral, spatial, or temporal. RAMEN shows all three can be handled jointly in a single architecture without sacrificing performance, and that resolution itself can be a controllable interface rather than a fixed design constraint. It also demonstrates that a smaller, well-designed model can beat larger ones when the inductive biases fit the data.

Real-world applications:

• Disaster response: Coarse, fast inference for rapid wildfire or flood mapping when time is critical, with the option to refine locally where needed.
• Precision agriculture: Fine-resolution outputs to delineate individual field parcels from satellite time series.
• Environmental monitoring: Detecting small pollutant patches or marine debris that require resolving fine spatial detail.
• Land-cover and urban mapping: One model serving many mapping products across different sensors and desired output scales.

Industry relevance. The compute/performance dial is directly useful for operators who pay for inference. A single pretrained checkpoint that accepts new sensors and adjustable resolutions reduces engineering overhead for companies building geospatial analytics, and lowers the barrier to deploying Earth observation models on resource-constrained platforms.

Future Directions

• Adaptive resolution policies. Could the model learn to choose its own target resolution per region or per task, rather than requiring a user-specified value?
• Extending to new modality types. How well does the sensor-agnostic design transfer to hyperspectral, LiDAR, or thermal sensors beyond the optical, radar, and elevation families tested?
• Joint spectral and temporal adjustability. Spatial resolution is controllable at inference; whether spectral bands and temporal sampling could be made similarly adjustable remains open.
• Longer temporal sequences. Current temporal evaluation covers relatively short time series; scaling to years of dense revisit data raises questions about efficiency and attention design.
• Reducing reconstruction overhead. The quadratic cost of transformer attention still limits the finest resolutions, motivating efficient attention or token-merging strategies.

Target Audience

Researchers and graduate students in computer vision and remote sensing who work on foundation models, self-supervised pretraining, or multimodal representation learning. Also valuable for applied practitioners and engineers in geospatial analytics who need models that generalize across sensors and can be tuned for a specific accuracy-versus-cost budget. Readers without transformer familiarity will find the high-level motivation and benchmark results accessible, but the architectural details assume comfort with Vision Transformers and masked image modeling.

Authors’ abstract

Earth observation (EO) data spans a wide range of spatial, spectral, and temporal resolutions, from high-resolution optical imagery to low resolution multispectral products or radar time series. While recent foundation models have improved multimodal integration for learning meaningful representations, they often expect fixed input resolutions or are based on sensor-specific encoders limiting generalization across heterogeneous EO modalities. To overcome these limitations we introduce RAMEN, a resolution-adjustable multimodal encoder that learns a shared visual representation across EO data in a fully sensor-agnostic manner. RAMEN treats the modality and spatial and temporal resolutions as key input data features, enabling coherent analysis across modalities within a unified latent space. Its main methodological contribution is to define spatial resolution as a controllable output parameter, giving users direct control over the desired level of detail at inference and allowing explicit trade-offs between spatial precision and computational cost. We train a single, unified transformer encoder reconstructing masked multimodal EO data drawn from diverse sources, ensuring generalization across sensors and resolutions. Once pretrained, RAMEN transfers effectively to both known and unseen sensor configurations and outperforms larger state-of-the-art models on the community-standard PANGAEA benchmark, containing various multi-sensor and multi-resolution downstream tasks. Our code and pretrained model are available at https://github.com/nicolashoudre/RAMEN.