GeoCR: Learning a Generalist Cloud Removal Prior from Heterogeneous Observations

AI summary

Overview

• Research area: Remote sensing computer vision — cloud removal (CR) from satellite imagery, spanning generative modeling, multi-sensor fusion, and transfer across heterogeneous datasets.
• Technical level: Advanced. The paper assumes familiarity with latent autoencoders, flow matching / diffusion-style generative models, parameter-efficient fine-tuning (LoRA), and remote sensing sensor configurations (Sentinel-2, Landsat-8, Sentinel-1 SAR, TOA/BOA reflectance).
• Scope: The paper introduces GeoCR, a single generalist model pretrained on ten cloud removal datasets (883,331 cloud-free target images) that handles RGB-only and multispectral cloud removal from single- or multi-temporal cloudy observations with optional SAR guidance, evaluated on five primary benchmarks.

What This Paper Is About

Cloud removal methods are usually built and trained for one dataset and one input configuration, so a model tuned for, say, three-frame Sentinel-2 multispectral input cannot be reused when only a single RGB image or SAR guidance is available. GeoCR asks whether one pretrained checkpoint can learn a shared cloud removal prior that covers RGB-only and multispectral reconstruction, from single- or multi-temporal observations, with or without SAR. The goal is a reusable generative model that works directly on new datasets without per-dataset fine-tuning, and that can optionally be adapted cheaply with LoRA.

Key Contributions

1. A generalist cloud removal model. GeoCR unifies RGB-only-based CR and multispectral-based CR across single- and multi-temporal observations with optional SAR guidance inside a single network, which the authors state is the first time this question has been addressed.
2. A unified embedding and conditioning framework. Compact input and output stems extend frozen pretrained RGB encoder–decoder trunks, and observation-specific conditioning streams (each cloudy frame separately, plus optional SAR) are combined within one flow transformer.
3. A consolidated pretraining corpus. Ten CR datasets are pooled into a joint pretraining corpus of 883,331 cloud-free target images spanning Sentinel-2, Landsat-8, and high-resolution imagery, with RGB and multispectral targets, single- and multi-temporal cloudy inputs, and optional SAR.
4. State-of-the-art results across settings. GeoCR achieves the best FID and DISTS on three of five primary benchmarks (full-band SEN12MS-CR and Sen2_MTC_New, RGB-only CUHK-CR2), with the same checkpoint used across all five benchmarks in the no-fine-tuning mode.

Main Findings

• Best FID and DISTS on three benchmarks: On full-band SEN12MS-CR, GeoCR (w/o FT) reaches FID 28.4 and DISTS 0.202 versus the strongest competing baseline (EMRDM: FID 75.7, DISTS 0.274). On full-band Sen2_MTC_New, GeoCR (w/o FT) reaches FID 52.9 and DISTS 0.182 versus GACR (FID 87.4, DISTS 0.223). On RGB-only CUHK-CR2, GeoCR (w/o FT) reaches FID 93.6 and DISTS 0.153 versus EMRDM (FID 104.4, DISTS 0.167).
• Large relative FID reduction on SEN12MS-CR: The paper reports that GeoCR (w/o FT) reduces FID by over 60% relative to the strongest competing baseline on full-band SEN12MS-CR.
• Direct inference works without dataset-specific fine-tuning: GeoCR (w/o FT) uses one jointly pretrained checkpoint across all five primary benchmarks, and the qualitative improvements are already visible in this mode.
• LoRA gives modest, selective gains: LoRA adaptation (23.1M parameters, 0.60% of the model) improves selected datasets — for example Sen2_MTC_New full-band FID 52.7 versus 52.9 without fine-tuning — but the paper states it provides selective improvements rather than consistent gains over direct inference.
• Not a clean sweep: On RGB-only SEN12MS-CR, GeoCR leads FID but pix2pixHD retains lower DISTS and LPIPS. On T-CLOUD, CUHK-CR1, and WHUS2-CRv (Appendix), specialized baselines retain advantages in some metrics: EMRDM leads WHUS2-CRv native and CUHK-CR1 perceptual/pixel metrics, and HI-Diff leads WHUS2-CRv RGB-only.
• Stem training is what makes non-RGB routes work: With trunks frozen, stem training raises reconstruction PSNR by 21.24 dB for ten-band non-RGB inputs and 10.74 dB for SAR. The RGB path is unchanged (+0.00 dB) because it already reconstructs at 45.78 dB.
• The NIR route needs no stem training: The initialized single-band NIR stem achieves 42.06 dB on Sen2_MTC_New, 40.41 dB on CUHK-CR, and 49.51 dB on Sen2_MTC_Old without further optimization.
• More temporal frames help the same checkpoint: On Sen2_MTC_New with GeoCR (w/o FT), PSNR/SSIM rise from 16.390/0.4595 with 1 cloudy frame to 19.319/0.5963 with 2 and 20.522/0.6568 with 3.
• SAR guidance helps: On SEN12MS-CR with identical cloudy optical inputs, adding SAR raises PSNR from 28.596 to 29.442 and SSIM from 0.8572 to 0.8703.
• Robust across cloud coverage: Using coverage groups estimated with s2cloudless on SEN12MS-CR, both GeoCR variants show lower DISTS and LPIPS than compared baselines; in nearly clear scenes GeoCR improves over the unrestored input while all compared baselines worsen DISTS.
• Failure cases remain: When all optical observations are heavily obscured, generated details may look plausible but differ from the reference; SAR adds structural evidence but does not determine a unique optical appearance, and fine structures under complete cloud remain hard to recover.

Methodology in Plain English

GeoCR is built in two pretraining stages on top of a frozen pretrained RGB autoencoder (FLUX.2 VAE) and a pretrained image flow transformer (FLUX.2 [klein] 4B Base).

Stage 1 — teaching the frozen codec new sensors. Optical bands differ spectrally but share spatial structure such as scene layout and object boundaries, and SAR provides complementary structural evidence through a different sensing mechanism. So instead of retraining the encoder and decoder, the authors attach a single convolutional input stem and a single convolutional output stem for each non-RGB configuration (ten-band TOA, ten-band BOA, dual-polarization SAR) and train only those, using an ℓ1 reconstruction loss on each route's own observation. The pretrained encoder and decoder trunks and the original RGB path stay frozen. The added stems contain 53.5K parameters, of which 51.1K are trained; only 51.1K stem parameters are optimized in this stage.

Stage 2 — learning one cloud removal prior. Each cloud-free target and each cloudy observation is encoded into the shared latent space: RGB and non-RGB latents are concatenated along channels into a joint multispectral representation, and a missing non-RGB component is replaced by a zero latent so RGB-only and multispectral samples share one input layout. SAR, when present, is encoded separately. A flow transformer is then trained with conditional flow matching: an independent Gaussian source is linearly interpolated toward the target latent, and the generator predicts the corresponding velocity, conditioned on the available observations. The noisy target, each of the K cloudy frames, and the optional SAR input form separate token streams (1 + K, plus one when SAR is available), with separate parameter sets for target, optical-conditioning, and SAR tokens and parameters shared across the K optical-conditioning streams; double-stream blocks let streams interact through joint attention, and subsequent single-stream blocks process the combined sequence. Only target tokens reach the final velocity head.

Inference and adaptation. One shared Stage 2 generator is used with all autoencoder trunks and stems frozen. Either it is used directly (GeoCR w/o FT) or adapted per dataset with LoRA on all 80 transformer linear layers. Inference integrates the learned flow from Gaussian noise and decodes the resulting latents through the frozen decoder routes, using only four Euler steps on a uniform time grid without classifier-free guidance and a fixed per-sample noise seed.

Why This Matters

Impact on research. The paper reframes cloud removal from a per-dataset engineering problem into a transfer problem: heterogeneous sensors, band sets, temporal coverage, and SAR availability are unified through one latent interface rather than through separate models. It provides a concrete recipe — freeze the codec, adapt lightweight stems, share a single generative prior, adapt optionally with LoRA — and a consolidated ten-dataset, 883,331-target corpus that supports the claim that a shared CR prior is learnable. It also documents where generalist performance still trails specialized baselines, giving the field clear targets.

Real-world applications:

• Agricultural and forestry monitoring, where consistent cloud-free optical time series are needed to track crop or canopy condition.
• Disaster response, where SAR can see through cloud but optical context is needed for interpretation.
• Land-cover and urban mapping, where cloud contamination interrupts regular revisit coverage.
• Climate and surface-change analysis, where long multisensor archives must be made temporally consistent.

Industry relevance. A single checkpoint that serves multiple sensor and band configurations reduces the cost of deploying cloud removal across different satellite products and archives, and LoRA adaptation (0.60% of parameters) offers a cheap per-customer or per-region customization path. The direct-inference mode matters operationally because it avoids retraining per new dataset.

Future Directions

• Recovering fine structure under complete occlusion. The paper's failure analysis shows that when every optical observation is heavily obscured, plausible-looking generated detail can diverge from the reference; improving reliability in this regime is an open problem.
• Making SAR guidance more decisive. SAR provides complementary structural evidence but does not determine a unique optical appearance, so better ways to exploit SAR for appearance reconstruction remain unexplored.
• Handling temporal surface change and photometric mismatch. The listed challenging cases include occlusion across all temporal observations, temporal surface change, fine urban structure, and photometric mismatch.
• Extending and balancing the shared prior. GeoCR remains competitive rather than best on T-CLOUD, CUHK-CR1, and WHUS2-CRv (EMRDM and HI-Diff lead in those settings), and LoRA yields only selective improvements, so broader sensor coverage and stronger adaptation are natural next steps.

Target Audience

Researchers and graduate students in remote sensing and generative vision, especially those working on multi-sensor fusion, image restoration, or foundation-style models for Earth observation. It is also relevant to practitioners building operational cloud removal pipelines across multiple satellite products, and to readers interested in how frozen pretrained generative backbones can be repurposed across heterogeneous sensing domains.

Authors’ abstract

Cloud removal methods are typically specialized to individual datasets and input configurations, limiting reuse across sensors, spectral bands, and observation settings. We introduce GeoCR, a generalist model that unifies RGB-only-based CR and multispectral-based CR from single- or multi-temporal cloudy observations, with optional SAR guidance, within a single network. To accommodate different spectral and sensing domains, compact input and output stems extend a pretrained RGB autoencoder while keeping its encoder and decoder trunks frozen. This shared latent interface enables a single flow transformer to jointly model clean RGB and non-RGB latents, conditioned on separate cloudy-observation streams and optional SAR tokens. Through joint pretraining on the training splits of ten datasets comprising 883,331 cloud-free target images, GeoCR learns a shared cloud removal prior across these heterogeneous configurations. The same pretrained checkpoint supports direct inference without dataset-specific fine-tuning and efficient adaptation through low-rank adaptation (LoRA). We evaluate GeoCR against general image restoration and cloud removal methods on test splits of the contributing datasets under full-band and RGB-only settings. GeoCR achieves the best FID and DISTS on full-band SEN12MS-CR and Sen2_MTC_New and RGB-only CUHK-CR2, outperforming existing models and demonstrating the effectiveness of a reusable generative model across diverse settings.