GeoSET: Generalist Foundation Model for SAR-to-EO Image Translation

AI summary

Overview

Research area: Computer vision / remote sensing — generative image-to-image translation between synthetic aperture radar (SAR) and electro-optical (EO) satellite imagery.

Technical level: Advanced. The paper assumes familiarity with latent diffusion, flow matching, transformer backbones, and parameter-efficient fine-tuning.

Scope: The paper introduces GeoSET, a single generalist SAR-to-EO translation model pretrained on over three million curated SAR–EO pairs and then adapted to six downstream benchmarks at low cost.

What This Paper Is About

SAR satellites image the Earth regardless of sunlight or cloud cover, but the resulting speckle-heavy imagery is hard for humans to read and unlike the natural images used to pretrain vision models. SAR-to-EO image translation (SET) converts SAR observations into intuitive EO-like pictures, but existing methods train a separate model from scratch on each small target dataset, so no reusable SET prior is ever learned. This paper asks whether one generalist SET model can be pretrained once on heterogeneous SAR–EO sources and then adapted many times across different sensors, resolutions, and regions under a common protocol.

Key Contributions

1. GeoSET, a generalist SET model under a pretrain-once, adapt-many framework. The authors describe it as the first generalist model for SAR-to-EO image translation, pretrained on over three million SAR–EO pairs curated from heterogeneous sources through source-aware filtering.
2. A two-stage pretraining strategy. Stage 1 learns a decoder-compatible, speckle-robust SAR representation; Stage 2 pretrains a SAR-conditioned flow model on the multi-source corpus.
3. A domain-aware speckle augmentation scheme. SAR inputs are perturbed with a Gamma-distributed multiplicative field routed according to each source's representation domain (linear intensity, linear amplitude, logarithmic measurement, display imagery, or unknown), applied during pretraining and adaptation and disabled at inference.
4. Extensive evaluation across six downstream benchmarks. GeoSET reaches state-of-the-art FID and DISTS results, while its LoRA variant updates only 0.60% of the generator parameters and remains competitive with full fine-tuning.

Main Findings

• Best FID on all six benchmarks, best DISTS on five. Across full fine-tuning and LoRA, GeoSET achieves the best reported FID on all six downstream benchmarks and the best DISTS on five.
• Large FID margin on SAR2EO. GeoSET with full fine-tuning reaches FID 24.5 versus 52.7 for the strongest competing baseline, cBBDM — a reduction of over 50%. Its DISTS is 0.181.
• LoRA is nearly as strong as full fine-tuning and sometimes better. On SpaceNet6, GeoSET (LoRA) achieves the strongest FID and DISTS (87.9 and 0.189) compared with GeoSET (full FT) at 94.1 and 0.192. On QXS-SAROPT, GeoSET (full FT) reaches FID 16.9 and DISTS 0.244 versus LoRA at 19.3 and 0.254; C-DiffSET retains the best DISTS there at 0.233.
• Other public benchmark numbers. SAR2Opt: GeoSET (full FT) FID 71.1 / DISTS 0.196, LoRA 74.6 / 0.200. QXS-SAROPT LoRA FID 19.3. On the two private datasets, the Appendix reports that GeoSET achieves the best FID, DISTS, KID, and DINO similarity on both CAP-BSG and KOMPSAT, with CAP-BSG full FT FID 58.9 and LoRA FID 64.7, and KOMPSAT full FT FID 69.8 and LoRA FID 72.0.
• Pixel-level metrics favor other methods. Competing methods often achieve stronger LPIPS, PSNR, or SSIM. The authors note that SAR backscatter does not uniquely determine optical color and texture, and residual misregistration can penalize otherwise plausible outputs, so they emphasize FID for distributional realism and DISTS for paired structural and textural similarity.
• Stage 1 encoder training improves SAR reconstruction. Average PSNR rises by 3.54 dB on original SAR inputs and 5.23 dB on perturbed inputs. The larger gain under perturbation supports speckle robustness; the frozen decoder's EO reconstruction PSNR averages 32.20.
• Cheap adaptation. LoRA updates 23.10M of 3.852B generator parameters, i.e. 0.5997%, cutting per-dataset storage from 30.8 GB to 185 MB, peak memory from 96.9 GB to 49.1 GB, and raising throughput from 67 img/s to 94 img/s. Wall time per dataset is 1.34 h for full fine-tuning and 0.98 h for LoRA.
• Every component helps. A cumulative comparison on SAR2EO and SpaceNet6 shows FLUX.2 LoRA at FID 71.2 / 117.4, adding corpus pretraining improves to 50.5 / 107.9, adding the SAR encoder to 39.8 / 94.2, and adding speckle augmentation yields the full GeoSET (LoRA) at 29.5 / 87.9.
• The model genuinely uses the SAR condition. With true SAR conditioning, PSNR and LPIPS beat both shuffled and null conditions on GUSO, SARLO-80, and TerraMesh (for example, GUSO PSNR 13.36 true versus 10.24 shuffled and 10.58 null).
• No consistent zero-shot transfer. The authors state that GeoSET does not consistently achieve strong zero-shot performance across unseen SAR–EO domains; its generalist capability lies in a reusable prior enabling efficient downstream adaptation.

Methodology in Plain English

The authors gathered SAR–EO pairs from five sources — GUSO, TerraMesh, SARLO-80, SAR-1M, and 3MOS — spanning different sensors, polarizations, resolutions, ground sampling distances, and regions. Starting from 10,114,514 SAR observations, they applied source-specific eligibility rules (for example, keeping TerraMesh pairs with zero cloud cover and an acquisition-time difference of at most one day, and SAR-1M samples with an available EO counterpart) to reach 3,281,393 eligible pairs, then removed pairs with featureless content, cloud contamination, misregistration, compression artifacts, missing tiles, or dead regions. The final corpus contains 3,204,744 pairs. Source sampling probabilities were set from counts of non-overlapping 256×256 crop equivalents rather than raw tile counts, with 3MOS tile counts divided by four to offset overlapping-tile redundancy.

Training proceeds in two pretraining stages. In Stage 1, a SAR encoder is initialized from a pretrained image autoencoder and trained to reconstruct the original SAR observation from a speckle-perturbed version of it, while the shared decoder stays frozen. Keeping the decoder fixed prevents the encoder and decoder from forming an unconstrained SAR-specific codec and pushes the SAR features to remain compatible with the pretrained latent space. This stage needs only SAR images, no paired EO data.

In Stage 2, a pretrained text-to-image flow transformer is repurposed: its text encoder is removed and the text-conditioning stream is replaced by a spatial SAR stream initialized from the corresponding pretrained image-stream parameters. The SAR and EO encoders and the decoder stay frozen, and the 3.852B-parameter generator is trained with conditional flow matching to move from Gaussian noise to the EO latent, using classifier-free guidance with a conditioning dropout probability of 0.1. At inference, the endpoint is decoded with the frozen decoder using 50 integration steps and two-pass CFG with a guidance weight of 2.0.

In Stage 3, the same generalist checkpoint is adapted to each benchmark either by full fine-tuning or by LoRA on the attention and MLP projections of both the double-stream and single-stream blocks, using rank 16 and scaling factor 16.

Why This Matters

Impact on research. Most SET work treats each dataset as an isolated task, so nothing accumulates. This paper shows that heterogeneous SAR–EO sources can be consolidated into a shared generative prior, and that adapting that prior costs roughly an hour per dataset while updating under 1% of the model. It also demonstrates a practical route for making natural-image-pretrained generators work with radar data by adapting the conditioning encoder rather than rebuilding the pipeline.

Real-world applications:

• Rapid visual triage during disasters, when cloud cover or darkness blocks optical satellites and only SAR is available.
• Flood, infrastructure, and land-cover monitoring, where an EO-like rendering helps analysts interpret SAR scenes without specialized radar training.
• Feeding downstream vision models that were pretrained on natural imagery, giving them a familiar input interface for SAR observations.
• Fusion of SAR and optical archives into consistent products across sensors and resolutions.

Industry relevance. The economics matter as much as the accuracy: 185 MB per adapted dataset instead of 30.8 GB, 49.1 GB peak memory instead of 96.9 GB, and roughly one hour of adaptation on one GPU make it feasible for an operator to maintain many dataset-specific variants from one shared parent rather than training and storing a separate full model for each region or sensor.

Future Directions

• Closing the zero-shot gap: GeoSET still does not transfer reliably to unseen SAR–EO domains, so stronger generalist behavior — or explicit mechanisms for handling unseen sensors, acquisition geometry, and radiometric processing — remains open.
• Reducing the reliance on paired target-domain data, since LoRA adaptation still requires paired SAR–EO data from the target domain.
• Narrowing the pixel-fidelity gap: competing methods retain advantages in LPIPS, PSNR, and SSIM, which the authors partly attribute to the ambiguity of SAR backscatter and residual misregistration rather than to generation quality alone.
• Extending the corpus and filtering pipeline to more sensors and domains, and testing whether the Speckle-robust encoder plus flow-matching recipe transfers to other cross-modal remote sensing translation tasks.

Target Audience

Researchers and engineers working on remote sensing, SAR–EO fusion, and conditional image generation; practitioners who need to deploy SAR interpretation tools across many sensors; and readers interested in how foundation-model pretraining and parameter-efficient adaptation (LoRA) apply to geospatial modalities rather than natural images. The paper is written for readers comfortable with latent generative models and diffusion-style training, so it is best suited to an advanced audience.

Authors’ abstract

Paired synthetic aperture radar (SAR) and electro-optical (EO) imagery is increasingly available across sensors, resolutions, and geographic regions. Yet existing SAR-to-EO image translation (SET) methods are typically trained on a single, limited-scale dataset, producing models specialized to particular sensing conditions. We introduce GeoSET, the first generalist model for SET, built around a single pretrained parent that is adapted to downstream datasets under a common protocol. We curate over 3 million high-quality SAR--EO pairs from a collection of more than 10 million SAR observations, spanning diverse sensors, spatial resolutions, and ground sampling distances. To bridge the modality gap between SAR observations and a pretrained image generator, we develop a speckle-robust SAR encoder and pretrain the conditional generator on this heterogeneous corpus. The resulting parent supports efficient adaptation across downstream datasets through low-rank adaptation (LoRA), updating only 0.60% of the generator parameters and requiring approximately one hour per dataset. Across six downstream benchmarks, GeoSET achieves state-of-the-art results in FID and DISTS with full fine-tuning or LoRA, demonstrating effective transfer across heterogeneous SAR-EO domains.