SARMAE: Masked Autoencoder for SAR Representation Learning

AI summary

Overview

Research area: Computer vision / remote sensing — self-supervised representation learning for Synthetic Aperture Radar (SAR) imagery.

Technical level: Advanced. The paper assumes familiarity with masked autoencoders, Vision Transformers, self-supervised pretraining, and the statistical physics of SAR speckle noise.

Scope: This paper introduces SAR-1M (a 1.3M-image SAR pretraining corpus with paired optical data) and SARMAE, a noise-aware masked autoencoder that combines physically grounded speckle corruption with cross-modal semantic alignment to achieve state-of-the-art SAR classification, detection, and segmentation.

What This Paper Is About

SAR imagery works day and night and through cloud cover, but it is expensive to acquire and inherently grainy due to speckle noise, so models trained on SAR data struggle to learn rich, transferable features. Existing pretraining datasets are small (180k–560k images) and borrowed optical-imagery recipes ignore the physics of SAR noise. The authors build a million-scale SAR dataset and a pretraining framework that explicitly models speckle statistics while using paired optical imagery as a semantic teacher.

Key Contributions

1. SAR-1M dataset — The first million-scale SAR pretraining corpus, aggregating 18 public datasets into 1.3 million SAR images plus roughly 1 million geographically aligned optical counterparts (2.3M samples total). It spans 57 categories, five sensors (Sentinel-1, Gaofen-3, RadarSat-2, Terma A/S, TerraSAR-X), four frequency bands (C/X/Ku/Ka), multiple polarizations, and resolutions from 0.1 m to 60 m.

2. Speckle-Aware Representation Enhancement (SARE) — A pretraining objective that corrupts input patches with physically grounded noise (Gamma-distributed multiplicative speckle, plus Rayleigh, Gaussian, and Uniform variants) and trains the model to reconstruct the clean original, forcing the encoder to learn noise-invariant features.

3. Semantic Anchor Representation Constraint (SARC) — A cross-modal alignment loss that pulls SAR patch embeddings toward corresponding patch embeddings from a frozen DINOv3 optical encoder via cosine distance, injecting semantic structure that SAR data alone does not provide.

4. State-of-the-art results across three task families — Classification, horizontal and oriented object detection, and semantic segmentation, validated with extensive ablations isolating each component.

Main Findings

• SAR-1M pretraining beats generic and optical-domain pretraining. A standard MAE pretrained on SAR-1M reaches 82.22% on FUSAR-SHIP versus 75.40% for ImageNet-pretrained MAE and 80.16% for MillionAID-pretrained MAE, with similar gains in detection and segmentation.

• SARE delivers large, consistent gains. Adding speckle-aware reconstruction to the SAR-1M baseline raises FUSAR-SHIP classification from 82.22% to 86.80% (+4.58%) and AIR-PolSAR-Seg mIoU from 64.36 to 65.15. Attention maps show the model attending to subtle objects that baseline models miss.

• SARC is especially valuable for detection. Adding optical alignment pushes SSDD detection from 64.40 to 68.10 mAP (+3.7%). The authors attribute this to reduced false alarms — models trained on SAR alone tend to mistake speckle for targets. Notably, simply fine-tuning DINOv3 on SAR yields poor results (74.25% FUSAR, 61.60 mAP SSDD), showing the benefit comes from explicit alignment, not from the optical backbone's strength.

• Classification performance. SARMAE achieves 89.30% (ViT-B) / 90.86% (ViT-L) on FUSAR-SHIP 40-shot, 92.92% on FUSAR-SHIP 30% labels, 96.70%/97.24% on MSTAR 40-shot, 99.61%/98.92% on MSTAR 30%, and 95.06%/95.63% on SAR-ACD. The 30% FUSAR result exceeds the prior best by 21.01 points.

• Detection performance. 57.9 mAP (ViT-B) on SARDet-100k with a plain ViT backbone, edging out SARATR-X's 57.3 mAP built on the architecturally stronger HiViT. On oriented detection (RSAR), SARMAE reaches 66.8% (ViT-B) and 72.2% (ViT-L), with ViT-L adding +5.4 mAP over ViT-B.

• Segmentation performance. On AIR-PolSAR-Seg, SARMAE achieves 65.87/65.84 mIoU for multiclass (versus roughly 48 from prior methods) and 92.31/93.06 IoU for water extraction, with notably large gains on the Industrial Area class (29.20 vs ~1.0 for baselines).

• Data scale matters but the method matters more. Training on only 30% of SAR-1M (300k images), SARMAE-S still outperforms SUMMIT (560k images) on both FUSAR_30% (88.27 vs 71.91) and SAR-ACD (93.76 vs 84.25).

• Training details. 300 epochs, AdamW with lr 1e-3 and weight decay 0.05, batch size 1024, cosine schedule, on NVIDIA A800 GPUs. SARC weight λ = 0.1. Noise injection is applied probabilistically at 50% per iteration, not to every sample.

Methodology in Plain English

The framework has two branches. The SAR branch is a standard masked autoencoder: 75% of image patches are hidden, and a ViT encoder plus a small decoder must reconstruct the missing content. The optical branch is a frozen DINOv3 network that never updates.

The twist is what the SAR branch is asked to reconstruct. Rather than reconstructing the already-noisy input, the researchers first corrupt the input further using a statistical model of how speckle actually arises physically — speckle comes from many independent radar echoes averaging together within a single pixel, which mathematically produces a Gamma distribution whose shape depends on the "number of looks." By sampling a noisier version with an artificially low look number and asking the model to recover the original, the encoder is forced to learn which fluctuations are noise and which are real scene content. Four noise families are used in rotation (Gamma, Rayleigh, Gaussian, Uniform) so the model generalizes across noise types.

For images that have a matching optical counterpart, an extra loss is applied. Optical patches carry much clearer semantic structure than SAR patches of the same location. The model computes embeddings for visible SAR patches and compares each one, using cosine distance, to the frozen DINOv3 embedding of the spatially corresponding optical patch. Training minimizes that distance, effectively transferring optical semantic organization into the SAR feature space. Images without optical pairs simply skip this loss.

The final pretraining objective adds these two losses together with a small weight on the alignment term. The resulting encoder is then fine-tuned separately for classification (linear probe on pooled features), detection (integrated into Faster R-CNN or Oriented R-CNN), and segmentation (UperNet).

Why This Matters

Research impact. This is the first million-scale SAR pretraining resource paired with optical imagery, which shifts the SAR community from task-specific small-model training toward foundation-model pretraining. The SARE idea — treating the noise process itself as a training signal rather than an obstacle — is a general template applicable to other noisy sensing modalities (sonar, ultrasound, low-light imaging). The SARC result also provides clean evidence that cross-modal distillation helps even when the teacher performs poorly on the target modality alone.

Real-world applications.

• Maritime surveillance: ship detection, illegal fishing monitoring, and port activity tracking independent of weather or daylight.
• Disaster response: rapid flood, earthquake, and landslide mapping where cloud cover blocks optical satellites.
• Defense and security: aircraft, vehicle, and infrastructure recognition in contested or obscured environments.
• Environmental and agricultural monitoring: land-cover classification and water body extraction over large regions.

Industry relevance. Any organization operating SAR satellites or SAR analytics (commercial remote sensing firms, national mapping agencies, maritime authorities, defense contractors) can use the released dataset, code, and models as a starting point, reducing the labeling cost that currently bottlenecks SAR product development. The finding that a plain ViT backbone can beat specialized hierarchical architectures suggests deployment simplicity is not sacrificed for accuracy.

Future Directions

• Scaling behavior beyond 1M images. The paper shows 300k already beats larger competitors; whether SAR follows optical scaling laws, and where returns diminish, remains open. Larger and more diverse SAR corpora would test this.

• Better use of the optical pairing. SARC currently uses a single frozen teacher and a simple cosine loss on visible patches. Alternatives such as contrastive objectives, masked-region alignment, or jointly trained cross-modal teachers could extract more from the 1M optical pairs.

• Generalizing to other SAR modalities and bands. The dataset covers C/X/Ku/Ka but the framework has not been evaluated on polarimetric or interferometric SAR products, which carry additional physical information the current pipeline discards.

• Reducing the optical dependency. Roughly a quarter of SAR-1M lacks optical pairs, so the framework cannot exploit SARC there. Methods that learn comparable semantic anchors without paired data would broaden applicability to regions never imaged optically.

• Task-specific fine-tuning efficiency. All downstream evaluations rely on full fine-tuning; whether SARMAE features support parameter-efficient adaptation (adapters, LoRA, few-shot prompting) is an unresolved practical question.

Target Audience

Researchers and graduate students in remote sensing, computer vision, and multimodal representation learning; engineers building SAR-based detection or segmentation pipelines who need strong pretrained backbones; and practitioners in defense, maritime, disaster response, and geospatial analytics who want to understand what foundation-model pretraining can now deliver for radar imagery. Readers without background in self-supervised learning or SAR physics will find the methodology sections dense, though the motivation and results are accessible.

Authors’ abstract

Synthetic Aperture Radar (SAR) imagery plays a critical role in all-weather, day-and-night remote sensing applications. However, existing SAR-oriented deep learning is constrained by data scarcity, while the physically grounded speckle noise in SAR imagery further hampers fine-grained semantic representation learning. To address these challenges, we propose SARMAE, a Noise-Aware Masked Autoencoder for self-supervised SAR representation learning. Specifically, we construct SAR-1M, the first million-scale SAR dataset, with additional paired optical images, to enable large-scale pre-training. Building upon this, we design Speckle-Aware Representation Enhancement (SARE), which injects SAR-specific speckle noise into masked autoencoders to facilitate noise-aware and robust representation learning. Furthermore, we introduce Semantic Anchor Representation Constraint (SARC), which leverages paired optical priors to align SAR features and ensure semantic consistency. Extensive experiments across multiple SAR datasets demonstrate that SARMAE achieves state-of-the-art performance on classification, detection, and segmentation tasks. Code and models will be available at https://github.com/MiliLab/SARMAE.