A comprehensive and trustworthy benchmark of AI methods for change detection in Earth observation

AI summary

Overview

• Research area: Deep learning for change detection in Earth observation (EO), with an emphasis on benchmarking methodology, trustworthy AI, and FAIR data/software practices.
• Technical level: Intermediate. Familiarity with CNNs, vision transformers, semantic segmentation metrics (mIoU, F1), and standard training protocols is helpful, but the paper is structured around experimental design rather than novel architecture derivations.
• Scope: The paper builds and releases a standardized, open-source benchmark that evaluates ten representative deep learning change detection architectures across ten heterogeneous EO change detection datasets under identical training and evaluation protocols, reporting predictive performance alongside computational efficiency.

What This Paper Is About

Change detection in EO — identifying semantic differences in a geographic area by comparing co-registered bi-temporal image pairs — has produced a rapid stream of new architectures (CNNs, vision transformers, state-space models), each claiming state-of-the-art (SOTA) performance. The authors argue that these claims are frequently built on inconsistent evaluation setups: non-standardized train/validation/test splits, varying patch resolutions, non-uniform loss functions, and disparate hyperparameter configurations, while computational overhead such as FLOPs, parameter counts, and memory footprints is largely unreported. The paper's goal is to eliminate this protocol bias by running ten model architectures across ten datasets in a single strictly controlled pipeline, and by releasing every artifact (splits, code, checkpoints, logs) under FAIR principles so results can be audited and reused.

Key Contributions

1. A strictly controlled experimental protocol. Fixed, reproducible dataset splits, unified data processing, unified training and evaluation pipelines, and standardized loss functions minimize setup discrepancies and evaluation bias across all models and datasets.
2. A comprehensive evaluation. Ten representative AI methods for change detection are benchmarked across ten heterogeneous datasets covering urban development, land-cover change, and post-disaster damage.
3. Study-design principles and practical guidance. The authors derive principles for training and evaluating AI methods for change detection and discuss common practical aspects affecting their application and performance.
4. A fully open-source, FAIR benchmarking framework. Released through the AiTLAS toolbox, including standardized data splits, data loaders, training scripts, schemas and ontology on GitHub, plus trained model checkpoints and TensorBoard logs on Hugging Face and Zenodo.

Main Findings

• Classical architectures remain highly competitive. SiamCRNN achieves the highest average mIoU when trained from scratch (0.7392), followed closely by ChangeViT (0.7381) and U-Net SiamConc (0.7224). Under pre-training, U-Net SiamConc leads the benchmark with an average mIoU of 0.7931, followed by SiamCRNN (0.7797) and ChangeViT (0.7709).
• Pre-training delivers a consistent boost at no inference cost. Transfer learning raises the benchmark-wide average mIoU from 0.6980 to 0.7283 across most architectures, with the authors noting this comes with no additional inference cost. ChangeFormerV6 and CSSM are exceptions: no pre-trained weights were available for them, so they were trained from scratch only.
• The pre-trained U-Net SiamConc ranks first on seven of ten datasets. It achieves the best result on BANDON (0.6114), CLCD (0.7521), DSIFN (0.8569), LEVIR-CD+ (0.7478), MSOSCD (0.8169), OMBRIA (0.7114), and Season-varying CDD (0.9407).
• SiamCRNN is the strongest model from scratch and the leading model on two datasets under pre-training. It secures the highest average mIoU when trained from scratch (0.7392) and is the best-performing model on MSBC (0.8950) and EGY-BCD (0.8445) in the pre-trained regime.
• From-scratch leaders vary by dataset. U-Net SiamConc leads CLCD (0.7048) and LEVIR-CD+ (0.7201) when trained without pre-training.
• CSSM underperforms in this benchmark. The state-space model records the lowest average mIoU when trained from scratch (0.6170), with notably low per-dataset scores such as CLCD (0.4596) and MSOSCD (0.5330).
• Computational efficiency is treated as a first-class axis. Model capacity ranges from TinyCD at 0.29 M parameters and 1.46 GFLOPs to ChangeFormerV6 at 41.03 M parameters and 138.77 GFLOPs, measured for a single forward pass on a 2 × 3 × 256 × 256 bi-temporal input tensor. Detailed latency, activation memory, and training time results are not reported in the available content of this paper.
• The motivating "reality check" evidence is substantial. The authors cite prior work showing that a popular random split of WHU-CD leaked roughly 85% of test patches into the training set, inflating scores, and that U-Net models with ImageNet-pretrained ResNet-50 backbones can match or outperform transformer-based models such as BIT and ChangeFormer on benchmarks like LEVIR-CD and WHU-CD.

Methodology in Plain English

Datasets. Ten change detection datasets were selected to span different sensors, resolutions, and scene dynamics: BANDON (2,283 image pairs, off-nadir aerial, 0.6 m), CLCD (600, GaoFen-2), DSIFN (3,988, Google Earth), EGY-BCD (6,091, Google Earth, 0.25 m), LEVIR-CD+ (985, Google Earth, 0.5 m), MSBC (3,769, GaoFen-2 / Sentinel-2 / Sentinel-1), MSOSCD (5,107, same sensor mix), OMBRIA (694, Sentinel-1 and Sentinel-2, 10 m), Season-Varying CDD (16,000, Google Earth), and SYSU-CD (20,000, aerial, 0.5 m). Each provides co-registered pre-event (T1) and post-event (T2) image pairs with binary pixel-level ground truth masks. None of these datasets were used for pre-training, only for benchmarking. The datasets deliberately cover three challenges: large ground sample distance spread (sub-meter aerial imagery up to 10 m/px spaceborne), multi-sensor inputs including SAR (testing resilience to geometric misregistration and speckle noise), off-nadir imagery with building side-wall relief displacement, and seasonal or illumination variation creating high intra-class variance.

Models. Ten architectures were benchmarked across four families: convolutional networks (U-Net SiamConc 40.35 M params / 19.37 GFLOPs, HRNet SiamConc 21.48 M / 9.55, SiamCRNN 28.51 M / 65.49, TinyCD 0.29 M / 1.46), attention and prior-guided networks (STANet 12.21 M / 19.15, CGNet 33.68 M / 87.55), vision transformers (BIT 3.50 M / 10.88, ChangeFormerV6 41.03 M / 138.77, ChangeViT 20.66 M / 26.36), and state-space models (CSSM 2.59 M / 2.78).

Protocol. A 60/20/20 train/validation/test split was applied everywhere. Season-Varying CDD, SYSU-CD, and CLCD already provide such a split from their original authors and were used as-is; splits for LEVIR-CD+, DSIFN, MSBC, MSOSCD, EGY-BCD, BANDON, and OMBRIA were generated by random sampling. Inputs were per-channel min-max normalized to [0,1] with an epsilon of 10⁻⁷ to prevent numerical instability, zeroed out where channels had zero variance, permuted from channels-last to channels-first, and resized to 256 × 256 with antialiasing. Ground truth maps were similarly converted and resized.

Training. All models were implemented and trained in the AiTLAS toolbox, using a maximum of 100 epochs, batch size 16, initial learning rate 1 × 10⁻⁴, and 4 worker threads. Training data was shuffled each epoch, while validation and test ordering was deterministic. The loss was an equally weighted combination (w1 = w2 = 0.5) of focal loss (focusing parameter gamma = 2, class balancing weight alpha = 0.25) and dice loss. Early stopping after 10 epochs was applied on the validation split, and the checkpoint with the lowest validation loss was evaluated on the test split.

Initialization regimes. Each architecture was trained twice where possible: from random initialization, and with weights pre-trained on ImageNet-1K. Pre-trained weights were applied strictly to model backbones, since full end-to-end pre-trained checkpoints are generally unavailable; fusion modules and prediction heads were randomly initialized. Most backbone weights came from the torchvision repository. ChangeViT is a special case, combining a ResNet18 encoder (weights from torchvision) with a DeiT encoder (weights from the original repository).

Evaluation. Continuous probability maps were thresholded at tau = 0.5 into binary change masks. Because changed pixels represent a small fraction of total area, overall accuracy was considered misleading, so mIoU and F1 score served as the primary metrics. Efficiency was profiled with parameter counts (millions), FLOPs for a single forward pass on a 2 × 3 × 256 × 256 tensor, latency in milliseconds per image pair at batch size 1 on a single GPU, peak activation memory in MB at training batch size 16, and total training time in minutes until convergence under the early-stopping protocol.

Reproducibility infrastructure. All runs were tracked with MLflow, which logged execution metadata, dataset split identifiers, and hyperparameters, while step-wise loss and metric trajectories were recorded as TensorBoard logs and archived as run artifacts. Best-performing checkpoints and final evaluation summaries were archived automatically. All experiments ran on identical hardware: NVIDIA A100-PCIe (40 GB) GPUs with CUDA version 12.9.

Trustworthy benchmarking pillars. The design rests on four pillars: methodological fairness (identical pipelines isolating architectural effects), transparency and reproducibility (fixed released splits, complete logging, open release), reliability (cross-dataset evaluation and reported variance rather than single-dataset optima), and FAIRness of released artifacts. The artifacts are described with an ontological schema in four layers — Dataset, Method, Experiment, and Evaluation — anchored in the AI4QC ontology, the FAIR-EO vocabulary, and MLDCAT-AP, cross-referenced with DCAT, ML Schema, and the W3C Data Quality Vocabulary, and released as turtle-annotated files.

Why This Matters

Impact on research. The paper directly addresses an evaluation-bias problem in EO change detection: published SOTA claims that fail to survive re-evaluation under unified optimization. By fixing splits, losses, preprocessing, and hardware, it isolates architectural effects from protocol differences, and by releasing all artifacts under FAIR principles with machine-actionable ontological descriptions, it makes results auditable and reusable rather than merely human-readable. The finding that pre-training consistently improves performance with no additional inference cost gives the community a concrete, low-risk lever for improving results.

Real-world applications:

• Urban growth and infrastructure tracking: monitoring building construction and demolition, as captured in LEVIR-CD+, EGY-BCD, and BANDON.
• Post-disaster damage mapping: rapid assessment of flood inundation and disaster-induced surface transformation, as captured in OMBRIA.
• Agricultural monitoring: distinguishing true cropland conversion from background soil variation and seasonal crop growth, as captured in CLCD.
• Environmental conservation and land-cover monitoring: suppressing phenological and illumination-driven apparent change to find genuine semantic change, as captured in Season-Varying CDD and CLCD.

Industry relevance. The benchmark reports parameter counts, FLOPs, latency, activation memory, and training time alongside accuracy, which matters for operational deployment. The comparison shows a wide efficiency spread — from TinyCD at 0.29 M parameters and 1.46 GFLOPs to ChangeFormerV6 at 41.03 M parameters and 138.77 GFLOPs — which is decision-relevant for edge platforms and large-scale processing pipelines. The standardised release also lowers the barrier for organizations to reproduce, audit, or extend published results without re-running an entire benchmark.

Future Directions

• Extending the analysis to efficiency metrics in depth. The paper's framing calls for profiling latency, activation memory, and training time, but those detailed results are not reported in the available content, leaving the full accuracy-versus-cost trade-off partially open.
• Building out the semantic catalogue. The authors explicitly state that this change detection benchmark is a first step toward a broad semantic catalogue of EO datasets, methods, and experiments, and that a full description of the catalogue and its ontology is the subject of ongoing work.
• Broadening dataset and task coverage. The benchmark covers binary pixel-level change detection; the related work notes that the literature predominantly focuses on binary over multiclass change detection, and that 3D change detection (LiDAR, point clouds) remains in its infancy due to acquisition costs, data noise, and computational intensity.
• Addressing the reported operational bottlenecks. Cited work identifies annotation cost, label availability, and domain shift — rather than model capacity — as the binding constraints on operational progress, suggesting that benchmark extensions around data efficiency and cross-domain robustness are natural next steps.

Target Audience

Researchers and practitioners who build, select, or evaluate deep learning models for Earth observation change detection, particularly those who need trustworthy, reproducible comparisons rather than single-model SOTA claims. It is also valuable for benchmark and reproducibility researchers interested in FAIR artifact release and ontological annotation of experiments, for remote sensing engineers weighing accuracy against deployment cost on edge or large-scale pipelines, and for reviewers and journal editors who need a template for what a rigorous, auditable change detection comparison should look like.

Authors’ abstract

Change detection in Earth observation (EO) is critical for monitoring land surface transformations, yet recent research in the field is constrained by inconsistent evaluation protocols and a narrow focus on predictive accuracy without regard for computational efficiency. To address this, we present a standardized, open-source benchmark for evaluating state-of-the-art (SOTA) deep learning methods for Earth observation change detection. We conduct a comprehensive analysis of ten representative model architectures, ranging from convolutional networks (CNNs) to vision transformers (ViTs), across ten heterogeneous change detection datasets. We rigorously evaluate these models with identical experimental protocols, comparing models trained from scratch against those utilizing pre-trained weights. Furthermore, we evaluate predictive performance alongside computational efficiency, including parameter counts and inference latency. Our findings reveal that well-optimized classical architectures, such as Siamese U-Nets, frequently outperform more complex contemporary models when computational efficiency is factored in, and that pre-training consistently provides a significant performance boost with no additional inference cost. To ensure complete transparency and reproducibility, all experimental resources, including standardized data splits, training scripts, training logs, and model checkpoints are publicly available and adhere to FAIR principles (Findable, Accessible, Interoperable, and Reusable).