Transfer Learning for Onboard Cloud Segmentation in Thermal Earth Observation: From Landsat to a CubeSat Constellation

AI summary

Overview

Research area: Computer vision and transfer learning applied to thermal Earth observation (EO), specifically onboard cloud segmentation for a CubeSat constellation. Technical level: Intermediate — the reader should be comfortable with segmentation architectures (UNet, MobileNet), transfer learning terminology, and standard classification metrics, but the paper is written with practical deployment context. Scope: The paper studies whether a model pretrained on a large public Landsat-7 cloud-mask dataset can be adapted, through joint training on roughly 500 mission-specific crops, to perform single-band thermal cloud segmentation onboard a FOREST-2 CubeSat.

Note on the byline: the paper content lists Lukas Liesenhoff (OroraTech GmbH) as an author, while the prompt's header names Lukas Kondmann.

What This Paper Is About

CubeSats that image in the thermal infrared usually carry only one usable thermal band and have little or no labeled data in their early mission phases, which makes conventional cloud masking — designed for multispectral or visible imagery — difficult to apply. Clouds corrupt the measured thermal signal and block the ground, so an accurate cloud mask is a prerequisite for downstream applications such as wildfire detection. The paper's goal is to show that pretraining on a large public Landsat-7 Cloud Cover Assessment Dataset, then fine-tuning with a small set of FOREST-2 CubeSat samples, produces an accurate and fast enough cloud segmentation model to run directly on constrained onboard hardware.

Key Contributions

1. Transfer learning for single-band thermal cloud segmentation. The authors define and compare three inductive transfer learning setups — l7-l7 (train and test on Landsat-7), l7-f2 (train on Landsat-7, test zero-shot on FOREST-2), and joint-f2 (train jointly on Landsat-7 plus each FOREST-2 fold, test on the held-out FOREST-2 fold) — against a FOREST-2-only baseline (f2-f2).
2. A demonstrated gain from mixing public and mission-specific data. Joint training with Landsat-7 raises the macro F1 from 0.850 to 0.877 and accuracy from 0.889 to 0.910 on FOREST-2, even though only about 10% of all training samples come from FOREST-2.
3. An onboard deployment pipeline. The trained PyTorch model is exported to ONNX and compiled to a TensorRT engine, then run with a tiling-and-stitching scheme on an NVIDIA Jetson Nano with 4 GB RAM, achieving full-image (2691x1762 px) inference in under 5 seconds.
4. A cross-sensor domain-adaptation recipe. Only spatial domain adaptation is applied — Landsat-7 Band 6 crops are bilinearly resampled to the 200 m ground sampling distance and 256x256 px size of FOREST-2 — justified by the close match between Landsat-7 Band 6 and the FOREST-2 LWIR2 band (10.5-12.6 µm).

Main Findings

• Joint training beats the target-domain-only baseline: MobileNet joint-f2 reaches a macro F1 of 0.877 and accuracy of 0.910, versus 0.850 and 0.889 for MobileNet f2-f2. The paper reports this as an improvement of 2.7% F1 and 2.1% accuracy.
• Landsat-only performance sets the ceiling: MobileNet l7-l7 achieves the highest scores of the study — macro F1 0.903 and accuracy 0.917 on the Landsat-7 test set.
• Zero-shot cross-sensor transfer works but leaves a gap: MobileNet l7-f2, trained only on Landsat-7 and evaluated on FOREST-2 without any fine-tuning, still obtains a macro F1 of 0.796 and accuracy of 0.830.
• Threshold-independent curves agree with the ranking: the joint-f2 model reaches an AP of 0.92 and AUC of 0.96, compared with 0.90 and 0.94 for f2-f2; l7-f2 scores AP 0.78 and AUC 0.87; l7-l7 scores AP 0.95 and AUC 0.98.
• Joint training reduces fold-to-fold variability: the authors report that variability across the six folds is reduced in terms of F1, which they attribute to the greater diversity of the training data.
• Qualitative behavior on thin clouds: the joint-trained model performs better in visually complex regions and detects thin cloud structures more reliably, which the authors suggest is due to the higher representation of such cases in the Landsat-7 training data.
• Onboard inference is feasible: with overlapping 512x512 px tiles where only the central 256x256 region of each tile is kept for stitching, the TensorRT model runs a full image in under 5 seconds, with segmentation performance within 1% of the on-ground implementation that processes the whole image in a single pass.
• Class balance context: the dataset is slightly imbalanced at roughly two-thirds clear pixels, which is why the authors report macro-averaged F1 alongside overall accuracy.

Methodology in Plain English

The authors use a UNet segmentation network with a custom MobileNet-v3-large encoder that was pretrained on ImageNet; this combination had previously been identified as the best trade-off between accuracy and computational cost for thermal cloud detection on FOREST-2 imagery. The encoder weights are frozen during training and only the decoder is updated, using a binary cross-entropy loss and the Adam optimizer. The model always sees a single input band: FOREST-2's LWIR2 band, and Landsat-7's spectrally matching Band 6, each Z-score normalized with the mean and standard deviation of its own dataset.

Two datasets supply the training signal. The mission dataset contains 528 crops of 256x256 pixels drawn from 24 hand-labeled FOREST-2 thermal scenes, which are concentrated over wildfire-prone regions such as Australia and North America. The public dataset is the Landsat-7 Cloud Cover Assessment Dataset: 6,000 crops extracted from 206 globally distributed scenes covering diverse biomes, resampled from the sensor's 60 m thermal resolution down to 200 m to match FOREST-2.

Evaluation is split by dataset. FOREST-2 uses a 6-fold spatial cross-validation with spatial blocking to limit the effects of spatial autocorrelation and the small sample size; on each iteration four folds train, one validates, and one is held out for testing, and results are averaged over the six test scores. Landsat-7, being large and globally representative, uses a traditional train-test split.

For deployment, the PyTorch model is converted to ONNX to reduce dependencies, then compiled into a TensorRT engine for CUDA-enabled embedded hardware. Because a full image does not fit in memory on the target device, the input is cut into overlapping 512x512 px tiles processed independently, with only each tile's central 256x256 px region assembled into the final mask to avoid edge artifacts and seams.

Why This Matters

Research impact. The work pushes cloud segmentation into a relatively underexplored regime — thermal-only, single-band input on a resource-constrained platform — and provides a concrete, reproducible comparison of intra-dataset, zero-shot cross-dataset, and joint-training regimes for EO. It follows and extends cross-mission adaptation work such as the Landsat-8 to Proba-V study by Mateo-García et al. (2020), and shows that public labeled datasets can substitute for the large mission-specific datasets that new satellites simply do not have yet.

Real-world applications.

• Wildfire detection and monitoring, which is the primary motivation cited and the reason FOREST-2 data is denser over fire-prone regions.
• Urban heat monitoring, listed among the thermal EO applications that depend on clean, cloud-free thermal imagery.
• Agricultural drought assessment, another downstream use of the thermal signal.
• Disaster response and natural hazard monitoring, where the OTC-P1 constellation's twice-daily global revisit is intended to deliver actionable, timely imagery.

Industry relevance. For satellite operators, the practical claim is economic: labeling effort is reduced, prelaunch training enables cloud masking from the first day of operation, and the model already fits the compute and memory budget of an NVIDIA Jetson Nano representative of FOREST-2 hardware. Onboard cloud masking also enables data prioritization and pre-filtering before downlink, which matters directly for missions with limited bandwidth and power budgets. The authors note the method was previously validated in a similar architectural study by Wölki et al. (2024), situating this paper within an ongoing operational program.

Future Directions

• Expand and diversify validation. The current evaluation is limited by the size and geographic diversity of the FOREST-2 dataset, and the authors note that operational benchmarks are validated across conditions such as snow-covered terrain, coastal regions, and various biomes — a broader and more professionally labeled validation set is needed for robust benchmarking.
• Bridge the residual domain gap. The gap between l7-f2 and the in-domain models could narrow with improved domain adaptation, for which the authors name histogram matching, spectral response function and point spread function matching, and adversarial training.
• Grow the mission-specific dataset. Since joint training already surpasses the FOREST-2-only baseline with only about 10% of samples from FOREST-2, a larger and globally representative FOREST-2 dataset combined with better domain adaptation is expected to improve robustness to varying environmental conditions and cloud types.
• Test additional spectral input. Future work could examine whether combining LWIR2 with additional bands or indices improves performance beyond the single-band setting studied here.

Target Audience

This paper is most useful to engineers and researchers building onboard processing pipelines for small satellites, particularly those working with thermal-only or otherwise spectrally limited instruments and tight compute, memory, power, and bandwidth budgets. It also speaks to applied machine learning practitioners interested in transfer learning and domain adaptation across remote sensing sensors, to mission planners at EO companies evaluating prelaunch training and post-launch fine-tuning strategies, and to students or newcomers looking for a compact, end-to-end example that runs from dataset selection through architecture choice, cross-validation design, and embedded deployment. The topic is intermediate: the reader benefits from familiarity with segmentation metrics and transfer learning terminology, but the paper's emphasis on practical constraints keeps it accessible to a technically grounded non-specialist.

Authors’ abstract

Onboard cloud segmentation is a critical yet underexplored task in thermal Earth observation (EO), particularly for CubeSat missions constrained by limited hardware and spectral information. CubeSats often rely on a single thermal band and lack sufficient labeled data, making conventional cloud masking techniques infeasible. This work addresses these challenges by applying transfer learning to thermal cloud segmentation for the FOREST-2 CubeSat, using a UNet with a lightweight MobileNet encoder. We pretrain the model on the public Landsat-7 Cloud Cover Assessment Dataset and fine-tune it with a small set of mission-specific samples in a joint-training setup, improving the macro F1 from 0.850 to 0.877 over FOREST-2-only baselines. We convert the model to a TensorRT engine and demonstrate full-image inference in under 5 seconds on an NVIDIA Jetson Nano. These results show that leveraging public datasets and lightweight architectures can enable accurate, efficient thermal-only cloud masking on-orbit, supporting real-time decision-making in data-limited EO missions.