Exploring Federated Learning for Thermal Urban Feature Segmentation -- A Comparison of Centralized and Decentralized Approaches

AI summary

Overview

Research area: Federated Learning (FL) applied to semantic segmentation of thermal urban features in unmanned aerial vehicle (UAV) imagery, with a focus on real-world (non-simulated) deployment.

Technical level: Intermediate. The core ideas are accessible, but the paper assumes familiarity with deep learning segmentation models (U-Net), FL aggregation algorithms, HPC job scheduling, and energy measurement tooling.

Scope: A real-world comparison of centralized learning (CL), centralized FL aggregation algorithms, and decentralized FL workflows for U-Net-based thermal feature segmentation on UAV images from two German cities, evaluated on accuracy, training time, communication overhead, and energy consumption.

What This Paper Is About

Training a shared segmentation model usually requires pooling all data in one place, which is difficult when images come from different locations and cannot be shared for privacy or technical reasons. This paper asks whether Federated Learning can match a centrally trained model on a real UAV thermal imaging task, where the data from Munich (MU) and Karlsruhe (KA) are naturally distributed and strongly imbalanced. The goal is to test FL not in simulation but in a genuine deployment across two High-Performance Computing (HPC) sites, measuring not only model quality but also time and energy cost.

Key Contributions

1. A real-world demonstration of FL for UAV-based thermal urban feature segmentation using non-IID, imbalanced data from two German cities, rather than a simulated FL setup.
2. An evaluation of FL aggregation strategies (FedAvg, FedProx, FedOpt, Scaffold) tailored to non-IID segmentation data, in both a 2-client and a 5-client configuration.
3. A head-to-head comparison of FL against a centralized learning baseline on model accuracy (mACC, mwP, mwF1, mwIoU), training time, memory consumption, and energy use.
4. A comparison of centralized FL workflows (Scatter & Gather) against decentralized ones — Swarm Learning, Decentralized Cyclic Weight Transfer (DCWT), and Cyclic Weight Transfer (CWT) — including client-ordering effects.
5. An exploration of using HPC systems (HoreKa and HAICORE) as FL clients and the practical challenges this introduces.

Main Findings

• FedAvg and Scaffold match centralized learning in the 2-client scenario. Across both clients overall, FedAvg reached mwP 0.935 ± 0.005 versus CL at 0.939 ± 0.002, and Scaffold reached mwP 0.931 ± 0.001. FedAvg's overall mACC was 0.930 ± 0.007 versus 0.931 ± 0.002 for CL.
• FedAvg was strongest on the larger Munich client. On Client-1 (MU), FedAvg recorded mwP 0.945 ± 0.003, mACC 0.940 ± 0.007, and mwF1 0.937 ± 0.005, while CL recorded 0.952 ± 0.017, 0.949 ± 0.003, and 0.944 ± 0.002.
• Scaffold was strongest on the smaller Karlsruhe client. On Client-2 (KA), Scaffold achieved mACC 0.866 ± 0.007, mwF1 0.858 ± 0.007, and mwIoU 0.777 ± 0.009, versus FedAvg's mwF1 0.810 ± 0.014 and mwIoU 0.737 ± 0.014. (The narrative text reports a mACC comparison of 0.846 ± 0.009 versus 0.66 ± 0.007, which does not match the 0.846 ± 0.009 / 0.866 ± 0.007 split shown in Table 3.)
• FedProx and FedOpt lagged in the 2-client setting. FedProx was weakest on both clients — mACC 0.92 ± 0.01 and mwIoU 0.85 ± 0.02 on Client-1, and overall mwP 0.913 ± 0.011. FedOpt was consistently worst overall (mwP 0.905 ± 0.002, mACC 0.873 ± 0.008, mwIoU 0.812 ± 0.009).
• FedOpt and Scaffold required a normalization change. With standard Batch Normalization in the U-Net backbone, their performance dropped after two rounds and kept declining; replacing Batch Normalization with Group Normalization resolved this.
• Results shift in the 5-client scenario. FedProx improved and became comparable to CL (mwP 0.932 ± 0.003, mACC 0.926 ± 0.006, mwF1 0.918 ± 0.008, mwIoU 0.872 ± 0.009 versus CL's 0.939 ± 0.002, 0.931 ± 0.002, 0.928 ± 0.002, 0.884 ± 0.003). Scaffold dropped by 3–5% on each metric compared to CL, ending at mwP 0.900 ± 0.002. FedAvg reached mwP 0.919 ± 0.012.
• Splitting data across five clients slows convergence. With each client holding one fifth of the data, smaller per-round updates require more communication to reach comparable performance. The authors settled on nine local epochs and seven rounds as a balanced setting after also testing five local epochs with 13 rounds.
• Scaffold is the most expensive algorithm. It had the highest execution time at an average of 103.33 minutes with a standard deviation of 8.84 minutes, and the largest energy consumption across all sites.
• Centralized learning was dramatically cheaper than FedAvg. CL consumed 382.279 kJ versus FedAvg's 1008.9152 kJ, a percentage reduction of approximately 163.97%; CL's runtime was 643.245 seconds versus 1687.0624 seconds for FedAvg, approximately 162.3%. Server-side energy consumption per algorithm was very low, at 14–29 kJ.
• Decentralized workflows cut time but not accuracy. In the 2-client setting, Swarm Learning matched Scatter & Gather performance (mwP 0.940 ± 0.020 versus 0.935 ± 0.023) while FedAvg recorded a mean execution time of 2906.07 seconds versus 949.86 seconds for Swarm Learning, an approximately 205.95% increase. Swarm Learning also had a mean execution time of 952.42 seconds versus 1949.69 seconds for DCWT, roughly a 105% increase.
• Client order matters when dataset sizes differ. Training first on the smaller KA dataset and then on MU (CWT KA-MU) consistently outperformed the reverse order (CWT MU-KA), which the authors attribute to overfitting the smaller KA dataset.
• With five clients, workflow choice matters less. All frameworks showed statistically consistent performance across key metrics, and Swarm Learning again led slightly (mwP 0.939 ± 0.003, mACC 0.934 ± 0.003, mwF1 0.923 ± 0.005, mwIoU 0.883 ± 0.005).

Methodology in Plain English

The researchers started from an existing centralized segmentation pipeline for detecting thermal anomalies in UAV imagery and converted it into a federated one using NVIDIA FLARE (NVFlare). The dataset, from a publicly available multispectral collection, contains 793 images from 14 UAV flights, with most images from Munich and the rest from Karlsruhe, captured at a nadir 90° pitch angle and 60 m height using a DJI Zenmuse XT2 thermal infrared and RGB camera. Images were annotated with seven classes of common thermal urban features, and the class distribution is heavily imbalanced — for example, Street lamp (cold) has only 100 objects in total while Car (cold) has 2532.

The model is a U-Net with a ResNet-152 encoder, trained with a sigmoid focal cross-entropy loss and the Adam optimizer at a learning rate of 0.001, with an 80% training / 20% testing split. Each client trained on its own data and only model updates were exchanged.

Two client configurations were tested. The first assigned one client per city: Client-1 (MU) with 700 images and Client-2 (KA) with 93 images. The second split the same data into five heterogeneous subsets of 159, 159, 160, 164, and 151 images. Four aggregation algorithms — FedAvg, FedProx, FedOpt, and Scaffold — were run in each configuration, and each experiment was repeated five times, with results reported as averages plus standard uncertainty. Aggregated overall scores were computed as weighted averages across clients.

Infrastructure was deliberately realistic: Client-1 ran on a GPU node of HoreKa and Client-2 on a GPU node of HAICORE, both offering 2 Intel Xeon Platinum 8368 processors (76 CPU cores, 152 CPU threads), 512 GB main memory, 4 NVIDIA A100-40 GPUs with 40 GB each, and a 960 GB local NVMe SSD, though only one GPU per node was used per client. The server initially ran on a bwCloud virtual machine with 32 GB main memory and 40 GB storage. NVFlare's launching scripts were modified to work with the SLURM job scheduler. MLflow, provided through the AI4EOSC project, was used for experiment tracking, and the authors modified NVFlare source code so that the global model would also be logged.

Energy use was measured with the perun package, tracking CPU, GPU, and memory power draw on clients and server. Because the virtual machine gave no access to hardware performance counters, the server was moved to bare-metal hardware with energy measurement capabilities for the energy experiments. In the 5-client setup, where four clients shared one node, energy was measured for only two clients. Carbon intensity calculations were deliberately not reported because they vary strongly by region.

Why This Matters

This is one of the comparatively few FL studies executed as a real deployment rather than a simulation, using genuinely distributed, imbalanced, real-world image data. It shows that FL can approach centralized accuracy for a segmentation task while also documenting the practical costs — especially energy and runtime — that simulations rarely surface. Its findings on normalization layers, client count, client ordering, and decentralized workflow trade-offs are directly actionable for anyone deploying FL on infrastructure rather than in a notebook.

Real-world applications:

• District heating and utility maintenance: detecting thermal anomalies in urban environments so that false positives from buildings, cars, manholes, and street lamps do not obscure locations that actually need repair.
• Privacy-constrained aerial or satellite surveying: municipalities, utilities, or regulators that cannot pool aerial imagery centrally can collaborate on a shared model without moving raw data.
• Cross-city or cross-region infrastructure analytics: the two-city MU/KA design mirrors how operators of separate service areas would realistically share model improvements.
• Edge and drone-based inference: the U-Net configuration and the HPC/edge deployment discussion inform settings where drone-collected data stays local.
• Energy-aware AI infrastructure planning: the measured energy gap between CL and FL is relevant to organizations that must justify the environmental cost of distributed training.

Industry relevance centers on the fact that FL's accuracy benefits come with measurable operational penalties. The paper quantifies both sides, which matters for cloud and HPC operators, energy and utility companies, and teams subject to data protection rules such as GDPR and the EU AI Act.

Future Directions

• Understanding and fixing the FedProx reversal. FedProx was weakest in the 2-client setting but competitive with CL once data was split across five clients, which the authors attribute to reduced proximal penalties. A deeper analysis of this effect, and of when it generalizes, remains open.
• Scaffold's cost–benefit trade-off. Scaffold achieved the best single-client results but the highest execution time and energy consumption, and it degraded by 3–5% when data per client shrank. Finding ways to retain its benefits at lower cost is an open question.
• Decentralized deployments at larger scale. Swarm Learning and DCWT were tested with two and five clients; how these workflows behave with many more geographically dispersed participants is not reported here.
• Energy and carbon reporting. The authors explicitly omitted carbon intensity calculations because emission factors vary by region, leaving a standardized, region-aware comparison of FL energy footprints as future work.
• HPC-as-client integration. The paper raises challenges and opportunities of using HPC systems as FL clients, including SLURM integration and shared-node energy attribution, without fully resolving them.

Target Audience

Researchers and practitioners working on federated learning, distributed deep learning, or privacy-preserving machine learning who need evidence from real deployments rather than simulations. It is also valuable to applied machine learning engineers in remote sensing, UAV imaging, and urban infrastructure analytics, and to HPC and MLOps teams responsible for running distributed training on shared clusters and measuring its energy cost. Readers seeking the underlying segmentation data pipeline and annotation details are pointed by the authors to the original dataset publication, since this paper reuses that model configuration rather than introducing it.

Authors’ abstract

Federated Learning (FL) is an approach for training a shared Machine Learning (ML) model with distributed training data and multiple participants. FL allows bypassing limitations of the traditional Centralized Machine Learning CL if data cannot be shared or stored centrally due to privacy or technical restrictions -- the participants train the model locally with their training data and do not need to share it among the other participants. This paper investigates the practical implementation and effectiveness of FL in a real-world scenario, specifically focusing on unmanned aerial vehicle (UAV)-based thermal images for common thermal feature detection in urban environments. The distributed nature of the data arises naturally and makes it suitable for FL applications, as images captured in two German cities are available. This application presents unique challenges due to non-identical distribution and feature characteristics of data captured at both locations. The study makes several key contributions by evaluating FL algorithms in real deployment scenarios rather than simulation. We compare several FL approaches with a centralized learning baseline across key performance metrics such as model accuracy, training time, communication overhead, and energy usage. This paper also explores various FL workflows, comparing client-controlled workflows and server-controlled workflows. The findings of this work serve as a valuable reference for understanding the practical application and limitations of the FL methods in segmentation tasks in UAV-based imaging.