The MICCAI Federated Tumor Segmentation (FeTS) Challenge 2024: Efficient and Robust Aggregation Methods for Federated Learning

AI summary

Overview

• Research area: Federated learning (FL) for medical image analysis, specifically multi-parametric MRI segmentation of glioma sub-regions, with a focus on model weight aggregation algorithms.
• Technical level: Intermediate — readers benefit from familiarity with federated learning terminology (rounds, collaborators, aggregators) and segmentation metrics (Dice, Hausdorff distance), though the paper explains each.
• Scope (one sentence): This paper reports the design, participating methods, evaluation protocol, and results of the third iteration of the FeTS Challenge (2024), a benchmarking competition for efficient and robust weight aggregation in federated brain tumor segmentation.

What This Paper Is About

Multi-institutional medical AI is limited by privacy regulations (HIPAA, GDPR) that discourage sharing patient data, so federated learning is used to train models locally at each site and combine only model weights. The central open problem is how to combine those weights well, since naive averaging can be unstable or wasteful when sites differ in data size and distribution. The FeTS Challenge 2024 is organized specifically to benchmark novel aggregation methods — allowing innovations in hyperparameter tuning, collaborator selection, and the aggregation function itself — under a standardized, single-machine simulation of a federation using BraTS glioma MRI data.

Key Contributions

1. A standardized aggregation-focused benchmark: The challenge restricted innovation to three "hotspots" — hyperparameter tuning, collaborator selection, and the weight aggregation function — while keeping the segmentation architecture (U-Net) and the software stack fixed for all participants.
2. A multi-faceted scoring system: Teams were ranked using a cumulative score combining segmentation quality (DSC and HD95 for ET, TC, WT) with a convergence score that simulates training, validation, download, and upload times over a one-week simulated budget.
3. A curated multi-institutional dataset configuration: The challenge used BraTS 2021-derived data partitioned into 23 sites (by institution) or 33 sites (artificial repartitioning of the five largest institutions by whole tumor size), with 1,251 training, 219 validation, and 570 hidden test cases.
4. Benchmarking of six submitted methods: Submissions from SNU, rigg, HTTUAS (two methods), Flair, and ReMIC were evaluated, with a PID-controller-based approach emerging as the top overall performer.

Main Findings

• Winner of the challenge: The FedPOD method of SNU ranked first in the Overall Ranking (w = 3) with a score of 2.317, and second in the Segmentation-Only ranking (w = 0) with a score of 2.976.
• Best segmentation accuracy: SNU (FedPOD) achieved the best DSC across all tumor types — 0.733 (ET), 0.761 (TC), and 0.751 (WT) — with corresponding HD95 values of 33.922 mm (ET), 33.623 mm (TC), and 32.309 mm (WT).
• Best boundary precision is distributed: rigg achieved the lowest HD95 for ET (32.246 mm) and TC (31.705 mm), while HTTUAS (Rec) attained the lowest for WT (28.228 mm). No single method dominated every metric.
• Best communication efficiency: SNU (FedPOD) also had the best convergence score "by a significant margin," and the abstract reports a convergence score of 0.764 for the top method.
• Segmentation-only ranking changes the outcome: When convergence score is excluded (w = 0), rigg ranks first (2.809), SNU second (2.976), and HTTUAS (Rec) third (3.116).
• PID-controller foundations are effective: Both rigg's FedPID and SNU's FedPOD build on FedPIDAvg (featuring in FeTS 2022, itself following FedCostWAvg from FeTS 2021) and both remain in the top three whether or not communication efficiency is counted.
• Stability varies across methods: rigg and SNU exhibited the lowest variance in performance distributions across samples; Flair showed good DSC but higher variability and below-average communication efficiency.
• Trade-offs are real: HTTUAS (Rec) performed competitively in DSC with lower variability in HD95, while HTTUAS (RL) had slightly lower DSC; Flair's clustering and dynamic hyperparameter tuning did not appear to justify its computational overhead.
• Ranking scale: 570 test subjects × 3 tumor regions × 2 segmentation metrics = 3,420 individual rankings, rising to 5,130 rankings per team when convergence score is weighted in with w = 3.

Methodology in Plain English

The challenge provided a pre-built federated learning pipeline so participants did not need to construct one from scratch. Data came from the BraTS 2021 glioma dataset, comprising four MRI scans per patient (T1, T1-Gd, T2, T2-FLAIR) that had been registered to a common atlas (SRI-24) at 1 mm³ resolution, skull-stripped, and annotated for enhancing tumor (ET), necrotic tumor core (NCR), tumor core (TC), and whole tumor (WT) regions under a peer-reviewed protocol with board-certified neuroradiologist approval.

Each "site" in the federation sees only its own partition of data. In every round, each collaborator validates the model received from the central aggregator, trains it on local data, and returns the updated weights plus validation results. The aggregator combines all returned updates into a new consensus model, which is sent back out. Participants modified the aggregator's logic, collaborator selection, or hyperparameters.

Evaluation used three measures: Dice Similarity Coefficient (overlap between predicted and reference masks), 95th percentile Hausdorff Distance (boundary disagreement, robust to outlier pixels), and a convergence score. The convergence score is the area under the projected best-so-far DSC-over-time curve, where per-round times are simulated by sampling training, validation, download, and upload times from normal distributions based on real measurements from the largest real-world FL study to date. Runs were capped at one week of simulated time. All submitted algorithms were re-trained by the organizers before test-set evaluation to ensure reproducibility and prevent data leakage. Rankings were computed per test case with average-rank tie-breaking, then averaged across cases.

Why This Matters

• Research impact: The challenge establishes a reproducible benchmark for a component of federated learning — weight aggregation — that is often treated as a fixed default, showing that PID-controller-inspired aggregation can exceed standard approaches.
• Real-world applications:
  • Multi-hospital training of glioma and glioblastoma segmentation models without moving patient data off-site.
  • Enabling collaboration on rare diseases or underserved populations where no single institution has enough data.
  • Transfer of the aggregation techniques to other clinical domains such as radiology, cardiology, and pathology.
  • Deployment in regulated healthcare environments where HIPAA and GDPR compliance is mandatory.
• Industry relevance: The work sits at the intersection of privacy-preserving infrastructure (OpenFL, GaNDLF, Kubernetes auto-scaling in FedPOD), medical device/AI vendors, and clinical networks, all of which need efficient communication to make federated training financially and operationally viable.

Future Directions

1. Bridging simulation and real-world deployment: The authors explicitly frame the single-machine setup as a preliminary step toward benchmarking under realistic communication conditions (variable bandwidth, latency, dropped connections, asynchronous updates).
2. Modeling hardware heterogeneity: GPU memory limits, CPU load, and system failures were abstracted away but strongly affect real-world performance.
3. Integrating security protocols: Secure aggregation, differential privacy, homomorphic encryption, and confidential computing are noted as complementary but unrepresented in the challenge setup.
4. Extending aggregation research: Open questions remain around handling skewed data distributions, including outlier nodes that would otherwise be excluded (a motivation behind FedPOD) and methods that do not require historical participant data.

Target Audience

Federated learning researchers and engineers, medical imaging and clinical AI scientists, challenge organizers, and healthcare technology teams responsible for deploying privacy-preserving multi-institutional AI. It is also useful for graduate students looking for a concrete, reproducible benchmark with public code, and for regulators or ethics reviewers seeking a worked example of privacy-preserving collaborative model development.

Note: The paper reports "six teams" in the abstract and "6 submissions by 5 teams from four continents" in Section 7 — both figures are stated as written, and the discrepancy is not resolved in the provided text. The percentages of enhancement status, the number of challenge registrants, or the identity of team affiliations beyond institution names are not reported in the content available.

Authors’ abstract

We present the design and results of the MICCAI Federated Tumor Segmentation (FeTS) Challenge 2024, which focuses on federated learning (FL) for glioma sub-region segmentation in multi-parametric MRI and evaluates new weight aggregation methods aimed at improving robustness and efficiency. Six participating teams were evaluated using a standardized FL setup and a multi-institutional dataset derived from the BraTS glioma benchmark, consisting of 1,251 training cases, 219 validation cases, and 570 hidden test cases with segmentations for enhancing tumor (ET), tumor core (TC), and whole tumor (WT). Teams were ranked using a cumulative scoring system that considered both segmentation performance, measured by Dice Similarity Coefficient (DSC) and the 95th percentile Hausdorff Distance (HD95), and communication efficiency assessed through the convergence score. A PID-controller-based method achieved the top overall ranking, obtaining mean DSC values of 0.733, 0.761, and 0.751 for ET, TC, and WT, respectively, with corresponding HD95 values of 33.922 mm, 33.623 mm, and 32.309 mm, while also demonstrating the highest communication efficiency with a convergence score of 0.764. These findings advance the state of federated learning for medical imaging, surpassing top-performing methods from previous challenge iterations and highlighting PID controllers as effective mechanisms for stabilizing and optimizing weight aggregation in FL. The challenge code is available at https://github.com/FeTS-AI/Challenge.