Finally Outshining the Random Baseline: A Simple and Effective Solution for Active Learning in 3D Biomedical Imaging

AI summary

Overview

Research area: Active learning (AL) for 3D biomedical image segmentation, evaluated within the nnActive framework and benchmark using nnU-Net as the segmentation backbone.

Technical level: Advanced. The paper assumes familiarity with uncertainty-based active learning, U-Net architectures, partial-loss training, and ranking-based statistical evaluation of methods.

Scope: The paper proposes and empirically validates a single query strategy, Class-stratified Scheduled Power Predictive Entropy (ClaSP PE), across a 24-setting benchmark and a four-dataset "roll-out" study, arguing that it is the first AL method for 3D biomedical segmentation to consistently outperform improved random baselines.

What This Paper Is About

Active learning promises to cut annotation costs in 3D biomedical imaging by querying only the most informative data for expert labeling, but existing AL methods have repeatedly failed to beat random sampling baselines that are properly adapted to 3D data. The authors address this long-standing gap with a deliberately simple method that adds two modifications to standard uncertainty-based querying, and they back the claim with a large-scale evaluation plus a simulated real-world deployment. The goal is not methodological novelty but a reliable, generalizing recipe that practitioners can apply to unseen tasks without per-dataset tuning.

Key Contributions

1. ClaSP PE, a simple and effective query method. It systematically addresses two key limitations of current uncertainty-based AL methods, class imbalance and redundancy in early selections, by combining class-stratified sampling with scheduled power noising.

2. A large-scale benchmark evaluation. The authors demonstrate that ClaSP PE brings reliable performance improvements over standard and improved random sampling baselines for 3D biomedical image segmentation on the nnActive benchmark, spanning four datasets and six annotation budgets each (the paper also describes the evaluation as spanning three Label Regimes and two query patch sizes per dataset, giving 24 distinct settings).

3. A Roll-Out study simulating real-world use. ClaSP PE is tested on four additional, previously unseen datasets with all parameters set according to the authors' Guidelines for Real-World Deployment and no manual adaptation, providing evidence for generalization.

4. Open-source implementation and deployment guidelines. The code is released within the nnActive framework at https://github.com/MIC-DKFZ/nnActive, together with clear deployment guidelines for practitioners.

Main Findings

• ClaSP PE leads on average aggregation. Across all 24 distinct AL settings of the nnActive benchmark, ClaSP PE achieves the best average results: AUBC 67.62 ± 0.33 and Final Dice 72.81 ± 0.30, compared with Random 66% FG (67.14 ± 0.22 and 71.14 ± 0.22), Random 33% FG (66.00 ± 0.27 and 69.74 ± 0.32), and plain Random (60.57 ± 0.39 and 61.65 ± 0.43). The uncertainty baselines are PowerPE (64.85 ± 0.35, 68.01 ± 0.38), PowerBALD (64.81 ± 0.35, 67.93 ± 0.29), SoftrankBALD (63.74 ± 0.32, 67.32 ± 0.28), Predictive Entropy (63.27 ± 0.40, 67.35 ± 0.58), and BALD (62.39 ± 0.30, 65.43 ± 0.41).

• ClaSP PE is the only AL method that improves on improved random strategies. The authors state that on average Final Dice and AUBC over all settings, ClaSP PE is the only AL method that improves over improved random strategies, while also remaining annotation efficient.

• Statistically distinct top group. Using a Nemenyi post-hoc test at significance level p = 0.05, ClaSP PE forms a distinct top-performing group for AUBC and Final Dice, while the naive random baseline is consistently ranked lowest and is significantly outperformed by all other methods. SoftrankBALD also appears in the top Nemenyi group, but ClaSP PE shows a clearer overall advantage in average rankings and absolute performance.

• Pairwise Penalty Matrix dominance with one exception. On the nnActive benchmark, ClaSP PE outperforms all random and AL baselines more frequently than it is outperformed. In roughly 20% of comparisons, Random 66% FG surpasses ClaSP PE, with those cases concentrated almost exclusively on the AMOS dataset under Low-Label Regimes. The PPM trends persist across different p-values and under the Bonferroni-Holm method.

• Longer training amplifies the advantage. On AMOS, comparing 200 versus 500 training epochs, at 200 epochs ClaSP PE loses on 60% of experiments to Random 66% FG and ties in the rest; at 500 epochs it outperforms Random 66% FG in 20%, ties in 48%, and loses in only 32%. At 500 epochs it consistently outperforms Random 66% FG in the High-Label Regime, while the Low-Label Regime is still dominated by lose-cases. Against Random 33% FG it shows clear and consistent gains in both Medium- and High-Label Regimes.

• The AMOS failure mode is explained by class count and budget. AMOS has 15 classes, and the 200-patch Low-Label budget, evenly spaced across all classes, could capture less than 14 examples per class (compared to 67 on KiTS, for 3 classes). Class-level Dice analysis attributes the low-label losses mainly to segmentation performance on the right and left adrenal gland, which is also less frequently queried compared to Random 66% FG.

• Ablations isolate each component. Both Cla PE 66% and Cla PE 33% (class-balanced querying without power noising) outperform standard Predictive Entropy on all evaluation metrics, and raising stratification from 33% to 66% improves AUBC and Final Dice with only a minor decrease in FG-Eff. Adding constant power noising (ClaP PE) substantially improves FG-Eff but leads to a reduction in AUBC and Final Dice. Gradually decayed power noising ranks among the best across all three metrics, giving the best trade-off.

• Robust out-of-the-box performance in the Roll-Out. On LiTS (n=99), ClaSP PE reaches AUBC 60.30, Final Dice 65.80, FG-Eff 39.60 versus Predictive Entropy's 57.81, 65.38, 38.94 and Random's 51.23, 52.38, 46.25. On WORD (n=90), ClaSP PE scores 78.27 / 78.42 / 1.33 versus Predictive Entropy 78.43 / 78.96 / 0.91. On Tooth Fairy 2 (n=360), ClaSP PE scores 67.32 / 71.49 / 20.07 versus Predictive Entropy 66.65 / 71.97 / 16.25 and Random 66% FG 65.30 / 68.61 / 10.85. On MAMA MIA (n=1130), ClaSP PE scores 63.85 / 68.62 / 57.36 versus Predictive Entropy 59.07 / 64.74 / 9.43 and Random 66% FG 44.38 / 45.10 / -4.67. Corresponding 100% Data Dice values are 77.3, 80.7, 72.6 and 71.0.

• Roll-Out PPM. ClaSP PE is never significantly outperformed by Random and Random 66% FG, wins in over 50% of all cases, significantly outperforms Predictive Entropy in 25% of cases, and is significantly outperformed by Predictive Entropy in 5%.

• High FG-Eff from Random can be misleading. On LiTS and WORD, Random shows high FG-Eff because it queries only a very small amount of foreground, which artificially inflates FG-Eff without translating into segmentation performance gains.

Methodology in Plain English

The method starts from the standard practice of scoring unlabeled images by how uncertain the model is about them, then picking the top-scoring patches for annotation. Predictive Entropy provides this uncertainty map, following evidence from the nnActive benchmark that it performs best among AL methods there. The authors then apply two changes.

First, instead of letting globally high-uncertainty regions dominate, they compute a class-specific uncertainty by multiplying the predicted probability for a class by the overall uncertainty, rank patches per class by that quantity, and select an equal number of patches from each predicted class. This ensures underrepresented structures are covered. Only a fraction of the budget is selected this way; the remainder uses the standard uncertainty map so that highly uncertain examples are still prioritized regardless of class distribution. In the final method, the stratified fraction is α = 66%.

Second, to prevent the same redundant regions being queried again and again, they perturb the patch scores with Gumbel noise before taking the top-k. The noise scale follows an exponential decay schedule from β₀ = 1 to β_max = 100 across the AL cycles, so early cycles are more exploratory and later cycles focus on hard cases. These two additions are agnostic to the underlying uncertainty function.

Evaluation uses the nnActive framework with nnU-Net as the backbone, rather than querying full volumes or restricting to 2D slices. Annotations are made as 3D patches of fixed size combined with a partial loss, which lets the model exploit partial annotations. The benchmark covers AMOS2022 (challenge task 2), Medical Segmentation Decathlon–Hippocampus, KiTS2021, and ACDC, under three Label Regimes (Low-, Medium-, High-Label) defined as annotation budgets in total patches, and two query patch sizes (Main and Patch ×1/2, the latter half the size along each dimension). This yields 24 distinct AL experiments, trained with four random seeds, a fixed test split, a custom nnU-Net trainer for 200 epochs in the 3D full-resolution configuration, and five AL cycles per experiment, totaling over 1000 nnU-Net training runs including ablations.

Performance is tracked with mean Dice-based metrics: Final Dice, Area Under Budget Curve (AUBC), Foreground Efficiency (FG-Eff), which relates the number of queried foreground voxels to model performance via an exponential fit, and the Pairwise Penalty Matrix (PPM), which counts how often one method significantly outperforms another based on paired t-tests, performed without family-wise error rate correction following prior work.

For the Roll-Out, the authors keep all nnActive design decisions, such as starting budget and preprocessing, but add two deployment-specific components: selecting the query patch size systematically from the median connected component sizes of the target structures, and normalizing query budgets to 50 or 100 patches per class depending on task complexity. Four unseen datasets are used, LiTS (liver and tumor, two foreground classes), WORD (16-class organ segmentation), Tooth Fairy 2 (42 dental structures), and MAMA MIA (single-class lesion segmentation), with a fixed 75% train-and-pool / 25% test split shared across four random seeds.

Why This Matters

The paper attacks a specific credibility problem in medical image computing: AL is widely proposed as a way to reduce annotation cost, but rigorous 3D evaluations keep finding that random sampling, once adapted, is hard to beat, and prior work noted that further research is necessary to prove the effectiveness of active learning for medical image segmentation. The authors argue that their dual evaluation, benchmark plus Roll-Out, provides compelling evidence that an AL method can consistently outperform random baselines adapted to 3D segmentation in terms of both performance and annotation efficiency in a realistic, close-to-production scenario. Because annotation cost is described as the main bottleneck for new clinical and research applications, a method that works without per-dataset tuning is directly useful.

Real-world applications:

• Segmentation of abdominal organs and tumors (AMOS2022, KiTS2021, LiTS) where labeling requires specialized radiological expertise.
• Brain structure segmentation on MRI (Medical Segmentation Decathlon–Hippocampus, ACDC cardiac MRI) where budgets for expert annotation are constrained.
• Dental structure labeling (Tooth Fairy 2, 42 structures) where dense annotation is labor-intensive.
• Breast cancer lesion segmentation (MAMA MIA) where large-scale datasets make full annotation expensive.

Industry relevance: The method is released as open source in the nnActive framework with deployment guidelines, uses the widely adopted nnU-Net as backbone, and adds no additional tuning burden, which lowers adoption barriers for clinical and commercial pipelines that need to decide which cases to annotate next.

Future Directions

• Extending beyond overlap metrics. The authors limit their evaluation to average Dice, the default overlap-based metric for semantic segmentation, and note results do not necessarily extend to boundary-based evaluation metrics or to settings where only specific classes are of interest.
• Closing the AMOS Low-Label gap. Since losses concentrate on AMOS at Low-Label Regimes with 15 classes, and the authors emphasize adapting the annotation budget to the number of classes for practitioners, determining principled budget-per-class rules remains open.
• Handling weak initial models. Stratified querying depends on the predictive capacity of the underlying model and becomes less effective when initial segmentation quality is insufficient; the paper's guidelines mitigate but do not remove this risk, and the discussion of pretrained approaches is cut off in the provided text.
• Reducing benchmark-overfitting risk. All AL methods face the risk of benchmark-specific overfitting because design decisions must be validated empirically; the dual benchmark-plus-Roll-Out evaluation mitigates but cannot fully eliminate this concern. The paper also notes open questions around obtaining representations for 3D patches, which limits applicability of representation-based query methods such as Core-Set and USIM.

Target Audience

Researchers and engineers working on active learning, annotation-efficient deep learning, and 3D medical image segmentation, particularly those using nnU-Net or the nnActive framework. It is also relevant to practitioners who must decide which volumes or patches to send to expert annotators, and to benchmark designers who need to understand why improved 3D-adapted random baselines are the correct comparison point rather than naive random sampling.

Authors’ abstract

Active learning (AL) has the potential to drastically reduce annotation costs in 3D biomedical image segmentation, where expert labeling of volumetric data is both time-consuming and expensive. Yet, existing AL methods are unable to consistently outperform improved random sampling baselines adapted to 3D data, leaving the field without a reliable solution. We introduce Class-stratified Scheduled Power Predictive Entropy (ClaSP PE), a simple and effective query strategy that addresses two key limitations of standard uncertainty-based AL methods: class imbalance and redundancy in early selections. ClaSP PE combines class-stratified querying to ensure coverage of underrepresented structures and log-scale power noising with a decaying schedule to enforce query diversity in early-stage AL and encourage exploitation later. In our evaluation on 24 experimental settings using four 3D biomedical datasets within the comprehensive nnActive benchmark, ClaSP PE is the only method that generally outperforms improved random baselines in terms of both segmentation quality with statistically significant gains, whilst remaining annotation efficient. Furthermore, we explicitly simulate the real-world application by testing our method on four previously unseen datasets without manual adaptation, where all experiment parameters are set according to predefined guidelines. The results confirm that ClaSP PE robustly generalizes to novel tasks without requiring dataset-specific tuning. Within the nnActive framework, we present compelling evidence that an AL method can consistently outperform random baselines adapted to 3D segmentation, in terms of both performance and annotation efficiency in a realistic, close-to-production scenario. Our open-source implementation and clear deployment guidelines make it readily applicable in practice. Code is at https://github.com/MIC-DKFZ/nnActive.