nnActive: A Framework for Evaluation of Active Learning in 3D Biomedical Segmentation

AI summary

Overview

Research area: Active Learning (AL) for semantic segmentation in 3D biomedical imaging (CT, MRI, microscopy).

Technical level: Advanced. The paper assumes familiarity with active learning terminology, nnU-Net, uncertainty-based query methods, and segmentation evaluation metrics.

One-sentence scope: The paper introduces nnActive, an open-source active-learning extension of nnU-Net, and uses it to run the largest study to date of uncertainty-based query methods for 3D biomedical segmentation, showing that a properly adapted "Foreground Aware Random" baseline is very hard for AL methods to beat.

What This Paper Is About

Training 3D biomedical segmentation models requires large, expensive, expert-produced annotation masks. Active Learning promises to cut that cost by selectively querying only the most informative samples, but the field has no shared benchmark and no consensus on whether AL actually beats simply sampling at random. The paper identifies four recurring evaluation pitfalls in prior 3D biomedical AL work and builds a framework and a large-scale study designed to avoid all four.

Key Contributions

1. nnActive framework: A highly configurable AL extension for nnU-Net that trains on partial annotations in the form of 3D patches, keeping state-of-the-art segmentation performance and adapting out-of-the-box to new segmentation tasks. Code is released at https://github.com/MIC-DKFZ/nnActive.
2. Foreground Aware Random sampling: New baseline strategies that address the foreground-background class imbalance typical of 3D medical images, by ensuring foreground is present in a set percentage of all queries.
3. Largest AL study to date: Over 7500 nnU-Net trainings covering 12 dataset-settings from four datasets with three Label Regimes each, evaluating 8 query methods, plus four large-scale ablation studies.
4. Foreground Efficiency (FG-Eff): A novel metric that measures annotation efficiency while accounting for the fact that annotating background costs far less effort than annotating foreground, unlike metrics that treat voxels as a proxy for effort.

Main Findings

• AL beats plain Random: All AL methods consistently outperform standard Random sampling on performance metrics comparing patch budgets. Random is consistently among the worst-performing methods for Final Dice, and all AL strategies beat it in over 37% of all evaluated budgets. However, Random draws the least amount of foreground voxels, giving it a good ranking on FG-Eff despite bad Final Dice and AUBC rankings.
• AL does not reliably beat Foreground Aware Random: Random 66% FG shows the best overall AUBC mean rank and a positive Win-/Lose ratio against all AL methods except Predictive Entropy. Random 33% FG generally performs slightly worse than most AL methods in AUBC and Dice. Foreground Aware Random also outperforms plain Random across all measured metrics except FG-Eff.
• Predictive Entropy is the strongest AL method: It achieves the best mean rank in both AUBC and Final Dice among AL methods, and is the only AL method with a positive win-loss ratio against Random 66% FG. Its gains concentrate in later AL stages, but its queries focus heavily on foreground, giving it relatively low FG-Eff compared to all other methods.
• Dataset properties drive AL benefit: On Hippocampus and KiTS, AL is beneficial against Random 66% FG; on ACDC the trend is neutral; on AMOS all AL methods are generally outperformed by Random strategies. On AMOS, models struggle with small organs such as adrenal glands, sometimes reaching a Final Dice of 0.
• Low budgets are hard for Greedy methods: Greedy query methods (BALD, Predictive Entropy) suffer from redundant queries at low annotation budgets, while Noisy query methods (PowerBALD, SoftrankBALD, PowerPE) are more diversified and are never outperformed by Random. On larger budgets and later stages, Noisy methods do not perform as well as their Greedy counterparts.
• Smaller query sizes help: Smaller query sizes with more AL loops improve performance over larger query sizes with fewer loops, an effect more pronounced for Greedy QMs. No setting showed a smaller query size causing a significant performance decrease, but the benefit comes at increased computational cost scaling inversely proportional to query size.
• Longer training improves query quality: Training for 500 epochs produced significantly better queries than the 200-epoch main-study setting, even after accounting for the general Dice gains of longer training (the Precomputed setting). On AMOS — where AL does not beat Random — the ranking shifts toward favoring AL methods; on KiTS, where AL already wins, the ranking stays similar.

Methodology in Plain English

The authors first state four requirements for trustworthy AL evaluation and match them to four pitfalls found in prior work:

• P1 — too few settings: They evaluate on four datasets (AMOS2022 challenge task 2, Medical Segmentation Decathlon Hippocampus, KiTS2021, ACDC) and three annotation budgets per dataset (Low, Medium, High Label Regime), each spanning 5 AL loops, using a 75% training & pool split and a 25% test split that is identical across all seeds and experiments.
• P2 — 2D models on 3D data without partial annotations: They use the 3D full-resolution nnU-Net configuration trained for 200 epochs, with a region sampling enhancement that enriches the observed region with unlabeled context, and train with the partial loss on 3D patches rather than slices. Models are trained as an ensemble of five via 5-fold cross-validation, fully retrained from scratch at every AL loop, averaged over four seeds.
• P3 — unadapted Random baseline: They add Random 33% FG and Random 66% FG, which mix fully random patches with foreground-oversampled patches (half centered on a randomly chosen foreground class, half centered on a foreground class border).
• P4 — voxel-only cost measurement: They add FG-Eff, derived from an exponential decay fitted to the performance gap against a model trained on the entire dataset, plotted against the number of foreground voxels.

Query methods are drawn in two steps: first a set of best patches per image is selected by an uncertainty function with maximum allowed overlap, then the final query is drawn across the training & pool dataset. The study uses no allowed overlap (o = 0) and mean aggregation with aggregation size equal to the query patch size; the beta parameter for PowerBALD, SoftrankBALD and PowerPE is set to 1. Query patch sizes were chosen from median image size and structure size: AMOS 32×74×74, KiTS 64×64×64, ACDC 4×40×40, Hippocampus 20×20×20. Total budgets: ACDC 150/300/450 patches, AMOS 200/1000/2500, KiTS 200/1000/2500, Hippocampus 100/200/300, with starting budget and query size equal to 20% of the full budget of each regime.

Evaluation uses Mean Dice per 3D image, Final Dice of the last loop, AUBC, FG-Eff, and a Pairwise Penalty Matrix based on t-tests at α = 0.05. The eight evaluated query methods are five uncertainty-based methods — Predictive Entropy, BALD, PowerBALD, SoftrankBALD, PowerPE — plus the three Random strategies. Ablations cover query size (three settings: halved, identical, doubled, on ACDC/AMOS/KiTS Low- and High-Label Regimes, measured with Kendall's τ at α = 0.1), training length (500 epochs, 200 epochs, and 500 epochs on precomputed query trajectories, on AMOS and KiTS Medium- and High-Label Regimes, t-test at α = 0.1), and noise strength in Noisy QMs (an ablation of PowerBALD reducing noise over 6 steps on ACDC, AMOS and KiTS across Low, Medium and High Label Regimes). The fourth ablation is not described in the provided content, which is truncated mid-sentence.

Why This Matters

Impact on research: The paper shows that a large share of prior 3D biomedical AL evaluations used too few datasets, 2D slice models, and a plain Random baseline, and that fixing these issues changes the verdict — most AL methods cannot reliably beat a foreground-aware Random baseline when querying 3D patches. It provides a shared, reproducible benchmark and a metrics suite that distinguishes absolute performance, relative performance, and annotation efficiency.

Real-world applications:

• Clinical delineation of organs and pathologies in CT and MRI, where specialized personnel perform the annotation.
• Radiotherapy planning workflows that depend on accurate structure contours.
• Microscopy-based biomedical research pipelines that require dense pixel-level labels.
• Curation of large hospital scan archives, where only a small fraction of the stored data can realistically be annotated.

Industry relevance: nnU-Net is widely used and extended by an ecosystem of downstream projects, so an AL layer built directly on it is practical to integrate into existing medical imaging toolchains. The finding that annotation efficiency depends on the amount of foreground queried, not just voxel counts, informs how annotation vendors and clinical teams estimate labelling effort and budget.

Future Directions

• Combine AL with orthogonal efficiency approaches: Requirement R2 calls for measuring whether AL gains hold up in combination with self-supervised or semi-supervised learning, and the authors note that semi-supervised pretraining and pretrained models have been used in related work.
• Test on held-out rollout datasets: The authors recommend evaluating the best-performing AL methods on separate held-out test datasets independent of the development datasets to prevent overfitting and to measure a generalization gap.
• Resolve the compute-versus-benefit trade-off: Smaller query sizes and longer training both improve AL performance but increase computational cost, so the amortization of AL setup and compute costs against the annotation savings remains an open question.
• Explain the dataset dependence of AL benefits: The differing outcomes on AMOS, KiTS, ACDC and Hippocampus, tied to number of classes, class size, class location and foreground-to-background ratio, are discussed qualitatively and leave room for a more systematic characterization of when AL pays off.

Target Audience

Researchers and practitioners working on 3D biomedical image segmentation and annotation-efficient learning; developers extending nnU-Net or building AL tooling for medical imaging; clinical and industrial teams deciding whether to invest in AL rather than random or foreground-aware sampling; and methodologists interested in rigorous benchmarking design and in metrics that reflect real annotation effort.

Authors’ abstract

Semantic segmentation is crucial for various biomedical applications, yet its reliance on large annotated datasets presents a bottleneck due to the high cost and specialized expertise required for manual labeling. Active Learning (AL) aims to mitigate this challenge by querying only the most informative samples, thereby reducing annotation effort. However, in the domain of 3D biomedical imaging, there is no consensus on whether AL consistently outperforms Random sampling. Four evaluation pitfalls hinder the current methodological assessment. These are (1) restriction to too few datasets and annotation budgets, (2) using 2D models on 3D images without partial annotations, (3) Random baseline not being adapted to the task, and (4) measuring annotation cost only in voxels. In this work, we introduce nnActive, an open-source AL framework that overcomes these pitfalls by (1) means of a large scale study spanning four biomedical imaging datasets and three label regimes, (2) extending nnU-Net by using partial annotations for training with 3D patch-based query selection, (3) proposing Foreground Aware Random sampling strategies tackling the foreground-background class imbalance of medical images and (4) propose the foreground efficiency metric, which captures the low annotation cost of background-regions. We reveal the following findings: (A) while all AL methods outperform standard Random sampling, none reliably surpasses an improved Foreground Aware Random sampling; (B) benefits of AL depend on task specific parameters; (C) Predictive Entropy is overall the best performing AL method, but likely requires the most annotation effort; (D) AL performance can be improved with more compute intensive design choices. As a holistic, open-source framework, nnActive can serve as a catalyst for research and application of AL in 3D biomedical imaging. Code is at: https://github.com/MIC-DKFZ/nnActive