BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation

AI summary

Overview

Research area: Biomedical image segmentation (computer vision applied to medical imaging), with a theoretical component rooted in information theory and asymptotic statistics.

Technical level: Intermediate overall, with an Advanced theoretical section (expected Fisher Information, Loewner ordering, MLE asymptotics, softmax Fisher decomposition). The experimental sections are accessible to anyone familiar with Dice, HD-95, and NSD metrics.

Scope in one sentence: The paper argues that sub-dividing the single "background" class of a lesion segmentation task into semantically meaningful auxiliary structures (organs, tissues) improves small-lesion segmentation, and supports this with an information-theoretic proof plus experiments on five datasets, three architectures, and several ways of obtaining the auxiliary labels.

What This Paper Is About

Most lesion segmentation models are trained as binary problems: lesion pixels versus everything else, with all non-lesion pixels collapsed into one "background" class. The authors argue that this discards the anatomical context in which lesions appear, and that this coarsening is part of why small, sparse lesions are so hard to segment reliably.

Their proposal, called BackSplit, simply keeps those non-lesion regions as separate classes during training (for example, kidney parenchyma alongside a kidney cyst), trains the model jointly on the target and the auxiliary structures, and at inference still predicts only the target class. The goal is better lesion delineation without changing the network or paying any extra inference cost.

Key Contributions

1. A theoretical result on label coarsening. The authors prove (Theorem 1) that the expected Fisher Information of the full multi-class problem equals that of the collapsed binary problem plus a non-negative "missing information" term, so binary training can only lose information relative to multi-class training. A companion corollary (Corollary 1) states that the multi-class maximum-likelihood estimator is asymptotically no less efficient for any smooth function of the parameters, including the target-class posterior.

2. A closed-form softmax analysis. Proposition 1 instantiates the general result for the softmax output layer used by most segmentation networks, writing the multi-class Fisher Information as the binary Fisher Information plus an explicit extra term involving the distribution over non-target classes.

3. Empirical validation across five datasets and three architectures. BackSplit is tested on KiTS23 (489 CT scans, target cyst), PANTHER (92 MRI scans, target tumor), NSCLC-Radiomics (415 CT scans, target GTV), AutoPET 3 (1,611 paired CT and PET scans, target tumors), and MSWAL (484 abdominal CT scans, multi-lesion). Backbones are U-Net, ResEnc U-Net (via nnU-Net), and SegResNet.

4. Robustness to imperfect auxiliary labels. Gains persist when auxiliary structures come from automatically generated masks rather than manual annotations: a U-Net pretrained on AbdomenAtlas1.0Mini, large pretrained models TotalSegmentator and VIBE-Segmentator, and noisy pseudo-labels from the interactive model nnInteractive using 7 and 10 simulated clicks.

Main Findings

• Structured background supervision substantially improves cyst segmentation. On KiTS23 with the U-Net backbone, binary training reached Dice 0.1787, HD-95 428.4097, and NSD 0.1695; BackSplit reached Dice 0.4573, HD-95 267.2722, and NSD 0.6004. With ResEnc U-Net, Dice moved from 0.2532 to 0.4614; with SegResNet, from 0.2914 to 0.4290.

• Gains appear across modalities and targets. PANTHER-MR (U-Net) went from Dice 0.4784 to 0.5251. NSCLC-Radiomics (U-Net) went from Dice 0.4969 to 0.5256; with ResEnc U-Net, from 0.502 to 0.5386; with SegResNet, from 0.5061 to 0.5131. These were run with 5-fold cross-validation and manual auxiliary labels.

• Model parameters and inference cost are unchanged. Parameter counts are identical between baseline and BackSplit for a given architecture (for example, U-Net 31.19M on KiTS23; ResEnc U-Net 102.35M; SegResNet 18.79M).

• Automatically generated auxiliary masks are enough. On AutoPET, a U-Net with organ masks derived from a model pretrained on AbdomenAtlas1.0Mini improved Dice from 0.3881 to 0.4435 and NSD from 0.3478 to 0.4128. On MSWAL, the mean over all lesion types improved from Dice 0.2724 to 0.3190 and NSD from 0.5577 to 0.6218.

• MSWAL per-lesion results are mixed in Dice but consistent in surface metrics. Gallstone Dice rose from 0.2665 to 0.3497, pancreatic cancer from 0.1836 to 0.3228, liver tumor from 0.2666 to 0.3312, liver cyst from 0.317 to 0.3571, kidney cyst from 0.4931 to 0.5104, and kidney tumor from 0.1855 to 0.1979. Kidney stone Dice decreased from 0.195 to 0.1639 while its NSD improved from 0.6247 to 0.5401 (lower HD-95 is better: 471.4653 to 484.3217 for HD-95, which worsened slightly).

• Large pretrained segmenters work as label sources. In a single-fold AutoPET evaluation, regular training gave Dice 0.3921, BackSplit gave 0.4537, BackSplit with TotalSegmentator masks gave 0.4456, and BackSplit with VIBE-Segmentator masks gave 0.4314. On MSWAL, regular training gave Dice 0.2518, BackSplit 0.2978, with TotalSegmentator 0.2843, and with VIBE-Segmentator 0.2646.

• Interactive, noisy pseudo-labels preserve the benefit. Using nnInteractive with 7 and 10 simulated clicks as auxiliary supervision, BackSplit still outperformed regular training on all five datasets in single-fold tests. On KiTS23, regular training gave Dice 0.2033, BackSplit 0.5297, BackSplit with 7 clicks 0.4919, and with 10 clicks 0.4921. On NSCLC-Radiomics, regular training gave 0.5279 and BackSplit with 7 clicks reached 0.5869. The authors describe the nnInteractive auxiliary segmentations as exhibiting only modest accuracy.

• Fine-tuning an existing binary model helps quickly. On KiTS23 with cyst as the target, fine-tuning a pretrained binary U-Net with auxiliary structures improved Dice immediately, even after 50 epochs, and the score nearly doubled by 250 epochs relative to the original binary model.

• Partial auxiliary supervision has a non-monotonic start. When training from scratch with only a small fraction of samples carrying auxiliary labels, performance initially dropped, which the authors attribute to confusion between target and auxiliary structures; as the proportion of auxiliary labels increased linearly, the model recovered and approached full BackSplit performance.

• Theory explains the design choice. The score of the coarsened problem is the conditional expectation of the full multi-class score (Lemma 1). Collapsing non-target classes to a single background averages gradient directions that would otherwise differ (for example, organ A versus organ B), removing curvature directions in parameter space. Equality in Theorem 1 holds only when the coarsened score is fully determined by the coarsened labels — for instance when K = 2 or the non-target distribution is deterministic.

Methodology in Plain English

The authors start from a modeling observation rather than an architecture change. In standard practice, a training label map has one value for the lesion and one value for everything else. BackSplit instead keeps the auxiliary anatomical classes visible in the label map during training, so the network's softmax head learns to separate lesion from organ A, organ B, and so on. At test time only the lesion output is read out; the auxiliary classes act purely as a richer training signal, so nothing changes at deployment.

To justify this, they set up the problem statistically: there is a full multi-class label Y, and a coarsened binary label Z that marks only whether the pixel belongs to the target class c. They apply the missing-information principle to show that the score (gradient of the log-likelihood) of the coarsened problem is the conditional expectation of the full score, and then decompose the full Fisher Information into the coarsened Fisher Information plus a non-negative conditional variance term. Passing through the standard MLE central limit theorem and the delta method, they conclude that prediction variance for the target posterior is no larger, and generally strictly smaller, under multi-class training. They then write out the explicit matrices for a softmax head.

Empirically, they run controlled baseline-versus-BackSplit pairs where everything except the label space is held constant: identical nnU-Net configurations, identical hyperparameters, the same backbone. The experiments are organized in escalating levels of realism: manual auxiliary labels (KiTS23, PANTHER, NSCLC-Radiomics), automatically generated labels from an organ model trained on AbdomenAtlas1.0Mini (AutoPET, MSWAL), labels from TotalSegmentator and VIBE-Segmentator, and finally deliberately noisy labels from nnInteractive seeded with 7 or 10 clicks. They also test adoption paths — fine-tuning an already-trained binary model, and training from scratch when only some cases have auxiliary annotations.

Why This Matters

Impact on research. The paper reframes a practical segmentation problem as a label-granularity problem and backs the reframing with a Fisher Information argument. That connects the medical segmentation literature to earlier work on label coarsening and provides a principled reason, rather than only empirical evidence, for why context-aware and auxiliary-supervision methods help. It also suggests that the auxiliary network used in prior work is a design choice rather than a necessity: the same benefit comes from the label space alone, with no architectural branch or extra loss term.

Real-world applications:

• Improving cyst, tumor, and lesion detection on CT and MRI where small targets currently produce many false positives, without retraining or redeploying a heavier model.
• Reusing organ segmentations that hospitals or research groups already have (from TotalSegmentator or similar tools) as free auxiliary supervision for lesion tasks.
• Building interactive annotation workflows where a few clicks per structure generate pseudo-labels good enough to train a better lesion model, as demonstrated with nnInteractive.
• Enriching partially annotated datasets: both the fine-tuning experiment and the partial-supervision experiment speak to settings where only some cases or some structures are labeled.

Industry relevance. Because the paradigm adds zero parameters and zero inference cost, it can be dropped into existing nnU-Net pipelines and regulatory-cleared segmentation workflows without changing the deployed model's footprint or latency. This matters for clinical deployment, where false positives and unstable predictions are the stated obstacles. The reliance on widely available pretrained organ models also lowers the annotation cost barrier that usually blocks richer supervision schemes.

Future Directions

• Small-data behavior. The authors acknowledge that their analysis assumes a large-sample regime where finer label granularity increases Fisher curvature, and that in small-data settings this could amplify sampling noise and risk overfitting. They report not observing this within typical medical segmentation dataset sizes, but the boundary is left open.

• Full exploration of partial supervision. The paper shows a non-monotonic trend when only a fraction of samples include auxiliary annotations and does not report a full characterization of how much auxiliary coverage is needed before gains appear.

• Choosing which structures to split. The support structures in these experiments are selected per dataset (kidney and tumors for KiTS23, pancreas for PANTHER, multiple organs for NSCLC-Radiomics, AutoPET, and MSWAL). The paper does not report a general rule for selecting or ranking candidate auxiliary structures.

• Extension beyond the target-posterior analysis. The theory covers smooth functionals of the parameters and the softmax parameterization specifically; the paper does not report how the argument behaves for other output parameterizations or for the metric-based objectives (Dice-like losses) actually used in training.

Note: the provided content is truncated mid-sentence in the conclusion ("In futu"), so the authors' own explicit list of future work is not reported.

Target Audience

Medical imaging and computer vision researchers working on lesion or small-object segmentation; practitioners building clinical segmentation pipelines with nnU-Net or similar frameworks who want accuracy gains without added inference cost; methodologists interested in information-theoretic or Fisher-Information analyses of supervision granularity; and graduate students who want an example of connecting a simple dataset-labeling change to an asymptotic statistical argument and then validating it across many datasets and backbones. Readers mainly looking for a new architecture or loss function will find this paper deliberately takes the opposite route.

Authors’ abstract

Segmenting small lesions in medical images remains notoriously difficult. Most prior work tackles this challenge by either designing better architectures, loss functions, or data augmentation schemes; and collecting more labeled data. We take a different view, arguing that part of the problem lies in how the background is modeled. Common lesion segmentation collapses all non-lesion pixels into a single "background" class, ignoring the rich anatomical context in which lesions appear. In reality, the background is highly heterogeneous-composed of tissues, organs, and other structures that can now be labeled manually or inferred automatically using existing segmentation models. In this paper, we argue that training with fine-grained labels that sub-divide the background class, which we call BackSplit, is a simple yet powerful paradigm that can offer a significant performance boost without increasing inference costs. From an information theoretic standpoint, we prove that BackSplit increases the expected Fisher Information relative to conventional binary training, leading to tighter asymptotic bounds and more stable optimization. With extensive experiments across multiple datasets and architectures, we empirically show that BackSplit consistently boosts small-lesion segmentation performance, even when auxiliary labels are generated automatically using pretrained segmentation models. Additionally, we demonstrate that auxiliary labels derived from interactive segmentation frameworks exhibit the same beneficial effect, demonstrating its robustness, simplicity, and broad applicability.