Foundation Model-Enabled Efficient Data Sampling (FEEDS): A label-efficient training strategy for pan-cancer, multi-tracer PET/CT datasets

AI summary

Overview

• Research area: Medical computer vision — whole-body lesion segmentation on PET/CT scans, with a focus on label-efficient training in pan-cancer, multi-tracer settings.
• Technical level: Advanced (requires familiarity with foundation model embeddings, nnU-Net, Dice/volume metrics, and semi-supervised/active learning terminology), though the central idea is intuitive.
• Scope in one sentence: The paper proposes FEEDS, a one-step sampling strategy that uses DinoV2 foundation model embeddings to pick the most diverse unlabeled PET/CT cases for expert annotation, and shows it reaches performance comparable to fully labeled training with a 30% labeling budget across three datasets.

What This Paper Is About

Training deep learning models to automatically outline (segment) cancerous lesions on whole-body PET/CT scans requires large numbers of expert-drawn annotations, which are slow and expensive to produce because lesion number, size, and appearance vary widely by cancer type and tracer. The authors ask a practical question: given a large archive of unannotated scans and a small budget for expert labeling, which cases should a clinician annotate first? FEEDS answers this by using vision foundation model embeddings to select the cases that are least represented in the current labeled set, adding them to training in a single round rather than through iterative active-learning or semi-supervised loops.

Key Contributions

1. A one-step, label- and compute-efficient sampling paradigm (FEEDS). It extracts DinoV2 features from PET maximum-intensity projection (MIP) images, computes each unlabeled case's minimum cosine distance to the labeled pool separately for FDG and PSMA tracers, and selects the farthest (most diverse) cases for annotation — requiring no iterative training.
2. A pan-cancer, multi-tracer benchmark. The method is trained and validated on AutoPET-III and evaluated on three held-out sets: a held-out AutoPET-III test set (321 scans; FDG and PSMA; lung cancer, lymphoma, melanoma, prostate cancer, and no-cancer cases), Deep-PSMA (200 scans; prostate cancer), and an internal Dartmouth Hitchcock Medical Center (DHMC) cohort (23 PSMA scans; prostate cancer).
3. Evaluation beyond voxel-level Dice. The paper adds lesion-level (sensitivity, PPV) and anatomical-region-level assessment (liver, lung, bone, high-risk bone) using a pretrained TotalSegmentator model, explicitly framed around treatment planning and high-risk disease.
4. A matched comparison of sampling strategies. FEEDS is compared against random sampling (five iterations per budget), Determinantal Point Process (DPP)-based sampling using the same foundation model features, and pseudo-label-based semi-supervised learning, plus 10% and 100% labeled-data baselines.

Main Findings

• Reduced annotation burden: FEEDS matched fully-labeled training performance with a 30% labeling budget — the abstract describes this as "matching fully-labeled (100%) training performance with 70% less annotation burden."
• Ablation results on the AutoPET-III validation set (N=247): The 10% fixed labeled baseline scored Dice 0.594, FPVol 20.38, FNVol 23.72. Averaged over five random iterations, mean Dice was 0.637, 0.639, and 0.648 at the 10+10%, 10+20%, and 10+30% budgets; FEEDS reached 0.637, 0.645, and 0.655 at the same budgets, with lower FNVol in every case (12.95 vs. 14.49; 11.25 vs. 14.59; 12.10 vs. 14.39). The 100% labeled upper bound reached Dice 0.671, FPVol 22.12, FNVol 8.64.
• DPP and pseudo-label comparison: DPP with the same features performed comparably to FEEDS but did not achieve the lowest FNVol at each budget. Semi-supervised learning using pseudolabels from a 10%-trained model scored Dice 0.634, below both FEEDS and a 10+10% strongly labeled model, suggesting the pseudolabels were noisy.
• AutoPET-III held-out test set: The 10+20% FEEDS model achieved overall Dice 0.61 vs. 0.64 for 100% labeled training, FNVol 11.20 vs. 9.01 cc, but better FPVol (6.59 vs. 9.35 cc) and better than random sampling across all metrics except PSMA FPVol.
• Disease-stratified test results: FEEDS Dice was comparable to 100% for all diseases — lung cancer 0.73 vs. 0.75, lymphoma 0.69 vs. 0.73, melanoma 0.66 vs. 0.65, prostate cancer 0.54 vs. 0.57 — and FEEDS reduced FPVol in negative (disease-free) cases to 5.61, versus 8.11 for Rand-Mean and 9.85 for 100%.
• Lesion-level test results: FEEDS lesion sensitivity 0.743 and PPV 0.821, versus 0.728/0.807 for random sampling and 0.758/0.784 for 100% labeled training — the best PPV of the compared methods.
• Region-level test results: FEEDS achieved region-level Dice higher than or equal to the 100% model within anatomical regions, the lowest FPVol in the liver, and higher bone specificity than 100% labeled training (0.94 vs. 0.84) despite using only 30% of the labels.
• Unseen dataset, Deep-PSMA (N=200): FEEDS achieved the lowest FPVol (3.389 vs. 5.556 for Rand-Mean and 7.520 for 100%), better FNVol than random sampling (14.979 vs. 17.069), and higher lesion sensitivity (0.658 vs. 0.624), approaching the fully supervised model (Dice 0.645 vs. 0.655 for 100%).
• Unseen dataset, DHMC (N=23): The 10% labeled model had the highest Dice (0.536) and lowest FPVol (16.034); FEEDS was second on both (Dice 0.534, FPVol 19.081) and achieved the best FNVol (2.721) and second-best lesion sensitivity (0.766, behind the 100% model's 0.774). The 100% AutoPET-trained model performed worst here (Dice 0.489, FPVol 34.728), which the authors attribute to scanner and protocol differences — notably older scanners in AutoPET-III versus the newer Siemens Biograph Vision 600 used at DHMC.
• Statistical analysis: Using a two-sided Wilcoxon signed-rank test with FDR-BH correction (α = 0.05), FEEDS significantly reduced false-positive volume versus random sampling in all datasets (pooled p = 5.57 × 10⁻⁴⁴, Δ = −1.64). Non-inferiority against the 100% model was assessed with a one-sided paired t-test; all pooled comparisons passed except DeepPSMA FN volume. The text states non-inferiority margins of 0.05 for Dice and 5 cc for FP/FN volumes, while the Table 6 caption states margins of 0.02 Dice and 2 mL.
• Interpretability check: Pairwise inspection of embedding space showed the closest cases were often longitudinal scans of the same patient or patients with similar disease distribution, while farthest cases showed wide variability in cancer spread — supporting the diversity rationale.

Methodology in Plain English

FEEDS proceeds in three steps. First, for each 3D PET image the authors compute a maximum-intensity projection, normalize it, and pass it through the pretrained DinoV2 encoder (a vision foundation model trained on 142 million natural images) to obtain a 768-dimensional image-level token representation. Second, for each unlabeled case they measure the minimum cosine distance to any already-labeled case, doing this separately for FDG and PSMA scans because the two tracers form distinct clusters. The cases with the largest distances — the ones least like anything already labeled — are chosen for expert annotation, which also preserves the FDG:PSMA ratio. Third, those newly annotated cases are combined with the small fixed labeled pool and used to train an nnU-Net segmentation model for 280 epochs, with preprocessing following Rokuss et al.

Training used 1,043 AutoPET-III scans, of which 103 (66 FDG, 37 PSMA) were the fixed 10% labeled pool and 940 formed the unlabeled pool; validation used 247 AutoPET-III cases, with patient-level splits to prevent leakage from longitudinal scans. The authors compared FEEDS against random sampling (five repeated iterations per budget to capture variability), DPP-based sampling on the same features, pseudo-label semi-supervised learning trained on 10% strong labels plus 90% pseudolabels, and 10% and 100% labeled baselines. Based on validation ablations they selected the 10+20% budget (30% of total labels) for all test-set evaluation. PET intensities were clipped below the 5th and above the 95th percentile and z-score normalized, and anatomical regions for the region-level analysis were segmented from CT using the pretrained TotalSegmentator model.

Why This Matters

The work reframes annotation budgeting as a data-selection problem that can be solved once, upfront, without iterative retraining — making label-efficient PET/CT segmentation more practical for clinical research groups with limited compute.

Real-world applications:

• Annotation queue prioritization: Clinical archives hold large volumes of unannotated PET/CT scans; FEEDS ranks them by distance from a small labeled set so radiologists annotate the most informative cases first.
• Pan-cancer, multi-tracer model development: A single training workflow handles FDG and PSMA scans across lung cancer, lymphoma, melanoma, and prostate cancer, rather than requiring one model per tracer or disease.
• Treatment planning support: Lesion counting matters for PSMA-PET-guided decisions (the paper cites the ≤5-lesion oligometastatic threshold) and region-level detection in liver, lung, and high-risk bone informs radiation therapy decisions aimed at reducing skeletal-related events.
• Tumor-burden tracking and longitudinal monitoring: The paper frames voxel-level segmentation as the basis for tumor-burden computation and longitudinal tracking, and the reduced false-positive volume in disease-free patients is presented as a clinically important property.

Industry relevance: the method needs no task-specific training for sample selection, uses a deployment-friendly foundation model, is tracer-agnostic, and is described as scaling to large unannotated repositories — properties relevant to imaging software vendors, clinical trial imaging cores, and health systems building automated staging pipelines.

Future Directions

• Move from 2D to 3D features: FEEDS currently uses DinoV2 features from 2D PET maximum intensity projections rather than 3D volumes, which may lose spatial information; the authors plan to explore 3D-native foundation models.
• Compare with domain-specific foundation models: DinoV2 is trained on natural images, not radiology or PET data; the authors state they will compare against other domain-specific foundation models.
• Incorporate CT information: Selection currently uses PET features only; adding CT features alongside PET is listed as a future direction.
• Extend beyond one-time training: The authors suggest FEEDS can support few-shot learning as new unlabeled data becomes available, and ask which cases to select next in an ongoing annotation campaign.

Target Audience

Researchers and practitioners in medical image analysis, computational radiology, and nuclear medicine who work with PET/CT data and face limited annotation budgets; machine learning engineers interested in foundation-model-driven data selection; and clinician-scientists in radiation oncology or urology who need to understand what automated lesion segmentation can and cannot currently deliver for staging, treatment planning, and lesion counting. Readers without a machine learning background will find the central concept accessible, but the quantitative results require familiarity with Dice, false-positive/false-negative volumes, sensitivity, and PPV.

Authors’ abstract

Automated lesion segmentation in whole-body PET/CT imaging can assist clinicians with cancer detection, staging, and treatment planning across radiotracers and cancer types. However, training lesion segmentation models that capture variations in lesion size, distribution, and appearance requires large annotated datasets, whose creation is both time- and expertise-intensive. As a result, models trained on limited labeled PET/CT data often lack the accuracy and generalizability needed for clinical use. We present FEEDS (Foundation model-Enabled Efficient Data Sampling), a label- and compute-efficient learning strategy that uses vision foundation model embeddings to select the most informative and diverse unlabeled cases for expert annotation. Unlike unsupervised, semi-supervised, and active learning approaches, FEEDS is a one-step training paradigm requiring only a limited, representative training set, making it label- and compute-efficient. We train and validate FEEDS using the AutoPET-III dataset. We test its accuracy and generalizability on three held-out sets: AutoPET-III, DeepPSMA, and an internal Dartmouth-Hitchcock Medical Center dataset. We evaluate clinical utility at the voxel, lesion, and anatomic region level to assess performance in high-risk areas and treatment planning utility. FEEDS outperforms random-sampling-based labeling, pseudolabel-based semi-supervised learning, and training with limited labeled data alone. It generalizes across all three test sets, FDG and PSMA tracers, and multiple diseases, matching fully-labeled (100\%) training performance with 70\% less annotation burden. FEEDS addresses the challenge of label scarcity in an automatic lesion segmentation framework by providing a practical approach for constructing representative and diverse annotation queues from large, unannotated clinical repositories.