Unsupervised Adaptation of 3D CT Foundation Models for 3D CBCT Segmentation

AI summary

Overview

• Research area: Medical image analysis / computer vision — unsupervised domain adaptation (UDA) and 3D foundation models applied to CT-to-CBCT liver segmentation.
• Technical level: Advanced. The paper assumes familiarity with 3D segmentation architectures (3D U-Net, nnU-Net, SAM-Med3D, VISTA-3D), adversarial domain adaptation, and representation-learning objectives such as Barlow Twins redundancy reduction.
• Scope (one sentence): The paper proposes a lightweight, architecture-agnostic feature-alignment framework that adapts pretrained 3D CT segmentation models — both CNN- and ViT-based — to cone-beam CT (CBCT) liver segmentation using no target-domain annotations and no inference-time adaptation.

What This Paper Is About

CBCT is used in interventional procedures and radiation therapy, but segmenting it with deep learning is hard because annotated CBCT data are scarce and because CBCT differs substantially from diagnostic CT (limited reconstructed field of view, acquisition-geometry artifacts, scatter and beam hardening, and locally injected iodine contrast producing high-intensity regions). These differences mean that large pretrained 3D CT foundation models do not transfer reliably to CBCT, and preprocessing pipelines built for full-field CT often fail on CBCT. The goal is therefore to adapt CT-trained models to CBCT using only labeled CT data (source) and unlabeled CBCT data (target), without collecting any CBCT annotations.

Key Contributions

1. A 3D feature-level unsupervised domain adaptation framework for adapting foundation-model representations from CT to CBCT, designed to work with both CNN-based and ViT-based architectures.
2. An evaluation on two complementary CT–CBCT liver segmentation benchmarks — one radiation therapy (DR) and one interventional (DI) — showing consistent gains over existing UDA methods and zero-shot foundation models.
3. The release of manually curated liver segmentations for the publicly available Pancreatic 3D CBCT dataset (Pancreatic-CT-CBCT-SEG), which originally provided annotations for several organs at risk but no liver masks.
4. Public release of the code, trained models, and weights at https://github.com/mirabll/FARR3D/.

Main Findings

• Source-only baseline is weak: A source-only (CT-trained) nnU-Net reaches F1 of 66.5 on the radiation therapy CBCT dataset (DR) and 80.1 on the interventional dataset (DI), showing the CT-to-CBCT gap is real.
• Ours improves on the source-only baseline: nnU-Net + Ours reaches F1 78.0 (DR) and 86.0 (DI), adding only 0.08M additional trainable parameters.
• Best among UDA methods on both datasets: DA-nnUNet (2024) reaches 73.3 (DR) / 84.6 (DI) with 6.62M extra parameters; MDD-UNet (2023) reaches 66.7 / 80.6 with 0.09M; SIFA-3D (2023) reaches 55.2 / 64.7 with 37.69M; MAPSeg (2024) reaches 59.9 / 70.2 with 210.96M.
• Image-alignment and self-training struggle: SIFA-3D underperforms because of the large field-of-view discrepancy between domains, and MAPSeg fails to surpass the source-only baseline, which the authors attribute to sensitivity to poor initialization and error propagation during self-training.
• Zero-shot foundation models are mostly insufficient in automatic mode: MA-SAM 15.4 (DR) / 61.8 (DI); Merlin 18.8 / 33.7; TotalSegmentator 64.1 / 73.8; VISTA-3D automatic 42.7 / 48.3.
• Prompting helps but is not enough alone: With 10 points, VISTA-3D reaches 80.0 (DR) / 74.9 (DI) and SAM-Med3D reaches 73.8 / 65.3; with 1 point, MedSAM2 reaches 40.4 / 30.7 and SAM-Med3D 44.2 / 53.6.
• Adapting prompted foundation models yields large gains: VISTA-3D + Ours (10 pts) reaches 90.0 (DR) / 81.5 (DI) with 0.99M extra parameters; SAM-Med3D + Ours (10 pts) reaches 77.6 / 74.0 with 11.37M extra parameters.
• DI is harder for foundation models than DR: VISTA-3D (interactive branch) and MedSAM2 perform worse on DI than on DR, which the authors link to iodine-induced high intensities falling outside the range expected by default preprocessing (min–max normalization or CT-specific intensity scaling); nnU-Net benefits from percentile normalization (0.5th–99.5th), which clips such outliers.
• Hyperparameter robustness: On dataset DR, an ablation over α and γ shows a large contiguous region with F1 ≥ 88.5%, with the best run reaching F1 = 90.0%. The paper reports single-run values rather than standard deviations or significance tests.
• Remaining headroom: A model trained on target data only (nnU-Net, Target Only) reaches 91.4 (DR) / 93.7 (DI), which upper-bounds what adaptation achieves.
• Qualitative behavior: MedSAM2 tends to over-segment and shows reduced sensitivity to high-intensity structures compared to VISTA-3D; after adaptation, the source-only model becomes more robust to high-intensity artifacts and field-of-view variations.

Methodology in Plain English

The method keeps the underlying segmentation network untouched in structure and instead re-organizes it into three parts: a feature encoder ψ that extracts the shared representation z = ψ(x), a representation head f, and a task-specific prediction head g for the segmentation output. An extra adversary f′ — initialized as a copy of f — is added alongside f, and it is thrown away at inference time.

Training alternates between three objectives:

• Task loss on source data: f and g learn to segment CT correctly using the labeled CT data.
• Adversary update: f′ is trained to produce representations similar to f on source samples (alignment) but dissimilar to f on target CBCT samples (separation). A hyperparameter γ controls the separation strength.
• Encoder update: ψ is trained on the task loss plus alignment terms — one for source features weighted by α, one for target features weighted by γ — so that the representations of f and f′ become indistinguishable on both domains.

The alignment and separation losses are built by revisiting the Barlow Twins redundancy-reduction mechanism. Each batch of features is reshaped, centered, and L2-normalized, then a cross-correlation matrix is computed between f(z) and f′(z). The alignment loss pushes the diagonal of that matrix toward 1 and off-diagonal entries toward 0 (perfect correlation along matched feature dimensions, no redundancy across dimensions); the separation loss pushes everything, including the diagonal, toward 0 (complete decorrelation). Both losses are task-agnostic and operate in the space of f rather than g, which is what lets the method drop the task-specific adversarial head used by MDD-style approaches.

The authors also provide a theoretical argument: with a capacity assumption (Assumption 1), the adversary that attains the closed-form minimizer of the adversary objective (Lemma 1) is a global optimum, and minimizers of ψ then align the marginal feature distributions of source and target, that is p(z) = q(z) at optimum (Theorem 2.1).

Experimental setup: all volumes are resampled to isotropic 1.8 mm voxel spacing; all UDA methods share the same 3D U-Net backbone (five resolution stages, 64 base channels, two 3×3×3 convolutions per stage, LeakyReLU activations, skip connections) with standard spatial and intensity augmentations (rotation, scaling, flipping, contrast modification, additive noise). Splits are at the patient level, with 2/3 of patients for training and validation and 1/3 for testing. For nnU-Net and VISTA-3D, f corresponds to the penultimate stage; for SAM-Med3D, to the final 3D attention block. Prompted variants use 1 or 10 randomly sampled liver points.

Why This Matters

• Impact on research: The results argue that even large-scale pretrained 3D foundation models need explicit feature-space bridging, not just bigger pretraining, to cross acquisition modalities. The framework's independence from the backbone architecture gives a single adaptation recipe for CNN and ViT models, and the released liver masks for Pancreatic-CT-CBCT-SEG create a shared benchmark where none previously existed.
• Interventional radiology: Robust liver segmentation on interventional CBCT supports procedural guidance and vascular work, where iodine contrast and truncated fields of view have been the main obstacles.
• Radiation therapy planning: CBCT is routinely acquired on treatment machines; reliable liver delineation without new annotations supports planning and image guidance where slice-by-slice contouring is expensive.
• Label-scarce clinical sites: Because the method needs no target-domain labels and no inference-time optimization, hospitals can reuse existing CT-trained models on their CBCT scanners.
• Industry relevance: The work was partially funded by a CIFRE grant and supported by GE HealthCare, with two authors employed there, indicating direct interest from a medical imaging vendor in deploying adapted segmentation in commercial systems.
• Beyond the liver: The authors argue the approach generalizes to multi-organ tasks and could benefit other label-scarce interventional applications such as artifact correction, denoising, or pose estimation where domain shift limits performance.

Future Directions

• Extension to multi-organ segmentation: The paper demonstrates the method only on liver segmentation; whether the same redundancy-reducing alignment scales to whole-abdomen or multi-structure tasks is untested.
• Application to other interventional tasks: Artifact correction, denoising, and pose estimation are named as potential beneficiaries of explicit representation alignment, but no experiments are reported for them.
• Closing the gap to target-only training: On DR, adaptation reaches F1 90.0 versus 91.4 for a target-only nnU-Net; on DI, 86.0 versus 93.7. The remaining gap and the hardest interventional cases motivate further work.
• Richer evaluation practice: The reported results are single-run F1 values on two datasets with no variance or significance reporting, so multi-run, multi-site validation and a study of iodine-related preprocessing (as a complement to or replacement for percentile normalization) are natural next steps.

Target Audience

Researchers and engineers working on medical image segmentation, domain adaptation, or foundation models for 3D imaging — particularly those dealing with cross-modality transfer, CT/CBCT data, or label-scarce interventional settings. It is also relevant to clinical translation teams at medical imaging companies evaluating whether pretrained 3D models can be reused across scanner modalities, and to benchmarking groups interested in the newly released Pancreatic-CT-CBCT-SEG liver annotations. Readers without a background in adversarial adaptation or representation learning will need to work through the method section carefully.

Authors’ abstract

Accurate 3D segmentation of cone-beam CT (CBCT) is critical for interventional and radiation therapy applications, yet it remains limited by two compounding challenges: the scarcity of annotated CBCT data and the large domain shift from diagnostic CT. Interventional CBCT exhibits fundamental modality differences from conventional CT, driven by acquisition and physics effects as well as contrast-specific vascular content, thereby limiting effective cross-modality model transfer. We propose a novel unsupervised domain adaptation (UDA) framework based on redundancy-reducing feature alignment, enabling 3D CBCT segmentation with no target-domain annotations or inference-time adaptation. Our framework is architecture-agnostic, seamlessly adapting both CNN-based and ViT-based foundation models. We evaluate our method on two challenging CT-CBCT liver segmentation benchmarks: one for interventional vascular procedures and one for radiation therapy, demonstrating that even large-scale pretrained segmentation networks require explicit feature-space bridging to generalize across acquisition modalities, and that our approach consistently outperforms existing pretrained foundation model and UDA strategies. To support reproducibility and benchmarking, we release the liver segmentations for a public CBCT dataset, along with the code, trained models, and weights.