Anatomy Contextualized Adaption of CT Foundation Models

AI summary

Overview

• Research area: Medical computer vision — vision-language foundation models for 3D CT imaging, with a focus on anatomy-level (fine-grained) representation learning for zero-shot finding classification.
• Technical level: Intermediate. The paper assumes familiarity with contrastive image-text pre-training, transformer encoders, CT segmentation tooling, and AUROC-based evaluation of diagnostic models.
• Scope: The paper introduces Anatomy Contextualized Adaptation (ACA), a lightweight framework that adapts the frozen representations of pretrained CT foundation models into anatomy-level embeddings that are contextualized across organs and aligned to both per-anatomy and whole-report text, and it evaluates that framework on the Merlin and CT-RATE datasets.

What This Paper Is About

CT vision-language foundation models such as Merlin and CT-CLIP compress an entire 3D scan into a single embedding, which dilutes signals from small or subtle anatomical structures. Fine-grained alternatives align anatomy-level image features with anatomy-specific text, but they discard global scan context and are typically trained from scratch, which is computationally expensive. ACA aims to get both: it reuses the broad frozen representations of an existing foundation model, adds lightweight trainable modules that produce anatomy-level embeddings, and uses an inter-anatomy transformer plus a scan-level report loss to restore global context.

Key Contributions

1. Anatomy Contextualized Adaptation (ACA): a framework that adapts frozen CT foundation model features into anatomy-level embeddings using segmentation-derived pooling, without retraining the backbone.
2. An inter-anatomy transformer: a transformer encoder augmented with learned spatial position encodings (a 44-dimensional vector of normalized inter-structure centroid distances) and learnable anatomy type embeddings, which contextualizes each anatomy against all other structures present in the scan.
3. A combined loss: an anatomy-level soft-target contrastive loss (L_anatomy) paired with a scan-level hard-diagonal report contrastive loss (L_scan) weighted by λ = 1, so that local anatomy-level and global scan-level supervision shape the same embedding space.
4. An inference-time anatomy-guided pooling strategy that upweights anatomies relevant to the finding being tested, plus an empirical demonstration (via attention heatmaps) that the learned cross-anatomy attention follows plausible anatomical structure and dataset scope.

Main Findings

• ACA beats global foundation models in every setting. Average AUROC rises from 0.7729 to 0.8213 on Merlin (in-distribution) and from 0.7082 to 0.7311 on CT-RATE (in-distribution), with larger gains out-of-distribution: 0.5822 to 0.6723 on Merlin* and 0.6919 to 0.7400 on CT-RATE*. Out-of-distribution evaluation is restricted to the 7 findings shared between the two datasets.
• ACA beats fine-grained baselines. Average AUROC boosts across settings of 0.061 versus MLP, 0.062 versus fVLM, and 0.044 versus ViSD-Boost, with gains observed up to 11.8 points in-distribution (CT-RATE, vs. MLP) and 11.4 points out-of-distribution (Merlin*, vs. fVLM). The one exception is a near-tie with MLP and ViSD-Boost on CT-RATE*.
• ACA beats global adaptation. Against the Spatial Transformer baseline, ACA shows an average increase of 0.068 AUROC.
• Both loss terms matter. Removing the anatomy-level loss lowers average AUROC by 0.033 and removing the scan-level loss lowers it by 0.029. ACA averaged 0.7412, ACA w/o L_anatomy averaged 0.7083, and ACA w/o L_scan averaged 0.7119.
• Anatomy-guided pooling adds a low-cost gain. For ACA, in-distribution AUROC rises from 0.8213 to 0.8372 on Merlin and from 0.7311 to 0.7413 on CT-RATE, and out-of-distribution AUROC rises from 0.7400 to 0.7530 on CT-RATE*, while Merlin* is essentially unchanged (0.6723 vs. 0.6718). This configuration reaches an overall average of 0.7508. The effect is larger for ACA w/o L_scan (in-distribution CT-RATE 0.6935 to 0.7365; CT-RATE* 0.7068 to 0.7451), which the authors attribute to the model lacking scan-level supervision to otherwise aggregate global context.
• Findings that benefit most are subtle, single-organ pathologies. Examples named include aortic valve calcification, renal cyst, and hepatomegaly; more diffuse or non-anatomy-specific findings such as free air, submucosal edema, and thrombosis show less benefit. Arterial wall calcification, which can occur in any artery, shows a 0.13 average AUROC boost (Table 12).
• The learned attention is anatomically plausible. In the Merlin-trained model, the largest weights are among the stomach, small bowel, and colon, and the liver and gallbladder attend to one another with the highest weight in each anatomy's row. In the CT-RATE-trained model, strong attention weights are directed to the lung and esophagus, and the spleen attends most strongly to the kidney while the kidney attends to the pancreas. Self-attention along the diagonal is relatively weak in both datasets.
• Contextual reasoning case study. In Figure 4, two Merlin scans contain spleens of comparable cross-sectional area, but only one is labeled splenomegaly; ACA predicts the positive case correctly while all fine-grained baselines predict negative.
• Frozen adaptation recovers most of the benefit of fine-tuning. A LoRA-finetuned end-to-end variant of the full Merlin-based ACA achieved an increase in average AUROC of only 0.008 on the Merlin test set compared to the original ACA formulation with the same hyperparameters.
• Training is cheap. ACA requires less than one hour of training once embeddings are cached.

Methodology in Plain English

The pipeline has three parts.

First, each CT volume is segmented by TotalSegmentator, whose original 117 classes are condensed into 44 anatomical groups (for example, left and right kidney become "kidney"). The scan is passed through a frozen foundation model — Merlin (I3D ResNet152 encoder, 224 × 224 × 160 voxel input, feature map of 2048 × 10 × 7 × 7) or CT-CLIP (CT-ViT encoder, 480 × 480 × 240 voxel input, feature map of 512 × 24 × 24 × 24). Segmentation masks are projected to the feature map resolution using non-overlapping max-pooling with a kernel matched to the backbone's effective patch size (32 × 32 × 16 for Merlin, 40 × 40 × 20 for CT-CLIP), and for each structure the features of patches it occupies are mean-pooled into one ℓ2-normalized embedding (d = 2048 for Merlin, d = 512 for CT-CLIP). Structures with empty segmentation masks are treated as absent and dropped from the sequence. Separately, anatomy-specific findings are extracted from radiology reports using Qwen3-4B-Instruct and encoded by the corresponding frozen text encoder; the full report yields a report-level text embedding.

Second, all present anatomy embeddings pass jointly through an inter-anatomy transformer. Each input token is the projected anatomy embedding plus an MLP applied to its 44-dimensional spatial position encoding plus a learnable anatomy type embedding.

Third, the contextualized embeddings are projected into a shared contrastive space and trained with two losses. The anatomy-level loss compares each structure's instances across the batch using a soft target matrix that treats same-patient pairs and normal-normal pairs as partial positives, and it skips structures where every instance is normal. The scan-level loss aligns the mean-pooled scan embedding to the full report using a standard hard-diagonal contrastive objective. Temperatures are learnable, initialized at 0.07, and clamped to [0.001, 0.5]. Evaluation uses macro-average AUROC with uncertainty estimated over 1000 bootstrap resamples, and each finding is treated as binary scan-level classification using positive and negative prompt sets.

Training uses AdamW with learning rate 1 × 10⁻⁴, batch size 64, and 50 epochs, with the best model selected by validation loss. Merlin uses creator-defined splits of 15,314 training, 5,055 validation, and 5,125 test scans; CT-RATE is partitioned into 37,545 training and 9,598 validation scans with a 3,039-scan test set. Unlike Merlin, negatives are not subsampled, so evaluation covers all samples.

Why This Matters

Impact on research. The paper argues that fine-grained alignment and global scan context are not mutually exclusive, and that adapting frozen foundation models can recover most of the benefit of full end-to-end training — the LoRA experiment showed only a 0.008 average AUROC gain over ACA on Merlin. It also offers a compute-efficient template: under one hour of training once embeddings are cached, and a module decoupled from the backbone so it can be applied to any CT foundation model.

Real-world applications:

• Zero-shot detection of subtle, single-organ abnormalities that whole-volume embeddings tend to wash out, such as renal cyst, aortic valve calcification, and hepatomegaly.
• Organomegaly assessment, where a structure must be judged relative to surrounding organs and body habitus rather than by absolute size — the splenomegaly example in Figure 4.
• Findings spanning multiple sites, such as arterial wall calcification, which requires integrating features across the scan.
• Cross-dataset and cross-backbone deployment, since the out-of-distribution experiments test generalization across both datasets and base foundation models.

Industry relevance. Hospitals and imaging vendors care about adapting existing pretrained models at low compute cost rather than retraining from scratch. Because ACA's adaptation module is decoupled from the backbone and its training is measured in under an hour once embeddings are cached, it is a plausible path to rapid customization of a CT model for new institutions, scanners, or clinical question sets. The authors also note the modularity allows application to other CT foundation models at low additional cost.

Future Directions

• Full end-to-end comparison. The authors state that a full end-to-end retraining of fVLM and ViSD-Boost would be needed to compare against their originally reported performance, since the comparisons here were reimplemented on frozen embeddings.
• Extending beyond zero-shot classification. The paper explicitly leaves segmentation, outcome prediction, and report generation to future work.
• Broadening evaluation. Only two datasets and two backbones were tested, and out-of-distribution comparisons were restricted to the 7 findings shared between Merlin and CT-RATE, so broader generalization remains untested.
• Reducing label noise and vocabulary limits. Per-anatomy findings and normality labels were extracted automatically with an LLM rather than verified by radiologists, and anatomy decomposition depends on TotalSegmentator's vocabulary, so structures outside it are not directly represented.

Target Audience

Researchers and engineers working on medical imaging foundation models, CT vision-language pre-training, and parameter-efficient adaptation will get the most from this paper, along with clinical AI teams interested in anatomy-aware representation learning. Radiologists and clinical informatics groups evaluating zero-shot diagnostic models will also find the per-finding results useful, though readers without background in contrastive learning or CT segmentation will need to consult the referenced prior work (Merlin, CT-CLIP, fVLM, ViSD-Boost, TotalSegmentator).

Authors’ abstract

CT vision-language foundation models have demonstrated promising performance across downstream tasks, but are typically trained with whole-volume representations that dilute fine-grained anatomical signals. Fine-grained vision-language pre-training addresses this by aligning anatomy-level visual features with anatomy-specific text, but in doing so discards the global context that whole-volume models provide. Furthermore, existing fine-grained approaches train from scratch, making them computationally expensive. We introduce Anatomy Contextualized Adaptation (ACA), a lightweight framework that adapts frozen CT foundation model representations for anatomy-level vision-language alignment while enhancing global contextualization. ACA uses TotalSegmentator to decompose CT volumes into anatomy-level embeddings, which are refined via a transformer that captures cross-anatomy relationships, and aligned to both per-anatomy and scan-level text extracted from radiology reports. Evaluated on Merlin and CT-RATE, ACA consistently outperforms both the frozen foundation model baselines and existing fine-grained methods in zero-shot finding classification, while requiring less than one hour of training once embeddings are cached. The attention weights learned by ACA's inter-anatomy transformer additionally indicate plausible cross-anatomy context routing. Altogether, these results support ACA as a lightweight approach for adapting CT foundation models to anatomically grounded vision-language alignment while preserving and enhancing global anatomical context.