Organ-Aware Attention Improves CT Triage and Classification

AI summary

Overview

• Research area: Medical computer vision — supervised multi-label classification and triage of 3D computed tomography (CT) studies, with an emphasis on organ-aware aggregation of encoder features.
• Technical level: Intermediate. The paper uses standard deep learning vocabulary (encoders, attention pooling, multiple instance learning, calibration), but the core idea — pooling features inside organ masks instead of averaging the whole volume — is explained clearly enough for a reader with basic machine learning background.
• Scope in one sentence: The paper introduces ORACLE-CT, an encoder-agnostic, organ-aware pooling head that combines Organ-Masked Attention with Organ-Scalar Fusion, and benchmarks it against a carefully tuned Global Average Pooling baseline and prior CT vision-language models on chest (CT-RATE, RAD-ChestCT) and abdomen (MERLIN) CT datasets.

What This Paper Is About

CT produces large 3D volumes — hundreds of slices per study — that radiologists must read quickly across heterogeneous scanners and protocols, making triage and classification time-consuming and error-prone. Existing CT vision-language models (VLMs) struggle with 3D anatomy, protocol shifts, prompt sensitivity, and noisy report supervision, and they do not automatically produce well-calibrated clinical predictions.

The goal of this work is a supervised alternative: a lightweight, encoder-agnostic aggregation head that pools image features inside organ segmentation masks, learns directly from CT disease labels rather than text alignment, and yields study-level predictions with organ-localized evidence maps for auditing.

Key Contributions

1. Organ-masked softmax pooling for CT triage. A lightweight pooling head that assigns one unary score per local feature and normalizes the softmax within each organ mask, producing permutation-invariant organ embeddings and spatially localized weight maps. It uses a small per-location unary scorer with no query-key-value mixing, costing O(|Ω|) compute.
2. Organ-Scalar Fusion (OSF). A minimal extension that appends compact mask-derived scalars — normalized volume (z-scored over the training set), mean HU (clipped, then z-scored), and a binary border-contact flag for truncation — to the pooled organ feature, with a learned gate that softly down-weights spatial features for truncated organs.
3. An encoder-agnostic benchmarking protocol. One modular engine compares 2.5D token backbones (DINOv3, MedSigLIP) and native 3D voxel backbones (CT-Net, I3D-ResNet-121, MedNeXt-3D) under fixed preprocessing, label maps, splits, schedules, and calibration, enabling staged GAP vs. global attention vs. masked attention vs. masked attention + OSF comparisons.
4. Calibrated supervised state of the art. Under one fixed protocol, the authors report supervised state-of-the-art AUROC on CT-RATE (0.86 with masked attention) and MERLIN (0.85 with masked attention plus scalar fusion), outperforming reported linear-probe CT-VLM references. Source code is available at https://github.com/lavsendahal/oracle-ct.

Main Findings

• Supervised GAP baseline is already competitive. On CT-RATE (internal test), Inflated ResNet-121 with GAP led the backbone sweep at AUROC 85.74 / AUPRC 61.73 / F1 59.91 / BA 76.12, followed by DINOv3 (84.52 / 59.17 / 57.24 / 74.30) and MedNeXt-3D (84.41 / 58.37 / 57.76 / 75.75). On RAD-ChestCT (external), the same GAP baseline scored AUROC 76.45 / F1 51.80.
• Comparison against prior CT-VLMs on chest. Reported linear-probe references: CT-CLIP (61.21 CT-RATE / 54.05 RAD-ChestCT AUROC), CT-FM (82.14 / 68.49), Merlin (82.62 / 70.91), COLIPRI-C (84.15 / 72.66).
• Organ-masked attention improves chest results. Applied to the best GAP backbone (Inflated ResNet-121), masked attention gave ΔAUROC = +0.42 and ΔF1 = +0.47 on CT-RATE versus its GAP baseline, and matched AUROC with a small F1 gain on RAD-ChestCT. ORACLE-CT reached AUROC 0.86 on CT-RATE and 0.76 on RAD-ChestCT, compared with fVLM (0.78 / 0.68), SegVL (0.81 / 0.72), and Uniferum (0.83 on CT-RATE).
• Supervised models beat a reproduced zero-shot baseline on abdomen. On the official MERLIN test split, the reproduced zero-shot baseline scored AUROC 0.72 / AUPRC 0.50 / F1 0.47 / BA 0.64, while supervised GAP models across five backbones ranged from CT-Net (0.75 / 0.51 / 0.54 / 0.66) up to DINOv3 (0.83 / 0.64 / 0.63 / 0.72). MedSigLIP reached 0.82 / 0.61 / 0.59 / 0.70, ResNet-121 0.81 / 0.59 / 0.59 / 0.69, and MedNeXt 0.78 / 0.54 / 0.55 / 0.66.
• Masked attention plus scalar fusion is the best abdomen configuration. On DINOv3 across four modes: GAP (0.83 / 0.64 / 0.63 / 0.72), Global Attn (0.83 / 0.63 / 0.63 / 0.73), Masked Attn (0.84 / 0.66 / 0.63 / 0.73), and Masked Attn + OSF (0.85 / 0.68 / 0.66 / 0.74). Masked-based variants beat GAP and global attention, with the largest gains on size/morphology-driven findings such as "-megaly".
• Mask dependence cuts both ways. Because masked attention depends on segmentation quality, labels without reliable masks (for example appendicitis and gallbladder pathology) can favor GAP. Organ-masked attention lifts most organ-tied labels, while Masked+OSF shows the most consistent improvements, especially for size- and morphology-driven findings.
• Unfreezing backbones helps. Comparing frozen versus unfrozen backbones for DINOv3 and MedSigLIP, unfreezing consistently lifted AUROC/AUPRC, with larger and more uniform gains for DINOv3, concentrated on multi-organ, morphology/size-driven findings (for example "-megaly", bowel obstruction, hydronephrosis, surgically absent gallbladder).
• Scalar type matters by finding class. With the region fixed to the full segmentation mask, volume-only fusion tended to help size/morphology findings (cardiomegaly, hepatomegaly, splenomegaly, pancreatic atrophy), while HU-only fusion favored intensity/density patterns (calcifications, thrombosis).
• Masks versus bounding boxes is class-dependent. Both generally beat GAP, but for appendicitis a bounding box better captured peri-appendiceal context, and for biliary ductal dilation, thin boundary-adjacent ducts benefited from periportal context. The authors' rule of thumb: use bounding box for small/tubular or boundary-adjacent targets, masks for diffuse within-organ signals, and otherwise choose by validation.
• Qualitative evidence is organ-localized but not causal. Visualizations on four MERLIN studies showed attention confined to the lungs for atelectasis and pleural effusion, and kidney-confined attention for both a true-positive renal cyst and a false-positive renal cyst. The authors state these maps depict organ-level pooling weights used for study-level logits, and are not pixel-accurate lesion segmentations or guarantees of correctness.

Methodology in Plain English

The authors treat each CT study as a 3D volume (D axial slices, height H, width W) with multiple disease labels that can be positive, negative, or missing (encoded as −1). An encoder — either a 2.5D transformer producing a token lattice or a native 3D network producing a voxel lattice — turns the volume into a grid of local feature vectors. Everything the authors contribute lives in the head that aggregates those features.

Segmentation masks are computed once per study and merged into a small set of organ groups (for example, all lung lobes become "lungs"; left and right kidneys become "kidneys"). Each group is dilated by a fixed amount in millimeters (0.5 mm to 5.0 mm depending on the organ) to absorb boundary uncertainty, then resampled onto the encoder lattice. Every study-level label is anchored to exactly one organ group through a fixed mapping; labels with no stable anatomical region fall into an "other" group handled by a mask-free global head.

Four aggregation modes form a ladder. GAP averages all features uniformly. Global attention learns a scalar score per location and applies a softmax over the whole lattice. Organ-masked attention does the same but restricts the softmax to the voxels inside each organ mask, producing a separate pooled representation and classifier per organ group. Masked+OSF additionally appends the volume, mean HU, and border-contact scalars to that pooled feature.

Training uses per-label binary cross-entropy. Labels marked missing are ignored for an initial burn-in of n_burn epochs, then gradually introduced as weak negatives with a linearly increasing weight over n_ramp epochs up to a small cap, to avoid over-penalizing uncertain supervision. Class imbalance is handled by conservative positive-class reweighting. Model selection uses validation loss; per-label temperature scaling and F1-optimal thresholds are fit once on validation and frozen for all test and external evaluation, and metrics exclude missing targets from denominators. Intensities are clipped to [−1000, 1000] HU and min-max normalized to [0, 1].

Datasets: CT-RATE (43,738 train / 3,409 val / 3,040 test; 18 study-level findings; 4 masked organ groups plus "other"; 1.5 × 1.5 × 1.5 mm spacing; 224³ volumes), MERLIN (15,175 / 5,018 / 5,082; 30 labels; 13 masked organ groups plus "other"; 3 × 1.5 × 1.5 mm; 160 × 224 × 224), and RAD-ChestCT as an external test set of 3,630 volumes harmonized to the CT-RATE 16-label protocol with organ merges and dilations mirroring CT-RATE.

Why This Matters

• Research impact: The paper argues that a supervised, anatomy-guided pooling head can match or exceed linear-probe CT-VLM performance without large-scale vision-language pretraining, and it supplies a fixed, auditable benchmark protocol so that GAP, global attention, masked attention, and masked+OSF can be compared fairly. Segmentation is a shared once-per-study cost reusable across labels and models.
• Real-world applications:
  • Study-level CT triage: flagging studies that need urgent radiologist attention, given calibrated probabilities and fixed decision thresholds.
  • Chest reading support: organ-anchored evidence for findings such as pleural effusion, emphysema, atelectasis, and cardiomegaly.
  • Abdomen reading support: size- and density-driven findings such as hepatomegaly, splenomegaly, pancreatic atrophy, hydronephrosis, and thrombosis.
  • Audit and quality review: organ-level pooling weight maps that show where the model drew evidence for a study-level prediction.
• Industry relevance: The work targets the reading backlog problem — the paper notes over 90M CT scans were performed in the United States in 2023, with chest and abdomen-pelvis studies comprising over half, while the radiologist workforce has not grown proportionally. An encoder-agnostic head that plugs into existing 2.5D or 3D backbones and reuses standard multi-organ segmentation tools is directly relevant to clinical AI deployment, and the emphasis on fixed calibration and auditable evidence speaks to regulatory and safety expectations for triage tools.

Future Directions

• Disease-aware, gated scalars: replacing mean HU with more robust summaries and enabling sparse per-label fusion gates, since uniform fusion of z-scored volume and mean HU can dilute focal signal.
• Faithfulness and robustness testing: deletion-insertion curves, counterfactual mask edits, and calibration under masking, given that attention maps are described as supportive evidence rather than guaranteed faithful attributions.
• Mask reliability and fallbacks: estimating mask quality and marginalizing or switching across mask, padded box, or expanded regions, since segmentation errors on small, tubular, or truncated structures propagate into masked pooling and OSF.
• Tuning and scaling: systematic hyperparameter sweeps and parameter-efficient adapters, because the authors state they did not fully tune backbones or schedules, and masks remain fixed preprocessing with no uncertainty modeling or joint segmentation-classification training.

Target Audience

Researchers and engineers working on medical image analysis, 3D CT classification, multiple instance learning, and attention-based aggregation; clinical AI teams and radiologists interested in triage and interpretability of organ-localized evidence; and practitioners who want a reproducible, calibrated supervised baseline (with code, configs, and checkpoints planned for release) to compare against CT vision-language models under a shared protocol.

Authors’ abstract

There is an urgent need for triage and classification of high-volume medical imaging modalities such as computed tomography (CT), which can improve patient care and mitigate radiologist burnout. Study-level CT triage requires calibrated predictions with localized evidence; however, off-the-shelf Vision Language Models (VLM) struggle with 3D anatomy, protocol shifts, and noisy report supervision. This study used the two largest publicly available chest CT datasets: CT-RATE and RADCHEST-CT (held-out external test set). Our carefully tuned supervised baseline (instantiated as a simple Global Average Pooling head) establishes a new supervised state of the art, surpassing all reported linear-probe VLMs. Building on this baseline, we present ORACLE-CT, an encoder-agnostic, organ-aware head that pairs Organ-Masked Attention (mask-restricted, per-organ pooling that yields spatial evidence) with Organ-Scalar Fusion (lightweight fusion of normalized volume and mean-HU cues). In the chest setting, ORACLE-CT masked attention model achieves AUROC 0.86 on CT-RATE; in the abdomen setting, on MERLIN (30 findings), our supervised baseline exceeds a reproduced zero-shot VLM baseline obtained by running publicly released weights through our pipeline, and adding masked attention plus scalar fusion further improves performance to AUROC 0.85. Together, these results deliver state-of-the-art supervised classification performance across both chest and abdomen CT under a unified evaluation protocol. The source code is available at https://github.com/lavsendahal/oracle-ct.