RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation

AI summary

Overview

• Research area: Medical computer vision — 3D multimodal (vision–language) tumor segmentation for radiotherapy planning, combining CT volumes with narrative radiology reports.
• Technical level: Advanced (assumes familiarity with vision–language models, MLLM segmentation architectures such as LISA, and 3D medical image segmentation).
• Scope: The paper introduces RepConTUR, a native-3D early-fusion multimodal model that conditions abdominal tumor segmentation on a suborgan-level referring phrase plus an optional free-form radiology report, and evaluates it against text-promptable baselines and vision-only specialists in-domain and on four out-of-distribution cohorts.

What This Paper Is About

Automatic tumor delineation for radiotherapy still produces errors — predominantly false positives — that distort dose and lengthen manual editing time, and standard vision-only models are scored mainly by volumetric Dice, which hides poor instance-wise precision. The paper's goal is to make the segmentation model read the radiology report that accompanies the scan, since the report already localizes disease at the suborgan level, and to do so in native 3D rather than the 2D, short-phrase text prompting that most existing medical segmentation models use. The authors build a four-task dataset from AbdomenAtlas 3.0 and train one model that both segments and reasons over report text.

Key Contributions

1. Report-conditioned segmentation. Segmentation is conditioned on a suborgan-level referring phrase plus an optional free-form narrative report, rather than a short class name alone. The authors state this is, to their knowledge, the first 3D referring/grounded segmentation model to condition on such report context, and they introduce a suborgan-level mask and prompt design aligned to narrative radiology reports, implemented for abdominal tumors.

2. A native-3D MLLM for report-conditioned abdominal tumor segmentation. The vision encoder and the segmentation module are both 3D, and visual tokens are processed jointly with the report text from the first LLM layer, so vision and text are reasoned over jointly rather than fused late at a decoder.

3. A report-derived multitask dataset and multitask learning. From AbdomenAtlas 3.0 tumor masks and their reports, the authors construct a unified four-task dataset: referring segmentation (RES), report-grounded segmentation (GSEG), radiology report generation (RRG), and LLM-verified VQA. Report-matched suborgan localization produces the RES and GSEG targets, and all four tasks are trained jointly on a single backbone.

4. Systematic ablation of the design choices. The paper isolates the effect of the report text (RES vs. GSEG within the same model), of joint GSEG co-training on RES, of cumulative language-task co-training, and of replacing 2D visual and segmentation modules with 3D counterparts.

Main Findings

• Best text-promptable segmentation across organs. On the AbdomenAtlas 3.0 test split, RepConTUR attains the highest DSC, instance-wise precision, and detection F1 in every organ section (liver, pancreas, kidney) for both RES and GSEG. Mean DSC is 0.510 (RES) and 0.523 (GSEG), against 0.382/0.317 for VoxTell, 0.342/0.354 for UniBioMed, and 0.372 (RES) for BiomedParse v2.

• Precision is the headline gap. Mean instance-wise precision is 0.397 in RES and 0.433 in GSEG, versus ≤ 0.071 for every baseline. Mean detection F1 is 0.499 in RES against 0.130 for BiomedParse v2.

• The one baseline that wins on recall. BiomedParse v2 has higher recall in every organ, with mean recall 0.747 against RepConTUR's RES 0.674, but its much lower precision leaves it far behind on detection F1.

• The report effect is concentrated in precision. Prepending the report raises mean precision from RES 0.397 to GSEG 0.433 while mean recall stays comparable (0.674 vs. 0.677). Late-fusion VoxTell actually degrades with the report (macro F1 0.082 → 0.062), while MLLM-based UniBioMed (0.109 → 0.115) and RepConTUR (0.499 → 0.528) improve.

• Prompts control presence as well as location. In a multifocal liver case, switching the prompt between segments 5 and 6 moves the mask anterior-to-posterior, and querying segment 3 or the tumor-free kidney yields an empty mask. Qualitatively, baselines add other real but non-referred tumors, or segment the tumor-free contralateral kidney.

• In-domain, the report beats vision-only specialists on liver and pancreas. Detection F1 reaches 0.599 on liver (RES) and 0.475 on pancreas (GSEG), against R-Super at 0.485 and 0.452 and nnU-Net at 0.463 and 0.387. On kidney, where tumors are well delineated, nnU-Net is strongest at F1 0.661 and RepConTUR sits between at 0.575–0.584. Mean in-domain F1 is 0.543 (RES) and 0.548 (GSEG) vs. nnU-Net 0.504 and R-Super 0.499.

• Out of distribution, the margin grows. GSEG has the highest detection F1 in three of four external cohorts (0.412 on HCC-TACE-Seg, 0.506 on CPTAC-CCRCC, 0.687 on Colorectal-Liver-Metastases) and is comparable to R-Super on CPTAC-PDA (0.571 vs. 0.573). Mean OOD F1 is 0.544 (GSEG) and 0.501 (RES) against nnU-Net 0.415 and R-Super 0.426. Mean OOD precision is 0.432 (GSEG) and 0.390 (RES) against nnU-Net 0.307 and R-Super 0.322.

• Gains come from false-positive suppression under domain shift. Recall stays comparable across methods out of domain, while nnU-Net and R-Super precision collapse. The mean detection-F1 advantage over nnU-Net is larger out of domain (0.129 for GSEG, 0.086 for RES) than in domain (0.044 and 0.039).

• Ablation: 3D segmentation module matters most. Moving from 2D/2D (UniBioMed) to 3D/2D gives small detection gains and mixed Dice (GSEG 0.354 → 0.378, RES 0.342 → 0.331). Replacing the 2D segmentation module with the pretrained 3D one raises GSEG precision from 0.080 to 0.433 and GSEG F1 from 0.139 to 0.528.

• Ablation: GSEG co-training transfers to RES. Adding GSEG to a RES-only model lifts RES DSC from 0.481 to 0.505, precision from 0.345 to 0.385, and F1 from 0.450 to 0.482, even though RES uses no report text at test time. Adding RRG then VQA raises detection F1 further, from 0.482 → 0.493 → 0.499 in RES and 0.508 → 0.523 → 0.528 in GSEG, with other metrics not improving monotonically.

Methodology in Plain English

• Data curation. The authors start from AbdomenAtlas 3.0, which provides 9,262 abdominal CT volumes with liver, kidney, and pancreas tumor masks, organ masks including suborgan structures such as liver Couinaud segments, and a synthetically generated narrative radiology report for each volume (mean 225 tokens, maximum 1,163). They remove the 5,464 tumor-free volumes to match the radiotherapy setting, leaving 3,798 tumor-bearing volumes (1,508 liver-positive, 2,222 kidney-positive, 1,113 pancreas-positive).

• Splitting organ masks into suborgan targets. Because the original tumor masks are organ-level, the authors decompose them into per-suborgan instances by measuring each connected component's intersection over component (IoC) with each suborgan mask. Where the report states a tumor count for a suborgan, they keep the highest-IoC components up to that count; where the report is only organ-level, a fixed IoC threshold of 0.2 plus a spatial-proximity rule assign the component. Across 14,584 tumor components, 80.4% are suborgan-stated and assigned by IoC, 1.6% suborgan-stated by surface distance, 13.7% organ-level by IoC, and 4.3% organ-level by surface distance. Components assigned to the same suborgan are merged by voxel-wise union.

• Building VQA supervision. Up to 20 question–answer pairs per volume are generated by an LLM in three formats (yes/no, short answer, multiple choice), covering tumor presence, count, suborgan location, size band (queried as an interval rather than an exact diameter), enhancement pattern, and organ-level findings. A second LLM verifies each pair against the source reports and corrects or discards unsupported items. 71,783 pairs survive (18.9 per volume), of which 48,451 are yes/no, 13,433 short-answer, and 9,899 multiple choice; 64.9% of yes/no answers are affirmative.

• Defining the four tasks. RES uses a suborgan-level phrase drawn from a pool of phrasal variants inserted into a randomly sampled instruction template, plus a whole-organ aggregate target. GSEG uses the identical target mask and phrase but prepends the full narrative report, so the comparison isolates the report's contribution. RRG generates the report from the CT, and VQA answers the derived questions.

• Model architecture. The 3D visual encoder is CT-CLIP's CT-ViT (factorized spatial-then-through-plane attention), resampled to (1.5, 1.5, 3.0) mm and a 240×240×140 grid, and pretrained with a CLIP objective against Clinical-Longformer report embeddings, augmented with abdominal CT image–report pairs from Merlin to move beyond chest CT. An attentional pooler with 256 learnable queries produces 256 visual tokens for an InternVL2.5-1B language model initialized from UniBioMed weights, frozen and adapted with LoRA, with a [SEG] token requesting segmentation. The [SEG] hidden state is projected to a 2048-dimensional query that drives VoxTell's 3D segmentation module (frozen nnU-Net residual encoder plus a text-prompted decoder).

• Training and inference. 20 epochs on four NVIDIA H100 80GB GPUs in bf16 with DeepSpeed ZeRO-2, per-GPU batch 4 and gradient accumulation 8 (effective batch 128), maximum sequence length 8192, AdamW with learning rate 4×10⁻⁵ and weight decay 0.05, LoRA rank 64 and α = 128. Segmentation uses summed voxel-wise sigmoid cross-entropy and Dice loss; language tasks use standard cross-entropy. Full-volume inference uses sliding windows of 192³ with 50% overlap.

• Evaluation. Metrics are DSC plus instance-wise recall, precision, and their harmonic-mean F1, with a 3D connected-component predicted tumor counted as matching ground truth at IoU ≥ 0.10. Comparisons are against three text-promptable baselines (UniBioMed, BiomedParse v2, VoxTell, each finetuned on the tasks it supports) and two vision-only specialists (nnU-Net and R-Super). For the specialist comparison, an agentic pipeline has an LLM parse the report into tumor-bearing suborgans, segments each with RepConTUR, and unions the results.

• Out-of-distribution handling. Four public cohorts (HCC-TACE-Seg n=130, CPTAC-PDA n=108, CPTAC-CCRCC n=108, Colorectal-Liver-Metastases n=197) lack real reports, so reports are synthesized with the same construction pipeline used for AbdomenAtlas 3.0 and used both to drive inference and as GSEG grounding text.

Why This Matters

• Research impact. The paper reframes report-based supervision: instead of using the report only as a training-time signal (as R-Super and similar methods do) and discarding it at inference, it keeps the report in the loop at prediction time. It also supplies evidence that precision, not Dice, is where multimodal conditioning pays off, and it releases a four-task report-derived dataset and code.

• Radiotherapy planning. Fewer false-positive tumor contours mean less manual editing before dose planning, and the authors motivate this directly by the clinical cost of false positives (distorted dose, higher normal-tissue exposure, longer editing time). Target contouring already pairs the scan with its report, so the conditioning signal is available in routine workflow.

• Suborgan-level targeting. The model can be asked for a specific Couinaud segment or pancreatic subregion and returns an empty mask when no tumor is there, which matters for focal therapies and for sparing adjacent structures.

• Handling multifocal disease. On multifocal liver and pancreas cases, baselines segment other real but non-referred tumors; prompt-conditioned grounding lets a clinician ask for one lesion at a time.

• Industry relevance. The work is a collaboration between Oncosoft Inc. and Yonsei University College of Medicine, targeting a clinical workflow product. The interleaving of a frozen pretrained encoder, LoRA adaptation of a 1B-parameter LLM, and a pretrained 3D segmentation module is a practical recipe for building regulatory-adjacent clinical multimodal tools without training everything from scratch.

Future Directions

• Scaling beyond three organs. The model covers liver, kidney, and pancreas only; whether the approach generalizes to further abdominal organs and other body regions is explicitly stated as unestablished, and verifying class-scaling is named as a primary direction.

• Validation with real clinical reports. All training reports are synthetically generated from masks, and the four OOD cohorts were grounded with synthesized reports. Whether the same gain holds under independently authored clinical reports — with their stylistic variation and reporting inconsistencies — remains to be verified.

• Prospective editing-time and clinical studies. The clinical value of automatic segmentation rests on the editing time it saves, which the authors do not measure on their outputs; prospective clinical and editing-time studies are left as future work.

• Explaining the kidney regression. RepConTUR trails the nnU-Net specialist on kidney in-domain (F1 0.575–0.584 vs. 0.661) while beating it on the kidney OOD cohort, and R-Super fails on kidney through a precision collapse to 0.435 despite the highest kidney recall of 0.784. Understanding when report conditioning helps versus when vision-only training is already sufficient is an open question the results leave unresolved.

Target Audience

Researchers and engineers working on multimodal medical imaging, 3D vision–language models, and text-promptable or grounded segmentation; medical physicists and radiation oncology clinicians interested in automated target delineation and its failure modes; and applied ML practitioners building clinical products who want a concrete recipe for combining a frozen 3D CT encoder, a LoRA-adapted small LLM, and a pretrained 3D segmentation head trained jointly on segmentation, report generation, and VQA. Readers unfamiliar with MLLM segmentation architectures such as LISA or with instance-wise detection metrics will need background reading, since the paper assumes both.

Authors’ abstract

Tumor delineation is required for radiotherapy planning, yet automatic segmentation is limited by errors, predominantly false positives, that distort dose and increase editing time. The radiology report accompanies the scan and localizes disease at the suborgan level, but existing text-promptable models condition on short referring phrases, and most operate in 2D. We present RepConTUR, a single native-3D early-fusion multimodal model that conditions volumetric abdominal tumor segmentation on the narrative report. We extend AbdomenAtlas 3.0 to a suborgan-level promptable segmentation dataset and generate report-based VQA pairs, yielding four report-derived tasks on the same annotations: referring segmentation (RES), report-grounded segmentation (GSEG), report generation (RRG), and visual question answering (VQA). We train RepConTUR, comprising a 3D CT encoder, an early-fusion language model, and a native-3D segmentation module, jointly on the four tasks. Across liver, kidney and pancreas, RepConTUR attains the highest Dice similarity coefficient (DSC), instance-wise precision, and detection F1 among the text-promptable baselines, and adding the report raises mean precision while leaving recall comparable. Against two vision-only specialists, nnU-Net and the report-supervised R-Super, RepConTUR attains the highest value on every in-domain metric for liver and pancreas and the highest mean DSC, recall, precision, and detection F1 over the four out-of-distribution public cohorts. Ablations indicate that conditioning on the narrative report raises detection F1 from 0.499 to 0.528 within the same model, that joint GSEG training raises RES F1 from 0.450 to 0.482, and that the native-3D segmentation module raises GSEG F1 from 0.139 to 0.528 over its 2D counterpart. Code is available at https://github.com/Oncosoft-Research/RepConTUR.