Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset

AI summary

Overview

Research area: Medical computer vision — specifically referring image segmentation (RIS) applied to chest X-rays (CXRs), combining vision–language models (VLMs) with automated dataset construction.

Technical level: Intermediate. The core idea (instruction + image → mask + text) is intuitive, but the pipeline involves LLM-based report structuring, diffusion-based image editing, anatomy segmentation, and object detection, and the model builds on LISA, LLaVA, and SAM.

Scope: The paper introduces a new task (instruction-guided lesion segmentation, ILS), a fully automated pipeline for building a million-scale CXR lesion segmentation dataset (MIMIC-ILS), and a model (ROSALIA) trained on it.

What This Paper Is About

Existing CXR lesion segmentation models are limited because they cover few lesion types and require long, expert-level text prompts written after a radiologist has already reviewed the image. This paper asks how to build a large-scale dataset of lesion masks paired with simple, user-friendly instructions without any human annotation effort, and then trains a model that can segment lesions and explain them in response to plain instructions. The authors address this by deriving masks and instruction–answer pairs automatically from paired CXR images and radiology reports in MIMIC-CXR.

Key Contributions

1. An automated dataset construction pipeline. A two-stage, human-free method that generates grounded lesion masks and corresponding instruction–answer text directly from image–report pairs, using an LLM for report structuring and location mapping, plus pretrained vision models (RadEdit, CXAS, YOLO, HybridGNet) for spatial cues.

2. MIMIC-ILS, the first large-scale ILS dataset for CXRs. 1.1M instruction–answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types, with a reported expert acceptance rate of over 95%.

3. ROSALIA, the first VLM designed for ILS in CXRs. A LISA model fine-tuned on MIMIC-ILS that outputs both segmentation masks and textual explanations from simple instructions, including confirming the absence of a requested lesion.

4. A new evaluation setting. The paper defines instruction types (basic, global, lesion inference) and reports accuracy on a physician-verified test set, establishing baselines that existing general-domain and medical-domain segmentation models fail on.

Main Findings

• Baselines collapse on this task. On the MIMIC-ILS test set, general-domain models score far below the proposed model: LISA-7B (gIoU 8.3, cIoU 12.8, N-Acc. 0.7), LISA-13B (8.9, 12.2, 0.0), Text4Seg (6.1, 10.3, 20.6), PixelLM-7B (9.2, 11.8, 0.0), PixelLM-13B (12.8, 15.4, 0.0). Medical-domain baselines also fall short: BiomedParse (23.8, 18.5, 0.6), RecLMIS (22.4, 19.5, 0.0), IMIS-Net (9.8, 11.8, 21.6).

• ROSALIA substantially outperforms all baselines. It reaches gIoU 71.2, cIoU 75.6, and N-Acc. 91.8 — versus a best baseline gIoU of 23.8 and N-Acc. of 21.6.

• Performance varies by lesion type. Cardiomegaly is the strongest (gIoU 89.0, cIoU 89.0, N-Acc. 85.8), while pneumonia is weakest on IoU (gIoU 57.2, cIoU 60.4) but has the highest empty-target accuracy (97.1). Other types: atelectasis (60.2, 58.7, 91.7), opacity (60.5, 64.2, 85.0), consolidation (61.9, 65.6, 91.2), edema (64.8, 66.6, 92.2), effusion (60.3, 59.6, 90.4). Total: 71.2, 75.6, 91.8.

• Text responses are largely accurate. Overall accuracy across question types is 94.4%, with basic 96.8%, global 88.8%, and lesion inference 84.8%. Positive samples: overall 90.7, basic 95.4, global 93.7, lesion inference 75.1. Negative samples: overall 95.3, basic 96.9, global 82.3, lesion inference 90.6.

• Expert review supports dataset quality for negatives more than positives. Of 10.7K mask samples initially reviewed by four radiation oncologists, 96.4% were rated acceptable overall. The breakdown is negative 97.7% of 8,860 samples versus positive 90.1% of 1,841 samples. Per expert: A 96.1% (4,090), B 97.2% (4,028), C 98.7% (4,041), D 97.6% (4,065).

• Dataset composition and split. MIMIC-ILS contains 135K positive and 930K negative samples, split following the official MIMIC-CXR split into 1M training, 8.2K validation, and 12K test samples.

• Qualitative behavior. Baseline models either fail entirely or segment whole anatomical regions such as the left or right lung, whereas ROSALIA isolates the instructed lesion and handles empty-target cases; it also produces different outputs for different instructions applied to the same image.

Methodology in Plain English

The starting point is MIMIC-CXR, a public collection of chest X-rays each paired with a radiologist-written report. The pipeline turns each of these pairs into training data in two stages.

Stage one — producing lesion masks. Reports are first converted into structured tuples by an LLM (Mistral-Small-3.1-24B-Instruct-2503), capturing entity, sentence index, presence, certainty, location, and predicted lesion type; a medical-domain LLM (medgemma-27b-text-it) then maps the free-text location onto anatomical labels such as "right lung base." In parallel, three pretrained vision models extract spatial information: a diffusion model (RadEdit) removes the lesion using the prompt "No acute cardiopulmonary process," and the pixel difference between the original and edited images defines an anomaly map of hyperintense regions; an anatomy segmentation model (CXAS, 159 anatomical classes) produces anatomy masks for the reported locations; and a pretrained YOLO detector produces lesion bounding boxes with confidence scores. A pretrained HybridGNet supplies right lung, left lung, and heart masks.

Candidate lesion boxes are kept only if they satisfy four conditions — sufficient overlap with the anatomy masks, a high detection confidence, a high internal signal ratio from the anomaly map, and sufficient size relative to a lung mask. The connected components of the anomaly map intersecting the surviving boxes become the final lesion mask after noise removal and small refinements. A verification step then labels each region as a reported location, a grounded location (overlapping an actual mask), or an empty location, and the empty ones supply negative examples.

Stage two — producing instructions and answers. Using the grounded information, the authors generate three instruction types: basic ("Segment the [Target] in the [Location]."), global ("Segment the [Target]."), and lesion inference ("Segment the opacity in the [Location] and predict its type."). The global type is only generated when the grounded and reported locations match, and lesion inference is applied only to pneumonia, atelectasis, and edema, which are reframed as instances of the broader "opacity" concept. Answers contain a [SEG] token for the mask plus a textual description; negative pairs use an empty mask. Cardiomegaly uses the heart mask directly, while localized findings use the generated lesion masks.

Model training. ROSALIA adopts the LISA architecture, pairing a LLaVA vision-language backbone with SAM-H. The VLM takes the image and instruction and emits a [SEG] token plus text; the hidden embedding of [SEG] is fed to SAM's mask decoder together with the frozen image encoder's features to produce the mask. Training fine-tunes from the LISA-7B checkpoint with LoRA (rank 128, alpha 256) on the VLM and full fine-tuning of the mask decoder, for 15 epochs at an initial learning rate of 0.0003 with AdamW, total batch size 256, and a 1:1 positive-to-negative ratio in each mini-batch. The loss combines a text term and a mask term (binary cross-entropy plus DICE), with coefficients λ_txt = 0.5, λ_bce = 5, λ_dice = 1, and the DICE loss computed only on positive samples. Evaluation uses gIoU, cIoU, and empty-target accuracy (N-Acc.).

Why This Matters

Impact on research. Pixel-level lesion annotations in CXRs are scarce, and prior datasets provide either coarse bounding boxes or single-lesion-type masks at limited scale. MIMIC-ILS offers 91K masks and 1.1M instruction–answer pairs across seven lesion types with no manual annotation, and the strong baseline gap reported in Table 4 suggests the task is not solvable by simply transferring existing general-domain or medical-domain segmentation VLMs.

Real-world applications.

• Assisting radiologists in localizing and outlining lesions without requiring them to first write expert-level descriptions.
• Supporting non-experts or trainees who cannot interpret a CXR themselves but can ask a simple question such as whether pneumonia is present in a given region.
• Enabling consistent, template-driven lesion contouring in workflows where contouring is otherwise labor-intensive.
• Reporting absence of a finding ("There is no atelectasis in the left lung base"), which supports triage and prior-comparison workflows.

Industry relevance. Automated, report-driven annotation is directly relevant to companies building medical imaging software and clinical documentation tools, since it removes the main cost driver — expert mask annotation — from dataset creation. The dataset and model are released at https://github.com/checkoneee/ROSALIA under a CC BY 4.0 license.

Future Directions

• Extending beyond seven lesion types. The current dataset covers cardiomegaly, pneumonia, atelectasis, opacity, consolidation, edema, and effusion; whether the pipeline generalizes to rarer findings is an open question.
• Improving weaker lesion categories. Pneumonia and effusion trail cardiomegaly on IoU by a wide margin (57.2 and 60.3 versus 89.0), which points to localization difficulty for diffuse, variable-position abnormalities.
• Closing the gap between positive and negative quality. Expert acceptance was 90.1% for positive samples versus 97.7% for negatives, so mask precision for reported findings is the limiting factor.
• Testing the pipeline on other modalities and datasets. The method is described as general and relies on paired image–report data; applying it beyond MIMIC-CXR or beyond X-ray is a natural next step.

Target Audience

Researchers and practitioners in medical image analysis, particularly those working on segmentation, vision-language models, or dataset construction. It is also relevant to clinical AI teams looking to reduce annotation costs, and to radiologists or radiation oncologists interested in how automated lesion contouring performs relative to expert judgment. Readers should have some familiarity with segmentation metrics (IoU variants) and multimodal model architectures to get the most from the experiments section.

Authors’ abstract

The applicability of current lesion segmentation models for chest X-rays (CXRs) has been limited both by a small number of target labels and the reliance on complex, expert-level text inputs, creating a barrier to practical use. To address these limitations, we introduce instruction-guided lesion segmentation (ILS), a medical-domain adaptation of referring image segmentation (RIS) designed to segment diverse lesion types based on simple, user-friendly instructions. Under this task, we construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using our fully automated multimodal pipeline that generates annotations from CXR images and their corresponding reports. MIMIC-ILS contains 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types. To empirically demonstrate its utility, we present ROSALIA, a LISA model fine-tuned on the MIMIC-ILS dataset. ROSALIA can segment diverse lesions and provide textual explanations in response to user instructions. The model achieves high accuracy in our newly proposed task, highlighting the effectiveness of our pipeline and the value of MIMIC-ILS as a foundational resource for pixel-level CXR lesion grounding. The dataset and model are available at https://github.com/checkoneee/ROSALIA.