Sim4Seg: Boosting Multimodal Multi-disease Medical Diagnosis Segmentation with Region-Aware Vision-Language Similarity Masks

AI summary

Overview

Research area: Medical computer vision and vision-language modeling, specifically medical image segmentation combined with explainable diagnosis.

Technical level: Advanced. The paper assumes familiarity with large vision-language models (LVLMs), the Segment Anything Model (SAM), referring/reasoning expression segmentation, chain-of-thought prompting, and test-time scaling.

Scope: The paper introduces a new medical vision-language task (Medical Diagnosis Segmentation), the M3DS dataset that supports it, a framework (Sim4Seg) built around a region-aware vision-language similarity-to-mask module, and a test-time scaling strategy for that task.

What This Paper Is About

Existing medical image segmentation models are typically trained only to outline structures, lesions, or organs, and they cannot explain why a region is abnormal. The authors define a new task, Medical Diagnosis Segmentation (MDS), where a model receives a medical image plus a clinician-style query and must output both a segmentation mask and a diagnostic answer with reasoning. To make this task trainable, they build the M3DS dataset and a model framework, Sim4Seg, that reuses internal similarity signals from a large vision-language model to guide a segmentation decoder.

Key Contributions

1. A new task definition, Medical Diagnosis Segmentation (MDS). MDS requires a model to understand clinical queries about medical images and jointly produce segmentation masks and diagnostic results, rather than treating segmentation and diagnosis as separate problems.

2. The M3DS dataset. A Multimodal Multi-disease Medical Diagnosis Segmentation dataset containing 10 subsets spanning 5 modalities, with each sample containing an image, ground-truth mask, query, diagnosis result, and a diagnosis chain-of-thought. The CoT data is produced by an automated multi-role pipeline using the open-source medical LVLM HuatuoGPT-Vision, with a critical-assistant review loop and a human-assisted review phase. The splits contain 12,000 training, 2,284 validation, and 1,864 test samples.

3. The Sim4Seg framework and the Region-Aware Vision-Language Similarity to Mask (RVLS2M) module. RVLS2M derives region-level masks from the similarity between image tokens and a special segmentation token in the LVLM's last hidden state, and uses those masks as prompts for the segmentation model.

4. A test-time scaling strategy for the MDS task. At inference, the model samples multiple chain-of-thought reasoning paths and multiple segmentation masks, forming m × n candidate masks, and selects the final mask using a quality metric computed as the average of gIoU and cIoU.

Main Findings

• Main benchmark results on the M3DS test set (1,864 samples). Sim4Seg (ft-CoT) reaches 51.86 gIoU, 53.90 cIoU, and 69.04 diagnosis accuracy. Adding test-time scaling raises these to 53.11 gIoU, 55.83 cIoU, and 82.63 accuracy. For comparison, LISA (ft-CoT) reaches 45.90 gIoU, 45.92 cIoU, and 58.05 accuracy, and LISA (ft-diagnosis) reaches 45.87 gIoU, 46.05 cIoU, and 53.27 accuracy.

• Reported improvement over reasoning segmentation models. The paper states that Sim4Seg exceeds reasoning segmentation models by +57.3% in segmentation performance and +165.4% in diagnosis accuracy.

• Baselines on M3DS. LISA without fine-tuning scores 32.43 gIoU, 31.83 cIoU, and 4.71 accuracy; READ scores 13.37 gIoU, 25.75 cIoU, and 2.52 accuracy; SAM-Med2D scores 22.94 gIoU and 51.42 cIoU; LLaVA-Med scores 3.48 accuracy.

• Ablation averages (Table 3). Without RVLS2M: zero-shot 26.42 gIoU / 25.11 cIoU / 11.94 Acc; fine-tuned without diagnosis 33.09 / 33.27 / 0.00 Acc; fine-tuned with diagnosis 35.21 / 36.23 / 59.80 Acc; fine-tuned with diagnosis-CoT 35.16 / 35.35 / 64.64 Acc. With RVLS2M: fine-tuned with diagnosis 39.08 / 41.16 / 55.84 Acc; with diagnosis-CoT 40.34 / 41.68 / 69.90 Acc; with diagnosis-CoT plus test-time scaling 41.37 / 43.48 / 80.98 Acc.

• Fine-tuning with diagnosis text removes a failure mode. Fine-tuning without diagnosis text yields 0.00 average accuracy, while adding diagnosis text raises it to 59.80, and adding diagnostic chain-of-thought raises it to 64.64.

• RVLS2M works zero-shot as a prompt creator. Using LISA as the base model and adding RVLS2M without any training improves average performance by 11.6% (from 32.13 to 35.85 average; gIoU moves from 32.43 to 31.82 and cIoU from 31.83 to 39.88).

• An inverted U-shaped relationship with similarity-map granularity. Selecting 36 grid cells gives the best average (52.88), versus 39.11 at 12 cells, 45.85 at 24 cells, and 43.14 at 48 cells. At a 16 × 16 grid resolution, performance peaks at 52.88 average, compared with 33.04 at 8 × 8, 50.27 at 32 × 32, and 47.55 at 64 × 64.

• Test-time scaling parameters behave monotonically. Increasing the number of CoT paths m improves diagnosis accuracy, and increasing the number of segmentation masks n improves segmentation performance, as shown in the paper's scaling plots.

• Cross-modality generalization. When each modality is excluded from training in turn, Sim4Seg outperforms LISA on every listed modality: X-Ray 10.45 gIoU / 19.07 cIoU versus 6.34 / 14.01; Dermoscopy 44.97 / 34.96 versus 43.51 / 27.10; Endoscopy 41.38 / 31.12 versus 36.13 / 35.86; Ultrasound 27.08 / 23.69 versus 26.17 / 23.42; Fundus Photography 15.76 / 16.17 versus 13.37 / 13.56.

• Cross-dataset generalization. When datasets are excluded from training in turn, Sim4Seg leads on ISIC (60.44 gIoU / 56.83 cIoU versus 58.54 / 54.12), Kvasir-SEG (41.38 / 31.12 versus 36.13 / 35.86), TN3K (33.32 / 33.62 versus 29.06 / 25.40), ChestX-Det (7.78 / 9.13 versus 6.21 / 7.24), and FIVES (15.76 / 16.17 versus 13.37 / 13.56).

• Sub-dataset level detail. The ablation table reports per-subset results across FracAtlas, bone fracture, BFD, ISBI, ISIC, Kvasir-SEG, BUSI, TN3K, ChestX-Det, and FIVES, showing that the strongest gains from RVLS2M and CoT appear in subsets such as FracAtlas, TN3K, and Kvasir-SEG.

Methodology in Plain English

The authors first assembled a dataset from ten public medical segmentation sources covering X-Ray, dermoscopy, endoscopy, ultrasound, and fundus photography, including FracAtlas, a bone fracture dataset, Bone Fracture Detection (BFD), ISBI, ISIC, Kvasir-SEG, BUSI, TN3K, ChestX-Det, and FIVES. Each sample was paired with a query, a segmentation mask, a diagnosis, and a step-by-step diagnosis chain-of-thought.

The chain-of-thought text was generated automatically using HuatuoGPT-Vision in two roles: a medical assistant that writes the reasoning in a fixed format (identify the modality, analyze the image, reach a diagnosis), and a critical assistant that checks the output against criteria for step completeness, logical rigor, and medical reliability, returning pass or reject. Rejected outputs are looped back for regeneration up to a maximum number of review rounds, and a human-assisted review phase follows.

The model, Sim4Seg, couples a large vision-language model with a SAM-based segmentation decoder. Following LISA, a special segmentation token is added to the vocabulary. When the LVLM processes the image and text query, the last hidden state embedding of that special token is projected and treated as a query, while the image token embeddings act as keys; their similarity is computed and softmax-normalized. That similarity vector is reshaped into a 2D map, split into non-overlapping g × g grids, average-pooled within each cell, and thresholded to produce a binary region mask. This region mask, together with the projected token embedding and the image's visual features, is passed to the segmentation decoder to produce the final mask. Because the mask comes from the model's own internal vision-language similarity, it works as an extra prompt for the decoder.

Training uses a cross-entropy loss for text generation and a combined binary cross-entropy plus DICE loss for segmentation, with weighted averaging. The model was initialized from LISA, trained for four epochs with an AdamW optimizer, learning rate 3 × 10⁻⁴, weight decay 0.01, batch size 2, gradient accumulation of 10, loss weights λ_mask = 1.0, λ_txt = 1.0, λ_bce = 2.0, λ_dice = 0.5, on an NVIDIA H800 GPU.

At inference, the test-time scaling strategy samples m different reasoning paths and, for each, generates masks with stochastic perturbation, yielding m × n candidates; the final mask is chosen by maximizing a quality metric that averages gIoU and cIoU.

Why This Matters

Impact on research. The paper argues that segmentation and explainable diagnosis are rarely studied jointly, and provides both a task definition and a paired dataset so that future work can evaluate on one shared benchmark. It also shows that internal LVLM representations can serve as free, plug-and-play spatial prompts, which broadens how reasoning segmentation methods can be built for medical data.

Real-world applications.

• Clinical decision support that returns both a lesion outline and a written rationale, so a clinician can audit the reasoning behind a highlighted region.
• Radiology and pathology triage, where a query-driven mask plus a diagnosis answer can prioritize which scans need review first.
• Deployment across heterogeneous imaging equipment, since the cross-modality and cross-dataset experiments test generalization to X-Ray, dermoscopy, endoscopy, ultrasound, and fundus photography, as well as to datasets excluded from training.
• Medical education and reporting, where the chain-of-thought output documents step-by-step findings such as modality identification followed by feature analysis and a final diagnosis.

Industry relevance. Healthcare AI vendors and device makers that need a single model serving both localization and structured explanation can build on a framework that reuses an existing LVLM and a SAM-style decoder rather than training separate specialists. The plug-and-play behavior of RVLS2M, which improves LISA's average result by 11.6% with no training, suggests a low-cost upgrade path for systems already built on reasoning segmentation. The dataset, code, and test-time scaling strategy are released publicly via the linked GitHub repositories.

Future Directions

• Extending M3DS beyond the current ten subsets and five modalities. The dataset is built entirely from existing public segmentation datasets, with chain-of-thought text generated by an automated pipeline plus human-assisted review; the paper does not report an evaluation of that generated text independent of downstream model performance.

• Making test-time scaling efficient. The strategy produces m × n candidate masks for every input and selects among them using a quality metric computed against a mask. The paper does not report the inference cost or latency trade-offs of larger m and n.

• Exploring the granularity trade-off further. The paper documents an inverted U-shaped relationship with 36 grid cells and a 16 × 16 grid resolution performing best, explaining that coarse grids blur regions while fine grids retain spurious correlations, but does not provide a principled way to set this threshold per image or per modality.

• Broadening evaluation. Baseline coverage on M3DS includes LLaVA-Med, SAM-Med2D, READ, and LISA; the paper does not report results from the other reasoning segmentation models it discusses, such as GSVA, GLaMM, PixelLM, or BiomedParse.

Target Audience

Researchers working on medical image segmentation, medical vision-language models, reasoning segmentation, and multimodal clinical AI will get the most from this paper, particularly those interested in chain-of-thought supervision for pixel-level tasks. It is also useful for practitioners building explainable diagnostic systems who need a benchmark covering multiple imaging modalities, and for dataset builders interested in automated pipeline design with critical-assistant review. Readers without a background in vision-language architectures and promptable segmentation models will find the methodology section dense.

Authors’ abstract

Despite significant progress in pixel-level medical image analysis, existing medical image segmentation models rarely explore medical segmentation and diagnosis tasks jointly. However, it is crucial for patients that models can provide explainable diagnoses along with medical segmentation results. In this paper, we introduce a medical vision-language task named Medical Diagnosis Segmentation (MDS), which aims to understand clinical queries for medical images and generate the corresponding segmentation masks as well as diagnostic results. To facilitate this task, we first present the Multimodal Multi-disease Medical Diagnosis Segmentation (M3DS) dataset, containing diverse multimodal multi-disease medical images paired with their corresponding segmentation masks and diagnosis chain-of-thought, created via an automated diagnosis chain-of-thought generation pipeline. Moreover, we propose Sim4Seg, a novel framework that improves the performance of diagnosis segmentation by taking advantage of the Region-Aware Vision-Language Similarity to Mask (RVLS2M) module. To improve overall performance, we investigate a test-time scaling strategy for MDS tasks. Experimental results demonstrate that our method outperforms the baselines in both segmentation and diagnosis.