LungNoduleAgent: A Collaborative Multi-Agent System for Precision Diagnosis of Lung Nodules

AI summary

Overview

• Research area: Medical computer vision and multimodal AI — specifically CT-based lung nodule analysis, vision-language models (VLMs), and collaborative multi-agent clinical reasoning.
• Technical level: Advanced. The paper assumes familiarity with detection architectures (Mixture of Experts, DBSCAN clustering), vision-language models, retrieval-augmented generation, and knowledge graphs.
• Scope: The paper proposes and evaluates a three-module multi-agent system that detects lung nodules in CT scans, generates localized radiology-style reports, and grades malignancy, benchmarked against generalist VLMs, medical VLMs, and medical agent systems on two private datasets and the public LIDC-IDRI dataset.

What This Paper Is About

Diagnosing lung cancer requires radiologists to find nodules in CT scans, describe their morphology, and judge malignancy — a slow process subject to interobserver variability. General and medical VLMs can analyze these scans, but the authors argue they describe nodule morphology imprecisely and lack sufficient pathology knowledge, while existing medical agent systems reach only 40–50% accuracy on lung cancer-specific tasks (versus 75–80% on general medical tasks). LungNoduleAgent addresses this by decomposing the diagnostic workflow into specialized, sequential modules — nodule detection, localized CT report generation, and multi-agent malignancy reasoning grounded in a pathology knowledge base.

Key Contributions

1. A clinical-workflow-mimicking architecture. The system splits diagnosis into three sequential components: the "Nodule Spotter" for precise nodule detection, the "Simulated Radiologist" for detailed CT report generation, and the "Doctor Agent System (DAS)" for malignancy assessment using expert knowledge and multi-agent discussion.
2. A Focal Prompting Mechanism. This mechanism is designed to improve fine-grained visual perception, enabling description of nodule morphological features and dynamic evaluation of nodule characteristics across slices.
3. A shared Memory plus medical knowledge graph. Pathological information is stored in a shared Memory for the agent system, supporting collaborative multi-agent discussion and evidence-based quantitative analysis with medical prior knowledge.
4. Evaluation across three datasets. The authors report comprehensive comparisons on two private datasets and the public LIDC-IDRI dataset, showing advantages over VLMs, medical agentic systems, and expert models.

Main Findings

• CT report generation performance: On PrivateA, LungNoduleAgent (LLaVA3.2-11B base) reached a LungDLC-score of 81.9, an increase of 6.3 over the highest score among other methods; on PrivateB it reached 80.3 (improvement of 4.1); and on LIDC-IDRI it reached 83.5 (increase of 8.3). Corresponding LLM-scores were 87.6, 89.8, and 89.3.
• Large gains over base models: Against Qwen2.5-VL-7B, LungNoduleAgent improved LungDLC-score by +26.2 (PrivateA), +23.2 (PrivateB), and +22.8 (LIDC-IDRI), with Pos QA gains of +35.9, +25.4, and +31.5, and Neg QA gains of +16.5, +20.9, and +14.2. Against LLaVA3.2-11B, LungDLC gains were +23.8, +20.0, and +21.2.
• LLM-score caveat: GPT-4o showed marginally higher LLM-scores than LungNoduleAgent, which the authors attribute to evaluator bias; they argue their model's consistent superiority on objective metrics validates its clinical utility.
• Malignancy grading performance: On the 3-class tasks, the system attained Acc of 86.7% (PrivateA) and 81.2% (PrivateB) with F1-scores of 0.889 and 0.803, outperforming MedGemma by 15.9–24.4% in Acc. On the 2-class LIDC-IDRI benchmark it achieved 89.1% Acc and 0.871 F1, surpassing MedGemma by 15.9% in Acc and 0.185 in F1.
• Baseline comparison range: Generalist VLMs scored far lower on malignancy prediction — for example GPT-4o reached 46.2% Acc on PrivateA, 41.2% on PrivateB, and 64.1% on LIDC-IDRI — while medical agents such as MedGemma-27B reached 62.3%, 60.2%, and 73.2% respectively.
• Module ablation (PrivateA): Removing modules degraded both metrics. With only DAS active, Acc was 62.1 and LungDLC 57.9; with Nodule Spotter and Simulated Radiologist but no DAS, Acc was 66.7 and LungDLC 88.9; with Nodule Spotter and DAS but no Simulated Radiologist, Acc was 75.1 and LungDLC 67.3; the full system reached 86.7 Acc and 88.9 LungDLC. The Nodule Spotter cannot be ablated independently because the Simulated Radiologist depends on the masks it produces.
• Detection sub-component ablation: Adding Mask Clustering and then the Judging Panel raised mAP from 67.1 to 71.6 to 79.3, and F1 from 64.3 to 71.3 to 83.5 (×10⁻²), corresponding to mAP improvements of 4% and 8% and F1 enhancements of 0.07 and 0.12.
• Detection accuracy drives diagnosis: Artificially varying detection region accuracy and IoU against ground truth showed that higher detection accuracy consistently led to better diagnostic outcomes, which the authors frame as support for evidence-based quantitative analysis over experience-driven qualitative assessment.
• Agent count sweet spot: Malignancy grading accuracy increased with the number of medical agents, peaked at five agents, then became inconsistent; five agents were chosen as the balance between diagnostic performance and computational efficiency.

Methodology in Plain English

Step 1 — Finding the nodule (Nodule Spotter). The CT volume is divided into slices. Several specialized detection models run in parallel in a Mixture of Experts setup, each producing a candidate mask. Because these masks disagree, the system clusters them by spatial overlap using DBSCAN, where similarity is defined as 1 minus Intersection over Union (IoU), with neighborhood radius ε equivalent to an IoU threshold τ = 1 − ε and a minimum sample count. Masks in a cluster are averaged and binarized at a threshold of 0.5. Outliers are discarded. A "Judging Panel" of multiple independent VLMs then votes on each candidate: each gives a +1 approval or −1 rejection plus a confidence score, and the weighted total must exceed 0 for the candidate to survive as a true nodule rather than a false positive.

Step 2 — Describing the nodule (Simulated Radiologist). A Focal Prompting Mechanism crops the image and mask around the nodule while keeping surrounding context, encodes both the full image and the crop, and fuses global context via gated cross-attention. Sequence concatenation captures how the nodule changes across slices. A MedPrompt then instructs the VLM to focus only on annotated areas, use anatomically accurate terminology, and follow a clinical report format without speculation, producing the localized CT report.

Step 3 — Judging malignancy (Doctor Agent System). Each medical agent is a VLM augmented with a medical knowledge graph. Documents from authoritative pathology websites and literature are compiled into a graph, from which community-level summaries are derived and retrieved to answer queries — an approach the authors adopt because standard retrieval-augmented generation struggles with broad questions like "what are the primary characteristics of malignant lung nodules." Agents first produce independent diagnoses from the nodule image and CT report; a summarizer consolidates them; agents then revise their conclusions taking other agents' views into account, iterating until consensus yields the final diagnosis.

Memory. A shared Memory stores nodule images, measured nodule size as evidence, CT reports, and the multi-agent conversations and summaries during discussion.

Evaluation setup. Two private datasets of 1,616 and 386 axial lung CT slices (512×512 pixels) with bounding box masks, morphological descriptions (lobar location, density, shape, margin, cavitation/vacuolation), and malignancy classes (pre-invasive, minimally invasive, invasive adenocarcinoma), plus LIDC-IDRI with 1,018 lung CT scans at identical resolution, providing segmentation masks, semantic attributes, and malignancy ratings from 1 to 5 (above 3 malignant, below benign). Because localized reports lack precise ground truth, the authors build yes/no attribute questions from annotations — positive questions (Pos QA) verify present features, negative questions (Neg QA) catch fabricated details — and average the correctness rates into the LungDLC-score, inspired by MedDLC-score. Report quality is also scored by GPT-4o as an "LLM-as-a-judge" on fluency, relevance, consistency, and clinical rationality, averaged into the LLM-score. Malignancy grading uses Accuracy and F1-score.

Why This Matters

• Research impact: The paper argues that region-level semantic alignment and multi-agent collaboration matter more than raw model scale for nodule diagnosis, and it contributes a reference-free evaluation approach (LungDLC) for fine-grained regional description in medical image understanding. It positions collaborative multi-agent systems as a way to combine the generality of VLMs with the precision of task-specific tools.
• Real-world applications:
  • Computer-aided detection and triage of lung nodules in CT screening programs.
  • Automated drafting of structured radiology reports describing nodule location, shape, margin, density, and size.
  • Decision support for malignancy risk stratification, including distinguishing pre-invasive, minimally invasive, and invasive adenocarcinoma.
  • Quality-check tooling that flags fabricated or irrelevant descriptions in AI-generated reports via the Pos QA / Neg QA scheme.
• Industry relevance: The workflow decomposes into interchangeable parts (detection models, VLMs, agent roles), and the authors release code at https://github.com/ImYangC7/LungNoduleAgent, making the architecture adaptable to other clinical imaging pipelines. The reported gap between generalist VLMs (e.g., GPT-4o at 46.2% and 41.2% Acc on the two private datasets) and the proposed system indicates where general-purpose multimodal models remain unready for specialized clinical deployment.

Future Directions

• Longitudinal and temporal analysis. The related work notes that agent systems gain more modest benefits from temporal data than human radiologists, suggesting longitudinal CT comparison as an open area.
• Handling nodule heterogeneity. Mixed-density subtypes (solid, ground-glass, part-solid) remain difficult for existing systems; the paper cites this as a limitation of current agent approaches.
• Stronger pathology correlation. The paper reports that agent systems show weaker alignment with histopathological grades than radiologists, pointing to improved linkage between imaging-based reasoning and confirmed pathology.
• Scaling and efficiency of agent collaboration. Performance peaked at five agents and became inconsistent beyond that, leaving open how to add expertise without introducing redundancy or inconsistency, and how to balance this against computational cost.

Target Audience

• Medical imaging and computer vision researchers working on CT-based lung cancer detection, segmentation, or grading.
• Researchers building multimodal agents and multi-agent systems for clinical reasoning, especially those integrating knowledge graphs or external tools.
• Radiology and pulmonology researchers evaluating AI decision support and automated report generation.
• Machine learning engineers in medical AI who need a concrete architectural blueprint for combining detection models, VLMs, and collaborative agent discussion, and who may want to reuse the released code.

Authors’ abstract

Diagnosing lung cancer typically involves physicians identifying lung nodules in Computed tomography (CT) scans and generating diagnostic reports based on their morphological features and medical expertise. Although advancements have been made in using multimodal large language models for analyzing lung CT scans, challenges remain in accurately describing nodule morphology and incorporating medical expertise. These limitations affect the reliability and effectiveness of these models in clinical settings. Collaborative multi-agent systems offer a promising strategy for achieving a balance between generality and precision in medical applications, yet their potential in pathology has not been thoroughly explored. To bridge these gaps, we introduce LungNoduleAgent, an innovative collaborative multi-agent system specifically designed for analyzing lung CT scans. LungNoduleAgent streamlines the diagnostic process into sequential components, improving precision in describing nodules and grading malignancy through three primary modules. The first module, the Nodule Spotter, coordinates clinical detection models to accurately identify nodules. The second module, the Radiologist, integrates localized image description techniques to produce comprehensive CT reports. Finally, the Doctor Agent System performs malignancy reasoning by using images and CT reports, supported by a pathology knowledge base and a multi-agent system framework. Extensive testing on two private datasets and the public LIDC-IDRI dataset indicates that LungNoduleAgent surpasses mainstream vision-language models, agent systems, and advanced expert models. These results highlight the importance of region-level semantic alignment and multi-agent collaboration in diagnosing nodules. LungNoduleAgent stands out as a promising foundational tool for supporting clinical analyses of lung nodules.