ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

AI summary

Overview

• Research area: Medical multimodal large language models (MLLMs) — computer vision, clinical NLP, and medical AI evaluation.
• Technical level: Advanced. The paper combines vision-encoder architecture design, clinical benchmark construction, and expert validation; readers will benefit from familiarity with vision transformers, multimodal LLM training pipelines, and radiology reporting conventions.
• Scope: ClinFusion proposes a vision-centric medical MLLM with a compositional 2D/3D encoder and a clinician-aligned evaluation framework, validated on 24+ benchmarks and a blinded study by six board-certified radiologists.

What This Paper Is About

Deploying multimodal LLMs in medicine is fundamentally a vision problem: models must absorb knowledge from heterogeneous medical images (2D X-rays, pathology slides, fundus photos; 3D CT and MRI volumes) and be judged the way radiologists actually work. Existing medical MLLMs rely on a single monolithic vision encoder and are evaluated with text-matching metrics that ignore clinical context and cannot separate missed findings from hallucinations. ClinFusion addresses both problems with a new encoder architecture and a new evaluation protocol, then validates them against expert clinical judgment.

Key Contributions

1. A compositional and cascaded vision architecture. ClinFusion replaces the single-encoder design with a foundational, well-aligned Qwen ViT progressively enriched by specialist 2D encoders (DINOv2, ConvNeXt, Medsiglip) plus a dedicated native 3D encoder. The unifying mechanism is CaSL (Cascade Spatial-Aware Locality) Fusion, a local cross-attention operator that cascades features across encoders and uses aligned 2D representations to anchor alignment of 3D volumetric features.
2. A vision-grounded evaluation framework. Two components: MedIF-Bench, which measures whether a medical MLLM can actually follow complex clinical instructions (a prerequisite that domain fine-tuning often destroys), and an RoI-grounded report generation methodology that conditions generation on patient-specific clinical context ("Clinical Indication" and "Area of Focus") and uses an LLM-as-a-judge to decompose claims into matched, missed, and hallucinated findings.
3. State-of-the-art results across modalities, plus expert validation. ClinFusion outperforms leading open-source medical MLLMs (Hulu-Med, Lingshu) on 20 of 24 benchmarks and beats proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 of 16 benchmarks. A blinded study with six radiologists over 300 cases confirms ClinFusion produces the highest-ranked reports and that the RoI-grounded metric correlates best with expert judgment.
4. An agentic tool-use extension. Retrieval-augmented generation grounds answers in citable literature, while perception expert tools let the model invoke specialized AI models for tasks like organ segmentation or disease classification where dedicated systems remain stronger.

Main Findings

• 2D visual question answering is competitive with frontier proprietary models. ClinFusion-8B ranks first or second on all eight 2D VQA benchmarks and beats Gemini-3-Flash on six (OmniMedVQA, PMC-VQA, MedFrameQA, VQA-RAD, SLAKE, PathVQA). A remaining gap on the hardest reasoning benchmark, MedXpertQA, shrinks with scale.
• Report generation improves substantially. ClinFusion-8B reaches F1 of 37.8 on CheXpert-Plus and 57.3 on IU-XRAY, versus 31.9 and 46.5 for Hulu-Med-7B, matching or beating proprietary APIs — most clearly on CheXpert-Plus.
• Native 3D understanding is a decisive advantage. On AMOS-MCQ, ClinFusion-8B scores 80.2 versus 65.7 for Hulu-Med-7B and 73.9 for the much larger Hulu-Med-32B, and leads Gemini-3-Flash by 16 points. On CT-Rate Report, ClinFusion-32B attains the best F1 (23.9) across all models, ahead of Hulu-Med-32B (23.4) and GPT-5.2 (14.1). On AMOS Report, Gemini-3-Flash still leads overall (23.4 vs. 16.1), which the authors flag as open headroom.
• Instruction-following survives medical adaptation. ClinFusion scores 98.1 (8B) and 98.9 (32B) on MedIF-Bench, beating GPT-5.2 (96.0), Gemini-3-Flash (96.6), and Claude-Sonnet-4.5 (86.5). Critically, prior medical adaptations of the same backbones degrade badly (Lingshu-7B: 80.4; 32B: 82.6). MedGemma-1.5-4B-IT scores only 27.4, and its inability to produce answers in the requested format makes its other benchmark scores misleadingly low — evidence that instruction-following is a prerequisite for measuring latent knowledge at all.
• Textual medical knowledge is not sacrificed. Despite the vision-centric design, ClinFusion-32B leads all medical MLLMs on seven of eight text benchmarks, including MedXpertQA (26.7 vs. 19.8), Medbullets (74.8 vs. 67.5), and MedQA-MCMLE (93.8 vs. 87.0). Scaling from 8B to 32B narrows the gap to proprietary models noticeably (MedXpertQA 20.0 to 26.7; SuperGPQA 32.6 to 41.8).
• Radiologists prefer ClinFusion, and prefer it more with tools. Across 300 cases, ClinFusion with agentic tools ranks first on Overall, Accuracy, Completeness, and Operability, with statistically significant margins over both Hulu-Med and Gemini-3-Flash (p < 0.001). Standalone ClinFusion already beats both baselines.
• The RoI-grounded metric best matches expert judgment. Among eleven automatic metrics, it achieves the highest rank correlation with expert consensus (Kendall's tau = 0.511, Spearman's rho = 0.572, Top-1 accuracy 55.5%), with a higher mean and lower variance per case than all alternatives. Inter-annotator agreement (Kendall's W = 0.665 ± 0.275) confirms the annotations are reliable.
• Clinical context is necessary for meaningful report evaluation. Removing it collapses scores for all models and compresses the gap between them, because claims about regions the reference report never addressed cannot be distinguished from hallucinations and are penalized as false positives.
• RadGraph-F1 has two measurable failure modes. Synonymy insensitivity (e.g., "the liver appears enlarged" scoring zero against "hepatomegaly is present") and length bias (a deliberately verbose variant inflates RadGraph-F1 from 11.8 to 18.8 on CheXpert-Plus, while the LLM-judge correctly penalizes it, 16.8 to 15.1).
• Ablations confirm the design. Local cross-attention beats channel concatenation, global cross-attention, and mixture-of-vision-experts. DINOv2 is the best single specialist encoder (the only one within 0.3 of the best on both VQA and report generation). Cascade fusion beats parallel fusion on report generation (28.8 vs. 27.3). Returns diminish beyond one additional specialist encoder, and disabling the native 3D encoder hurts volumetric understanding.

Methodology in Plain English

The team started from the observation that medical images come in fundamentally different forms — flat 2D pictures and stacked 3D volumes — and that one vision encoder cannot serve both well. Rather than fine-tuning a general model harder on more data, they redesigned the vision front end.

A strong, already well-aligned vision transformer (Qwen ViT) serves as the base representation. Specialist encoders trained differently (DINOv2's self-supervised features, ConvNeXt, Medsiglip) then feed into it one after another through a local cross-attention operator, so each specialist refines only nearby spatial regions rather than overwriting the whole representation. Because the base stays aligned, the language model behind it does not need to relearn vision. For 3D volumes, a dedicated 3D encoder is aligned using the 2D features as reference points, keeping both modalities in a shared space.

On the evaluation side, the team argued that current scoring is clinically wrong. Radiologists read images with a specific question in mind, and their reports cover only the relevant anatomy — so forcing a model to write a comprehensive report and then scoring it against a reference punishes legitimate observations the original radiologist simply did not write down. ClinFusion's protocol extracts the clinical indication and area of focus, feeds them to the model, and limits evaluation to that anatomical scope. Findings are then sorted by an LLM judge into matched, missed, and hallucinated, giving a finer picture than string overlap.

Finally, they tested whether their metric reflects what doctors actually value. Six board-certified radiologists with at least six years of experience each ranked anonymized reports from four systems (ClinFusion with tools, ClinFusion standalone, Gemini-3-Flash, Hulu-Med) across 300 cases spanning chest X-ray, chest CT, and abdominal CT, scoring factual accuracy, completeness, and clinical utility. Those rankings were then correlated against eleven automatic metrics.

Why This Matters

Impact on research. The paper reframes medical MLLM progress as an architecture problem, not purely a data problem. Its ablation showing that cascade fusion beats parallel fusion on open-ended generation, and that a native 3D encoder is not optional, gives the field concrete design guidance. MedIF-Bench and the RoI-grounded metric give researchers evaluation tools that track expert judgment far better than BLEU, ROUGE, or RadGraph-F1 — which matters because misleading metrics steer entire research programs in the wrong direction.

Real-world applications.

• Radiology reporting assistance. Generating structured, region-scoped draft reports for chest X-ray, chest CT, and abdominal CT, with explicit pertinent negatives.
• Triage and second-read workflows. Flagging findings that generalist models miss on subtle diffuse density changes, such as mild fatty liver or cardiomegaly.
• Tool-augmented clinical decision support. Combining literature retrieval with calls to specialized segmentation or classification models for verifiable, traceable reasoning.
• Medical model procurement and regulation. The instruction-following benchmark and expert-correlated metric offer regulators and hospitals a defensible way to compare clinical AI systems.

Industry relevance. ClinFusion comes from DAMO Academy (Alibaba), Hupan Laboratory, Zhejiang University, and Tsinghua University, with released weights on GitHub and Hugging Face. An open model that outperforms GPT-5.2 and Gemini-3-Flash on 13 of 16 multimodal benchmarks directly challenges the assumption that closed proprietary APIs are the default for clinical AI.

Future Directions

• Closing the hard reasoning and CT-reporting gaps. MedXpertQA remains a weak point, and Gemini-3-Flash still leads on AMOS Report (23.4 vs. 16.1). Scaling from 8B to 32B helped substantially; whether further scaling or better 3D training data closes these gaps is untested.
• Validating on prospective, multi-site clinical data. The 300-case study is retrospective and drawn from existing benchmarks. Prospective deployment across hospitals with different scanners, protocols, and patient populations is the next bar.
• Extending the RoI-grounded protocol beyond radiology. The methodology is demonstrated on chest X-ray and CT reporting. Pathology, dermatology, and ophthalmology reports have different structures and would test how far the approach generalizes.
• Making agentic tool use clinically safe. Tool-augmented reasoning improved rankings consistently, but the paper does not quantify tool-selection error, latency, or failure modes — all critical for a system that would sit inside a clinical workflow.
• Understanding why instruction-following degrades. MedGemma-1.5-4B-IT shows that volume of curated medical data alone does not preserve format compliance. Identifying which training-stage choices cause the degradation would help the whole field.

Target Audience

Clinical AI researchers and engineers building medical multimodal models will find the encoder architecture and ablation studies most useful. Medical AI evaluation researchers and radiologists involved in AI validation will find MedIF-Bench and the RoI-grounded metric the most consequential contributions, since they address a well-known flaw in how report generation is scored. Regulators, hospital informatics teams, and industry product leaders evaluating medical MLLMs will benefit from the expert-validated comparison against proprietary systems. Graduate students entering medical vision-language research will find the framing of the field's two central challenges — heterogeneous visual knowledge absorption and vision-grounded evaluation — a useful orientation, though the architectural details are dense for readers without transformer background.

Authors’ abstract

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.