PulseMind: A Multi-Modal Medical Model for Real-World Clinical Diagnosis

AI summary

Overview

Research area: Medical multi-modal AI — vision-language models applied to real-world clinical diagnosis and physician-patient consultation.

Technical level: Advanced (requires familiarity with vision-language models, supervised fine-tuning, and reinforcement learning), though the core ideas are explained in plain language below.

Scope: The paper presents PulseMind, a family of multi-modal medical diagnostic models, together with a new consultation dataset (MediScope), a diagnostic evaluation benchmark (PulseMind Benchmark), and a comparison-based reinforcement learning method (CRPO).

What This Paper Is About

Most medical AI models are built for narrow image-analysis tasks such as dermatology, pathology, or radiology, and they evaluate single questions about single images. Real clinical diagnosis is different: it mixes many kinds of inputs (lab results, reports, prescriptions, scans, surgical records) and unfolds over multiple back-and-forth turns between a physician and a patient. The paper's goal is to build a model, dataset, benchmark, and training method that together handle this messier, more realistic diagnostic setting.

Key Contributions

1. MediScope dataset — A large-scale multi-modal clinical diagnostics dataset described as the first of its kind featuring real-world multi-turn diagnostic consultations. It contains 98,000 real-world multi-turn consultations and 601,500 medical images, spanning over 10 major clinical departments and more than 200 sub-specialties, and covering laboratory test results, examination reports, prescriptions, medical images, and surgical records. Construction followed four stages: collection, anonymization, expansion, and proofreading.

2. PulseMind Benchmark — A benchmark for evaluating clinical diagnostic capability in multi-turn consultations, with a four-dimensional evaluation protocol: proactiveness, accuracy, usefulness, and language quality. It combines two subsets: MedDiagnose (237 multi-modal samples collected from patient cases with expert-verified dialogues) and an expanded CMtMedQA-test (1,000 samples), totaling over 1,200 samples.

3. Comparison-based Reinforcement Policy Optimization (CRPO) — A reinforcement learning method that replaces absolute numerical score rewards with relative preference signals. Each candidate response is compared pairwise against responses from five counterpart models across four evaluation dimensions, producing a binary reward that is averaged over counterparts and dimensions.

4. PulseMind models — Medical multi-modal models built on Qwen2.5-VL-72B and Qwen2.5-VL-32B, reported to achieve competitive results on both the PulseMind Benchmark and 11 public medical question-answering datasets.

Main Findings

• Average win rate of 76%: On the PulseMind Benchmark, PulseMind achieves an average win rate of 76% against six baseline methods.

• Win rates on the multi-modal MedDiagnose subset: 94% against GPT-4o, 89% against o1, 54% against Gemini 2.5-Pro, 86% against Qwen2.5VL-72B, 83% against InternVL3, and 98% against the domain-specific medical model Lingshu.

• Win rates on the text-only CMtMedQA-test subset: 83% against o1, 73% against GPT-4o, 72% against Gemini 2.5-Pro, 54% against Qwen2.5VL-72B, 55% against InternVL3, and 71% against Lingshu.

• PulseMind-72B results on 11 medical QA benchmarks: MMMU Health & Medicine 69.4, VQA-RAD 87.1, PMC-VQA 70.3, SLAKE 85.6, PathVQA 64.9, DermaVQA 42.0, MedXpertQA-MM 36.7, MMLU clinical topics 88.7, MedMCQA 71.3, MedQA 94.8, and MedXpertQA-text 29.8. The paper states PulseMind-72B achieves the best results across all 11 benchmarks, surpassing peer open-source models and outperforming closed-source models in multiple tasks.

• PulseMind-32B results: MMMU 64.6, VQA-RAD 83.2, PMC-VQA 68.1, SLAKE 81.5, PathVQA 62.0, DermaVQA 32.0, MedXpertQA-MM 29.6, MMLU 85.6, MedMCQA 66.4, MedQA 92.9, and MedXpertQA-text 21.5.

• Absolute scoring compresses differences: When seven models were rated on a 5-point scale across four dimensions, average ratings clustered between 4.01 and 4.35, making models hard to distinguish. PulseMind's own average was 4.35.

• Relative scoring is more human-aligned: Sampling 10% of evaluation outputs and asking 50 medical experts to verify judgments, the relative scoring strategy reached an average consistency of 86.1% with expert assessments, versus 51.5% for absolute scoring.

• MediScope drives large gains on the consultation benchmark: Ablations with PulseMind-72B show the average win rate rising from 26.4% (public data only) to 65.2% (public plus MediScope). Public medical QA benchmarks improved modestly, e.g., MedXpertQA from 34.9% to 36.5%.

• Reinforcement learning adds further gains: Adding RL to SFT raised the PulseMind Benchmark average win rate from 65.2% to 76.0%, with marginal gains on medical QA benchmarks.

• CRPO outperforms GRPO: CRPO raised the PulseMind Benchmark average win rate from 54.7% (GRPO) to 76.0%, and improved MMMU Health & Medicine by 2.7%.

Methodology in Plain English

The researchers first assembled data. They collected real clinical material, then ran automated checks using Optical Character Recognition and Named Entity Recognition to strip out personally identifying information from both text and images. They used large language models to clean up and expand physician responses, and had medical experts and licensed physicians proofread the results for clinical validity, ethics, and empathy. This produced MediScope, which they combined with public text-only and multi-modal datasets for roughly 792,000 training samples in total.

For evaluation, they built a benchmark mixing multi-modal and text-only multi-turn consultations, and had GPT-4 act as an automatic judge. Each model response is compared against baselines along four dimensions — proactiveness, accuracy, usefulness, and language quality — and the outcome is recorded as win, tie, or loss, with win rate as the headline metric.

Training happened in two phases. First, supervised fine-tuning: the model was trained on Huatuo26M to inject medical knowledge, then on MediScope plus public datasets to unlock multi-modal and multi-turn dialogue ability. Second, reinforcement learning with CRPO. The insight behind CRPO is that people find it easier to say which of two answers is better than to assign an absolute quality score to one answer. So instead of scoring a single response, the method generates several candidate responses to a query and compares each against responses from five counterpart models across the four dimensions, awarding a 1 if the candidate is judged better and 0 otherwise, then averaging those binary scores into a reward. Training used LoRA (rank-64 adaptation matrices) on top of frozen Qwen2.5-VL-72B and Qwen2.5-VL-32B base models, on 128 NVIDIA A100 GPUs with HuggingFace Transformers, PEFT, DeepSpeed ZeRO-3, BF16 mixed precision, the AdamW optimizer, a cosine annealing learning rate schedule, and a dropout rate of 0.1.

Why This Matters

Impact on research. The paper argues that existing medical multi-modal work is fragmented around single specialties and single-image question answering, and that existing benchmarks fail to capture the complexity of real clinical workflows. It contributes a dataset, a benchmark, and an RL method in one package, and its CRPO experiments offer evidence that relative preference scoring is more consistent with expert judgment than absolute scoring for open-ended clinical dialogue — a design lesson that extends beyond medicine.

Real-world applications.

• Clinical consultation support where a system must ask follow-up questions to gather missing but critical information before suggesting a direction.
• Handling of heterogeneous patient records that mix lab results, examination reports, prescriptions, images, and surgical records in one conversation.
• Multi-turn triage and intake, where the model must maintain context across a long exchange rather than answer isolated questions.
• Imaging-adjacent diagnostic assistance across modality types including Ultrasound, CT/MRI, Pathology, Endoscopy, and Medical Records.

Industry relevance. The work is a collaboration involving Ant Group alongside Dalian University of Technology, Peking University, University of Hong Kong, and City University of Hong Kong, and was supported by the Ant Group Research Intern Program and the National Natural Science Foundation of China under Grant 62431004. Code is released at https://github.com/AQ-MedAI/PulseMind. The paper's authors frame PulseMind as a foundation for practical diagnostic dialogue applications.

Future Directions

• Extending to high-dimensional data. The authors identify limited ability to process specialized formats such as 3D medical imaging and other high-dimensional clinical modalities as a current limitation.
• Reducing compute cost. The authors note that training demands substantial computational resources and considerable time, which may constrain use in resource-limited environments.
• Scaling and validating the benchmark. The PulseMind Benchmark totals over 1,200 samples, with the multi-modal MedDiagnose subset at 237 samples; broader clinical coverage would be a natural extension.
• Testing whether relative rewards generalize. CRPO's advantage is demonstrated here on diagnostic dialogue; whether comparison-based rewards transfer to other open-ended clinical tasks remains an open question that the ablations do not settle.

Target Audience

Researchers and engineers working on medical vision-language models and clinical NLP, particularly those interested in multi-turn dialogue, evaluation benchmark design, and reinforcement learning from preference-style signals. It is also relevant to clinical informatics teams evaluating whether multi-modal models can support realistic consultation workflows, and to readers who follow how general-purpose VLMs are being adapted into domain-specific medical systems.

Authors’ abstract

Recent advances in medical multi-modal models focus on specialized image analysis like dermatology, pathology, or radiology. However, they do not fully capture the complexity of real-world clinical diagnostics, which involve heterogeneous inputs and require ongoing contextual understanding during patient-physician interactions. To bridge this gap, we introduce PulseMind, a new family of multi-modal diagnostic models that integrates a systematically curated dataset, a comprehensive evaluation benchmark, and a tailored training framework. Specifically, we first construct a diagnostic dataset, MediScope, which comprises 98,000 real-world multi-turn consultations and 601,500 medical images, spanning over 10 major clinical departments and more than 200 sub-specialties. Then, to better reflect the requirements of real-world clinical diagnosis, we develop the PulseMind Benchmark, a multi-turn diagnostic consultation benchmark with a four-dimensional evaluation protocol comprising proactiveness, accuracy, usefulness, and language quality. Finally, we design a training framework tailored for multi-modal clinical diagnostics, centered around a core component named Comparison-based Reinforcement Policy Optimization (CRPO). Compared to absolute score rewards, CRPO uses relative preference signals from multi-dimensional com-parisons to provide stable and human-aligned training guidance. Extensive experiments demonstrate that PulseMind achieves competitive performance on both the diagnostic consultation benchmark and public medical benchmarks.