PatientVLM Meets DocVLM: Pre-Consultation Dialogue Between Vision-Language Models for Efficient Diagnosis

AI summary

Overview

Research area: Medical computer vision and vision-language models (VLMs), specifically dialogue-driven clinical diagnosis. Technical level: Intermediate — the paper assumes familiarity with vision-language models, supervised fine-tuning, and medical imaging benchmarks, but the core idea is described in accessible terms. Scope: The paper introduces a framework that simulates multi-turn doctor-patient conversations between two VLMs to generate synthetic dialogue training data, then fine-tunes a diagnostic model on those conversations to improve medical image classification.

What This Paper Is About

Most AI diagnostic systems map a medical image directly to a diagnosis, ignoring the back-and-forth questioning that real doctors use to elicit symptoms, history, and context. Collecting real doctor-patient conversations at scale is impractical because it requires ethics approvals, consent from hospitals and patients, and raises privacy and medico-legal concerns. The authors propose the Pre-Consultation Dialogue Framework (PCDF), which simulates these conversations synthetically using two cooperating vision-language models in distinct roles, then uses the resulting image-dialogue-diagnosis triplets to train a better diagnostic model.

Key Contributions

1. PCDF framework: A model-agnostic training paradigm that pairs a DocVLM (which asks clinically relevant follow-up questions based on the image and dialogue history) with a PatientVLM (which answers using a symptom profile derived from the ground-truth diagnosis), producing realistic image-dialogue-diagnosis triplets across T turns.
2. Dialogue-conditioned fine-tuning: The demonstration that synthetic triplets can equip VLMs with dialogue-aware diagnostic capability, enabling contextual symptom reasoning without any real clinical transcripts.
3. Broad empirical evaluation: Consistent accuracy and F1 gains across four medical imaging benchmarks (DermaMNIST, PneumoniaMNIST, RetinaMNIST, PathMNIST) and multiple VLMs, including generic models (InternVL3-2B, Qwen2.5-VL-7B, Gemma3-4B) and a domain-adapted model (MedGemma3-4B).
4. Clinical validation of synthetic dialogues: A small-scale expert review of 210 randomly selected cases (1,680 question-answer pairs) assessing clinical relevance, symptom coverage, and dialogue realism, plus a GPT-5-based evaluation at larger scale.

Main Findings

• Consistent gains over image-only training: PCDF-enhanced VLMs achieved an average F1 improvement of 11.48 over image-only fine-tuned VLMs across the four benchmarks.
• Largest gains for generic VLMs: PCDF-enhanced InternVL3-2B recorded the highest absolute F1 gains of 37.2 (DermaMNIST), 23.4 (RetinaMNIST), and 14.6 (PathMNIST). PCDF-enhanced InternVL3-2B reached 89.6 accuracy / 73.7 F1 on DermaMNIST, 98.7 / 98.6 on PneumoniaMNIST, 72.2 / 54.9 on RetinaMNIST, and 95.7 / 85.5 on PathMNIST.
• Domain-adapted models also improve: MedGemma3-4B improved F1 from 71.2 to 81.3 on RetinaMNIST, and reached 94.4 accuracy / 86.4 F1 on DermaMNIST and 99.4 / 99.3 on PneumoniaMNIST.
• PCDF outperforms Chain-of-Thought prompting: In zero-shot mode without fine-tuning, PCDF dialogues outperformed CoT prompting in the majority of evaluated scenarios, with particularly significant F1 improvements for MedGemma3-4B. MedGemma achieved an average F1 gain of 23.6 and Qwen2.5-VL-7B an average F1 gain of 19.7 relative to the comparison settings reported in Table 2.
• Longer dialogues help: Extending dialogue length from 2 to 8 turns improved F1 by +18.4% on DermaMNIST, +20.2% on PneumoniaMNIST, +39.9% on RetinaMNIST, and +31.1% on PathMNIST (Gemma3 as DocVLM, mPLUG-Owl3 as PatientVLM).
• PatientVLM choice matters but all variants help: With Qwen2.5-VL-7B as DocVLM, mPLUG-Owl3 as PatientVLM achieved the highest average F1 of 73.3, and every PatientVLM variant substantially outperformed the image-only SFT baseline of 61.8 F1.
• Expert-rated dialogue quality: Of 1,680 expert-reviewed exchanges, 1,628 (96.9%) were rated clinically relevant and 52 (3.1%) not useful; average symptom coverage was 4.5 and dialogue realism 3.9. No diagnosis leakage was reported by experts.
• GPT-5 evaluation matched the trend: 1,589 exchanges (94.6%) rated clinically relevant, 91 (5.4%) not useful, with average symptom coverage 4.1 and dialogue realism 4.7.

Methodology in Plain English

The framework runs in two stages. In the dialogue simulation phase, the DocVLM receives a medical image, the dialogue history so far, and the list of all possible diagnoses, and is prompted to ask exactly one follow-up question about symptoms (location, severity, duration, changes over time). The PatientVLM then receives the image, the DocVLM's question, and the ground-truth diagnosis, and is prompted to answer in the first person as a patient in a single sentence of at most 15 words — without ever mentioning the diagnosis. This constraint preserves the information asymmetry of a real consultation. The loop repeats for up to T turns, with both models frozen throughout, producing an image-dialogue-diagnosis triplet for each training example.

In the fine-tuning phase, the DocVLM is instruction-tuned on these triplets: it is given the image and the full dialogue history and must generate the correct diagnosis, framing classification as text generation and optimizing a standard generation loss. The authors used mPLUG-Owl3 as the PatientVLM for key results, capped dialogue exchange at T = 8, and fine-tuned the DocVLM with LoRA for 10 epochs (batch size 8, LoRA rank 16, alpha 32, dropout 0.05) on three A6000 GPUs (48 GB each), using PyTorch and the Hugging Face Transformers library.

Baselines included CNNs (ResNet50, DenseNet201 fine-tuned end-to-end for 100 epochs, batch size 128, learning rate 1e-4) and CLIP-family models (CLIP, MedCLIP, PMC-CLIP, BioMedCLIP) evaluated zero-shot and with image-only supervised fine-tuning. The VLM baselines were evaluated zero-shot, with image-only SFT, and with PCDF.

Why This Matters

The work reframes medical image classification as a dialogue problem rather than a pure mapping problem, and shows that realistic symptom elicitation can be synthesized without touching sensitive real clinical conversations.

Real-world applications:

• Clinical decision support: A diagnostic assistant that asks the same follow-up questions a physician would before committing to a diagnosis.
• Triage and telehealth: Pre-consultation intake tools that gather symptom detail before a clinician sees the patient.
• Training data generation: Creating scarce image-dialogue-diagnosis resources for specialties where conversational data is unavailable.
• Benchmark and evaluation design: Providing a dialogue-conditioned evaluation setting for medical VLMs beyond image-only classification.

Industry relevance: the framework is model-agnostic and improves both generic and medical-domain VLMs, making it relevant to developers of medical AI products who already use open-weight models such as InternVL3, Qwen2.5-VL, Gemma3, and MedGemma. The authors state the system is intended as a diagnostic aid, not a replacement for professional medical judgment, and that no real patient-identifiable data were used.

Future Directions

• Expand clinical validation beyond the small-scale dermatology review to larger, more diverse patient populations, as the authors note their verification was limited by budget and availability of medical professionals.
• Refine question generation so follow-up questions are not overly technical and are understandable to layperson patients.
• Extend the framework beyond English to support multilingual healthcare settings.
• Conduct large-scale, rigorous clinical evaluations and trials by deploying and validating the model in real-world healthcare settings.

Target Audience

Researchers and practitioners working on medical vision-language models, clinical decision support, and multimodal dialogue systems, as well as clinicians interested in how synthetic patient simulation could support diagnostic AI. The paper is also useful for teams building training data pipelines in data-scarce clinical domains, and for those evaluating medical VLMs on benchmarks such as MedMNIST v2.

Authors’ abstract

Traditionally, AI research in medical diagnosis has largely centered on image analysis. While this has led to notable advancements, the absence of patient-reported symptoms continues to hinder diagnostic accuracy. To address this, we propose a Pre-Consultation Dialogue Framework (PCDF) that mimics real-world diagnostic procedures, where doctors iteratively query patients before reaching a conclusion. Specifically, we simulate diagnostic dialogues between two vision-language models (VLMs): a DocVLM, which generates follow-up questions based on the image and dialogue history, and a PatientVLM, which responds using a symptom profile derived from the ground-truth diagnosis. We additionally conducted a small-scale clinical validation of the synthetic symptoms generated by our framework, with licensed clinicians confirming their clinical relevance, symptom coverage, and overall realism. These findings indicate that the resulting DocVLM-PatientVLM interactions form coherent, multi-turn consultations paired with images and diagnoses, which we then use to fine-tune the DocVLM. This dialogue-based supervision leads to substantial gains over image-only training, highlighting the value of realistic symptom elicitation for diagnosis.