AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding

AI summary

Overview

Research area: Natural Language Processing / medical AI evaluation — specifically automated benchmarking of large language models (LLMs) acting as clinical conversational agents.

Technical level: Intermediate. The paper is readable without deep clinical or mathematical background, but understanding it requires familiarity with LLM agents, multi-agent simulation, and QA benchmarks. The CARE metric uses straightforward arithmetic rather than complex statistics.

Scope: The paper introduces AutoMedic, a fully automated multi-agent framework that converts existing static medical QA datasets into virtual patient simulations and scores LLM "doctor" agents with a four-part metric called CARE, validated against four licensed medical professionals.

What This Paper Is About

Existing medical benchmarks test LLMs with self-contained, single-turn questions, so they cannot measure how well a model actually interviews a patient, requests tests, or communicates with empathy. Formally evaluating interactive clinical dialogue is hard because the space of possible patient states and conversation trajectories is combinatorially vast. AutoMedic addresses this by automatically turning off-the-shelf medical QA datasets into virtual patient profiles and simulated clinical encounters, then scoring the LLM doctor agent with a quantitative, multi-faceted metric instead of human judgment or simple accuracy.

Key Contributions

1. Automatic conversion of static QA datasets into interactive virtual patient profiles. The framework requires no hand-crafted conversational dataset. A profile generator agent first filters out queries unsuitable for patient-specific simulation (for example, questions about research participants, abstract pathophysiology, general fact recall, image-dependent items, or cases where critical details appear only in the answer choices), then structures suitable items into demographics, basic information, and optional information.

2. A four-agent simulation of a realistic clinical encounter. A doctor agent (the only agent being evaluated) must actively question a patient agent and request results from a clinical staff agent before answering the final question from the original query, while a profile generator agent prepares the scenario.

3. The CARE metric, a novel multi-faceted quantitative evaluation standard. CARE stands for Conversation efficiency and strategy, Accuracy, Robustness, and Empathy, giving a four-dimension score from the conversation transcript and final answer.

4. Human expert validation of the framework and metric, using three separate studies with four licensed medical professionals (IRB No. 4-2025-0982).

Main Findings

• Filtering is reliable: Human experts showed substantial agreement on which queries were suitable for profile generation, with a Fleiss' Kappa of 0.7630, and agreement between the profile generator's classifications and the experts' majority vote reached a Cohen's Kappa of 0.8197 (excluding two tied cases).

• Generated patient profiles are clinically sound: Expert ratings of 21 sampled profiles showed a percent agreement of 0.9140 and a Gwet's AC2 of 0.8313, with an average quality score of 3.464 out of a possible 4 points.

• Datasets differ sharply in conversational suitability: MedBullets and MedQA had the highest proportion of appropriate samples (over 90%). MedXpertQA had a lower proportion (79.8%) but the largest absolute number of appropriate samples (1,955). MedMCQA, HEAD-QA, and MMLU-Pro showed significantly lower proportions, ranging from 10% to 15%.

• Informational richness varies by dataset: Demographics were generally well covered, except in MedMCQA, which provided this information for only about half of its appropriate samples. MedBullets, MedQA, and MedXpertQA consistently included basic clinical information such as symptom duration and past medical history for over 70% of their samples, while MedMCQA and HEAD-QA showed noticeable gaps and had the lowest inclusion of optional information such as vital signs or physical examination findings. MMLU-Pro, despite its low proportion of appropriate samples, provided patient information at a frequency similar to the more suitable datasets once a sample was deemed appropriate.

• Conversational accuracy drops for every model: Across all experiments, every evaluated model performed worse in the conversational setting than in static QA. The drop was particularly pronounced for DeepSeek-R1-70B and Med42-v2-70B. The authors conclude that a model's raw medical knowledge, as reflected in QA accuracy, does not directly translate into effective clinical conversational ability.

• The CARE metric aligns with expert judgment: On the comparison of Claude Sonnet 4 and Llama 3-70B, human expert scores and CARE scores moved in the same direction on all four dimensions. Accuracy had the highest expert agreement (Gwet's AC2 of 0.9460), where Claude Sonnet 4's average human score of 2.2333 exceeded Llama 3-70B's 1.6333, matching CARE values of 0.4699 versus 0.2866. On robustness, Llama 3-70B slightly exceeded Claude Sonnet 4 in average human score (2.7333 versus 2.6500), and CARE showed Llama 3-70B at a perfect 1.0000 versus Claude Sonnet 4 at 0.9333.

• Robustness differences between models were small: The portion of the results available reports that Llama 3-70B achieved the highest robustness score across the board, while noting that inter-model differences in robustness were generally minor. The provided text is truncated within this analysis, so the full rankings of all 11 models across the CARE dimensions are not available here.

• Qwen3-32B showed strong accuracy: Among open-source general models, the available text reports that Llama 3-70B and Qwen3-32B exhibited average performance across the overall CARE metric, with Qwen3-32B demonstrating strong accuracy, though the sentence is cut off mid-word in the provided content.

Methodology in Plain English

The framework has three stages, run by four LLM agents.

Stage 1 — Building the patient. A "profile generator" agent reads a standard medical multiple-choice question (context, question, options, correct answer) and decides whether it fits a patient-specific clinical scenario. If it does, the agent extracts demographics (age, sex), basic information (chief complaint, symptom duration, family and past medical history, lifestyle factors such as smoking and alcohol use), and optional information (vital signs, lab tests, physical exams, imaging). Missing basic information is filled in with plausible randomly generated values that do not change the correct answer. Optional information is never invented if absent, because many questions test whether the doctor knows which tests to order. Information is then split by role to mimic real knowledge boundaries: the doctor sees only demographics, the patient agent sees demographics plus basic information, and the clinical staff agent holds the optional test data.

Stage 2 — The conversation. The doctor agent drives the dialogue using tags: <patient> for questions to the patient and <clinical> for test requests to clinical staff. The conversation ends when the doctor issues </end> or hits a maximum of 20 turns, where one turn is a doctor query plus the corresponding response. Afterwards the doctor receives the original question and options (without the clinical vignette) and must answer using only what it gathered. For a baseline, the same model also answers the complete original query directly in a static QA setting.

Stage 3 — Scoring with CARE. Four scores are computed automatically.

• Accuracy (S_ACC): conversational accuracy multiplied by the ratio of conversational accuracy to static QA accuracy, so models are penalized for losing accuracy when moving from the ideal-information setting to dialogue.
• Conversational Efficiency & Strategy (S_CES): the reciprocal of the doctor's average words per turn, counted only for correctly answered cases, then averaged and multiplied by 100. This penalizes unnatural "checklist" turns that cram many questions together.
• Empathy (S_EMP): the patient agent rates the doctor's empathy on a 5-point scale after each conversation, and the ratings are averaged and normalized.
• Robustness (S_ROB): one minus the proportion of failed conversations, where failures are role-breaking (the doctor simulates other agents itself), abrupt termination (defined as 3 turns or fewer, often caused by missing tags or a misplaced </end>), or invalid answers not listed among the options.

Evaluation setup. Six QA datasets were used: MedBullets (308 questions, 5 options), MedQA (1,273 questions, 5 options), MedXpertQA (2,450 text-only questions, 10 options), MedMCQA (2,816 filtered single-choice questions from the validation split, 4 options), HEAD-QA (2,742 English test-split questions, 4 options), and MMLU-Pro (1,535 health and biology questions, 3 to 10 options). Eleven LLMs served as the doctor agent: open-source general models (Llama 3-70B, Qwen3-32B, DeepSeek-R1-70B, gpt-oss-120B), open-source biomedical models (Med42-v2-70B, OpenBioLLM-70B, HuatuoGPT-o1-72B), and proprietary general models (GPT-4o, Claude Sonnet 4). GPT-4o was used for the profile generator, patient, and clinical staff agents to keep the simulation stable.

Human validation. Four licensed medical professionals ran three studies: judging filtering suitability on 24 sampled queries (12 appropriate, 12 inappropriate), rating 21 generated patient profiles on a 4-point scale, and blindly rating 30 simulation results on a 3-point scale across the four CARE dimensions. The 30 results came from Llama 3-70B and Claude Sonnet 4 as doctor agents (15 each); the body text describes the source dataset as MedQA, while the note under Table 3 states MedBullets.

Why This Matters

Research impact. Most medical LLM evaluation remains static question answering. AutoMedic shows that a model's QA accuracy does not predict its conversational competence, since every evaluated model lost accuracy when it had to gather information through dialogue. It also offers a way to reuse the abundance of existing static QA datasets for interactive evaluation, avoiding the cost of building bespoke conversational benchmarks such as HealthBench, which required 5,000 physician-created conversations.

• Model selection for clinical deployment: Hospitals and health systems could use CARE scores to compare candidate models on accuracy, dialogue efficiency, empathy, and robustness rather than on benchmark accuracy alone.
• Pre-screening for safety and robustness: The robustness dimension catches failure modes such as role-breaking, premature conversation termination, and out-of-option answers, which are directly relevant to safety review before deployment.
• Reusing existing datasets: Institutions with their own medical QA collections can convert them into conversational test suites without commissioning new conversational data.
• Curating better benchmarks: The dataset suitability analysis identifies which existing sources are worth converting, helping benchmark builders target datasets with rich patient-level detail.

Industry relevance. Developers of both proprietary and open-source medical LLMs need automated, reproducible, multi-faceted evaluation that does not depend on expensive human raters or clinician actors. Because AutoMedic runs entirely without human intervention after setup, it can be integrated into model development pipelines as a regression test for conversational quality.

Future Directions

• Determining whether support agents must be state-of-the-art. The paper fixes GPT-4o as the profile generator, patient, and clinical staff agent to keep the simulation stable, and explicitly notes that other state-of-the-art LLMs could be used and their impact analyzed. How sensitive CARE results are to the choice of supporting agents remains an open question.

• Extending beyond diagnostic scenarios. The authors note that prior clinical conversational frameworks are often limited to diagnostic scenarios, and the same question applies to how far the CARE dimensions generalize to treatment planning, follow-up, and longitudinal care.

• Resolving the dataset attribution inconsistency. The main text describes the human CARE correlation study as using MedQA, while the Table 3 note says MedBullets; a follow-up should clarify which was used and whether the correlation holds across both.

• Completing the cross-model comparison. The provided content is truncated during the CARE performance analysis, so a full leaderboard of all 11 models across the four CARE dimensions, and which model types (general, biomedical, proprietary) tend to win on which dimension, is not yet available here.

Target Audience

This paper is most useful for researchers and engineers building or evaluating medical LLMs, especially those working on conversational agents, clinical NLP benchmarks, and multi-agent simulation. It also serves regulators and clinical informatics teams who need structured, reproducible evidence about how a model behaves in interactive patient encounters rather than on static exams. Benchmark designers will find the dataset suitability analysis directly actionable, and clinicians interested in how LLM communication skills are quantified will find the CARE rubric and its human validation useful. Readers need basic familiarity with LLMs and multiple-choice QA evaluation, but no specialist mathematics.

Authors’ abstract

Evaluating large language models (LLMs) has recently emerged as a critical issue for safe and trustworthy application of LLMs in the medical domain. Although a variety of static medical question-answering (QA) benchmarks have been proposed, many aspects remain underexplored, such as the effectiveness of LLMs in generating responses in dynamic, interactive clinical multi-turn conversation situations and the identification of multi-faceted evaluation strategies beyond simple accuracy. However, formally evaluating a dynamic, interactive clinical situation is hindered by its vast combinatorial space of possible patient states and interaction trajectories, making it difficult to standardize and quantitatively measure such scenarios. Here, we introduce AutoMedic, a multi-agent simulation framework that enables automated evaluation of LLMs as clinical conversational agents. AutoMedic transforms off-the-shelf static QA datasets into virtual patient profiles, enabling realistic and clinically grounded multi-turn clinical dialogues between LLM agents. The performance of various clinical conversational agents is then assessed based on our CARE metric, which provides a multi-faceted evaluation standard of clinical conversational accuracy, efficiency/strategy, empathy, and robustness. Our findings, validated by human experts, demonstrate the validity of AutoMedic as an automated evaluation framework for clinical conversational agents, offering practical guidelines for the effective development of LLMs in conversational medical applications.