A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform

AI summary

Overview

Research area: Human-Computer Interaction / medical education — specifically the evaluation of large language model (LLM)-based virtual standardized patients (VSPs) for OSCE-style clinical communication training.

Technical level: Intermediate. Readers should be comfortable with standard evaluation terminology (Likert scales, inter-rater reliability, mixed-effects models), but the paper's framing is methodological and clinical rather than deeply technical.

Scope (one sentence): A double-blind, fully crossed, seventeen-rater human evaluation of four LLMs against a human medical-education expert across three Korean OSCE cases and 70 tagged input prompts, paired with a deployed web platform that generates further training dialogues and a public dataset.

What This Paper Is About

Medical schools use the Objective Structured Clinical Examination (OSCE) to train students in history-taking and clinical communication with standardized patients (SPs), but human SP programs are expensive and hard to scale. LLM-based virtual standardized patients built by "persona injection" (a single system prompt casting the model in a patient role) are a cheap alternative, but prior evaluations typically test only one model, use small rater panels, or rely on LLM-as-a-judge scoring without an utterance-level human-expert reference. This paper's goal is to build a rigorous, stratified, multi-rater evaluation that can localize where each model fails, and to couple that evaluation to a live training platform.

Key Contributions

1. A fully crossed multi-rater evaluation against a human-expert reference. Four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, HyperCLOVA X) plus a human medical-education expert produced responses to the same 70 prompts across three Korean OSCE cases (hematuria, fever in pregnancy, chronic cough). Seventeen final-year medical-student raters generated 7,480 ratings under formal inter-rater reliability analysis.

2. A two-axis stratification of the question itself. Every student input prompt is tagged with an 11-category history-taking taxonomy grounded in AMEE Guide No. 81 and a three-level linguistic-form label (affirmative, negative, open-ended), yielding an 11 × 3 grid. This surfaces failure modes that aggregate Likert means hide, notably a Korean negative-form penalty on Fluency.

3. A deployed real-time VSP web platform on which students conduct free-form OSCE-style encounters, with the same taxonomy confirmed to apply to platform-collected dialogue (all eleven categories represented among 1,614 platform utterances).

4. A released consent-based dialogue corpus on Hugging Face under CC BY-NC-SA 4.0, with Croissant Core and Responsible-AI metadata, plus public platform code.

Main Findings

• The human expert led on all six encounter-level metrics. Reference means were 4.84 Comprehensibility, 4.49 Hallucination-free, 4.80 Coherence, 4.75 Consistency, 4.18 Engagement, and 4.61 Overall Satisfaction, each pooling 51 ratings (17 raters × 3 cases).

• Three of four LLMs stayed close but below the reference. Claude 3.5 Sonnet, GPT-4o, and HyperCLOVA X fell 0.08 to 0.47 Likert points below the human reference on Comprehensibility, Consistency, and Hallucination-freeness.

• Only four of 24 source × metric contrasts were non-significant. These were Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X (−0.02 [−0.31, 0.27]).

• Llama-3.1 70B was significantly below the reference on all six metrics. Its largest gap was on Coherence (−1.86 [−2.14, −1.59]), and Overall Satisfaction showed an additional 1.5–2 point spread below the reference. Its mean scores were the lowest on every encounter metric (e.g., Coherence 2.94, Engagement 2.94, Overall Satisfaction 2.71).

• Sentence-level performance was higher and more compressed than encounter-level. Pooled Accuracy was within 0.09 Likert points of the human reference across all four LLMs. Fluency separated sources most clearly: human 4.91, HyperCLOVA X 4.68, GPT-4o 4.39, Claude 3.5 Sonnet 4.31, Llama-3.1 70B 3.69.

• Fluency ordering anticipated Coherence ordering, suggesting local linguistic naturalness propagates into global encounter quality.

• Stratification exposed category-specific weaknesses. LLMs were notably weak on Clinical reasoning and Patient education — categories requiring synthesis across prior dialogue. Llama-3.1 70B's deficit was largest on Greeting & ID check and Clinical reasoning, exceeding 1 Likert point below the human reference on Fluency.

• A Korean negative-form penalty on Fluency appeared, and it did not track Korean proficiency. The extra drop from affirmative to negative-polarity questions was 0.149 [−0.286, −0.011] for Llama-3.1 70B, 0.124 [−0.261, 0.014] for GPT-4o, 0.093 [−0.231, 0.044] for HyperCLOVA X, and 0.033 [−0.170, 0.105] for Claude 3.5 Sonnet; the human reference showed +0.043 [−0.067, 0.153]. Only the Llama interval excludes zero. The penalty was smallest for Claude 3.5 Sonnet and larger for HyperCLOVA X, the only Korean-developed model, indicating the effect is one of question form rather than language proficiency.

• Rare severe failures were concentrated and model-specific. Sixteen of 350 utterances fell below 3.0 on at least one sentence-level metric, and fourteen of these were Fluency failures. In seven utterances (all Clinical reasoning or Patient education) the virtual patient restated the student's explanation in clinical register instead of answering as a patient; in three (all negative-polarity questions) it answered a question that had not been asked. No human-reference utterance fell below 3.0 on any metric.

• Reliability was high by ICC but low by Krippendorff's alpha. Encounter-level pooled ICC(2,k) ranged from 0.79 (Hallucination-free) to 0.94 (Coherence), with a fully pooled ICC(2,k) of 0.91 [0.88–0.94]. Sentence-level pooled ICC(2,k) per source ranged from 0.80 to 0.92. Krippendorff's alpha with ordinal weights was lower (0.08–0.41), though both measures produced the same source-level ordering.

• The platform pilot produced a substantial corpus. The 17 raters generated 49 sessions across 27 distinct clinical scenarios between August 2025 and May 2026, totaling approximately 1,763 student–VSP turns.

Methodology in Plain English

The team wrote three OSCE scenarios (hematuria, fever in pregnancy, chronic cough) from a Korean OSCE/CPX training guide, refined by a board-certified internal medicine faculty member with thirteen years of clinical-skills assessment experience. Each scenario was turned into a single persona-injection system prompt of roughly 700–900 Korean characters with six fields (role declaration, demographic context, clinical state, history, conversational style, behavioral constraints). The same template was used for all three cases; only case-specific values changed.

They assembled 70 realistic student-doctor questions (23 hematuria, 23 fever, 24 cough) sampled from authentic OSCE training transcripts. Each of five sources — the four LLMs and the human expert — answered every question, giving 5 × 70 = 350 responses. The expert wrote blind to model outputs and at the level expected of a trained standardized patient, serving as a 1:1 utterance-aligned gold standard. Inference settings were held constant (temperature 0.7, top-p 0.95, maximum 1,024 tokens, one response per query, no retries or chain-of-thought), and all four models were queried within a single calendar week to limit drift.

Seventeen senior medical students (mean age 24.8 ± 1.9 years; 5 women, 29.4%) rated the corpus after a 60-minute calibration session using anchor examples. At the sentence level they scored Fluency, Accuracy, and Contextual Relevance on five-point Likert scales; at the encounter level they scored Comprehensibility, Hallucination-free, Coherence, Consistency, Engagement, and Overall Satisfaction. Responses were randomized within case and de-identified so raters did not know the source. This produced 5,950 sentence-level rating events (17 × 350) and 1,530 encounter-metric assessments (17 × 15 encounters × 6 metrics), totaling 7,480 rating actions.

For analysis, each of the 70 prompts was tagged by a rule-based tagger on two axes (with ambiguous cases resolved by two faculty members by consensus). Scores were analyzed with a linear mixed-effects model including Source, History-taking category, and Linguistic form as fixed effects with Source × category and Source × form interactions, rater as a random intercept, and utterance as a random intercept nested within case, estimated by REML. Descriptive means carry 95% nonparametric percentile bootstrap confidence intervals from 2,000 resamples. Analyses ran in R 4.5.2 and Python 3.11.

Separately, they built and deployed a web platform. It uses GPT-4.1 offline to generate scenario scripts (temperature 0.2) and GPT-4o for real-time patient role-play and automated checklist-based scoring (temperature 0.5, max 35 tokens for the patient; temperature 0 for scoring), all with seed = 42. Scenarios derive from a structured database of 80 core symptoms, each linked to expert-validated disease lists and three patient profiles per symptom (240 profiles total), organized as hierarchical JSON. Each generated narrative (≥1,500 characters) is retained only if a downstream diagnostic model predicts the intended disease. Sessions are anonymized and stored as structured records.

Why This Matters

Impact on research. Most published LLM-VSP evaluations use a single model, a small rater panel, or an LLM judge without an utterance-aligned expert reference. This paper demonstrates that a fully crossed multi-rater design with two-axis question stratification can reveal failure modes invisible to aggregate scores — the negative-polarity Fluency penalty being the clearest example, where the ordering does not follow Korean-language ability. It also shows that high ICC can coexist with low Krippendorff's alpha, a methodological caution for anyone reporting reliability in this space. The 7,480 fully crossed scores are positioned as a potential validation set for LLM-as-a-judge pipelines.

Real-world applications:

• Low-cost formative OSCE practice for medical students where human SP programs are unaffordable or unavailable, with automated checklist feedback at session close.
• Scenario authoring at scale from a symptom–disease–profile database, enabling practice across many more cases than a typical OSCE circuit.
• Curriculum quality assurance: identifying which question categories (Clinical reasoning, Patient education) a given model handles poorly before deploying it for training.
• Localization QA for clinical dialogue systems in non-English languages, where question polarity semantics differ from English.

Industry relevance. For vendors building clinical simulation, education, or triage dialogue products, the finding that within-utterance Accuracy is nearly saturated while encounter-level Coherence, Engagement, and Overall Satisfaction lag substantially is directly actionable: the bottleneck is multi-turn behavior, not single-response factuality. The fact that a Korean-developed model did not show the smallest Korean negative-form penalty is a caution against assuming domestic-language models are automatically better localized.

Future Directions

• Broaden the human reference pool to multiple OSCE educators across specialties and trained standardized patients, since the current reference was authored by one faculty member and any non-significant result is specific to that expert.
• Expand the model and rater panels. The burden of fully crossed human rating constrained the study to 17 raters and four LLMs; several open-source frontier models released after the rating window were excluded.
• Validate LLM-as-a-judge against the released human scores to reduce rater burden in future evaluations.
• Extend the simulator beyond text to speech, video, or VR, since the current system cannot reproduce prosody, facial expression, or gesture, and encounter-level Engagement and Comprehensibility may behave differently in spoken settings.
• Run a controlled comparative study of learning outcomes. The paper explicitly states that establishing the educational effect of VSP-based OSCE practice requires this, and it has not been done here.

Target Audience

Medical-education researchers and OSCE assessment designers; clinical NLP and clinical-dialogue researchers who need human-evaluation methodology for subjective, multi-turn outputs; medical AI developers choosing a backbone model for a patient simulator or judging whether single-utterance accuracy is a sufficient quality signal; and dataset builders interested in the consent, anonymization, and Croissant/Responsible-AI metadata workflow used for the platform corpus. The paper is also useful to statisticians and evaluation researchers interested in how stratification and mixed-effects modeling can localize model failures, and in the divergence between ICC and Krippendorff's alpha.

Authors’ abstract

The Objective Structured Clinical Examination (OSCE) is widely used to assess clinical communication skills in medical education, requiring trainees to take histories and reason clinically with standardized patients (SPs). Large language model (LLM)-based virtual standardized patients (VSPs), typically built by persona-injected prompting, offer a practical alternative. However, existing evaluations rely on a single backbone LLM, small human rater panels, or LLM-as-a-judge scoring without an utterance-aligned human-expert reference, and so cannot identify the specific shortcomings of individual models. We evaluated four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X), spanning commercial, open-source and Korean-developed models, against a human-expert reference across three Korean OSCE cases (hematuria, fever in pregnancy, and chronic cough). In a double-blind, fully crossed design, 17 senior medical-student raters scored every model utterance on three sentence-level and six encounter-level Likert metrics, yielding 7,480 data points. We analyzed the scores using a stratified linear mixed-effects model based on an 11-category history-taking and 3-level linguistic-form taxonomy. At the encounter level, three of the four LLMs fell 0.08 to 0.47 Likert points below the human-expert reference on Comprehensibility, Consistency and Hallucination-freeness. The difference was not statistically significant for Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X. Llama-3.1 70B fell significantly below the reference on all six metrics. We also introduce a web-based OSCE platform that supports virtual practice across diverse clinical cases and continuously collects student-VSP dialogues under research and privacy consent.