Shallow Robustness, Deep Vulnerabilities: Multi-Turn Evaluation of Medical LLMs

AI summary

Overview

Research area: Medical question answering with large language models; robustness and safety evaluation.

Technical level: Intermediate. The paper is accessible to readers familiar with LLM benchmarking, but the multi-turn taxonomy and compounding analysis require careful reading.

Scope: The paper introduces MedQA-Followup, a framework and dataset for measuring whether medical LLMs hold onto correct diagnoses when their initial answers are challenged across multiple conversation turns.

What This Paper Is About

Medical LLMs are being deployed clinically, yet almost all evaluation is single-turn: a model answers a USMLE-style question once, and that answer is graded. Real consultations are not like that. A clinician may push back, a patient may bring misleading internet research, or a retrieved document may quietly favor the wrong diagnosis. This paper asks whether models keep their correct answers when challenged — and finds they mostly do not, especially when the challenge arrives as indirect context rather than an explicit suggestion of a wrong answer.

Key Contributions

1. A robustness taxonomy for medical Q&A. The authors organize prior and new work along two axes: temporal scope (single-turn/shallow vs. multi-turn/deep) and intervention intent (indirect framing vs. direct suggestion). The taxonomy cleanly separates neutral re-evaluation, plausible wrong options, context manipulation, and explicit wrong suggestions.

2. MedQA-Followup, a multi-turn evaluation dataset. Built on 1,273 MedQA (USMLE) questions, with intervention techniques spanning four categories and 4–5 templates each, enabling controlled, reproducible follow-up and compounding evaluations.

3. Empirical evidence of a shallow–deep robustness gap. Five state-of-the-art models are benchmarked under single-turn, follow-up, and compounding interventions, revealing that models resist shallow perturbations but collapse under multi-turn context manipulation.

4. The counterintuitive finding that indirect beats direct. Context-based interventions that never assert a wrong answer is correct degrade accuracy more than explicit appeals to authority — a result with direct implications for how clinical systems should be probed and defended.

Main Findings

• Shallow robustness largely holds. Single-turn direct suggestions (BiasMedQA-style) reduce accuracy by only ~5.5% on average. GPT-4.1 and GPT-4.1 mini stay within ~1% relative decline. Plausible-wrong-option distractors (KGGD) cap out at 13.6% relative drop.

• Deep robustness fails badly. Claude Sonnet 4 falls from 91.2% baseline to 13.5% under RAG-style follow-up context — an 85.2% relative collapse. Before that, it was the strongest baseline model, making it the most vulnerable under intervention.

• Indirect context is more dangerous than direct suggestion. Across all five models, the context category produces over 30% average relative accuracy drops (average 39.6%), while explicit wrong-answer suggestions produce far less. A misleading context framing is harder to reject than an authority claim, because it never announces itself as an attempt to change the answer.

• Model families fail differently. GPT models resist explicit suggestions (only 3.5% average decline) but are brittle to added context. MedGemma models show the opposite pattern: more sensitive to shallow and direct bias, comparatively sturdier under contextual framing.

• Clinical reasoning questions are more fragile than basic science. Step 2&3 (clinical application) questions degrade 6–13.5% more than Step 1 (basic science) under context interventions. Patient scenarios appear more susceptible to misleading framing than fact recall.

• Context length matters unevenly. For GPT-4.1 and MedGemma 27B, longer misleading passages amplify degradation. Claude Sonnet 4 shows diminishing returns with length and actually recovers accuracy on longer edge-case and alternative contexts, apparently discounting them as irrelevant.

• Compounding has limits. Across sequenced multi-turn interventions, 85% of combinations were sub-additive — the damage was less than a worst-case sum of the individual effects. MedGemma 4B consistently recovered accuracy under compounding, though no consistent pattern held for other models.

• Neutral re-evaluation is a valid control. Prompts asking the model to rethink without bias produced near-zero deviations, confirming that extra turns alone do not harm accuracy — the damage comes from the content of the intervention.

Methodology in Plain English

The researchers took the MedQA dataset of USMLE multiple-choice questions and built three families of follow-up interventions on top of it.

First, they establish a baseline by asking each model the question and recording its answer. Then, in a follow-up turn, they inject an intervention: either a neutral prompt asking the model to re-evaluate (the control), an explicit suggestion that some authority or colleague believes a different answer is correct, or a generated block of clinical-sounding context that subtly supports an incorrect option. The generated contexts are created by GPT-4.1 under strict constraints — they must be medically accurate, must not contradict the correct answer, and must supply plausible clinical reasoning — so the intervention is a framing effect rather than an outright falsehood.

For the context category, four variants are used: "misleading context" presented as clinical evidence, "RAG style context" framed as retrieved knowledge-base material, "alternative context" supporting a diagnosis not even in the option set, and "edge case context" highlighting atypical presentations. A separate ablation regenerates contexts at fixed lengths of 1, 10, and 20 sentences to see whether verbosity amplifies the effect.

Compounding experiments add further turns on top: an authority or social-proof suggestion first, then a context manipulation, and in some setups a third intervention. The authors compare the observed combined drop against an expected additive drop to classify each combination as sub-additive or super-additive.

All runs use deterministic decoding (temperature 0, fixed seed 42). Five models are tested — GPT-4.1, GPT-4.1 mini, Claude Sonnet 4, MedGemma 27B, and MedGemma 4B — spanning general-purpose vs. domain-specialist and large vs. small within families.

Why This Matters

For research: The paper reframes medical LLM evaluation around a distinction the field has largely ignored — resisting a bad prompt is not the same as retaining a correct conclusion under conversational pressure. It also supplies a reusable taxonomy and dataset that future robustness work can build on, and its counterintuitive indirect-beats-direct result challenges the assumption that explicit adversarial prompting is the hardest test.

Real-world applications:

• Clinical decision support deployment. Systems that surface a differential diagnosis and then accept follow-up input from clinicians need to be evaluated under the exact conditions where they fail — successive turns of plausible but misleading framing.
• Retrieval-augmented medical tools. The RAG-style context intervention produced the single worst result in the study (Claude Sonnet 4 at 13.5%). Any clinical RAG pipeline must test whether retrieved documents can silently override a correct diagnosis.
• Second-opinion and referral workflows. The authority and social-proof interventions model real consultations where a senior clinician disagrees. Models that flip on authority alone are unsafe for settings with hierarchy.
• Evaluating domain-specialized models. MedGemma's opposite vulnerability profile (weak to direct bias, sturdier under context) means a single benchmark score does not transfer to deployment risk.

Industry relevance: Any organization shipping medical AI — hospital systems, EHR vendors, health-focused model providers, and regulators drafting evaluation standards — gains a concrete argument that accuracy benchmarks on single-turn exam questions are insufficient evidence of clinical safety. The paper's recommendation to expose raw retrieved evidence rather than model-interpreted summaries, and to flag large answer shifts for human review, is directly actionable for product and compliance teams.

Future Directions

1. Adversarial training on multi-turn dialogues. The authors explicitly call for training regimes that expose models to sequential misleading context rather than isolated adversarial prompts, since the failure mode is conversational rather than lexical.

2. Confidence-weighted resistance. A model that correctly computes an answer and then abandons it under social pressure needs a mechanism to weight its own verified reasoning above external framing — a calibration problem distinct from raw accuracy.

3. Extending beyond multiple choice. The controlled four-option format is a lower bound on vulnerability. Open-ended clinical dialogue involves nuanced reasoning and interpretation that this framework cannot capture, and evaluating it is an open problem.

4. Understanding why indirect framing wins. The mechanism by which unflagged, medically accurate context overrides a correct answer is not explained. Whether it is attention to recency, perceived authority of retrieved text, or a general deference to additional information remains unresolved.

Target Audience

This paper is most useful to clinical AI safety researchers, evaluation engineers, and regulatory or compliance teams responsible for validating medical LLM deployments. It also serves model developers building healthcare-specific systems, and clinician-informaticists who need to understand where an AI assistant is likely to abandon a correct assessment. Readers looking for a concrete, reproducible benchmark rather than a purely conceptual safety argument will get the most out of it.

Authors’ abstract

Large language models (LLMs) are rapidly transitioning into medical clinical use, yet their reliability under realistic, multi-turn interactions remains poorly understood. Existing evaluation frameworks typically assess single-turn question answering under idealized conditions, overlooking the complexities of medical consultations where conflicting input, misleading context, and authority influence are common. We introduce MedQA-Followup, a framework for systematically evaluating multi-turn robustness in medical question answering. Our approach distinguishes between shallow robustness (resisting misleading initial context) and deep robustness (maintaining accuracy when answers are challenged across turns), while also introducing an indirect-direct axis that separates contextual framing (indirect) from explicit suggestion (direct). Using controlled interventions on the MedQA dataset, we evaluate five state-of-the-art LLMs and find that while models perform reasonably well under shallow perturbations, they exhibit severe vulnerabilities in multi-turn settings, with accuracy dropping from 91.2% to as low as 13.5% for Claude Sonnet 4. Counterintuitively, indirect, context-based interventions are often more harmful than direct suggestions, yielding larger accuracy drops across models and exposing a significant vulnerability for clinical deployment. Further compounding analyses reveal model differences, with some showing additional performance drops under repeated interventions while others partially recovering or even improving. These findings highlight multi-turn robustness as a critical but underexplored dimension for safe and reliable deployment of medical LLMs.