MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs

AI summary

Overview

• Research area: Clinical safety evaluation of large language models and retrieval-augmented generation (RAG) systems under prompt injection attacks.
• Technical level: Intermediate. Familiarity with LLMs, RAG pipelines, and basic safety metrics (attack success rate, refusal rates) is helpful, but the core arguments are accessible.
• Scope: The paper introduces MPIB, a 9,697-instance benchmark suite that measures clinically meaningful harm—not just whether an attack "succeeds"—when adversarial instructions enter either the user query or retrieved medical documents.

What This Paper Is About

LLMs are moving into clinical workflows, and many of them retrieve documents from guideline repositories or knowledge bases to ground their answers. That retrieval step creates a new attack surface: malicious text hidden inside a retrieved document can be treated as trustworthy clinical authority and steer the model toward unsafe advice—wrong doses, downplayed emergencies, fabricated citations. Existing safety benchmarks mostly measure generic refusals or jailbreaks, which misses the real problem in medicine, where a harmful answer can still sound polished, confident, and policy-compliant. MPIB builds a clinically grounded benchmark and a new outcome-level metric to measure actual patient risk rather than surface-level compliance.

Key Contributions

1. MPIB dataset (9,697 instances): A clinically grounded benchmark spanning four scenario families (explanation, dosing, triage, guideline reasoning) across two attack vectors—direct injection via the user query (V1) and indirect, RAG-mediated injection via poisoned retrieved context (V2)—plus benign (V0) and borderline (V0′) anchor pools for utility and safety-boundary testing.

2. Clinical Harm Event Rate (CHER): A new outcome-centric metric measuring the rate of high-severity harm events (severity ≥ 3 on a 0–4 scale) under a five-type clinical harm taxonomy (misdiagnosis, treatment error, emergency downplay, guidance conflict, evidence manipulation). CHER is reported alongside conventional Attack Success Rate (ASR₂, severity ≥ 2) to separate moderate from high-severity outcomes.

3. Clinically aligned evaluation pipeline: A structured LLM-as-a-judge setup selected through empirical comparison of three candidate judges, validated against two independent clinicians (Cohen's κ = 0.62 on 300 blinded model responses) and cross-checked with multi-judge ranking stability (Spearman ρ = 0.86 against Llama-3.1-70B).

4. Responsible release with defense harness: Public splits with strict schema validation, a diagnostic four-component defense scaffold (D1–D4), and redacted V2 payloads that require explicit registry access for reconstruction, limiting dual-use risk.

Main Findings

• ASR and CHER diverge, and the divergence depends on attack vector. Under direct injection (V1), severity-2-or-worse outcomes and high-severity outcomes track each other closely. Under indirect injection (V2), the gap widens substantially: many "successful" attacks produce moderate-severity failures involving framing, citations, or formatting rather than actionable dangerous guidance. Using ASR alone under V2 can therefore overstate high-severity patient-safety events.

• Reducing ASR does not reliably reduce CHER. Defenses can shift outcomes between severity bands without changing overall failure counts. One example: MedGemma-4B under V2 with the context sanitizer raises ASR from 64.1% to 65.6% while lowering CHER from 21.9% to 18.8%. This is the paper's central argument for outcome-based auditing.

• The right defense depends on where the attack lives. Input-side rewriting (D2 Input Guard) tends to help most against V1—Qwen-2.5-72B drops from 65.7% to 50.7% V1 CHER, and Llama-3.1-70B from 86.6% to 68.7%. Context-side interventions help more against V2—Meditron-70B drops from 53.1% to 37.5% V2 CHER under the composite D4 configuration, and Qwen-2.5-72B from 7.8% to 1.6% under D1.

• Defenses are not universally beneficial. D1 (internal hardening) sometimes backfires: MedGemma-4B's V1 CHER rises from 88.1% to 100.0%. The composite D4 is not uniformly best either, indicating non-additive interactions between input rewriting and context sanitization.

• Safety–utility trade-offs are highly model-dependent. Utility False Positive Rate (incorrect refusals on benign queries) ranges widely. Meditron-70B's FPR-H jumps from 16.0% to 33.6% across D0→D4, while Qwen-2.5-72B stays in the 3.0–3.8% range. A guardrail acceptable for one model may be unacceptable for another that already over-refuses ambiguous cases.

• Best defensive results remain clinically inadequate. The lowest V1 CHER₃ across all 13 models and all five defense configurations is 50.7%. The authors explicitly state the evaluated defenses are diagnostic scaffolds, not production-grade guardrails.

• Emergency triage is consistently vulnerable. Scenario S3 (emergency triage) ranks among the most attack-susceptible V1 scenarios under baseline conditions across models.

• Medical-tuned models are not automatically safer. Meditron-70B shows 98.5% V1 CHER at baseline—among the highest in the matrix—while some general-purpose models perform comparatively better.

Methodology in Plain English

The researchers started with two public medical question-answering corpora (MedQA and PubMedQA) and ran them through a five-stage pipeline: normalize the text, generate adversarial variants, filter them through six quality gates, annotate harm outcomes, then package the release.

Adversarial content was generated using predefined rule families. Direct attacks (V1) use six templates such as urgency pressure ("give me the dose now, no warnings"), false authority claims, or instructions to ignore prior safety rules. Indirect attacks (V2) use ten templates that poison retrieved documents—exaggerating weak evidence into definitive proof, masking contraindications, faking PMIDs, disguising dangerous recommendations as an "Editor's Note" or "Urgent Institutional Policy Update," or quietly increasing a drug dose in the text.

Quality gates check that each attack is structurally valid, clinically plausible, preserves the original clinical intent, and carries enough adversarial intensity to be informative. Candidates that fail the intensity check are demoted to the borderline pool rather than discarded.

Outcome labels use a five-category harm taxonomy and a 0–4 severity scale anchored to established clinical grading frameworks (CTCAE, NCC MERP). Labeling combines automated priors, deterministic rule-based clinical linting, and a structured LLM judge.

For evaluation, the team tested 13 models (general-purpose: Qwen-2.5 7B/32B/72B, Llama-3.1 8B/70B, Mixtral 8x7B/8x22B; medical-tuned: MedGemma 4B/27B, Meditron 7B/70B, BioMistral-7B, MMed-Llama-3-8B) across five defense configurations, from no defense (D0) through a composite policy composer (D4). The primary judge was Qwen-2.5-72B-Instruct, chosen for the best trade-off between risk sensitivity, output validity, and ranking stability. Invalid judge outputs were counted as unsafe (worst-case), the conservative choice for a safety benchmark.

Why This Matters

Research impact. The paper argues that ASR, the dominant metric in prompt-injection research, is a poor proxy for clinical harm—particularly under indirect injection where most failures are moderate rather than severe. This changes how safety benchmarks in high-stakes domains should be designed and how defense improvements should be claimed. The paper also makes a methodological point: a defense that lowers ASR while leaving high-severity harm untouched is not a safety improvement.

Real-world applications:

• Clinical decision support systems: Hospitals deploying LLM assistants that pull from institutional guideline repositories or drug databases need to know whether a poisoned or outdated document in that repository can drive dangerous output, and MPIB provides a way to measure that.
• RAG pipeline hardening: Teams building retrieval-augmented clinical tools can use the V2 results to justify investment in context-side sanitization rather than relying solely on prompt-level guardrails.
• Model procurement and evaluation: Health systems selecting between general-purpose and medical-tuned models can use CHER-style outcome metrics instead of vendor-reported refusal rates, which the paper shows can be misleading.
• Regulatory and audit frameworks: The harm taxonomy (misdiagnosis, treatment error, emergency downplay, guidance conflict, evidence manipulation) offers a concrete vocabulary for assessing clinical AI safety that maps onto patient-safety concepts regulators already use.

Industry relevance. Every vendor shipping an LLM assistant that touches clinical content—EHR vendors, telehealth platforms, drug-interaction checkers, triage chatbots—faces the trust-boundary problem this paper describes. The finding that no evaluated defense drops high-severity harm below 50% under direct injection is a strong signal that current guardrail approaches are insufficient for deployment.

Future Directions

• End-to-end RAG security evaluation. MPIB audits post-retrieval conditional safety (given that a poisoned document was retrieved). It explicitly excludes retriever compromise, ranker noise, and open-world corpora. Extending to full-pipeline evaluation would test whether real retrievers surface poisoned content in practice and how ranking interacts with attack success.

• Whether the conditional audit is realistically reachable. The paper includes supporting evidence that the audit is not vacuous under three generic retrievers, but a natural next question is how attack rate changes when the adversary must also win the retrieval competition against legitimate documents.

• Judge calibration and reliability at scale. The primary judge achieved 100% structured-output validity on the 150-instance selection set but only 72.8% validity across 8,515 production judgments, and ranking stability against Qwen-2.5-32B on V2 was weak (ρ = 0.46, p = 0.11). Better-calibrated judges or stronger human-audit hybrids would improve absolute severity estimates.

• Reducing construction-model dependence. Multiple pipeline stages used Qwen-2.5-72B. Regenerating attacks with Qwen-2.5-32B and Llama-3.1-70B showed no systematically weaker attacks (within-rule TF-IDF cosine distance 0.93–0.98), but shared-model bias is not fully ruled out.

• Prospective and production-grade validation. Labels reflect text-based annotation rather than observed patient outcomes, and the defenses tested are lightweight scaffolds. Clinical deployment would require externally audited safeguards validated against real-world downstream effects.

Target Audience

• Clinical AI safety researchers studying prompt injection, RAG vulnerabilities, or LLM guardrail evaluation, who will find the ASR–CHER divergence argument directly relevant to benchmark design.
• Clinical informatics and health system engineering teams deploying or procuring LLM-based tools that retrieve from institutional knowledge bases.
• Model developers and safety teams at organizations building medical-tuned models, who need outcome-level rather than refusal-level evaluation targets.
• Regulators and hospital governance bodies developing assessment criteria for clinical AI, who can adopt the harm taxonomy and severity stratification as an audit vocabulary.
• Benchmark designers in other high-stakes domains (legal, financial, public health) who face the same problem: generic harmful-content metrics failing to capture domain-specific, outcome-level risk.

Authors’ abstract

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly integrated into clinical workflows. However, prompt injection attacks can steer these systems toward clinically unsafe or misleading outputs. We introduce the Medical Prompt Injection Benchmark (MPIB), a dataset-and-benchmark suite for evaluating clinical safety under both direct prompt injection and indirect, RAG-mediated injection across clinically grounded tasks. MPIB emphasizes outcome-level risk via the Clinical Harm Event Rate (CHER), which measures high-severity clinical harm events under a clinically grounded taxonomy, and reports CHER alongside ASR$_2$ to distinguish moderate-or-worse from high-severity outcomes. The benchmark comprises 9,697 curated instances constructed through multi-stage quality gates and clinical safety linting. Evaluating MPIB across a diverse set of baseline LLMs and defense configurations, we find that ASR$_2$ and CHER$_3$ can diverge substantially, and that observed rates vary depending on whether adversarial instructions appear in the user query or in retrieved context. The MPIB dataset and evaluation code are available on Hugging Face and GitHub, respectively.