Expert-Guided Prompting and Retrieval-Augmented Generation for Emergency Medical Service Question Answering

AI summary

Overview

• Research area: Natural Language Processing / medical question answering, applied to Emergency Medical Services (EMS) education and certification preparation.
• Technical level: Intermediate. The paper assumes familiarity with chain-of-thought prompting, retrieval-augmented generation (RAG), and parameter-efficient fine-tuning (LoRA), though each component is described in accessible terms.
• Scope in one sentence: The paper introduces EMSQA, a 24.3K-question EMS multiple-choice dataset with certification-level and subject-area labels, plus a knowledge base and patient-record corpus, and proposes two expertise-guided methods (Expert-CoT and ExpertRAG) that use those labels to improve LLM reasoning and retrieval.

What This Paper Is About

Large language models can answer medical questions, but they typically reason and retrieve information without knowing what kind of medical question they are facing—for example, whether a question is about trauma, airway management, or pharmacology, or whether it is pitched at the Emergency Medical Responder (EMR) level or the Paramedic level. Real EMS professionals start from exactly that kind of context. This paper builds a dataset and two methods that first infer a question's clinical subject area and certification level, then feed that expertise back into the model's reasoning and document retrieval.

Key Contributions

1. EMSQA dataset: Described as the first EMS multiple-choice question answering dataset, containing 24.3K questions curated from public and private sources, spanning 10 subject areas and 4 certification levels, with per-question subject-area labels and explanations where available.
2. Curated knowledge resources: A subject area-aligned EMS knowledge base described in the abstract as 40K documents and 2M tokens (reported in detail as 39,652 sections, 2,545,192 tokens, and 34,110 unique vocabularies), drawn from 16 open-access EMS education resources, plus 4,003,430 real-world patient care records from the NEMSIS 2021 public research dataset.
3. Expert-CoT: A prompting strategy that conditions chain-of-thought reasoning on the predicted subject area and certification level of the question.
4. ExpertRAG: A retrieval-augmented generation pipeline with a trained "Filter" module that predicts expertise attributes and drives subject-area-specific retrieval, tested in "Filter then Retrieve" (FTR) and "Retrieve then Filter" (RTF) variants, alongside a computer-adaptive NREMT certification simulation study.

Main Findings

• Expert-CoT beats standard CoT: Conditioning chain-of-thought on predicted subject area and certification level improves performance by up to 2.05% over vanilla CoT prompting.
• ExpertRAG plus Expert-CoT beats standard RAG: The best configuration yields up to a 4.59% accuracy gain and 3.69% F1 gain over standard RAG baselines.
• Predicted expertise is nearly as good as ground truth: Expert-CoT using Filter-predicted attributes performs comparably to using ground-truth annotations across the evaluated models.
• The Filter module outperforms LLM baselines at classification: On the public split, the LoRA-trained Filter reaches micro-F1 80.72 and macro-F1 71.92 for subject area, and micro-F1 65.87 and macro-F1 63.45 for certification level; on the private split, 79.06 / 70.48 and 65.54 / 63.50. Qwen3-4B zero-shot baselines scored far lower (e.g., public subject area micro-F1 55.43).
• Closed-source models lead on raw accuracy: OpenAI-o3 achieved the highest overall accuracy at 92.39 (public, 0-shot), and Gemini-2.5 reached 89.36. Among open-source models, Qwen3-32B reached 85.70 accuracy with Expert-CoT using ground-truth expertise on the public split.
• More few-shot examples help only up to a point: Varying exemplars from 0 to 64 (Qwen3-32B: 0-shot 83.55, 4-shot 84.41, 32-shot 81.13, 64-shot 82.48 on public) showed gains from a few examples and diminishing or negative returns beyond that.
• Models do worse on the "easiest" certification level: Accuracy was lowest for the entry-level EMR questions and highest for Paramedic, which the authors attribute partly to smaller data volume for EMR and the procedural nature of those questions.
• Core NREMT domains are hardest: Models handled "pharmacology" and "anatomy" relatively well but struggled on "pediatrics" and core areas such as "trauma," "airway," "EMS operations," and "cardiology."
• Retrieval source matters: Adding the knowledge base produced larger gains than adding patient records, and combining both gave the best result. RTF generally outperformed FTR and global retrieval.
• The 32B models pass all four certification simulations: ExpertRAG-32B with RTF passed EMR (score 1350, accuracy 92.22), EMT (1328, 92.32), AEMT (1356, 92.31), and Paramedic (1276, 88.04). The 4B models failed at one or more certification levels, though they benefited most from expertise augmentation.
• KB coverage is good: Semantic similarity between questions and their nearest KB document was 79.21% on public and 80.75% on private data; syntactic concept hit rates ranged from 41.65% to 90.89%. Patient-record overlap was semantically high (66.45% / 75.35%) but syntactically low (8.87%–28.26%).

Methodology in Plain English

The researchers first assembled the dataset. They crawled 17 websites offering National Registry of Emergency Medical Technicians (NREMT) practice tests, collecting 18,602 public and 5,669 private questions (24.3K total, matching the paper's reported "24.3K"). They cleaned the data with heuristics, removed image- and table-dependent questions, and deleted duplicates using Levenshtein distance with a similarity threshold of 0.9. Questions were manually labeled with subject areas and proofread; a sample of 100 questions and KB documents was verified by an EMT expert. Private questions were held out for testing only, and public questions were split into 13,021 train, 1,860 validation, and 3,721 test examples.

They then built two retrieval corpora. The knowledge base came from 16 open-access EMS sources (YouTube transcripts, guidelines, textbooks, lecture slides, and flashcards), converted to text and reorganized into sections using GPT-4o, then manually audited—39,652 sections, 2,545,192 tokens, 34,110 unique vocabularies, labeled into 10 subject areas. The patient record corpus came from the NEMSIS 2021 public research dataset: records were stripped of "NA" and similar fields, whole records were dropped if more than 30% of fields were discarded, and the remainder was converted to text and categorized into 6 subject areas.

The core method has three parts. First, a lightweight "Filter" is trained with LoRA (rank 8, scaling factor 16, dropout 0.05, sequence length 128 tokens) plus two classification heads—one for subject area (multi-label, binary cross-entropy, decision threshold 0.5) and one for certification level (multi-class, cross-entropy)—combined with dynamic weight adjustment (DWA, T=2) so neither task dominates. Second, Expert-CoT inserts the predicted subject area and certification level into the prompt so the model reasons from that starting perspective. Third, ExpertRAG uses the predicted subject area to steer retrieval, comparing keeping the whole corpus (Global), filtering before retrieving (FTR), and retrieving a larger candidate set before filtering (RTF). MedCPT serves as the retriever, with 32 KB documents and 8 patient records retrieved, and documents chunked at 512 tokens with 128-token overlap. Training used AdamW with weight decay 0.01 and a fixed random seed of 42 on NVIDIA H200 GPUs.

Evaluation covered four LLMs in the prompting study (OpenBioLLM-70B, Llama-3.3-70B, Qwen3-32B, and OpenAI-o3, plus Gemini-2.5 in the 0-shot comparison), several RAG baselines (MedRAG, i-MedRAG, Self-BioRAG, and vanilla RAG over KB, patient records, or both using Qwen3-4B), and a subscription-based computer-adaptive NREMT simulation exam (80–150 adaptively selected questions, 2.5-hour limit, 100–1500 scoring scale with 950 as the passing threshold).

Why This Matters

• Research impact: EMSQA and ExpertRAG shift medical QA research away from treating retrieval and reasoning as undifferentiated processes, showing that explicit, question-level expertise attributes are a usable signal. The framework is presented as generalizable to other medical MCQA datasets with similar or other expertise attributes.
• Benchmarking resource: The paper reports that existing MCQA datasets are limited to a single certification level (MedQA at 12.7K USMLE questions, MedMCQA at 193K AIIMS & NEET PG questions), while EMSQA spans 4 certification levels, 10 subject areas, and comes with a categorized knowledge base. Public-source data and the full EMS knowledge base are released.

Real-world applications named or implied by the paper:

• EMS certification preparation for EMR, EMT, AEMT, and Paramedic candidates, including simulation testing against the NREMT computer-adaptive format.
• Medical education and training support for EMS providers at different certification tiers.
• Clinical decision support in high-stakes prehospital settings, where accurate and reliable decisions are critical—though the authors explicitly state the models are research prototypes, not approved medical devices.
• Retrieval over real-world patient care data, using the NEMSIS 2021 corpus of 4,003,430 records as a knowledge source for case-grounded answers.

Industry relevance: The results suggest that small, separately trained expertise classifiers can make retrieval and prompting pipelines meaningfully more accurate without replacing the underlying LLM, which matters for cost-sensitive deployments. The finding that 4B models gained the most from expertise augmentation, while only 32B models passed all four certification simulations, gives a concrete sense of the capability floor for this kind of application.

Future Directions

• Extending the framework beyond EMS: The paper proposes applying expertise-guided prompting and retrieval to other medical MCQA datasets with different expertise attributes, but does not test that transfer.
• Adding certification-level labels to the knowledge base: The authors note the KB lacks certification information from its sources, so chapters could only be categorized into 10 subject areas. Certification-aware KB labeling is an open gap.
• Closing the accuracy gap between the Filter and ground-truth expertise: ExpertRAG-Filter+Expert-CoT still falls slightly below the ground-truth variant, so improving the Filter—particularly for smaller LLMs, which failed some certification levels—remains open.
• Understanding and addressing model errors: The paper defers error-type and per-subject/per-certification error analysis to appendix A.9, leaving the detailed failure modes of these systems as a target for follow-up work.

Target Audience

• EMS educators and training programs interested in LLM-assisted question banks and certification preparation.
• NLP and medical AI researchers building retrieval-augmented or expertise-conditioned question answering systems.
• Benchmark and dataset builders looking for an example of expertise-annotated, multi-certification-level medical MCQA data with aligned knowledge bases.
• Practitioners evaluating LLMs for high-stakes domains, who will find the certification-simulation results and the explicit ethics caveats (research prototypes, not approved medical devices, not for sole diagnostic or treatment use) most relevant.

Authors’ abstract

Large language models (LLMs) have shown promise in medical question answering, yet they often overlook the domain-specific expertise that professionals depend on, such as the clinical subject areas (e.g., trauma, airway) and the certification level (e.g., EMT, Paramedic). Existing approaches typically apply general-purpose prompting or retrieval strategies without leveraging this structured context, limiting performance in high-stakes settings. We address this gap with EMSQA, an 24.3K-question multiple-choice dataset spanning 10 clinical subject areas and 4 certification levels, accompanied by curated, subject area-aligned knowledge bases (40K documents and 2M tokens). Building on EMSQA, we introduce (i) Expert-CoT, a prompting strategy that conditions chain-of-thought (CoT) reasoning on specific clinical subject area and certification level, and (ii) ExpertRAG, a retrieval-augmented generation pipeline that grounds responses in subject area-aligned documents and real-world patient data. Experiments on 4 LLMs show that Expert-CoT improves up to 2.05% over vanilla CoT prompting. Additionally, combining Expert-CoT with ExpertRAG yields up to a 4.59% accuracy gain over standard RAG baselines. Notably, the 32B expertise-augmented LLMs pass all the computer-adaptive EMS certification simulation exams.