EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

AI summary

Overview

Research area: Natural Language Processing / biomedical NLP — specifically benchmark construction and evaluation of large language models on epidemiological question answering over research literature.

Technical level: Advanced.

Scope: The paper introduces EpiQAL, a three-subset multiple-choice benchmark built from open-access epidemiology articles, and reports a fifteen-model evaluation showing that current LLMs handle text-grounded recall far better than multi-step epidemiological inference.

What This Paper Is About

Existing medical QA benchmarks mostly test clinical knowledge or patient-level reasoning, not the population-level statistical and causal inference that epidemiology requires. The authors build EpiQAL, which they describe as the first benchmark to systematically evaluate evidence-grounded epidemiological QA over research literature, using documents from PLOS Neglected Tropical Diseases. The goal is to diagnose whether models can recall documented facts, integrate multiple findings into an inference, and reconstruct an article's stated conclusions when the Discussion section is hidden.

Key Contributions

1. Framing epidemiological QA as a distinct problem. The paper argues that epidemiological reasoning operates at the population level over heterogeneous study evidence, which differs from clinical or exam-style medical knowledge, and it distinguishes EpiQAL from prior resources such as MedQA, MedMCQA, BioASQ, PubMedQA, COVID-QA, CoQUAD, and EPIC-QA.

2. An expert-curated epidemiology taxonomy. Developed with a faculty member in biostatistics and epidemiology through five rounds of iterative refinement, the taxonomy organizes questions into six high-level classes and 25 topics, giving finer-grained control over question intent than existing study-design or disease-pathology frameworks.

3. An automated construction framework with quality control. The pipeline combines constrained generation, multi-LLM option-level verification with vote-ratio aggregation and targeted human review, plus difficulty judging and stem refinement for the two reasoning subsets.

4. A released benchmark and a fifteen-model evaluation. EpiQAL ships with three subsets (475, 478, and 479 instances for A, B, and C respectively) and a set-based evaluation protocol using F1 and Exact Match, benchmarked across 15 models from seven families ranging from 3B to over 100B parameters.

Main Findings

• Epidemiological reasoning remains hard for current LLMs. The strongest model, DeepSeek-V3.2-Thinking, reaches 0.928 Exact Match (0.970 F1) on EpiQAL-A with Chain-of-Thought, essentially tied with DeepSeek-V4-Flash-Thinking at 0.935 (0.970 F1), and leads EpiQAL-B at 0.868 (0.909 F1) with CoT. On EpiQAL-C, Mistral-7B-Instruct-v0.3 leads at 0.812 (0.816 F1) with CoT.

• EpiQAL-B is the most challenging subset under the evaluation setup. Without CoT, only four of fifteen models exceed 0.70 EM on EpiQAL-B, compared with twelve exceeding 0.76 on EpiQAL-A and ten exceeding 0.50 on EpiQAL-C. Distractor deception rates on B are uniformly high across all four categories (15.9–19.2%).

• Model rankings are subset-dependent. Spearman correlation across the fifteen models under zero-shot no-CoT is ρ=0.482 for A vs. B, ρ=0.615 for A vs. C, and ρ=0.760 for B vs. C, suggesting the subsets capture partially distinct behaviors. DeepSeek-V3.2-Thinking ranks first on A and B but second on C, behind Mistral-7B.

• Scale alone does not predict success. Mistral-7B-Instruct-v0.3 outperforms Mistral-Large-Instruct-2411 on both EpiQAL-B (0.789 vs. 0.644) and EpiQAL-C (0.808 vs. 0.685) despite far fewer parameters. Llama-3.2-3B-Instruct collapses on the reasoning subsets (0.157 on B, 0.088 on C), which the authors read as evidence of a capability threshold rather than a smooth scaling trend.

• The two DeepSeek variants diverge on reasoning. DeepSeek-V4-Flash-Thinking nearly matches V3.2-Thinking on EpiQAL-A (0.926 vs. 0.928) but drops on B (0.703 vs. 0.818) and C (0.666 vs. 0.720).

• CoT helps multi-step inference but is mixed elsewhere. Llama-3.1-8B-Instruct improves from 0.318 to 0.665 EM on B (+0.347) and GPT-4o-mini from 0.222 to 0.531 (+0.309). On EpiQAL-A gains are marginal, and on EpiQAL-C GPT-5-mini degrades from 0.599 to 0.555 while Llama-3.1-8B improves from 0.190 to 0.384.

• Distractor error analysis exposes over-selection. On EpiQAL-B without CoT, the F1–EM gap is Δ=0.429 for Phi-4-mini-instruct and Δ=0.428 for GPT-4o-mini, versus Δ=0.019 for Mistral-7B on B and Δ=0.014 on C. On EpiQAL-A, semantic near-miss distractors are roughly twice as deceptive as other categories (16.8% vs. 9.4%, 8.1%, and 8.0%). On EpiQAL-C, speculation/limitation leads at 21.0%; the reported 39.3% for background only is based on only 8 distractors and is flagged as unreliable.

• Verification resolves most options automatically. Each checker runs 3 times at temperature 1.0 for 9 votes per option. Option-level acceptance is 86.3% (A), 94.1% (B), and 89.7% (C); human review is required for 3.9%, 1.5%, and 2.8% respectively; 3.6%, 2.4%, and 2.2% of instances are discarded. The pipeline automatically resolves over 96% of options without human intervention.

• Human evaluation supports benchmark quality. Three computer science PhD students with biomedical training scored 120 questions (40 per subset, 8.4% of the benchmark) on a 1–3 scale. Answer Correctness is at least 2.81 and Evidence Sufficiency at least 2.75 across subsets; Exact Agreement ranges from 47.5% to 68.8%, but Binary Agreement exceeds 91% and Severe Disagreement stays below 8%. A biostatistics and epidemiology faculty member scored 30 questions blind, reaching Binary Agreement of 97.5% (A), 96.0% (B), and 98.3% (C).

• Grounding evidence is strong on A and C, partial on B. Replacing the correct article with a random one drops Exact Match to near zero on A and C, below the question-only baseline. Option-only accuracy is 0.31 (A), 0.21 (B), 0.22 (C), not exceeding the empirical single-pick reference. On B, DeepSeek-V4-Flash-Thinking and GPT-4.1-nano gain 0.196 and 0.240 EM from the correct passage over question-only, while GPT-5-mini and GPT-4o-mini show little measurable benefit. Residual question-only performance is comparable before and after stem refinement (B: 0.353 vs. 0.356; C: 0.253 vs. 0.246).

• Robustness checks. An out-of-distribution test set from 96 post-release articles yields Spearman rank correlation of at least 0.94 on all subsets. A cross-source evaluation on 82 articles from the International Journal of Epidemiology reproduces the main qualitative patterns. A temperature ablation across 12 models finds that switching to T=0 changes mean per-model EM by less than 1% with Spearman rank correlation exceeding 0.99; a multi-run analysis over three runs shows mean standard deviation under 1%.

Methodology in Plain English

The authors first built a corpus of approximately 10,600 open-access articles from the Journal Archive of PLOS Neglected Tropical Diseases, covering over 20 diseases across viral, parasitic, bacterial, and fungal pathogens, and used a randomly sampled subset of 500 articles for the main experiments.

They then defined three subsets. EpiQAL-A asks questions whose correct answers appear verbatim in the article, with confusable distractors drawn from the same document. EpiQAL-B asks questions whose answers require linking several findings in the document with epidemiological knowledge; for this subset, knowledge-graph relations were optionally supplied during construction but withheld at evaluation. EpiQAL-C asks about the authors' stated conclusions after the Discussion section is masked, so the model must reconstruct what the paper concluded from the rest of the text.

Questions were generated by four models acting as a generator (Qwen3-30B-A3B-Instruct-2507) under a constraint schema specifying topic scope, reasoning type, and option construction rules. Because automatically generated items can contain errors, each option was independently checked by three LLMs from different families (GPT-5-mini, DeepSeek-V3.2-Thinking, GLM-4.5-Air) running three times each at temperature 1.0, yielding nine votes per option. Checkers judge options individually rather than re-solving the question, and they evaluate against the test-time input only, so validity does not depend on information available only during construction. Options below a lower vote threshold are rejected automatically, options above an upper threshold are accepted, and options in between go to a human reviewer. The paper gives a concrete example with nine votes: fewer than 5 keep votes means rejection, exactly 5 means human review, and 6 or more means acceptance.

For EpiQAL-B and EpiQAL-C, a difficulty score (DiffScore = 1 − (α·F1 + (1−α)·EM), with α controlling the trade-off) is averaged over a four-model pool (GPT-5-mini, DeepSeek-V3.2-Thinking, Qwen3-32B, Phi-4-mini-instruct) using α=0.3 and threshold θ_d=0.2. Items judged too easy go through stem refinement, which replaces salient entities in the question with descriptive phrases retrieved from web sources, iterating until DiffScore exceeds the threshold or a maximum of T_r=3 iterations is reached. No retrieved text is given to models at evaluation time. Refinement affected a minority of items: 317 of 478 EpiQAL-B instances kept their original stem.

At evaluation, all models receive the test-time document (or the document without its Discussion for EpiQAL-C), the question, and the combined option set in a single prompt, and output selected option indices as JSON. The prompt states that a question may have one, several, or no correct options, so answer cardinality is never disclosed. Scoring uses set-based F1, which rewards partial overlap, and Exact Match, which requires exact set recovery. Temperature is 0.3 except for reasoning models (1.0). Reported Exact Match is a pass rate over roughly 478 items per subset, with 95% Wilson intervals having a half-width of at most about 0.045.

Why This Matters

Impact on research. The paper argues that medical QA benchmarks have largely measured clinical knowledge or patient-level reasoning, leaving population-level epidemiological inference unmeasured. EpiQAL provides fine-grained diagnostic signals for three separable capabilities—evidence grounding, inferential reasoning, and conclusion reconstruction—and the finding that rankings shift across subsets means a single aggregate medical QA score would obscure these differences. The distractor taxonomy also supplies a template for analyzing why models fail, not just how often.

Real-world applications:

• Evaluating whether an LLM can reliably summarize or answer questions about epidemiological study findings before it is used in evidence-synthesis workflows.
• Screening LLM outputs used in public health literature review assistance, where false positives (over-selection of unsupported claims) carry real costs such as unnecessary interventions.
• Testing model generalization across neglected tropical diseases and, via the cross-source check on the International Journal of Epidemiology, chronic and environmental epidemiology.
• Serving as a diagnostic for developers choosing between model sizes or prompting strategies for reasoning-heavy tasks.

Industry relevance. The paper's findings that a 7B model can beat a much larger one on some subsets, and that lighter model variants retain retrieval performance while disproportionately losing reasoning capability, matter directly for model selection and deployment cost in health-adjacent AI products. The paper is explicit that EpiQAL is not a clinical or public health decision-making tool and that strong scores should not be read as domain competence.

Future Directions

• Separating epidemiological reasoning from general scientific reading. The authors state that their controls do not establish that success requires epidemiology-specific knowledge, and they name three controls they did not run: a matched non-epidemiology benchmark built by the same pipeline, a comparison of epidemiologist versus generalist human performance, and per-item annotation of which epidemiological principle each question depends on.
• Broadening corpus and language coverage. The corpus is drawn solely from PLOS Neglected Tropical Diseases and is entirely in English, which the authors say may underrepresent other epidemiological domains and multilingual settings. The cross-source check uses the same generator and checkers, so it tests pipeline portability rather than coverage of epidemiology as a whole.
• Full-scale generator ablation. The benchmark is generated by a single 30B-parameter model; a preliminary check with a stronger cross-family generator on a 50-article sample per subset largely preserved per-model rankings, but no matched comparison of distractor deception rates was done, so a controlled generator ablation at full scale remains future work.
• Better controls for grounding on EpiQAL-B and for contamination. The authors treat evidence grounding on EpiQAL-B as only partial because two of four API models gain little measurable accuracy from the correct passage, and they cannot fully rule out indirect contamination through derivative training data despite the out-of-distribution evaluation.
• Moving beyond single-article reasoning. EpiQAL is described as a proxy for real-world public health analysis, which often requires integrating multiple documents and incorporating temporal and geographic context.

Target Audience

This paper is most useful to NLP and machine learning researchers working on benchmark construction, evaluation methodology, and LLM reasoning; to biomedical informatics researchers interested in evidence-grounded QA over scientific literature; and to applied AI practitioners selecting or prompting models for health-adjacent reasoning tasks. Epidemiologists and public health researchers may value the taxonomy and the negative results about current model reliability, though the evaluation is described by the authors as a proxy rather than a domain-competence measure.

Authors’ abstract

Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effects at the population level. Existing medical question answering benchmarks primarily emphasize clinical knowledge or patient-level reasoning, yet few systematically evaluate evidence-grounded epidemiological inference. We present EpiQAL, to our knowledge the first diagnostic benchmark for epidemiological question answering over research literature, comprising three subsets built from open-access articles across diverse diseases. The three subsets progressively test factual recall, multi-step inference, and conclusion reconstruction under incomplete information, and are constructed through a quality-controlled pipeline combining taxonomy guidance, multi-model verification, and difficulty screening. Experiments on fifteen models spanning open-source and proprietary systems reveal that current LLMs show limited performance on epidemiological reasoning, with multi-step inference posing the greatest challenge. Model rankings shift across subsets, and scale alone does not predict success. Chain-of-Thought prompting benefits multi-step inference but yields mixed results elsewhere. EpiQAL provides fine-grained diagnostic signals for evidence-grounding, inferential reasoning, and conclusion reconstruction.