CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field

AI summary

Overview

Research area: Natural Language Processing / biomedical question answering and evaluation benchmark construction.

Technical level: Intermediate. The paper assumes familiarity with multiple-choice QA evaluation metrics (Exact Match, F1, Hamming score), LLM benchmarking practices, and basic notions of medical study design.

Scope: The paper introduces CareMedEval, a French dataset of 534 multiple-choice questions grounded in 37 biomedical research articles, and benchmarks state-of-the-art generalist and biomedical-specialized LLMs on critical appraisal and reasoning over those articles.

What This Paper Is About

Medical professionals must be able to critically appraise scientific articles — judging study design, methodology, statistics, and limitations — but no existing biomedical benchmark explicitly measures this skill; existing datasets such as PubMedQA, MedQA, and SciDQA mostly test factual comprehension or domain knowledge. The authors build CareMedEval from authentic French medical school Lecture Critique d'Articles (LCA) exams, where sixth-year students answer multiple-choice questions grounded in real scientific papers. The goal is to test whether current LLMs can perform this kind of grounded critical appraisal, and to provide baseline results that expose where they fail.

Key Contributions

1. A new dataset for critical appraisal: CareMedEval, 534 questions linked to 37 scientific articles, derived from official national LCA exams and CNET mock exams, with 204 questions carrying expert-written justifications explaining correct and incorrect answers.
2. A skill-based annotation scheme: Every question was manually annotated by a medical expert with one or more labels — design (105), statistics (239), methodology (219), limitations (132), applicability (115) — each tied to the cognitive skill required (information retrieval, general knowledge, general conceptual understanding, or contextual reasoning).
3. A dedicated benchmark with four metrics: Exact Match Ratio, F1-score, Hamming score, and a custom "LCA score" modeled on the real exam's grading, including penalties for missing essential answers (23 questions) or selecting unacceptable answers (19 questions).
4. A systematic model evaluation: Baselines across generalist and biomedical-specialized models under three context conditions (full article, abstract only, no context) and two reasoning-effort settings.

Main Findings

• Models fail badly overall: Only four models surpass an Exact Match Rate of 0.25, and none surpasses an LCA score of 0.70, which is the minimum passing mark (14/20) for human candidates on the real exam. The best model, GPT-4.1, reaches EMR 0.49, F1 0.84, Hamming 0.78, and LCA 0.68 with the full article.

• Context matters a lot: Providing the full article gives the highest scores (GPT-4.1 EMR 0.49, Qwen3-32B EMR 0.36). Using only the abstract causes a slight drop, but performance stays competitive (GPT-4o-mini in particular), while removing context entirely lowers EMR by 5 to 15 points across models.

• Specialized models do not beat generalist ones: Biomedical-tuned models perform comparably at best. MedGemma-27B-text-IT (EMR 0.28) is close to Gemma3-27B-text-IT (EMR 0.27), and Qwen3-8B (EMR 0.19) outperforms II-Medical-8B (EMR 0.13). A McNemar's test showed these gaps are not statistically significant (p ≥ 0.05) except between Qwen3-8B and II-Medical-8B.

• Reasoning tokens help substantially: Generating intermediate reasoning steps improves every metric. GPT-OSS-120b goes from EMR 0.46 to 0.54, GPT-OSS-20b from 0.36 to 0.49, Qwen3-8B from 0.19 to 0.35, Qwen3-32B from 0.37 to 0.45, and GPT-4.1 from 0.49 to 0.53. The average reasoning trace was 879 tokens, ranging from 36 to 20,019 tokens.

• Hardest and easiest skill categories: Models struggle most with questions labeled limitations (reviewing biases and study weaknesses) and statistics (interpreting statistical results). They perform best on design and methodology questions. The statistics difficulty is partly attributed to articles being provided as plain text without figures.

• Multiple correct answers are a known difficulty: About 29% of questions have two or three correct options, 20% have four, 19% have only one, and around 3% have five, with an average of 2.60 correct answers per question. The authors note models do better when a question has a single correct answer, consistent with prior work on French medical MCQs.

• Reference comparison: GPT-4.1 reaches an EMR of 0.79 on the FrenchMedMCQA dataset, far above its 0.49 on CareMedEval, underscoring the added difficulty of article-grounded critical appraisal.

• A context-dependency subset: 16 questions were annotated with a requires_context field. Some questions (e.g., "This is a study of:") consistently require the article; others were answered more accurately without the article, which the authors suggest reflects a bias rather than a benefit from context.

Methodology in Plain English

The authors collected official national LCA exams from the ECN website and mock LCA exams with professional corrections from the CNET website. They manually corrected questions where official answer keys were unavailable, working with a general practitioner. Each question was annotated with skill labels by a medical expert and linked to its source article via an article ID. Articles were stored as PDFs and as plain text extracted with the PyMuPDF library, manually cleaned to fix formatting; abstracts were also extracted separately. Dataset statistics were computed using the tiktoken Byte-Pair Encoding tokenizer (articles average 5,675 tokens; abstracts average 1,019 tokens).

For evaluation, each prompt contained a fixed French instruction framing the model as a physician, the question, and the answer options, with either the full article, the abstract only, or no article inserted in between. Inference used vLLM and Ollama in float16 precision on NVIDIA L40-48GB and A100-80GB GPUs, with deterministic generation (temperature 0.0, top-p 1.0), a maximum of 8,000 generated tokens, and a maximum prompt length of 31,000 tokens with truncation applied only to article content. Models tested included Qwen3-8B/32B, Qwen2.5-3B-Instruct, Qwen2.5-3B-GRPO-medical-reasoning, II-Medical-8B, Gemma3-27B-text-IT, MedGemma-27B-text-IT, GPT-4.1, GPT-4o-mini, and GPT-OSS-20B/120B. Predictions were scored with Exact Match, F1, Hamming, and the LCA metric, and results were analyzed by context condition, by skill label, and by reasoning-effort setting.

Why This Matters

Impact on research: CareMedEval fills a gap that factual biomedical QA benchmarks leave open — grounded critical appraisal of methodology, bias, and limitations. It shows that high performance on factual medical QA (e.g., EMR 0.79 on FrenchMedMCQA) does not transfer to critical appraisal, and it provides a reproducible baseline for future work on grounded reasoning. It also offers expert justifications that can later be used to evaluate the quality of model reasoning traces, not just final answers.

Real-world applications:

• Supporting physicians and researchers who must appraise new publications, including preprints, as part of continuing medical education.
• Assisting systematic review workflows, where screening and quality assessment are time-consuming and where human reviewers still outperform automated methods.
• Training and assessment tools in medical education, since the dataset originates from the exact exam format used to certify French medical students.
• Evidence-based decision support, where a tool must judge whether a study's design and statistics actually support its clinical claims.

Industry relevance: The results matter to anyone deploying LLMs in clinical or scientific settings under regulatory scrutiny such as the EU AI Act, which the authors reference. The finding that domain-specialized models do not reliably beat generalist models, and that reasoning tokens measurably improve accuracy, gives concrete guidance on model selection and inference configuration. The LCA score provides an exam-anchored threshold (0.70) that maps model performance onto a real professional standard, which is more interpretable for deployment decisions than raw accuracy.

Future Directions

• Multimodal evaluation: Extending the benchmark to vision LLMs that can use figures and tables in the PDFs. Articles average about 10 pages with roughly 3.3 figures each, estimated at 123 figures total, and this visual content is often needed for statistics or limitation questions.
• Retrieval-augmented generation: Dynamically selecting the most relevant article sections based on question keywords, and testing how modifying or perturbing article content affects a model's ability to question what it reads.
• Evaluating reasoning quality, not just answers: Building a framework to compare model-generated reasoning traces against the 204 expert-written justifications in the dataset. The authors note this is non-trivial and leave manual evaluation of reasoning to future work.
• Dataset expansion and annotation reliability: Growing the dataset beyond 534 questions and beyond the French LCA exam context, and increasing the number and diversity of annotators to reduce variability and bias in the ground-truth justifications. The authors also flag that articles are publicly available online and may be in models' training data, so results could be inflated by memorization.

Target Audience

Researchers in biomedical NLP and clinical AI evaluation; builders of medical question-answering and retrieval systems; medical educators and assessment designers interested in measuring critical appraisal skills; and clinicians or health informaticians evaluating whether LLMs can be trusted to support evidence-based decisions. Readers with a general NLP background will follow it comfortably; readers focused on French-language or domain-specialized medical benchmarks will find the comparison to FrenchMedMCQA, MedFrenchmark, and DrBenchmark particularly relevant.

Authors’ abstract

Critical appraisal of scientific literature is an essential skill in the biomedical field. While large language models (LLMs) can offer promising support in this task, their reliability remains limited, particularly for critical reasoning in specialized domains. We introduce CareMedEval, an original dataset designed to evaluate LLMs on biomedical critical appraisal and reasoning tasks. Derived from authentic exams taken by French medical students, the dataset contains 534 questions based on 37 scientific articles. Unlike existing benchmarks, CareMedEval explicitly evaluates critical reading and reasoning grounded in scientific papers. Benchmarking state-of-the-art generalist and biomedical-specialized LLMs under various context conditions reveals the difficulty of the task: open and commercial models fail to exceed an Exact Match Rate of 0.5 even though generating intermediate reasoning tokens considerably improves the results. Yet, models remain challenged especially on questions about study limitations and statistical analysis. CareMedEval provides a challenging benchmark for grounded reasoning, exposing current LLM limitations and paving the way for future development of automated support for critical appraisal.