CGBench: Benchmarking Language Model Scientific Reasoning for Clinical Genetics Research

AI summary

Overview

• Research area: AI for science / benchmarking of large language models on clinical genetics literature interpretation (variant curation and gene curation).
• Technical level: Intermediate. Readers should be comfortable with LLM benchmarking terminology (precision/recall, F1, zero-shot, in-context learning, LLM-as-a-judge); domain knowledge of clinical genetics helps but is not required.
• Scope: The paper introduces CGBench, a benchmark built from ClinGen expert-curated data that tests whether language models can extract, score, verify, and explain scientific evidence in the way human clinical genetics curators do.

What This Paper Is About

Determining whether a gene causes a disease (gene curation) and whether a specific genetic variant is pathogenic (variant curation) requires curators to read mountains of publications and apply detailed, disease- and gene-specific protocols. This is manual, labor-intensive work, and existing LLM benchmarks rely on narrow, contrived tasks such as multiple-choice questions that do not reflect real research practice. CGBench was built from ClinGen's expert-curated resources to measure whether LMs can perform the actual evidence-interpretation steps those curators perform, and to reveal where current models fall short.

Key Contributions

1. A new benchmark sourced from ClinGen. CGBench aggregates expert-reviewed entries from the ClinGen Evidence Repository (ERepo), including the Variant Curation Interface (VCI) and Gene Curation Interface (GCI), together with VCEP criteria specifications and GCI standards of practice (SOPs).
2. Three grounded reasoning tasks. The benchmark defines evidence scoring (E-Score), evidence verification (E-Ver), and GCI experimental evidence extraction (EE), each formulated as a function of a query (disease, variant or gene, mode of inheritance), a PubMed article, and the applicable guideline specification.
3. Empirical evaluation of 8 language models (closed and open-weight, reasoning and non-reasoning) under zero-shot and in-context prompting, reporting precision/recall on code hierarchies, binary classification metrics, and scoring-error metrics.
4. Explanation-level evaluation. Using 2680 expert curator explanations, the authors apply a calibrated LLM-as-a-judge method to check whether model explanations refer to the same evidence and reach the same conclusions as ClinGen curators.

Main Findings

• Models struggle most on fine-grained level of evidence codes. On VCI E-Score, tertiary-code performance was poor overall: the best model (o4-mini) reached only 0.420 Precision@5 and 0.495 Recall@5, whereas top performance on primary codes was much higher (GPT-4o: 0.861 Precision@5, 0.878 Recall@5).
• Reasoning models lead on tertiary codes. The best-performing models on tertiary code prediction were reasoning models (o4-mini at 0.420 Precision@5; DeepSeek R1 at 0.418 Precision@5 and the best tertiary Recall@5 at 0.517). The authors note reasoning models showed lower precision but higher recall on primary codes, and became better in precision at the tertiary level.
• Smaller non-reasoning models underperform across the board. GPT-4o-mini and Qwen2.5 72B underperformed larger models at every hierarchy level in E-Score.
• In-context examples help E-Score, with diminishing returns. Adding description-only in-context examples improved secondary and tertiary code performance monotonically for Llama-4 and o4-mini, with smaller gains from 20-shot to 30-shot. GPT-4o peaked at 20-shot and declined at 30-shot. More shots tended to equalize performance across the three models tested.
• Evidence verification is hard, and models are overconfident about codes being "met." GPT-4o had the best zero-shot F1 (0.634), with DeepSeek-R1 just 0.005 F1 behind; o4-mini (0.509 F1) and Qwen2.5 (0.486 F1) performed close to random (0.500 F1). Many models predicted roughly two-thirds of samples as "met" when the test set positive rate was 0.434.
• In-context prompting does not help E-Ver. GPT-4o's best variant used 10 in-context examples (0.657), dropping to 0.633 and 0.639 at 20 and 30 examples. For DeepSeek-R1, all description-only in-context settings scored below zero-shot, with the best variant being 5-shot with full text (0.641).
• Models over-extract evidence in the GCI task. GPT-4o had the highest category-matching precision (0.493); o4-mini had the highest recall (0.835), but also predicted the most evidence pieces on average (3.397 vs. 2.307 for GPT-4o, 2.450 for DeepSeek-R1, and 3.140 for Llama-4). The ground-truth average was 1.273.
• Structure adherence is high except for DeepSeek R1. Success rates were 98.81% (GPT-4o), 99.40% (Llama 4), and 96.73% (o4-mini), while DeepSeek R1 broke the desired structure about 40% of the time (61.61% success).
• Scoring of evidence strength remains weak. Best Normalized MAE was o4-mini at 0.186, close to GPT-4o at 0.196; but o4-mini's ΔStrength (0.445) is under 50%, meaning its ability to determine whether evidence is stronger or weaker than a default score is roughly random. DeepSeek R1 (0.346) and GPT-4o (0.342) followed.
• Explanations often fail to match human ones even when the classification is right. With the calibrated task-aware judge (0.744 F1 against manual review, versus 0.723 for evidence-aware and 0.615 for task-agnostic), GPT-4o 30-shot reached 70.4% agreement with ClinGen explanations, o4-mini 30-shot 68.3%, Llama-4 30-shot 56.4%, GPT-4o zero-shot 48.6%, Llama-4 zero-shot 53.4%, and o4-mini zero-shot 65.7%. In-context prompting improved explanation agreement across all models, with GPT-4o jumping 44.8% from zero-shot.

Methodology in Plain English

The authors gathered human curator-reviewed entries from ClinGen, which supplies not only labels but also the expert explanations behind them. They turned this into three tasks. In evidence scoring, a model is given a disease, a variant (which encodes the gene via HGVS nomenclature), the mode of inheritance, the full text of a paper, and the set of evidence codes allowed by the relevant Variant Curation Expert Panel, and must pick the right code. Evidence verification flips the question: given a specific code, the model must decide whether it is "met" or "not met" by the paper. In gene-level extraction, the model must pull out experimental evidence units from a paper in a structured format, including an evidence category, a written explanation, a numeric score, and a reason if the score differs from the default in the applicable standard of practice.

Prompts use chain-of-thought, a role-playing instruction telling the model it is skilled in clinical genetics, and full context including gene/variant identifiers and the paper text. Genes use HGNC names and variants use HGVS nomenclature. For scoring, the authors report precision and recall over the code hierarchies; for verification, they report true positive rate, true negative rate, and F1; for extraction, they report category-matching precision/recall, structured-output parsing success, Normalized MAE (MAE normalized to each category's score range), and ΔStrength (whether the model correctly raised, lowered, or held the score relative to the default). Normalized MAE and ΔStrength are computed only on category-matched samples.

To compare explanations, they used an LLM-as-a-judge setup with three prompt variants (task-agnostic, task-aware, evidence-aware), calibrated against a manual review assisted by a medical trainee with clinical genetics experience. The judge asks whether the model explanation and the curator explanation refer to the same piece of evidence and reach the same conclusion, not which explanation is better. In-context demonstrations omit full paper text, showing only the codes or extracted evidence and explanations, which allowed up to 30 examples for E-Score and 13 examples for GCI extraction (covering at least one instance of every evidence category).

Why This Matters

The paper shows that strong general-purpose language models can partially assist with clinical genetics literature interpretation but are not reliable enough for the precise, guideline-following judgments that clinical curation demands, particularly at fine-grained levels of evidence and in judging evidence strength. It also introduces explanation-level evaluation as a complement to classification accuracy, revealing that correct labels can coexist with hallucinated or misinterpreted reasoning. This pushes benchmarking away from multiple-choice proxies and toward tasks that mirror real translational research.

Real-world applications:

• Assisting variant curation pipelines by pre-scoring literature against VCEP-specific evidence codes, prioritizing papers for human expert review.
• Assisting gene-disease validity curation by extracting structured experimental evidence tables (category, explanation, score, reason for score change) from publications.
• Triage and quality control in clinical genetics laboratories, where models could flag which evidence codes appear met versus not met and surface supporting passages.
• Benchmarking and model selection for biomedical natural language processing, giving developers a harder, expert-grounded target than existing question-answering datasets.

Industry relevance: any organization deploying LLMs in genomics, clinical diagnostics, or evidence-synthesis workflows (for example, clinical labs, biopharma translational research groups, and health AI developers) can use CGBench as a stress test to see where models fail on guideline-following before deployment. The release of code and data as open-source resources makes it usable by teams outside the authors' institution.

Future Directions

• Improving performance on highly specialized, precise instructions. The authors explicitly ask how LM performance can be improved when guidelines vary across variants, genes, and diseases.
• Aligning model interpretations with human evidence interpretation. The roughly random ΔStrength scores and imperfect explanation agreement motivate new methods for aligning LMs to how curators weigh evidence.
• Prompt tuning and memory approaches. The authors suggest using the scale of the GCI data to explore methods that can encode many demonstrations or general rules learned from many VCI/GCI entries.
• Extending evaluation beyond structured classification. The presence of 2680 expert explanations is offered as a spur for future work evaluating LMs on free-text explanation quality rather than classification alone, as well as for adding multimodal content (tables, figures, supplementary material) and multi-document reasoning for codes that require multiple studies.

Target Audience

Researchers working at the intersection of AI and biomedicine, especially those building or evaluating LLM-based systems for literature synthesis and clinical genomics. It is also relevant to benchmark designers interested in expert-grounded, real-world tasks; to clinical genetics curators and informatics teams considering AI assistance; and to model developers seeking a demanding evaluation beyond standard multiple-choice scientific question answering.

Authors’ abstract

Variant and gene interpretation are fundamental to personalized medicine and translational biomedicine. However, traditional approaches are manual and labor-intensive. Generative language models (LMs) can facilitate this process, accelerating the translation of fundamental research into clinically-actionable insights. While existing benchmarks have attempted to quantify the capabilities of LMs for interpreting scientific data, these studies focus on narrow tasks that do not translate to real-world research. To meet these challenges, we introduce CGBench, a robust benchmark that tests reasoning capabilities of LMs on scientific publications. CGBench is built from ClinGen, a resource of expert-curated literature interpretations in clinical genetics. CGBench measures the ability to 1) extract relevant experimental results following precise protocols and guidelines, 2) judge the strength of evidence, and 3) categorize and describe the relevant outcome of experiments. We test 8 different LMs and find that while models show promise, substantial gaps exist in literature interpretation, especially on fine-grained instructions. Reasoning models excel in fine-grained tasks but non-reasoning models are better at high-level interpretations. Finally, we measure LM explanations against human explanations with an LM judge approach, revealing that models often hallucinate or misinterpret results even when correctly classifying evidence. CGBench reveals strengths and weaknesses of LMs for precise interpretation of scientific publications, opening avenues for future research in AI for clinical genetics and science more broadly.