Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphs

AI summary

Overview

• Research area: Automated factuality evaluation of large language model outputs in healthcare, using medical knowledge graphs (KGs).
• Technical level: Intermediate. Readers should be comfortable with LLM evaluation concepts, knowledge graphs, and correlation-based benchmarking.
• Scope: The paper introduces FAITH, an unsupervised, reference-free framework that decomposes LLM medical responses into claims, links them to a medical KG, and scores their factuality via evidence paths, then validates it against clinician judgments across multiple medical QA, summarization, and fact-verification tasks.

What This Paper Is About

Deploying LLMs in healthcare requires checking whether their medical statements are actually true, but existing automated metrics such as BLEU-4, ROUGE-L, and BERTScore depend on reference answers that are rarely available and correlate poorly with clinician judgment. The authors ask whether structured medical knowledge graphs can instead serve as the grounding for automated fact-checking, and they build FAITH to test this systematically. The goal is to determine how reliable, robust, and explainable such a KG-based evaluator can be for real medical content.

Key Contributions

1. The FAITH framework. An unsupervised, reference-free factuality evaluator for LLM-generated medical text that extracts atomic claims from responses, resolves their entities to a medical KG via UMLS, finds the shortest evidence paths between matched nodes, and scores each claim using path length, relation semantic similarity, entity centrality (PageRank), and relation co-occurrence from contracted line graphs.
2. A multi-dimensional validation of KG-grounded evaluation. The paper tests FAITH on four medical QA datasets with five LLMs, using both quantitative comparisons against nine baseline metrics and subjective evaluation by 20 UK-based clinicians, and additionally tests robustness to paraphrasing.
3. Evidence for explainability. FAITH's per-claim scores are evaluated against clinician annotations of the single most incorrect claim in a response (precision 0.65, recall 0.59, F1 0.62), and the authors use its outputs to build an error typology of GPT-4o's medical mistakes from KG relation types.
4. Demonstrated practical and broader utility. FAITH is used as a thresholding mechanism to trigger Reject-to-Answer and retrieval-augmented generation interventions, and is applied beyond QA to medical summarization and medical fact verification, with ablation studies on KG choice, KG noise, and the claim extraction module.

Main Findings

• FAITH distinguishes model capability; most baselines do not. FAITH consistently separated responses from all five LLMs (paired t-test p < 0.004). ROUGE-L, BERTScore, and TransE on average rated all five models similarly (p > 0.05), and FActScore failed to distinguish Llama 3.1 from GPT-4o-mini on MedQA.
• FAITH is robust to paraphrasing. Using 10 paraphrased versions per response generated by GPT-4o, FAITH's mean ± s.d. coefficient of variation was 0.014 ± 0.005, versus 0.910 ± 0.862 for BLEU-4.
• FAITH aligns best with clinicians. Across 20 clinicians with at least 5 years of clinical practice experience rating factuality, relevance, and potential harm on 5-point Likert scales, FAITH reached Pearson's ρ = 0.696 with clinician judgments, compared with ρ = 0.081 for BLEU-4. Cohen's κ was 0.64 for inter-rater agreement. Clinicians ranked official answers highest, then GPT-4o, then Llama 3.1.
• FAITH's error localization is reasonably faithful. Against clinician-identified most-incorrect claims, FAITH achieved precision 0.65, recall 0.59, and F1 0.62, and in 83.6% of justifications the clinician-flagged erroneous statement appeared among FAITH's top-5 lowest-scoring claims.
• FAITH exposes a characteristic error pattern. Among the top-5 most frequent KG edge types linked to incorrect claims in GPT-4o's responses, nearly half involve phenotypical features of diseases, with frequent errors also in causal relations between diseases and exposures.
• FAITH enables useful interventions. Using FAITH scores as a threshold (percentiles from 5% to 50%) to trigger Reject-to-Answer improved both answer accuracy and justification factuality for GPT-4o more than uncertainty-based thresholding, and using it to trigger RAG was also effective, offering a more cost-effective alternative to applying RAG to every question.
• It generalizes beyond QA. On the FactPICO summarization benchmark (345 LLM-generated summaries of 115 randomized controlled trials across Alpaca, Llama-2, and GPT-4), FAITH achieved the highest Pearson correlation with expert judgments (ρ = 0.61) and aligned with expert-rated model rankings. On the HealthFC (750 claims) and BEAR-FACT (300 claims) fact-verification datasets, FAITH distinguished true from false statements.
• KG quality is a prerequisite. Substituting PrimeKG or OGBL-biokg preserved the relative ranking of LLMs but changed performance. Perturbing UMLS with 20% random edge deletion, 20% random node deletion, or 20% random noisy edge insertion showed that node deletion or noisy edge insertion degraded performance.
• The extraction module matters and is well-designed. All LLM-based extractor variants outperformed QuickUMLS on BioRED and DDI13. Multi-round conversation mainly improved recall, critical-analysis prompting mainly preserved precision, and their combination was most effective. Cheaper open-source models such as Llama 3 and Llama 3.1 caused only a slight decrease versus GPT-4o, and GPT-4o's gain over GPT-4o-mini was marginal.

Methodology in Plain English

FAITH runs in four stages over an LLM-generated medical response. First, it uses GPT-4o with a multi-phase prompt, five clinician-crafted in-context examples, multi-round prompting for recall, and a "critical analysis" prompt to guard against hallucination, to break the response into simple factual triplets of subject, relation, and object. Second, it maps the medical entities in those triplets and the nodes of the knowledge graph to standard UMLS concept unique identifiers, so that wording differences like "haemoptysis" and "Hemoptysis" still match; claims whose entities cannot be matched are labeled "unverifiable" and excluded, a deliberately conservative choice. Third, because exhaustive path search in a dense medical graph is intractable, it finds only the shortest path between the two matched entities, treating that as the strongest evidence. Fourth, it scores each claim by combining how semantically similar the relations along the path are to the claim's predicate, how generic the intermediate entities are (via PageRank centrality), and how often the path's relations co-occur with the claim's predicate in a contracted line graph. Scores run from -1 to 1, where positive means alignment with established medical knowledge and negative means contradiction, and the response-level score is the average over its claims.

For evaluation, the authors compared FAITH against NLP metrics (BLEU-4, ROUGE-L, BERTScore with voyage-3-large, MEDCON), computational fact-checking methods (KL, KL-REL, TransE), and LLM-based evaluators (FActScore, imapScore) over five LLMs (GPT-4o-mini, GPT-4o, Llama 3-8B, Llama 3.1-8B, OpenBioLLM) on four datasets (MedQA, MMLU, MS-AKT, LiveQA). The knowledge graph used was UMLS 2025AA, covering over 200 biomedical vocabulary sources, more than 23 million relationships, and 3.4 million concepts. Human validation involved 20 clinicians, each assessing 5 responses drawn from a pool of 16 MS-AKT questions, with each response rated by at least two clinicians.

Why This Matters

This work tests whether a structured, curated medical knowledge base can replace both reference answers and LLM-as-judge approaches in high-stakes evaluation, and shows that the KG-grounded route correlates far better with clinician judgment than lexical metrics that are still widely reported.

Real-world applications:

• Clinical deployment guardrails. FAITH can act as a filter that rejects or escalates low-factuality LLM responses before they reach clinicians or patients.
• Selective retrieval augmentation. Using FAITH scores to decide when to invoke retrieval from a medical KG avoids the cost of running RAG on every query.
• Model refinement feedback. Per-claim, per-relation error breakdowns give developers a targeted picture of which medical reasoning patterns, such as disease-phenotype and disease-exposure relations, a model repeatedly gets wrong.
• Evaluation of medical summarization. The same approach extends to summarizing randomized controlled trials, a task where the FactPICO results show it tracks expert rankings.

Industry relevance: healthcare providers, regulators, and medical AI vendors need scalable, explainable verification methods that do not require a reference answer for every query, and the framework's reference-free design and open-source release make it directly applicable to production evaluation pipelines. The authors also argue the approach is adaptable to other high-stakes domains such as law and finance.

Future Directions

• Reducing dependence on KG quality. The authors note that claims involving knowledge absent from the KG cannot be verified, producing potential false negatives, and that integrating multiple KGs or handling out-of-scope claims are open problems.
• Improving robustness to the upstream extractor. Because errors in the claim extraction module propagate through the whole pipeline, better or more robust extraction is a stated priority.
• Extending explainability analysis. The error typology built from KG relation types could be expanded into a more systematic diagnostic for guiding LLM refinement in complex medical applications.
• Establishing the boundaries of KG-based checking. Since performance shifted with PrimeKG and OGBL-biokg and degraded under KG noise, questions remain about how to characterize which KGs are sufficient for which kinds of medical claims.

Target Audience

Medical AI researchers and practitioners evaluating LLM factuality, clinical NLP groups building hallucination-detection and safety-oversight systems, health system and regulatory teams that need automated verification without reference answers, and knowledge-graph researchers working on fact-checking over structured biomedical knowledge. Readers with a general machine learning background will follow the framework, but some familiarity with LLM evaluation and knowledge graphs helps with the scoring formulation.

Authors’ abstract

The recent proliferation of large language models (LLMs) holds the potential to revolutionize healthcare, with strong capabilities in diverse medical tasks. Yet, deploying LLMs in high-stakes healthcare settings requires rigorous verification and validation to understand any potential harm. This paper investigates the reliability and viability of using medical knowledge graphs (KGs) for the automated factuality evaluation of LLM-generated responses. To ground this investigation, we introduce FAITH, a framework designed to systematically probe the strengths and limitations of this KG-based approach. FAITH operates without reference answers by decomposing responses into atomic claims, linking them to a medical KG, and scoring them based on evidence paths. Experiments on diverse medical tasks with human subjective evaluations demonstrate that KG-grounded evaluation achieves considerably higher correlations with clinician judgments and can effectively distinguish LLMs with varying capabilities. It is also robust to textual variances. The inherent explainability of its scoring can further help users understand and mitigate the limitations of current LLMs. We conclude that while limitations exist, leveraging KGs is a prominent direction for automated factuality assessment in healthcare.