Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification

AI summary

Overview

Research area: Automated fact-checking / claim verification, combining knowledge graphs, large language models, and retrieval agents (Natural Language Processing).

Technical level: Intermediate. The paper assumes familiarity with knowledge graphs, entity linking, retrieval-augmented generation, and natural language inference, but its architecture is described step by step.

Scope: The paper presents and evaluates a modular, zero-shot fact-checking pipeline that performs knowledge-graph lookups first, falls back to web search when the graph is insufficient, and studies whether claims labeled "Not Enough Information" in FEVER are genuinely unverifiable.

What This Paper Is About

Large language models can produce fluent judgments but lack grounded, verifiable evidence, while knowledge-graph fact-checkers offer precise structured evidence but suffer from limited coverage and latency. The authors build a hybrid pipeline that uses DBpedia knowledge-graph lookups as a fast first pass and invokes a real-time web search agent only when the KG pass returns "Not Enough Information." Their goal is a fact-checking system that is accurate, interpretable through cited evidence, and usable without task-specific fine-tuning.

Key Contributions

1. A modular, open-source, real-time fact-checking pipeline (available at github.com/AndriiLata/aiFactCheck) with three autonomous stages: KG retrieval over DBpedia, LM-based classification with a task-specific labeling prompt, and a web-search fallback agent triggered only on "Not Enough Information" outputs.
2. A KG-first, web-adaptive design that prioritizes precision and interpretability by grounding decisions in structured triples and reducing unnecessary web queries.
3. Empirical evaluation across FEVER, FEVER 2.0, and FactKG showing strong performance without task-specific fine-tuning, with an F1 of 0.93 on the Supported/Refuted split of FEVER reported in the abstract.
4. A targeted NEI reannotation study with two human annotators and one LLM reviewer, examining whether the pipeline uncovers valid evidence for claims originally labeled "Not Enough Information."

Main Findings

• Highest full-pipeline F1 on FEVER: Using GPT-4.1-mini for both stages (LLM, LLM) yielded precision 0.932, recall 0.931, and F1 0.931. With GPT-4o-mini, the combination of an LLM first stage and a DeBERTa-v3 classifier (LLM, DEBERTA) reached precision 0.930, recall 0.926, F1 0.927, versus 0.917 for LLM, LLM.
• Claim-only baselines are weaker: Zero-Shot GPT-4o mini reached F1 0.801 (precision 0.826, recall 0.790) and Zero-Shot 4.1 nano reached F1 0.734 (precision 0.816, recall 0.720). BERT-Base without retrieval scored F1 0.620, and Random Choice 0.500.
• KG-only and web-only stages have opposing error profiles: KG-only with an LLM achieved high precision 0.944 but lower recall 0.734 (F1 0.826); KG-only with DeBERTa reached precision 0.882, recall 0.620, F1 0.714. Web-only configurations were more balanced (LLM: precision 0.912, recall 0.908, F1 0.909; DeBERTa: precision 0.913, recall 0.878, F1 0.895).
• Combining stages helps: Full-pipeline configurations (LLM, LLM at 0.917; DEBERTA, LLM at 0.859; LLM, DEBERTA at 0.927; DEBERTA, DEBERTA at 0.860) outperformed the separate stages in F1.
• Generalization without fine-tuning: On FEVER 2.0 the pipeline reached precision 0.797, recall 0.769, F1 0.783; on FactKG, precision 0.791, recall 0.757, F1 0.774.
• Direct comparisons to prior work: The system reports 0.931 accuracy in the Supported/Refuted mode on FEVER versus 0.895 for Lewis et al. (2020); 0.702 in the Supported/Refuted/NEI mode versus 0.542 for Tan et al. (2023); and 0.732 on FEVER 2.0 in Supported/Refuted mode versus 0.733 for Yuan and Vlachos (2024).
• Fallback is used selectively: With the LLM and DeBERTa configuration, the web fallback was invoked in about 23% of all test cases.
• NEI claims often have findable evidence: In a sample of 150 NEI claims where the model surfaced evidence, over 70% of cases were judged "sufficient" by at least one human annotator. Inter-annotator agreement was moderate, with a Fleiss' kappa among humans of 0.385 and unanimous agreement in 70.7% of instances; LLM-human agreement varied.

Methodology in Plain English

Given a claim, the system predicts one of three labels — Supported, Refuted, or Not Enough Information (NEI) — and produces a small set of evidence items that justify the decision. It runs in a zero-shot inference mode and requires no task-specific training.

Stage 1 (Knowledge Graph first pass): ReFinED detects and disambiguates named entities in the claim and maps them to Wikidata Q-IDs, with spaCy's EntityLinker as a fallback if ReFinED produces nothing. Those IDs are mapped to DBpedia through owl:sameAs. For each linked entity, the system issues a one-hop SPARQL query to pull all RDF triples where the entity is the subject or the object, excluding triples with metadata predicates via a handcrafted blacklist. Each candidate triple is paired with the original claim and scored with the ms-marco-MiniLM-L6-v2 cross-encoder; the top k = 5 triples are kept. The claim plus those triples are passed to either GPT-4o mini or DeBERTa-v3 MNLI, which assigns a local label and a justification citing the triples. If the label is Supported or Refuted, the pipeline stops.

Stage 2 (Web-based fallback): If the label is NEI, GPT-4o mini rewrites the claim into 3 to 5 high-recall search queries, which are sent to the Google Programmable Search API. Up to 100 web snippets are collected, scored with the same MiniLM cross-encoder, and the top k = 5 are retained. Each claim-snippet pair is classified by a modular verifier (zero-shot GPT-4o mini or DeBERTa-v3 MNLI), and the final verdict is Supported, Refuted, or NEI. If NEI is returned here, the fallback is not triggered again.

DeBERTa as an NLI model: Claim-evidence pairs are formatted as [CLS] e_i [SEP] c [SEP] and the model outputs logits over Entailment, Neutral, and Contradiction; softmax picks the highest-probability label, which is mapped back to FEVER labels.

Evaluation setup: The main experiments use FEVER. To avoid ambiguity and enable fair comparison, the authors randomly sample 1,000 FEVER claims with all NEI-labeled instances explicitly removed. The system is built modularly behind a REST interface so components or models can be swapped.

Why This Matters

Impact on research: The paper argues that structured KG evidence and unstructured web evidence are complementary rather than competing, and it provides an open-source, modular baseline that other researchers can recombine. It also contributes a methodologically motivated critique of NEI labels in FEVER, showing that many claims labeled unverifiable do have retrievable evidence — which has implications for how benchmarks are built and how systems are scored.

Real-world applications:

• Newsroom and social-media moderation workflows that need a verifiable evidence trail behind each verdict rather than an opaque model score.
• Real-time monitoring of viral claims, where the KG-first pass resolves well-known entities quickly and web retrieval handles everything else.
• Assistive tools for researchers or analysts who need claim-by-claim evidence retrieval over structured encyclopedic sources.
• Auditing pipelines where interpretability requirements rule out a pure LLM verdict with no cited evidence.

Industry relevance: The system's REST-accessible, modular design and its ability to work without task-specific fine-tuning lower the barrier to deployment. The cost-saving argument is explicit: only about 23% of cases trigger the more expensive web retrieval path, keeping the common case fast and grounded.

Future Directions

• Multi-hop evidence retrieval: The authors identify combining information across multiple KG nodes or documents as a major remaining challenge, since the current system mainly uses single-hop paths.
• Handling truly unverifiable claims: The current design assumes supporting or refuting evidence can always be found and has no explicit mechanism for indicating missing evidence, which limits use on datasets where NEI is significant or required.
• Reducing error propagation: Small mistakes in entity linking, predicate selection, or evidence ranking can cascade into wrong final labels, motivating better early-stage components.
• Broader classifier and knowledge-source coverage: The authors plan to explore alternative classifiers and extend the approach to additional knowledge sources and datasets.

Target Audience

Researchers and practitioners working on automated fact-checking, retrieval-augmented generation, and knowledge-graph question answering will benefit most, particularly those interested in modular, interpretable pipelines and in the reliability of benchmark labels such as FEVER's NEI category. The paper is also relevant to engineers building production fact-verification or content-moderation systems who need a component they can swap in and out behind a service interface.

Authors’ abstract

Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information. At the same time, knowledge-graph-based fact-checkers deliver precise and interpretable evidence, yet suffer from limited coverage or latency. By integrating LLMs with knowledge graphs and real-time search agents, we introduce a hybrid fact-checking approach that leverages the individual strengths of each component. Our system comprises three autonomous steps: 1) a Knowledge Graph (KG) Retrieval for rapid one-hop lookups in DBpedia, 2) an LM-based classification guided by a task-specific labeling prompt, producing outputs with internal rule-based logic, and 3) a Web Search Agent invoked only when KG coverage is insufficient. Our pipeline achieves an F1 score of 0.93 on the FEVER benchmark on the Supported/Refuted split without task-specific fine-tuning. To address Not enough information cases, we conduct a targeted reannotation study showing that our approach frequently uncovers valid evidence for claims originally labeled as Not Enough Information (NEI), as confirmed by both expert annotators and LLM reviewers. With this paper, we present a modular, opensource fact-checking pipeline with fallback strategies and generalization across datasets.