ClaimDB: A Fact Verification Benchmark over Large Structured Data

AI summary

Overview

Research area: Natural Language Processing / fact verification over large-scale structured data (databases), sitting at the intersection of fact-checking benchmarks and neuro-symbolic program reasoning (text-to-SQL agents).

Technical level: Advanced. The paper assumes familiarity with NL-to-SQL benchmarks, LLM-as-a-judge evaluation, tool-calling agents, and standard classification metrics (precision, recall, F1).

Scope: The paper introduces ClaimDB, a fact-verification benchmark of 53,368 claims, each grounded in a real-world multi-table database averaging 11.3 tables and 4.6M records — roughly 110M tokens of evidence per claim — and evaluates 30 proprietary and open-source LLMs on it.

What This Paper Is About

Existing fact-verification benchmarks (FEVER, TabFact, SEM-TAB-FACTS, SCITAB and others) all share a simplifying assumption: the evidence is small enough to fit inside an LLM's context window, usually drawn from Wikipedia or scientific articles. ClaimDB attacks the setting those benchmarks ignore — claims that summarize official, publicly available structured datasets with millions of rows and many tables, like a statement about national inflation or city homicide rates. The goal is to build a benchmark where verification cannot be done by "reading" the evidence, and to measure how today's models perform when they must instead reason through executable programs.

Key Contributions

1. A benchmark at a new evidence scale. ClaimDB contains 53,368 claims, each paired with exactly one of 80 real-life databases from the BIRD benchmark, covering 11 high-level domains (Economy, Entertainment, Sports, Technology, Education, Gastronomy, Health, Environment, Transportation, Governance, Labor). Each claim's evidence context averages 11.3 tables, 4.6M records, and roughly 110M tokens — exceeding modern LLM context windows by two to three orders of magnitude (Google's flagship long-context models support up to 1M–2M tokens).

2. A three-way label design with a structured NEI taxonomy. Claims are labeled entailed (12,855), contradicted (16,529), or not-enough-info (23,984). The NEI category is subdivided into out-of-schema (12,644), counterfactual (5,786), and subjective (5,554) claims, giving the benchmark a way to test abstention rather than only true/false judgment.

3. A claim generation and quality-control pipeline. Starting from the 11k NL-to-SQL pairs in BIRD, the authors filter for compositional queries via SQL AST analysis, execute the queries to get gold answers, and prompt gpt-5 to generate claims. Quality is controlled by a three-model LLM judge panel (Phi-4, grok-3-mini, mistral-small) rather than OpenAI models, to avoid self-enhancement bias.

4. An empirical study of 30 LLMs on the benchmark. The evaluation uses tool-calling agents that can only execute SQL against the claim's database (limit: 20 tool calls), and shows that more than half of the models score below 55% accuracy.

Main Findings

• Most models fall below 55% accuracy. Out of 30 evaluated models, 17 stay below 55% on both accuracy and Macro-F1. The best performers are gpt-5-mini (accuracy 0.827, Macro-F1 0.828), followed by claude-haiku-4.5 (0.809 accuracy, 0.811 Macro-F1) and gemini-3-flash (0.801 accuracy, 0.800 Macro-F1).

• Open-source models lag, with one exception. gpt-oss:20b performs on par with the top proprietary models (0.740 accuracy, 0.739 Macro-F1). Aside from gpt-oss, none of the remaining 20 open-source models exceeds 68% accuracy or Macro-F1.

• Abstention is systematically broken, in two opposite ways. Proprietary models such as gpt-5-mini and claude-haiku-4.5 are strongly biased against predicting NEI — when the truth is entailed or contradicted they almost never predict NEI. Open-source models such as qwen3 and nemotron-3-nano over-predict NEI; for qwen3, NEI is predicted roughly half the time when the gold label is entailed or contradicted. The paper notes the closed/open gap is driven mostly by how NEI is handled.

• Long sessions hurt performance. Fitting a second-order polynomial to average tool calls over the 1,000 test examples, the top models average roughly 4–8 tool calls; both longer and shorter interaction lengths degrade performance.

• Scaling is marginal. Accuracy improves with model size, but the gains are marginal and roughly log-linear.

• Failures are not superficial. The two best models achieve SQL query success rates of 93% (gpt-5-mini) and 99% (claude-haiku-4.5), and remaining failed queries are evenly distributed across correct and incorrect predictions — pointing to breakdowns in reasoning and query composition rather than syntax errors.

• No evidence of data contamination. Evaluated without tools using parametric knowledge alone, gpt-5-mini achieves Macro-F1 0.253 and accuracy 0.367 (near random), suggesting ClaimDB cannot be solved from memorized knowledge.

• Judge panel validation. On a human-annotated test set of 75 NEI and 75 contradicted-or-entailed claims containing 4 claims flagged as problematic by PhD annotators, the judge panel discarded all four, achieving 100% recall.

• Filtering removed roughly 14% of claims. Across the full benchmark, 5.6% of claims were flagged for incorrect labels and 1.7% for not being self-contained; among NEI claims, 11% showed schema leakage and 5% had an incorrect category assignment.

Methodology in Plain English

The pipeline starts from BIRD, a text-to-SQL benchmark with about 11k question/SQL pairs over realistic, sometimes messy databases. The authors keep only queries that actually need to combine large parts of a database — queries whose syntax trees contain orderings or superlatives, aggregate functions like AVG and SUM, window functions, or joins of three or more tables — and discard any query returning more than ten records. This leaves roughly 6.5k pairs, which are executed to produce question/answer pairs in JSON.

Those answers are then handed to gpt-5, which writes claims: some entailed by the answer, some contradicted by it, and some in the NEI family (out-of-schema, subjective, counterfactual). Entailed and contradicted generation used one-shot prompting with medium reasoning; NEI generation used zero-shot prompting, because in-context examples biased the model toward repetitive patterns.

Because generating 64k claims cannot be hand-annotated, the authors use an LLM judge panel of three smaller models from different families (Phi-4, grok-3-mini, mistral-small), excluding OpenAI models to avoid self-preference bias. Judges answer binary rubrics — is the label correct, is the claim self-contained, and for NEI claims, is there schema leakage and is the category right. Any single negative judgment eliminates the claim — deliberately conservative. A prompt instruction, "If you are unsure, answer no," aims to maximize recall on claims humans would reject.

To keep NEI claims from being trivially easy, the authors embed the gold question/answer context and each claim with gemini-embedding-001 and compute semantic similarity as a proxy for how close a claim stays to the database concepts. For example, two out-of-schema claims from the Chicago Crime database scored 0.869 (top quartile) and 0.798 (lowest quartile). The test splits restrict out-of-schema and counterfactual NEI claims to the top quartile of similarity, while the training split keeps the full spectrum.

Evaluation gives each agent exactly one tool — executing SQL against the relevant database — via Google's MCP toolbox for databases, with a limit of 20 tool calls (exceeding it counts as a failed run). Performance is reported as per-label precision, recall, and F1, plus accuracy and Macro-F1.

Why This Matters

Impact on research. ClaimDB reframes fact verification as a program-execution problem rather than a reading-comprehension problem. Its 110M-token evidence scale makes in-context table prompting untenable and pushes the field toward Program of Thoughts, tool-calling agents, and recursive language models. The abstention findings also add a new evaluation axis — knowing when there is no evidence — that prior benchmarks did not stress.

Real-world applications:

• Fact-checking political and policy claims, such as the two motivating examples: Joe Biden's August 29, 2023 statement on U.S. inflation being "down close to 3" (checkable against Bureau of Labor Statistics consumer price index tables) and Donald Trump's August 11, 2025 statement that Washington, D.C. has "41 homicides per 100,000 people" (checkable against Metropolitan Police Department crime records).
• Auditing corporate or governmental reporting anchored in large tabular datasets.
• Journalism and research workflows where a claim summarizes millions of rows rather than a quotable document.
• Building agentic data-analysis systems that must answer questions over databases while knowing when to abstain.

Industry relevance. Any organization deploying LLM agents over data warehouses and business databases faces exactly this problem: the data does not fit in context, and confidently wrong answers are costly. The benchmark's finding that models either refuse to abstain or abstain excessively, plus its observation that tool-call length must sit in a narrow 4–8 range, gives practitioners concrete failure modes to design around.

Future Directions

• Beyond SQL as the sole tool. The paper explicitly limits evaluation to a single SQL tool and leaves comparison with coding agents that use Python and pandas to future work, noting prior evidence that such approaches are brittle for models below 70B.
• Multi-modal evidence. ClaimDB deliberately isolates structured data. Extending to combined evidence such as free-form text, reports, news articles, and charts is named as a natural future direction.
• Improving open-source reasoning over large data. Since no open-source model other than gpt-oss:20b exceeds 68% accuracy or Macro-F1, closing this gap is framed as a primary motivation for the benchmark.
• Fixing abstention. Both observed behaviors — proprietary models avoiding NEI and open-source models over-abstaining — are characterized as undesirable, leaving open the question of how to train models that abstain appropriately.

Target Audience

Researchers building fact-verification and claim-checking systems; NL-to-SQL and text-to-database researchers; practitioners designing tool-calling or agentic LLM systems over large databases; and evaluation specialists interested in benchmark construction with LLM-as-a-judge quality control and abstention metrics. Readers looking for a beginner-level introduction to fact verification will find the paper's scale, metrics, and agent architecture demanding, though the motivation sections are accessible.

Authors’ abstract

Real-world fact-checking often involves verifying claims grounded in structured data at scale. Despite substantial progress in fact-verification benchmarks, this setting remains largely underexplored. In this work, we introduce ClaimDB, a fact-verification benchmark where the evidence for claims is derived from compositions of millions of records and multiple tables. ClaimDB consists of 80 unique real-life databases covering a wide range of domains, from governance and healthcare to media, education and the natural sciences. At this scale, verification approaches that rely on "reading" the evidence break down, forcing a timely shift toward reasoning in executable programs. We conduct extensive experiments with 30 state-of-the-art proprietary and open-source (below 70B) LLMs and find that more than half score below 55% accuracy. Our analysis also reveals that both closed- and open-source models struggle with abstention -- the ability to admit that there is no evidence to decide -- raising doubts about their reliability in high-stakes data analysis tasks. We release the benchmark, code, and the LLM leaderboard at https://claimdb.github.io .