TSVer: A Benchmark for Fact Verification Against Time-Series Evidence

AI summary

Overview

Research area: Natural Language Processing / automated fact-checking, specifically fact verification grounded in structured numerical and temporal evidence (time series).

Technical level: Intermediate. The paper introduces a benchmark and baseline pipeline; understanding it fully requires familiarity with LLM prompting, retrieval evaluation, and fact-verification label schemes, but the core problem is easy to grasp.

Scope (one sentence): This paper introduces TSVer, a benchmark of 304 real-world claims aligned with 400 curated time series, plus baseline results showing that state-of-the-art reasoning models still struggle to verify claims against this evidence.

What This Paper Is About

Automated fact-checking systems have improved on text-based claims, but evaluating them against structured numerical and temporal evidence remains poorly supported: existing datasets often lack structured evidence, give little justification for verdicts, or rely on synthetic claims. The authors build TSVer, a benchmark where claims must be checked against real historical time series, and each claim is annotated with the relevant time frames, a verdict, and justifications explaining how the evidence was used. They also build a baseline pipeline and measure how well current large language models do on this task.

Key Contributions

1. TSVer, a new benchmark dataset: 304 real-world claims sourced from 41 fact-checking organizations, paired with a curated database of 400 time series drawn from Our World in Data, with annotations of relevant time frames, verdicts, and justifications. The authors state it is the first benchmark dataset for explainable fact verification grounded in time-series evidence.

2. An LLM-assisted, two-round annotation process: Claims were filtered for temporal and numerical content using HeidelTime and spaCy, pre-aligned to time-series categories via keyword heuristics, and then annotated in two Prolific rounds. Verdict agreement reached κ = 0.77 using Randolph's free marginal multi-rater kappa.

3. A baseline fact-checking pipeline and evaluation metrics: A two-component pipeline (time-series retrieval, then verdict and justification generation) is benchmarked on seven models. The paper introduces the Time Series Coverage Score (TSCS), which jointly measures time-series selection accuracy (dataset-level F1) and temporal overlap (Jaccard Index), and adapts the Ev²R scorer to evaluate verdict justifications.

4. An auxiliary synthetic dataset: By using gemini-2.5-pro to modify the countries and dates in claims and generate new labels, the authors produced 400 additional synthetic claims, released as a separate dataset within TSVer. These were not used in the main experiments reported in Section 5.

Main Findings

• Top models are still challenged: Gemini-2.5-pro-06-17 achieved 63.57 accuracy on verdicts and an Ev²R score of 47.36 on verdict justifications. GPT-5.2-2025-12-11 reached 62.86 accuracy and 46.51 Ev²R.

• A steep gap to smaller open-weight models: Ministral-8b-2512 reached 54.29 accuracy and Ministral-3b-2512 reached 49.29, while Llama-3.1-8B reached only 23.21. Llama-3.3-70B reached 55.36.

• Retrieval is a major bottleneck (TSCS): Gemini scored highest at 27.87 TSCS, GPT-5.2 scored 25.83, Mistral-large-2512 scored 20.01, Llama-3.3-70B scored 15.41, Ministral-3b-2512 scored 11.32, Ministral-8b-2512 scored 11.03, and Llama-3.1-8B scored 3.19.

• Smaller models over-retrieve: Gemini and GPT retrieved 2,728 and 2,143 time series total across the test set, versus 17,108 for Ministral-8B and 31,419 for Llama-3.1-8B. Using Gemini-2.5-Pro, only 31% of cases had retrieved evidence with fewer than 1 million tokens.

• Context length is a hard limit: Llama-3.1-8B failed on 42.50% of test instances by exceeding its 128k token limit (CL errors). Other CL error rates: Ministral-3b-2512 at 16.79%, Ministral-8b-2512 at 8.57%, Llama-3.3-70B at 8.21%, Mistral-large-2512 at 5.00%, GPT-5.2 at 4.64%, and Gemini-2.5-pro at 2.86%.

• METEOR fails to discriminate: Llama-3.1-8B and GPT-5.2 had comparable METEOR scores of 27.64 and 26.14 despite large gaps in verdict and retrieval performance. Mistral-large-2512 had the highest METEOR at 32.13. Ev²R was judged a more informative signal of factual alignment.

• A table-aware NLI model also struggles: PASTA, fine-tuned on TabFact and applied with all non-SUPPORTED labels collapsed into REFUTED for binary verification, achieved an F1 of 43.56.

• Dataset composition: REFUTES is the most common label at 53.95%. Africa Check (21.38%), Full Fact (15.13%), and PolitiFact (12.50%) were the most represented publishers. The most-discussed countries are the United States (29.93%), the United Kingdom (25.99%), Australia (17.11%), India (16.12%), and Nigeria (14.80%).

• Scale of evidence: Time series average around 11,000 records per instance, with some exceeding 80,000 records. The authors note that selecting different date ranges often leads to different verdicts, a practice they call cherry-picking.

Methodology in Plain English

The researchers started from roughly six thousand claims collected through the Google FactCheck Claim Search API, which aggregates the ClaimReview project. They filtered these with HeidelTime to find temporal expressions and spaCy to find numerical expressions, keeping claims likely to need reasoning over time-series data. Missing claim dates were filled in manually, country mentions were added in square brackets where absent, and Llama-3.1-8B was prompted to find country names inside the raw HTML of the source articles.

Evidence came from Our World in Data. The authors restricted themselves to 400 time series reported at annual resolution (or with meaningful annual aggregates), each paired with OWID metadata such as title, description, and units. Because many claims need multiple time series and the options are easily confused (e.g., cumulative versus per-capita versus total CO₂ emissions), they pre-aligned claims to broad semantic categories using keyword heuristics, and dropped claims that matched no category.

Final alignment was left to human annotators recruited through Prolific. In Phase 1, annotators saw a claim, its fact-checking article, and candidate time series, and selected relevant series, identified useful time ranges, and wrote explanations. In Phase 2, different annotators saw only the claim and the annotated evidence, without the articles, and assigned one of four labels along with a justification. They were shown precomputed statistics such as min/max values, averages, and trends. To improve justification quality, gpt-4o-2024-11-20 generated up to five numerically focused statements from the evidence, and annotators selected truthful ones to incorporate. Claims where the majority verdict disagreed with the reference fact-check were re-annotated in a second round but retained.

For the baseline, an LLM in a few-shot setup first proposes relevant time series based on metadata, then filters down to relevant time ranges and countries, and finally a verdict and justification are generated (zero-shot, with Chain-of-Thought prompting for non-reasoning models). Evaluated models needed context windows of at least 128k tokens. Generation settings were temperature 0.01, top-p 0.95, and maximum output length 4096 tokens.

Why This Matters

Impact on research. TSVer shifts fact-verification evaluation toward structured, high-volume numerical and temporal evidence, a modality the authors describe as underexplored and challenging for language models. It provides structured justifications (time frames plus explanatory notes) rather than the unstructured text found in earlier datasets, and it contributes two metrics: TSCS for retrieval with temporal alignment, and the demonstration that Ev²R transfers from evidence retrieval to justification evaluation.

Real-world applications:

• Assisting professional fact-checkers at organizations such as Africa Check, Full Fact, and PolitiFact by retrieving relevant data ranges and proposing verdicts with justifications.
• Detecting cherry-picking, where selective date ranges make a claim look true when the fuller series contradicts it, such as the example claim about the UK where 1990–2019 is used instead of 1990–2020.
• Monitoring public claims about global indicators such as emissions, GDP, population, and tax revenue against authoritative open data.
• Evaluating and auditing LLM-based verification tools before deployment in journalistic or policy settings.

Industry relevance. The results quantify how far current commercial and open-weight models are from reliable verification on this evidence type, and they show retrieval and context length, not just reasoning, are practical engineering constraints. The authors suggest quantization and dedicated time-series encoders as routes to more efficient representations.

Future Directions

• Multilingual and cross-lingual extension. TSVer contains only English claims and articles, so cross-lingual retrieval, multilingual reasoning, and language-specific numeral and date formats are untested.
• Broadening evidence modalities. The benchmark contains only time-series evidence with textual descriptions, while real claims often require reports, tables, charts, or multimedia.
• Better time-series representations. The authors point to quantization approaches (such as K-means clustering used in SpeechGPT and AudioLM, or VQ-VAE) and dedicated time-series encoders (such as GPT4TS and Time-LLM) as ways to encode temporal data within constrained context budgets.
• Improving retrieval, especially for smaller models. Given that retrieval quality is identified as a critical bottleneck and small models over-retrieve heavily, better selection and filtering methods are a clear next step. Using raw time-series data rather than only textual metadata for retrieval is noted as beyond the scope of this work.

Target Audience

Researchers and practitioners in automated fact-checking and evidence-based verification; NLP and LLM evaluation scientists working on numerical, temporal, or structured-data reasoning; time-series and tabular reasoning researchers interested in applying table-aware models to new domains; and professional fact-checking organizations or journalists who want to understand the current capabilities and limits of automated verification against open data.

Authors’ abstract

Reasoning over temporal and numerical data, such as time series, is a crucial aspect of fact-checking. While many systems have recently been developed to handle this form of evidence, their evaluation remains limited by existing datasets, which often lack structured evidence, provide insufficient justifications for verdicts, or rely on synthetic claims. In this paper, we introduce TSVer, a new benchmark dataset for fact verification focusing on temporal and numerical reasoning with time-series evidence. TSVer contains 304 real-world claims sourced from 41 fact-checking organizations and a curated database of 400 time series covering diverse domains. Each claim is annotated with time frames across all pertinent time series, along with a verdict and justifications reflecting how the evidence is used to reach the verdict. Using an LLM-assisted multi-step annotation process, we improve the quality of our annotations and achieve an inter-annotator agreement of kappa=0.77 on verdicts. We also develop a baseline for verifying claims against time-series evidence and show that even the state-of-the-art reasoning models like Gemini-2.5-Pro are challenged by time series, achieving a 63.57 accuracy score on verdicts and an Ev2R score of 47.36 on verdict justifications.