FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs

AI summary

Overview

• Research area: Natural Language Processing — automatic factuality evaluation of long-form text generated by large language models.
• Technical level: Intermediate (assumes familiarity with LLM prompting, claim decomposition, retrieval, and precision/recall style metrics).
• Scope: The paper diagnoses why existing "decompose-then-verify" factuality evaluators are slow and unreliable, then introduces FaStfact (a redesigned pipeline and metric) plus FaStfact-Bench (a 400-sample annotated benchmark) to show improved alignment with human judgment at lower token cost.

What This Paper Is About

Existing tools for checking whether a long LLM answer is factually correct follow the same recipe: split the answer into small claims, search for evidence, and verify each claim. The authors argue this recipe is both slow (sentence-by-sentence processing plus extra "check and revise" LLM calls) and error-prone (it produces unverifiable, redundant, and missing claims, and it supplies verifiers with only short search snippets). FaStfact is their replacement pipeline, together with a new scoring metric and a human-annotated benchmark built to test whether automated factuality scores actually track human judgment.

Key Contributions

1. A systematic failure analysis of prior pipelines. The paper categorizes three extraction failure modes — unverifiable claims (subjective, tautological, ambiguous), redundant claims (intra-sentence and inter-sentence), and missing claims (removed by relevance checks or missed by the extractor) — and attributes them to structural design choices such as sentence-level windows and post-hoc checks and revisions. It also identifies evidence-side failure: search pipelines that pass only short Serper snippets as the sole evidence.
2. The FaStfact evaluation framework. Four changes: chunk-window claim extraction with a configurable stride w (from 1 sentence up to the entire response); verifiable claim extraction folded into a single inference call instead of separate check-and-revise steps; confidence-based pre-verification that skips retrieval for easy claims; and document-level evidence collection with retrieval-augmented verification.
3. A new metric, F1@K'. It replaces SAFE's F1@K with a per-response ground-truth K' (the number of claims that should be extracted from that specific generation) and a symmetric recall penalty, R_K'(y) = 2 / (1 + e^{γ|S(y) − K'|}), that punishes both too few and too many claims.
4. FaStfact-Bench. An aggregated, manually annotated benchmark of 400 long-form QA pairs drawn from FActScore-Bio, Factcheck-Bench, ExpertQA, LongFact, and HelloBench (80 samples each), with sub-process ground truth such as the number of claims and per-claim verification labels. Code, data, and an annotation interface are released.

Main Findings

• Closest alignment with human scores. On FaStfact-Bench the ground-truth F1@K' is 0.792. FaStfact scores 0.780, giving |ΔF1@K'| = 0.012, versus 0.127 for ExpertQA, 0.195 for FacTool, 0.107 for VeriScore, and 0.168 for SAFE.
• Closest claim counts. FaStfact's |ΔK| (number of claims) is 3.35, compared with 21.26 (ExpertQA), 9.09 (FacTool), 7.32 (VeriScore), and 14.28 (SAFE).
• Token cost. Average token cost per sample is 5615 for FaStfact, against 7893 (ExpertQA), 4480 (FacTool), 22848 (VeriScore), and 49622 (SAFE). FaStfact is the second-lowest in raw tokens but far closer to ground truth than the cheaper FacTool; the baselines that use many more tokens are also less aligned.
• Severe extraction failure in prior work. A manual case inspection of SAFE on a GPT-3.5-Turbo generation of 19 sentences found 39 problematic claims out of 57 extracted, a 68% failure rate.
• Richer evidence. Rather than using search snippets of roughly 20–40 tokens (the paper measures the previous snippet average at 23.75 words), FaStfact scrapes full web pages that average 7054.29 words per document, which the authors say reduces "inconclusive" or "not enough evidence" verdicts.
• Human–FaStfact agreement is high across stages. Exact label agreement on verification is 85.3% (type-level agreement 92.9%). At the overall level, |Δ#supported| = 3.675, |Δ#unsupported| = 1.412, and |Δ#irrelevant| = 0.041.
• Longer chunks do not hurt extraction. In ablations, the chunk strides that best matched the ground-truth claim count ranged from 28 to MAX (the entire response), while token cost fell — the authors present this as evidence against the assumption that sentence-level extraction and accuracy must be traded off.
• Factuality does not track model scale. On the FaStfact-Bench leaderboard, gemini-2.0-flash-thinking tops the ranking, Gemini-flash-thinking surpasses Gemini-flash, and Qwen2.5-7B-Instruct outperforms the much larger Qwen2.5-72B.

Methodology in Plain English

The authors first audited existing pipelines by reading the claims they extracted and sorting the mistakes into categories, then designed a replacement around removing the steps that caused those mistakes.

The new pipeline works in three steps. First, instead of feeding one sentence at a time to the extractor, it feeds a chunk of sentences whose size is a tunable parameter, and asks the model in that same call to write verifiable atomic claims and to state how confident it is about each claim's factual status using labels ranging from "Supported" to "Unsure". Confidence is read off the model's own token probabilities (C = exp(target_logprob)), and only claims whose label is definite and whose confidence C clears a threshold are settled immediately; the rest go forward.

Second, unresolved claims are sent to a web search, and rather than using the short result snippets, the system opens each returned URL and downloads the full page content, building a document-level knowledge base.

Third, a verifier is given the claim plus the most relevant retrieved chunks from that knowledge base (selected with a BM2.5 retriever) and labels the claim from ["supported", "refuted", "conflicting evidence", "not enough evidence", "unverifiable"]. The multi-class labels keep "conflicting evidence" distinct from "not enough evidence" and let the verifier flag badly extracted claims for removal. Per-claim labels are then aggregated into the F1@K' score.

To test all of this, the team pooled 400 long-form QA pairs from five existing benchmarks and annotated them by hand with claim counts and per-claim labels, then ran FaStfact and four baselines (ExpertQA, FacTool, VeriScore, SAFE) with the same underlying LLM as extractor and verifier.

Why This Matters

Factuality evaluation is becoming a bottleneck as LLM outputs grow longer: if the checker costs tens of thousands of tokens per sample or disagrees with human raters, it cannot be used for leaderboards, regression testing, or deployment decisions. This paper argues that many of the field's standard pipeline components are inherited habits rather than necessities, and backs that claim with a benchmark that includes sub-process ground truth — which most prior factuality benchmarks lack, making head-to-head reliability comparisons difficult.

Real-world applications:

• Enterprise and developer QA validation: teams shipping long-form assistants can score answers against evidence with a pipeline that costs roughly 5615 tokens per sample rather than tens of thousands.
• Newsroom and content fact-checking: the document-level evidence base and the "conflicting evidence" versus "not enough evidence" distinction give reviewers a rationale trail rather than a single opaque verdict.
• Model selection and leaderboards: the FaStfact-Bench rankings, which separate Gemini-flash from Gemini-flash-thinking and Qwen2.5-7B-Instruct from Qwen2.5-72B, support choosing models on factuality rather than size.
• Benchmark and audit construction: the released annotation interface and F1@K' metric give auditors a way to build per-response ground truth instead of relying on auto-generated K values.

Industry relevance: the configurable chunk stride, confidence threshold, and search-result count let teams tune a cost–accuracy tradeoff against a fixed budget, and the reliance on the LLM's own log-probabilities means the method can be adopted without training new models.

Future Directions

• Evidence quality is the ceiling. The authors note that document-level scraping depends on the web being open and good; paywalled, scarce, or low-quality sources directly undermine verification.
• Domain coverage. The empirical results are limited to the datasets and long-form generation domains in the benchmark, and the authors explicitly invite the community to extend FaStfact to more diverse domains and tasks.
• Calibrating confidence. Threshold calibration and the robustness of confidence-based pre-verification to different thresholds are flagged as topics for ablation rather than settled questions.
• Reducing dependence on annotation. F1@K' requires human ground-truth claim counts per response; scaling this beyond the 400 annotated samples, or approximating it automatically, remains open.

Target Audience

Researchers and engineers working on LLM evaluation, factuality, and hallucination measurement; practitioners building or auditing long-form generation systems who need a cheaper and more human-aligned factuality score; and benchmark designers interested in sub-process ground truth and metric formulation, particularly the critique of SAFE's F1@K (verbosity blindspot and the circular dependency of K on auto-generated claims).

Authors’ abstract

Evaluating the factuality of long-form generations from Large Language Models (LLMs) remains challenging due to efficiency bottlenecks and reliability concerns. Prior efforts attempt this by decomposing text into claims, searching for evidence, and verifying claims, but suffer from critical drawbacks: (1) inefficiency due to overcomplicated pipeline components, and (2) ineffectiveness stemming from inaccurate claim sets and insufficient evidence. To address these limitations, we propose \textbf{FaStfact}, an evaluation framework that achieves the highest alignment with human evaluation and time/token efficiency among existing baselines. FaStfact first employs chunk-level claim extraction integrated with confidence-based pre-verification, significantly reducing the time and token cost while ensuring reliability. For searching and verification, it collects document-level evidence from crawled web-pages and selectively retrieves it during verification. Extensive experiments based on an annotated benchmark \textbf{FaStfact-Bench} demonstrate the reliability of FaStfact in both efficiently and effectively evaluating long-form factuality. Code, benchmark data, and annotation interface tool are available at https://github.com/Yingjia-Wan/FaStfact.