Stress Testing Factual Consistency Metrics for Long-Document Summarization

AI summary

Overview

Research area: Natural Language Processing, specifically evaluation of factual consistency (factuality) in abstractive summarization of long documents.

Technical level: Intermediate. The paper assumes familiarity with summarization metrics, retrieval-based scoring, sentence embeddings, and natural language inference (NLI), but its conclusions are stated in accessible terms.

Scope: A systematic stress test of six widely used reference-free factuality metrics under seven meaning-preserving perturbations, three retrieval context window sizes, and three long-document benchmark datasets. Paper: arXiv:2511.07689v2, by Zain Muhammad Mujahid, Dustin Wright, and Isabelle Augenstein (University of Copenhagen).

What This Paper Is About

Factuality metrics are supposed to check whether a summary is supported by its source, but most were designed and validated on short documents. Long documents break assumptions these metrics rely on: the source can be thousands of tokens long, relevant evidence is scattered across sections or even multiple documents, and summaries become highly compressed. The authors ask a simple but revealing question: if you rewrite a summary in a way that does not change any of its factual content, do these metrics give the same score? If not, the metric is reacting to surface form rather than to factual consistency.

Key Contributions

1. A systematic long-document stress test of six reference-free factuality metrics. The metrics are BARTScore, SummaC-Conv, SummaC-ZS, AlignScore, UniEval, and MiniCheck (specifically the Bespoke-MiniCheck-7B variant, which ranks highest on the LLM-AggreFact benchmark and has a 32k-token context window).
2. Seven factuality-preserving perturbations applied to long-document summaries using GPT-4o: Paraphrased, Simplified, Synonym Replaced, Less Diverse (vocabulary reduction), Negated (logically equivalent negations), Summarized (further compression), and Added Source Text (inserting an unrelated source sentence).
3. An analysis of long-document-specific factors, namely retrieval context window size and claim information density, that are absent from short-form robustness studies.
4. Cross-domain evaluation on three benchmarks (SQuALITY, LexAbSumm, ScholarQABench) covering science fiction, legal, and scientific domains, with released code, perturbed data, and reproduction scripts at https://github.com/zainmujahid/metricEval-longSum.

Main Findings

• No metric is invariant to meaning-preserving edits. All six metrics show varying levels of sensitivity to perturbations that introduce no factual errors; no metric consistently maintains factual alignment under long-context conditions.
• BARTScore is unstable in the legal domain. It shows clear negative score shifts across nearly all perturbations on LexAbSumm, while remaining relatively consistent on the other two datasets.
• MiniCheck is the most stable under most perturbations (very small changes across all perturbations and datasets) but struggles with logically equivalent negations, especially on LexAbSumm, which the authors attribute to possible domain mismatch.
• UniEval is sensitive to most perturbations and consistently fails to handle logically equivalent Negated summaries across all datasets, suggesting a lack of sensitivity to logical form.
• AlignScore degrades most in the legal domain, with large score drops under Paraphrased, Negated, and Summarized summaries, suggesting difficulty tracking sentence order and logical consistency in structured, formal text. AlignScore and SummaC-ZS are described as particularly unreliable across domains and perturbation types.
• NLI-based metrics are moderate but not invariant. SummaC-Conv and SummaC-ZS are affected by Summarized and Negated summaries, especially on SQuALITY and LexAbSumm.
• Broader retrieval context usually helps — but not for NLI metrics. Most metrics show consistent score improvements as the context window grows from w=0 to w=2. On LexAbSumm, AlignScore rises from 0.36 (w=0) to 0.52 (w=1) to 0.64 (w=2), and MiniCheck rises from 0.47 to 0.53 to 0.60. SummaC-ZS and SummaC-Conv display little sensitivity to larger context windows, implying they rely on more localized comparisons.
• Information density degrades reliability in legal and sci-fi domains. For LexAbSumm, metric scores consistently decrease as claim similarity to the source document increases; the same pattern appears, less pronounced, on SQuALITY. This means claims whose meaning overlaps with many parts of the source are harder for metrics to verify.
• AlignScore and BARTScore are the most affected by information density, showing sharp declines for high-similarity claims; SummaC-Conv and SummaC-ZS drop more gradually; UniEval and MiniCheck remain comparatively stable across similarity bins.
• The trend reverses on ScholarQABench. More general (high-similarity) statements are easier to fact-check there for many metrics, possibly because that benchmark is multi-document and similar sentences may be repeated instances of the same claim across documents.
• Perturbation faithfulness check (NLI-based). Contradiction rates between perturbed and original summaries were low for most perturbation types (between 0.001 and 0.052 depending on dataset and perturbation), but high for the Negated condition: 0.681 on SQuALITY, 0.560 on LexAbSumm, and 0.542 on ScholarQABench.
• Metrics were evaluated in their publicly released form, with no fine-tuning or task-specific adaptation, reflecting common practice in prior work.

Methodology in Plain English

The authors take existing summaries that are already known to be factually consistent with their sources, then have GPT-4o rewrite each summary in seven different ways that should not change any facts: paraphrasing, simplifying sentences, swapping in synonyms, using less varied vocabulary, adding logically equivalent negations, compressing the text further, and inserting an unrelated sentence copied from the source. If a metric is truly measuring factuality, the rewritten summary should get roughly the same score as the original.

Because long sources do not fit into most metrics' input limits, the authors use a retrieval-based workaround. They split the summary and the source document into sentences, embed them with SBERT (bert-base-nli-mean-tokens), and for each summary sentence retrieve the top-K most similar source sentences by cosine similarity. Each retrieved sentence is expanded into a snippet including w neighboring sentences on each side (window size), and the metric scores the summary sentence against each snippet; the maximum score is kept. Summary-level scores are the average of these sentence-level scores. The authors vary the window size across w=0, 1, 2 to see whether broader context makes metrics more stable.

Finally, they measure the "information density" of each summary sentence as the mean pairwise cosine similarity between that sentence's embedding and every sentence in the source document. High-similarity claims are broad statements that overlap semantically with many parts of the source; low-similarity claims are specific and locally grounded. They bin claims by similarity and average metric scores within each bin.

An NLI model (a DeBERTa-v3-large-mnli-fever-anli-ling-wanli checkpoint) is used as a sanity check to confirm that the perturbations did not introduce contradictions. The datasets used, with examples used, average summary sentences, average summary tokens, average document sentences, and average document tokens, are: SQuALITY (260 examples, 12.5 summary sentences, 273 summary tokens, 456.6 document sentences, 6,131 document tokens), LexAbSumm (351, 4.2, 169, 385.9, 10,840), and ScholarQABench (100, 43.2, 1,158, 575.4, 14,652). All three use human-written summaries.

Why This Matters

Impact on research: The paper shows that brittleness previously documented for short-document factuality metrics persists and is often amplified in long-form settings. It argues that evaluating factual consistency in long documents requires metrics that reason over multi-span evidence rather than relying on local or pairwise semantic alignment, and it identifies context-aware calibration and training on meaning-preserving variations as concrete directions.

Real-world applications:

• Scientific literature review and question answering — ScholarQABench models the task of answering expert queries from multiple open-access computer science papers, where hallucinated or unsupported statements are costly.
• Legal document summarization — LexAbSumm covers European Court of Human Rights judgments; the paper finds several metrics degrade most sharply in this domain.
• Long-form media and narrative summarization — SQuALITY consists of public-domain science fiction stories requiring both narrative abstraction and fine-grained detail.
• Internal evaluation pipelines for LLM-generated summaries — any system using a reference-free factuality metric to gate or filter outputs inherits the instabilities reported here.

Industry relevance: Reference-free factuality scores are commonly used as automated quality gates because human annotation is expensive. This paper shows those gates can shift substantially in response to edits that change nothing factual, and that reliability drops precisely for compressed, information-dense claims of the kind long-document summarizers produce. The authors also note that weak or biased metrics could overestimate the truthfulness of generated content in sensitive domains such as medicine or law.

Future Directions

• Multi-span reasoning metrics that can aggregate and reason over evidence distributed across multiple, dispersed source passages instead of scoring against a single retrieved snippet.
• Context-aware calibration to account for how much retrieval context a claim genuinely needs, and to make metrics responsive to broader evidence windows, which NLI-based metrics currently are not.
• Contrastive training on meaning-preserving perturbations so that metrics learn to be invariant to paraphrase, simplification, negation, and vocabulary changes.
• Human judgment studies and new benchmarks with human factuality annotations for long documents across multiple metrics, domains, and perturbation types, plus incorporation of human judgments to identify systematic weaknesses.
• Extending perturbation strategies to long-document phenomena, such as evidence relocation and cross-reference disruption, to stress-test long-range coherence and evidence tracking.
• Hybrid approaches combining reference-free with reference-based alignment signals, and evaluation of whether fine-tuning or adaptation mitigates the observed weaknesses.

Target Audience

Researchers and practitioners working on summarization evaluation, factuality and hallucination detection, and LLM evaluation pipelines. It is most useful to those who already use reference-free factuality metrics (BARTScore, SummaC, AlignScore, UniEval, MiniCheck) and want to understand where those metrics fail on long inputs, as well as to benchmark designers building long-document evaluation suites. Teams deploying long-document summarization in legal, scientific, or media domains will find the domain-specific degradation patterns directly actionable.

Stated limitations: The perturbations are automatically generated with GPT-4o and were not confirmed by human annotators to preserve factual correctness in every case; metric outputs are not evaluated against human factuality judgments in the long-document setting (the authors note such annotations are scarce and would require building a new benchmark); metrics are used in their publicly released form without fine-tuning or calibration; retrieval uses a fixed number of top-k similar sentences with only the context window varied, not dynamic query-based retrieval; and the analysis is confined to English-language datasets from three domains (science fiction, legal, scientific), so findings may not generalize to medical, financial, non-English, or low-resource settings.

Authors’ abstract

Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where conventional metrics struggle with input length limitations and long-range dependencies. In this work, we systematically evaluate the reliability of six widely used reference-free factuality metrics, originally proposed for short-form summarization, in the long-document setting. We probe metric robustness through seven factuality-preserving perturbations applied to summaries, namely paraphrasing, simplification, synonym replacement, logically equivalent negations, vocabulary reduction, compression, and source text insertion, and further analyze their sensitivity to retrieval context and claim information density. Across three long-form benchmark datasets spanning science fiction, legal, and scientific domains, our results reveal that existing short-form metrics produce inconsistent scores for semantically equivalent summaries and exhibit declining reliability for information-dense claims whose content is semantically similar to many parts of the source document. While expanding the retrieval context improves stability in some domains, no metric consistently maintains factual alignment under long-context conditions. Finally, our results highlight concrete directions for improving factuality evaluation, including multi-span reasoning, context-aware calibration, and training on meaning-preserving variations to enhance robustness in long-form summarization. We release all code, perturbed data, and scripts required to reproduce our results at https://github.com/zainmujahid/metricEval-longSum.