Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

AI summary

Overview

Research area: Natural Language Processing — automatic evaluation of factual recall and information completeness in large language model (LLM) outputs.

Technical level: Advanced (the paper assumes familiarity with natural language inference, atomic fact decomposition, and evaluation meta-metrics, though its central ideas are described conceptually).

Scope: The paper introduces and meta-evaluates three automated metrics — NLI-based, Q&A-based, and end-to-end — for scoring how much of the relevant factual content in a background corpus a model response actually covers.

What This Paper Is About

Most work on LLM factuality checks whether what a model says is true (precision), but a response can be entirely accurate and still be dangerously incomplete by omitting key facts or presenting only one side of a dispute. This paper tackles the harder, less-studied problem of measuring recall — how much of the relevant information in a set of reference texts a response actually includes. The authors build three automated metrics that do not just output a number, but identify the specific missing facts, and they test how reliable those metrics are.

Key Contributions

1. Three novel comprehensiveness metrics that evaluate an LLM response against a reference corpus: an NLI-based method (atomic statement extraction, relevance filtering, entailment graph construction, graph analysis), a Q&A-based method (question mining, refinement, answer generation, answer comparison), and an end-to-end method (an LLM directly returns covered and uncovered facts). Each outputs covered statements (A_in) and missing statements (A_out), combined into a score S = |A_in| / (|A_in| + |A_out|).
2. A meta-evaluation of 15 metric variants (3 strategies × 5 LLMs) on the WikiContradict and ConflictBank datasets, using bespoke label match rate (LMR) metrics, 95% confidence intervals via BCa bootstrap with 10,000 resamples, and paired permutation tests with Benjamini-Hochberg false-discovery-rate control.
3. A human evaluation on 50 randomly selected WikiContradict samples, annotating errors in the outputs of the three metrics to test whether the automated evaluation procedure is trustworthy.
4. An applied study of five popular open-weight LLMs answering 500 real-world questions from the r/explainlikeimfive (ELI5) Reddit forum, grounded in three retrieved web sources per question.

Main Findings

• NLI underperforms: Across both datasets, the Q&A and end-to-end (E2E) variants consistently outperformed the NLI-based pipeline for all models (p < 0.05). The authors attribute this to the NLI method's limited ability to incorporate broader context when judging entailment between atomic statements.

• E2E wins on WikiContradict: The E2E approach significantly outperformed Q&A across all models on WikiContradict (p < 0.05), except with gpt-oss-120b, where the difference was not statistically significant.

• ConflictBank flips the pattern: On ConflictBank, Q&A significantly outperformed E2E with gpt-oss-20b, gpt-oss-120b and Qwen 2.5 72B (p < 0.05), while E2E significantly outperformed Q&A with Llama 4 17Bx128E (p < 0.05). The difference with Llama 3.3 70B was not significant.

• Best average scores: E2E paired with Llama 4 17Bx128E achieved the best average result with an LMR of 0.85; Q&A performed best paired with gpt-oss-20b at an LMR of 0.81.

• Q&A is more robust: Q&A showed a standard deviation of only 0.009 in mean performance across models, versus 0.044 for E2E — meaning E2E's quality depends much more on which LLM runs it.

• Phrasing trips up E2E: A manual review of ConflictBank results for the E2E and Q&A metrics with gpt-oss-120b (where the gap was largest) found E2E sometimes failed to match logically equivalent statements phrased differently in the context and the answer. Q&A did not suffer from this, as its answering step produced consistent answers regardless of phrasing.

• Human evaluation confirms the ranking: On 50 WikiContradict samples, fully correct outputs were 48.0% (NLI), 66.0% (Q&A) and 88.0% (E2E). Agreement between LMR-based correctness judgements and human annotations was 81.3%.

• Real-world results: On the ELI5 dataset, both the gpt-oss-20b Q&A and Llama 4 17Bx128E E2E evaluators identified gpt-oss-120b as the most comprehensive model (0.71 Q&A, 0.83 E2E) and Qwen 2.5 72B as the least comprehensive (0.66 Q&A, 0.73 E2E). Differences between some models were relatively small.

• Metric scales differ: The Q&A metric produced significantly lower absolute scores than the E2E variant despite similar meta-evaluation performance, likely because its finer-grained questions and answers cover a larger number of facts in the background texts.

Methodology in Plain English

All three metrics take the same three inputs: the user prompt, the model response being judged, and a corpus of background texts assumed to contain the key information for that prompt. The corpus can optionally be summarized first to cut computation cost.

NLI-based approach. The system uses an LLM to break the response and every corpus text into "atomic statements" — simple, self-contained factual claims. It revises these to remove unresolved references (like "he" or "the event") and splits compound claims (e.g. "George Orwell wrote Animal Farm and 1984" becomes two claims). It then scores each claim for relevance to the query and discards those below a threshold. Next, an LLM judges entailment relations between all context–response, response–context and context–context statement pairs — note that this scales as 2 × |A_R| × |A_C| + |A_C| × (|A_C| − 1) relations, which is expensive. These relations form a directed "fact graph." The graph is condensed by contracting each strongly connected component (treated as logically equivalent statements) into one node. A corpus statement counts as covered if any response statement has a path to it; it counts as missing otherwise. The method also computes a minimal "uncovered context basis" — the smallest set of statements a response would need to add to be fully comprehensive — and it excludes response–response relations as irrelevant.

Q&A-based approach. Instead of extracting claims directly, the system mines open-ended questions from the response and each corpus text, then refines them: removing duplicates and off-topic questions and rewording the rest to be clearer and more self-contained, again filtering by a relevance score. The questions are then answered from each source separately, with confidence scores attached and low-confidence answers discarded. Answers to the same question are compared and labelled with relation types ("equivalent", "first implies second", "second implies first", "contradictory", "neutral"), with the first three converted into standard unidirectional entailments. The answers become graph nodes, the same condensation and path logic applies, and the score and uncovered basis are computed as before. The comparison module is an LLM augmented with a tool that uses the Pint library to compare physical quantities in different units, because the base LLM was found lacking at this.

End-to-end approach. A single LLM prompt receives the query, the corpus and the response together, and directly returns the covered and uncovered facts. No atom extraction, graph construction or pairwise classification is needed — making it the most computationally efficient but the least granular and interpretable.

Experiments. The metrics were tested on WikiContradict (using 1,200 samples from its HumanEval portion) and ConflictBank (a random subset of 500 samples), both of which provide conflicting background texts and responses with known completeness properties, allowing the authors to check whether the metric scores matched weak ground-truth labels. Thresholds were T_rel = 3.5 ("could be included in a comprehensive or extended answer, but is not necessary for a concise and focused response") and T_conf = 2 (even minority views with mild confidence should be counted). The five evaluator LLMs were gpt-oss-20b, gpt-oss-120b, Llama 3.3 70B, Llama 4 17Bx128E and Qwen 2.5 72B; reasoning effort was set to medium for the gpt-oss models and the FP8 quantized Llama 4 17Bx128E was used. For the ELI5 study, 500 questions were paired with three Google-Search-retrieved texts, cleaned and summarized with Mixtral-8x22B-v0.1, and each model was explicitly prompted to produce a comprehensive, grounded answer.

Why This Matters

Impact on research. Factuality evaluation has largely measured precision; this work pushes recall — the detection of omission — into the automated-evaluation toolkit, and it provides meta-evaluation evidence about which evaluation designs actually hold up. It also shows that a very simple, single-prompt LLM evaluator can rival elaborate multi-stage pipelines, which challenges assumptions about how much scaffolding evaluation methods need.

Real-world applications:

• Retrieval-augmented generation (RAG): flagging when a system ignores retrieved documents and answers from partial knowledge.
• Summarization and report generation: checking whether a summary drops key points from the source material.
• Safety-critical advisory settings (health, finance, law, public information): catching one-sided answers that omit conflicting evidence, which the authors argue can cause harm comparable to hallucinations.
• Real-time feedback and model correction: because the metrics name the specific missing facts and compute a minimal set of statements needed for full coverage, they can support diagnostics rather than just scores.

Industry relevance. The metrics apply to arbitrary queries and arbitrary reference corpora, not to fixed benchmarks, so they fit into continuous evaluation pipelines and retrieval-based systems. The efficiency-versus-granularity trade-off is directly practical: E2E is cheap, Q&A is more robust and interpretable, and NLI is expensive but the most fine-grained. The paper also warns that the metrics can penalize models unfairly if the corpus itself contains misinformation, and it notes concerns about evaluation circularity when LLMs judge LLM output.

Future Directions

1. Improving E2E robustness to phrasing. The end-to-end metric's main failure mode was not recognizing logically equivalent statements worded differently across the context and the answer; addressing this could combine E2E's efficiency with Q&A's reliability.
2. Reducing cost of fine-grained evaluation. The NLI pipeline's pairwise relation extraction scales quadratically with the number of context atoms, and the authors note the fact graphs may already be too simple to capture full information content — richer structures would likely be computationally infeasible.
3. Accounting for source reliability. The current metrics treat all corpus content as worth including; the authors suggest users must curate trustworthy corpora, leaving open how to weight or filter unreliable sources automatically.
4. Mitigating error propagation and circularity. Both multi-stage pipelines risk errors cascading through their steps, and using LLMs as evaluators raises circularity concerns — the paper argues these are only partially alleviated by grounding evaluation in a corpus and keeping LLM roles simple and isolated.

Target Audience

NLP and LLM evaluation researchers, practitioners building RAG or long-form generation systems, and teams responsible for accuracy and completeness audits in safety-critical deployments. Readers who mainly want a practical, ready-to-use completeness score will find the Q&A and end-to-end variants most directly applicable; readers interested in evaluation methodology and meta-evaluation design will benefit most from the experimental and human-annotation sections.

Authors’ abstract

Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&amp;A-based metric that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.