The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

AI summary

Overview

Research area: Evaluation methodology for large language models — specifically, how hallucination-detection benchmarks for open-domain question answering construct their ground-truth labels.

Technical level: Intermediate. The paper uses formal definitions and standard agreement statistics (Cohen's κ, McNemar's test, bootstrap confidence intervals) but the core argument is conceptual and can be followed without deep statistical background.

Scope: An empirical audit of how nine different automated labeling strategies — lexical similarity metrics, an NLI entailment baseline, and seven LLM judges — agree or disagree with 900 human-annotated question–answer pairs labeled for factual correctness.

What This Paper Is About

Hallucination-detection methods are typically benchmarked by having an LLM answer questions from a QA dataset and then using an automated strategy to label those answers as hallucinated or not, by comparing them against short reference answers. The paper argues this setup hides a choice between two different criteria: reference faithfulness (is the answer fully supported by the reference?) and factual correctness (is the answer free from contradictions and false specific claims?). The goal is to measure how much this criterion mismatch actually distorts automated hallucination labels.

Key Contributions

1. The authors construct a 900-pair human-annotated evaluation sample for auditing automated hallucination labelers in open-domain QA, with strong independent inter-annotator agreement (κ = 0.808, 95% CI: [0.729, 0.879]) on a 300-item overlap subset.
2. They formalize the distinction between reference faithfulness and factual correctness and show empirically that label sources approximating strict faithfulness produce false-positive-dominant errors against a factual-correctness benchmark.
3. They conduct a controlled prompt-ablation study showing that replacing a faithfulness-style criterion with a factual-correctness criterion substantially reduces this bias across most judges, and that frontier judges from different model families show capability alone does not determine the size of faithfulness-induced bias.
4. They analyze label sources in false-positive/false-negative error space, showing that lexical metrics, strict reference-entailment NLI, faithfulness-style prompts, and criterion-aligned prompts occupy distinct parts of that space.

Main Findings

• Faithfulness-style prompting over-labels hallucinations. Under the p1 (faithfulness-style) prompt, LLM judges are false-positive-dominant: they label answers as hallucinated that human annotators judged factually correct. This is the pattern the paper calls faithfulness-induced bias.

• Switching to a factual-correctness prompt helps most judges. Replacing p1 with p2 improved agreement with human labels and consistently reduced false-positive dominance for nearly all judges. For GPT-5-mini on the Llama-3-8B generator set, κ rose from 0.530 to 0.756 and false positives fell from 70 to 15.

• The effect is largest for the frontier judges on the faithfulness prompt. GPT-5.4 under p1 on the Mistral-7B set reached κ = 0.021 with a bias of +0.480 and 144 false positives against 0 false negatives (FP% = 100.0). Claude Opus 4.7 under p1 was already close to the p2 region (κ = 0.712 for Llama-3-8B, bias +0.037).

• Prompt structure is not the driver. In a factorial control, a p1 variant with p2-matching structure produced no detectable improvement, and condensing p2 to match p1's structure did not worsen agreement. Criterion choice, not prompt elaboration, drove the p1 to p2 gain.

• The extended p3 prompt does not consistently beat p2. Across judge families, p2 is usually the best or near-best prompt, suggesting the benefit of correctly specifying the criterion saturates between p1 and p2. No p2 to p3 difference was statistically significant for any generator, while for GPT-5-mini p2 beat p1 under McNemar's test for all three generators (p < 0.0001).

• Criterion-aligned LLM judging outperforms lexical and NLI baselines. Oracle-thresholded ROUGE-L reached κ of 0.403 (Llama-3-8B), 0.483 (Gemma-2-9B), and 0.412 (Mistral-7B). BERTScore reached 0.387, 0.486, and 0.415. Neither approached the p2 API-judge range.

• Strict NLI fails almost entirely in one direction. The T5-11B reference-entailment NLI model at τ = 0.5 produced FP% between 98.3 and 99.4 across generators, with κ of 0.053, 0.025, and 0.049. Even with an oracle threshold that collapsed to τ = 0.01 for all three generators, agreement stayed modest (κ = 0.380, 0.217, 0.395), so the problem is not merely threshold calibration.

• Label sources separate by error direction. ROUGE-L is mostly false-negative-prone, BERTScore is mixed across generators, strict NLI is false-positive-dominant, and the p1 to p2 movement for API judges is leftward — primarily a reduction in false positives.

• Smaller local judges behave heterogeneously. Gemma-2-9B and Qwen2.5-7B partly reproduce the p1 to p2 pattern at lower absolute agreement, while Llama-3-8B shifts toward false-negative dominance under p2/p3, reaching κ = 0.218 with bias −0.383 and only 2 false positives against 117 false negatives (FP% = 1.7) on Mistral-7B outputs.

• A controlled intervention supports the elaboration explanation. Removing elaborative material from p1-evaluated answers reduced the false-positive rate from 52.9% to 23.2%, with 45 FP to TN transitions against 4 TN to FP transitions.

• Hallucination rates vary by generator and dataset. Under the human A1 protocol, hallucination prevalence was 44.7% for Llama-3-8B, 38.0% for Gemma-2-9B, and 51.0% for Mistral-7B. At the dataset level it was 72.3% for HotpotQA, 30.3% for TriviaQA, and 30.6% for TruthfulQA.

Methodology in Plain English

The authors start by writing down two competing rules for what counts as a hallucination. Under the faithfulness rule, an answer is a hallucination if any part of it is not supported by the reference answers. Under the factual-correctness rule, an answer is a hallucination only if it contradicts the reference or contains a specific factual claim that is untrue. The two rules agree on most cases but diverge on a key one: an answer with a correct core plus a true extra detail that happens not to appear in the short reference. Faithfulness calls that a hallucination; factual correctness does not.

To measure how much this matters, they build a benchmark. They take questions from the validation splits of TriviaQA, HotpotQA, and TruthfulQA, generate one answer per question from each of three open-weight models (Llama-3-8B, Gemma-2-9B, Mistral-7B), and sample 300 questions with all three answers, giving 900 question–answer pairs. Sampling is done at the question level so the same questions can be compared across generators. A primary annotator labeled all 900 pairs for factual correctness, consulting external sources when the short reference was not enough to adjudicate a claim; a second annotator independently labeled 300 of them to check reliability.

They then run a range of automated labelers over the same 900 pairs and compare each to the human labels. The labelers are ROUGE-L and BERTScore (whose decision thresholds are either tuned directly on the evaluation set as an optimistic "oracle" bound, or set by 3-fold cross-validation), a T5-11B NLI entailment model used with the reference as premise and the answer as hypothesis, and seven LLM judges: GPT-5-mini, GPT-5-nano, GPT-5.4, Claude Opus 4.7, Gemma-2-9B, Llama-3-8B, and Qwen2.5-7B. The LLM judges are run under three prompt variants — p1 (faithfulness-style), p2 (factual-correctness), and p3 (an extended p2) — plus two auxiliary prompts, p1s and p2t, used to separate criterion from prompt structure.

Agreement is measured with Cohen's κ against the human labels, alongside positive-label bias (the automated hallucination rate minus the human rate), false-positive and false-negative counts, false positives as a share of all errors, and the false-positive and false-negative rates. For the GPT-5-mini ablation, paired comparisons use bootstrap confidence intervals for the difference in κ and McNemar's exact two-sided test, which is valid because all prompts are evaluated on the same pairs.

Why This Matters

Impact on research. The paper reframes label-source choice as a benchmark design decision rather than an implementation detail. Existing benchmark conclusions drawn from reference-similarity or entailment labels should, on this account, be read as performance under that specific operational criterion rather than as general evidence of hallucination-detection or factual-correctness performance. The paper also reports that automated labels agreed with a human annotator substantially less than two human annotators agreed with each other, which puts a ceiling on how much confidence automated benchmark scores can carry.

Real-world applications:

• Building QA and chatbot evaluation suites, where the choice of labeling rule directly determines which model responses get flagged as unreliable.
• Retrieval-augmented generation and summarization pipelines, where source faithfulness is often the genuinely intended criterion and the paper's caution runs in the opposite direction.
• Fact-checking and claim-verification tools, where separating "unsupported by the reference" from "factually false" is the core design problem.
• Model release and safety reporting, where hallucination rates are quoted as headline metrics and are sensitive to which labeler produced them.

Industry relevance. Teams that ship LLM products typically stand up an automated judge to score hallucination at scale. The paper shows that the same nominal judge prompt can yield very different error profiles depending on whether it treats absent information as hallucination, and that a more capable frontier model is not automatically better aligned with the intended criterion. It also provides a concrete, low-cost recommendation: validate the automated labeler on a small human-labeled set matched to the intended criterion before using it as ground truth.

Future Directions

• Extending the validation beyond English open-domain QA with short references and the three relatively small open-weight generators used here, to longer answers, other domains, and other generator models.
• Running the same comparison against human annotations collected under a reference-faithfulness criterion, which the authors explicitly leave to future work, since their reported false positives are errors relative to factual correctness rather than absolute errors.
• Identifying the mechanism behind the cross-model differences observed between GPT-5.4 and Claude Opus 4.7 under the same nominal p1 prompt, which the paper does not explain.
• Collecting consensus-adjudicated labels and logging how often annotators had to consult external sources, to quantify how often short references were insufficient and to strengthen the human reference itself.
• Improving robustness of smaller open-weight judges on realistic QA outputs, since Llama-3-8B, Gemma-2-9B, and Qwen2.5-7B showed varied and sometimes inverted error profiles under the criterion-aligned prompts.

Target Audience

This paper is most useful to researchers and engineers who build or consume hallucination-detection benchmarks: benchmark designers, evaluation and red-teaming teams at LLM labs, and NLP researchers working on factuality, faithfulness, or LLM-as-a-judge methodology. It is also relevant to practitioners running automated evaluation pipelines who need to decide what their judge should actually be measuring, and to reviewers and readers who want to know what the hallucination numbers in a model card or benchmark table are conditional on. The paper assumes familiarity with standard evaluation statistics but does not require expertise in any particular detection technique.

Authors’ abstract

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions within the QA dataset. Then, some automated labeling strategy is used to label these answers as hallucinated or not by comparing them with the reference answers in the dataset. This evaluation setting creates a methodological ambiguity between two criteria: reference faithfulness (whether the answer is fully supported by the reference) and factual correctness (whether the answer is free from contradictions and factually false specific claims). In practice, automated labelers may apply the former criterion even when the intended target is the latter. We study this potential criterion mismatch using 900 human-labeled question-answer pairs spanning three commonly used QA datasets and three generator models, with labels targeting answer-level factual correctness. We evaluate lexical similarity metrics, a reference-entailment NLI baseline, and seven LLM judges under controlled prompt variants as automated labelers. Our experiments reveal substantial disagreement both among automated labeling strategies and between these labels and human annotations. Many strategies also exhibit strong directional error biases, and for most judge-generator pairs, replacing a faithfulness-oriented prompt with a factual-correctness prompt improves agreement with human annotations and reduces false-positive dominance, indicating that automated hallucination labels depend strongly on how the target criterion is specified. Label-source choice should therefore be considered a fundamental part of benchmark design and made explicit, validated, and matched with the benchmark goal.