Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

AI summary

Overview

Research area: Natural Language Processing — evaluation and benchmarking of hallucination detection for large language models.

Technical level: Intermediate. Readers should be comfortable with concepts such as span-level annotation, F1 and IoU metrics, and black-box versus white-box detector access, but no mathematical background beyond that is required.

Scope: A meta-study that re-frames hallucination detection as a benchmark-conditioned measurement problem, organizing benchmarks along provenance, typology, and granularity, systematizing detectors, and empirically testing how metric choice changes span-level localization scores on Mu-SHROOM.

What This Paper Is About

Papers report "hallucination detection" scores as if they measured one shared capability, but the benchmarks behind those scores differ in how model outputs are collected, what error categories the labels encode, and what unit (whole response, span, claim, or multiple levels) is annotated and scored. The authors argue that a detector score characterizes a detector–benchmark pair rather than the detector alone, and that headline numbers from different benchmarks are therefore not directly comparable. The goal is to give researchers a common framework for interpreting existing results and to make explicit which annotation variations evaluation should and should not penalize.

Key Contributions

1. A shared analytical framework for hallucination detection built on three axes — provenance (how outputs and evidence conditions are obtained), typology (what failure distinctions the labels encode), and granularity (the unit at which hallucinations are annotated, predicted, and evaluated).
2. A comprehensive analysis of modern hallucination benchmarks, covering their construction, annotation protocols, evidence regimes, label taxonomies, and evaluation metrics, mapped onto the three axes.
3. A systematization of hallucination detectors by model access, signal source, detection mechanism, output granularity, transferability through a separate inspector model, and supervision/calibration requirements, related back to the benchmark conditions under which each is measured.
4. An empirical study of metric choice in span-level evaluation, showing how annotator boundary variation changes localization scores, plus a proposed complementary metric, Span Coverage F1, for containment-based matching.

Main Findings

• Scores measure a detector–benchmark pair, not a detector. Formally, a benchmark is defined by its instance distribution, provenance, granularity, label space, annotation rule, and metric; a detector's score is only comparable across benchmarks when target semantics, granularities, label spaces, alignment procedures, metrics, and instance distributions are sufficiently compatible.

• Granularity changes the prediction target. Response-level benchmarks assign one judgment to a full generation and evaluate whole-output discrimination, not localization. Span-level benchmarks vary further in resolution (character, word/token, entity, sentence). Claim-level benchmarks make decomposition part of the effective alignment, so results under different decomposition policies are not interchangeable even with similar verification metrics. Multi-level benchmarks can conflate localization, classification, and correction in a single scalar.

• Boundary disagreement in span annotation is common. In the English Mu-SHROOM test set (154 examples, 1,665 annotator-level span sets), 1,403 of 6,280 non-empty, non-identical annotation pairs (22.3%) are boundary-nested while preserving the same number of connected components.

• Metric choice produces opposite conclusions on the same predictions. Under leave-one-annotator-out evaluation, moving from a majority reference to a broader union reference raises Span Coverage F1 from 0.410 to 0.750 (+0.340) but lowers exact span F1 from 0.288 to 0.071 (−0.217), DetEval-1D from 0.574 to 0.239 (−0.335), character F1 from 0.653 to 0.568 (−0.085), character IoU from 0.561 to 0.462 (−0.099), and CLEval-1D from 0.626 to 0.559 (−0.067).

• The pattern holds under micro aggregation. Span Coverage F1 rises from 0.369 to 0.713, DetEval-1D falls from 0.534 to 0.198, and character IoU stays nearly unchanged (0.455 versus 0.461).

• Containment is real but narrowly scoped. On the nested-boundary subset (n = 376), Span Coverage F1 gives full credit by construction (1.000), versus 0.281 for DetEval-1D, 0.605 for CLEval-1D and character F1, 0.503 for character IoU, and 0.032 for exact span F1. It drops to 0.347 for partial overlaps and to zero for disjoint annotations, indicating the high score reflects containment specifically rather than general tolerance of disagreement.

• Containment has a failure mode. In a 1-character stress test, predicting one character inside each reference span covers only 3.1% of reference characters yet yields Span Coverage F1 = 1.000, while character IoU is 0.031. The authors therefore recommend reporting Span Coverage F1 alongside an overlap-sensitive metric such as character F1 or character IoU.

• Provenance shapes the error distribution. Natural benchmarks collect outputs under the intended task setting (e.g., RAGTruth, SHROOM, FRANK, and others); semi-natural benchmarks keep genuine model errors but arrange conditions to induce them (PsiloQA withholds required context, HaluEval-Wild selects failure-prone user queries); synthetic benchmarks create errors or adverse evidence directly (HADES perturbs source text, FAVA-Bench injects factual errors, FaithEval builds inconsistent, unanswerable, and counterfactual contexts, HaluEval generates and filters hallucinated responses). Synthetic performance does not necessarily transfer to natural failures.

• Typologies vary from binary to fine-grained, with many-to-many correspondences. Binary schemes (SHROOM, HaluEval) make contradictions, unsupported additions, entity substitutions, and invalid inferences observationally equivalent. Evidence-relation schemes (RAGTruth's Evident/Sublte Conflict and Evident/Subtle Baseless Information, ANAH's four-way scheme, FactCheckBench, FaithBench) and task-specific taxonomies (FRANK, FAVA-Bench, TofuEval, FELM, DiaHalu, HILT) recur in recognizable families — conflict-like errors, unsupported additions, local semantic errors, and discourse/reasoning failures — but the correspondences are many-to-many, not exact equivalences.

• Detector access and supervision trade off against portability. Black-box detectors are portable but may incur judge calls, retrieval, or repeated generation, and differ in whether they use evidence at all. White-box detectors use uncertainty, hidden representations, attention, or activation geometry, but require open models and are often tied to a specific architecture, layer, or checkpoint. Increasing supervision tends to improve in-distribution accuracy while reducing portability.

• Transfer across granularity is asymmetric. Localized predictions can usually be aggregated up to sentence or response level, but a response-level decision cannot be converted into faithful spans or claims without an additional localization or decomposition stage. Proxy inspection through a separate open-weight inspector model replaces the generator's internal signal and should be reported explicitly.

• No cross-benchmark leaderboard is reported. The paper offers a qualitative and conceptual synthesis of benchmarks and detectors; it does not report an aggregated quantitative comparison of detector performance across benchmarks.

Methodology in Plain English

The authors treat hallucination as a property of an evaluated setup — the input, the generated output, the available evidence, and the evaluation unit — rather than a property baked into a model. They write the reported score as an expectation over a benchmark's instance distribution, with an explicit "alignment" step that maps whatever a detector outputs (a probability, a set of spans, a set of claim verdicts) onto whatever the benchmark requires (a binary label, spans, claims). This notation lets them show that granularity affects the target and the alignment, provenance affects the distribution of examples, and typology affects the label space and annotation rule — three separate reasons why two scores can differ.

They then read across the benchmark literature, placing each benchmark on the three axes and cataloguing its annotation scheme and metrics, and separately catalogue detectors by access, signal, output unit, transferability, and supervision.

For the empirical part, they use the multi-annotator English Mu-SHROOM test set, exploiting the fact that it retains several span sets per output. They treat one annotator's spans as the prediction and the others as the reference, built either as a strict character-level majority (a narrower consensus) or as a union (a broader envelope). The same held-out prediction is thus scored as the reference changes. They compare six metrics spanning four evaluation semantics: exact span F1, character F1, character IoU, one-dimensional adaptations of DetEval and CLEval (with DetEval's 80% reference / 40% prediction coverage conditions and 0.8 credit for accepted splits and merges, and CLEval's 0.3 area-precision threshold), and their proposed Span Coverage F1. They also break results down by the relation between prediction and reference — nested boundary, partial overlap, and disjoint.

Why This Matters

Impact on research. The paper gives reviewers and authors a vocabulary for stating what a reported score actually measures, and shows concretely that "stricter versus more permissive" is the wrong way to rank span metrics: they encode different invariances and answer different questions. It motivates evaluating generalization across benchmark conditions rather than chasing higher in-benchmark scores.

Real-world applications:

• Retrieval-augmented generation: catching claims that are not grounded in retrieved context before they reach a user, where the relevant evidence is the retrieved material rather than world knowledge.
• Agentic and multi-agent systems: flagging unfaithful intermediate outputs that would otherwise be passed to downstream tools, stored in memory, or reused by other agents, letting a local error propagate into system-level failure.
• Output filtering and model comparison: response-level judgments remain useful for gating and for comparing models, provided their aggregation protocol is stated.
• Domain review workflows: typed error labels in summarization, dialogue, and knowledge-grounded settings (e.g., entity, relation, contradiction, and reasoning categories) support actionable diagnosis rather than a single pass/fail.

Industry relevance. Teams deploying detectors on closed commercial models depend on the black-box portability the paper discusses, while teams with open models may use white-box signals that require architecture-specific access. Because detector scores shift with the benchmark's provenance (natural, semi-natural, synthetic), practitioners need to know whether a vendor-reported number came from stress-test conditions or deployment-like data before trusting it.

Future Directions

1. Controlled generalization studies that measure detector robustness across data construction, error type, annotation unit, and metric semantics, instead of reporting single-benchmark scores.
2. Explicit invariance declarations in evaluation protocols — stating which annotation variations a metric is meant to tolerate and which count as detector errors.
3. Better metrics for non-unique span boundaries, since Span Coverage F1 accepts contained refinements but fails the 1-character stress test, showing no single metric captures all aspects of localization quality.
4. Standardized reporting of alignment and supervision — thresholding, aggregation rules, calibration protocols, decomposition policies, and whether white-box signals come from the generator or a proxy inspector — so that results from different setups can be compared meaningfully.

Target Audience

Researchers and practitioners who build, benchmark, or procure hallucination detectors: benchmark and evaluation-metric designers, NLP researchers working on factuality and faithfulness, and engineering teams deploying LLM systems in retrieval-augmented or agentic pipelines who need to interpret reported detection scores correctly. It is also a useful reference for reviewers assessing claims of detector superiority across benchmarks.

Authors’ abstract

Hallucination detection is increasingly important for evaluating and deploying large language models. However, existing methods are often compared across benchmarks that define and measure hallucination in substantially different ways, making reported scores difficult to interpret across settings. We study hallucination detection as a benchmark-conditioned measurement problem and organize existing benchmarks along three axes: provenance, which describes how model outputs and evidence conditions are obtained; typology, which captures the factuality and faithfulness failures represented by benchmark labels; and granularity, which specifies the unit at which hallucinations are annotated, predicted, and evaluated. Using this framework, we analyze how benchmark construction, annotation, and evaluation choices shape the underlying prediction task, and relate detector mechanisms and assumptions to the settings in which their performance is measured. We also examine span-level evaluation, where variation in annotation boundaries can substantially affect reported results, and propose Span Coverage F1 as a complementary containment-based metric. Our analysis clarifies what detector scores capture, highlights limitations of cross-benchmark comparisons, and motivates more explicit evaluation of generalization across benchmark conditions.