What do Geometric Hallucination Detection Metrics Actually Measure?

AI summary

Overview

• Research area: Hallucination detection in large language models using geometric statistics of internal model states (residual-stream hidden representations and attention maps).
• Technical level: Intermediate. Readers should be comfortable with LLM internals (hidden states, attention maps, layers), Gram matrices, entropy, and AUROC as an evaluation metric.
• Scope: This paper is a controlled, synthetic-data interrogation of what three popular geometric hallucination detector scores—Hidden Score (HS), Matrix Entropy (ME), and Attention Score (AS)—actually measure, and a proposed normalization that fixes their cross-domain failure.

What This Paper Is About

Hallucination detectors built on model internals typically produce a single number and treat "hallucination" as one undifferentiated category, even though a hallucination can be caused by factual incorrectness, under-confidence, irrelevance, incoherence, or incompleteness. This paper asks which specific property of a hallucination each geometric statistic is actually sensitive to, and how stable those sensitivities are across task domains. To answer this, the authors generate responses that vary each property independently at three severity levels across math, history, and counting questions, then measure the resulting HS, ME, and AS scores.

Key Contributions

1. A synthetic hallucination dataset with independent control of hallucination type and severity. The dataset contains 550 baseline correct QA pairs (400 math, 70 history, 80 counting) that are transformed into prompt–response pairs exhibiting each of five hallucination properties at three severity levels.

2. A systematic characterization of which geometric statistic responds to which hallucination property. The authors evaluate HS, ME, and AS across all hallucination types, severities, and domains, showing that different statistics capture different flavors of hallucination rather than a single shared notion of "hallucination."

3. Identification of domain shift as a first-order obstacle. Domain is shown to determine the optimal recording layer more than the choice of metric does, and combining domains collapses detection performance (e.g., HS and AS drop to AUROCs of 0.57 and 0.60 on the combined "all" set at level-3 incorrectness).

4. A perturbation normalization method with large AUROC gains. By computing each statistic relative to perturbed versions of the same response, the authors report AUROC gains of +34 to +40 points in multi-domain settings.

Main Findings

• Different statistics detect different hallucination types. HS and AS respond to irrelevant responses as hallucinations (AUROCs of 0.94 and 0.96 on the combined "all" set), while ME does not. Conversely, ME is highly receptive to incoherent responses, achieving near-perfect AUROC, while HS and AS are not.

• HS and AS invert their signal on incoherence. HS and AS produce lower scores for incoherent responses than for correct baselines—the opposite direction from factual incorrectness—so a "greater than or equal to" threshold tuned for incorrectness cannot detect incoherence. Their AUROCs for incoherence on "all" are 0.03 and 0.08.

• More severe hallucinations are easier to detect. Across statistics and domains, moving from level 1 to level 3 incorrectness improves AUROC.

• The best statistic depends on the domain. For math, HS and ME (the statistics computed from the Gram matrix of hidden states) perform best, reaching AUROC 0.92 at level 3. For history and counting, AS is the best detector, with AUROCs of 0.75 and 0.65 at level 3.

• Optimal recording layer is driven by domain, not by metric. Layers 30–31 are best for math, while layers 14–16 are best for history and counting across all three statistics.

• All statistics are only weakly aligned with verbal confidence. Under-confidence detection yields AUROCs of 0.59–0.69 on the "all" set.

• Domain shift severely degrades multi-domain detection. Although HS and AS perform well within individual domains for factual incorrectness, their combined "all" AUROCs at level 3 fall to 0.57 (HS) and 0.60 (AS). Domain-specific score distributions run parallel across layers, and HS/ME variance is much larger on counting than on math.

• Attention Score is the least domain-sensitive statistic. The authors conjecture this is because attention maps are normalized through the softmax function.

• Perturbation normalization recovers most of the lost performance. On the "all" set with level-1 hallucinations, HS-Norm and ME-Norm reach AUROC 0.96 (a 40-point gain) at layer 30, and AS-Norm reaches 0.89 (a 34-point gain) at layer 31. The normalization also reduces intra-domain variance, improving single-domain detection.

Methodology in Plain English

The authors do not wait for a model to hallucinate on its own. Instead they build a templated generator: a prompt question, a response question, an optional confidence modifier, an answer, and an answer offset. Starting from 550 known-correct question–answer pairs, they alter individual template slots to manufacture one hallucination property at a time:

• Factual incorrectness: shift the answer by a controlled offset (e.g., ±[1,9], ±[10,99], ±[100,999] for math).
• Under-confidence: insert phrases like "probably," "maybe," or "not."
• Irrelevance: mismatch the prompt question from the response question, with cross-domain mismatches representing the most severe case.
• Incoherence: repeat the response with inconsistent answers two, three, or four times.
• Incompleteness: truncate the response with an end-of-text token at 90%, 80%, or 70% of its length.

They feed each prompt–response pair into Llama 3.1-8B-Instruct using teacher forcing on a single Nvidia H100 GPU, recording hidden states and attention maps for every token across all 32 layers. From these they compute Hidden Score (a sequence-normalized log determinant of the hidden-state Gram matrix, interpretable as the log volume of the parallelepiped spanned by token representations), Matrix Entropy (a Shannon-entropy version of the eigenvalue spectrum of the same Gram matrix), and Attention Score (a sequence- and head-normalized log determinant of the attention maps, which informally measures how much tokens attend to themselves). Each statistic's AUROC for separating hallucinated from correct responses is measured across layers.

For the normalization fix, they locate the answer tokens, generate k perturbed versions of the response by adding −5, −2, −1, 1, 2, and 5 to the answer, compute the statistic on each perturbed version, and z-score the original statistic against the perturbed distribution. The intuition is that a correct answer should look like an outlier relative to its perturbed neighbors, while an incorrect answer should not.

Why This Matters

Impact on research. The paper reframes hallucination detection as a multi-property problem rather than a binary one. It provides evidence that geometric statistics are not interchangeable detectors, that aggregate AUROC numbers can hide domain-dependent behavior, and that a subset of the reported gains in prior geometry-based detection work may be artifacts of evaluating within a single domain. The perturbation-normalization trick is cheap and plugs into any statistic that operates on hidden states or attention maps.

Real-world applications.

• General-purpose chatbots and assistants that must handle arbitrary question domains, where a detector tuned on one domain silently underperforms on another.
• Retrieval-augmented and enterprise QA systems that need a lightweight, ground-truth-free hallucination signal because external verification is unavailable or expensive.
• High-stakes question answering in medicine, law, or finance, where the failure mode may be under-confident hedging or incoherent multi-part answers rather than a clean factual error.
• Model monitoring and evaluation pipelines that flag incoherent multi-sentence generations—a regime where HS and AS fail and ME succeeds, making statistic choice consequential.

Industry relevance. Internal-state detectors are attractive because they are computationally cheap and require no external knowledge at inference time. This paper shows that cheapness comes with a domain-alignment requirement that must be engineered for, and it supplies a concrete engineering fix. It also gives practitioners a vocabulary for choosing which statistic to deploy based on the hallucination failure mode they care about.

Future Directions

• Validate against naturally occurring hallucinations. The current study uses teacher forcing on programmatically constructed responses as a surrogate; whether the same statistic sensitivities hold for responses the model actually generates is an open question.
• Automate the answer-token localization and perturbation-set design. The paper notes that identifying the exact answer tokens is simple in its synthetic setting but would require an auxiliary LLM in real-world use, and that the perturbation set here (−5, −2, −1, 1, 2, 5) is domain-specific.
• Test generalization beyond this model, scale, and domain set. All experiments use Llama 3.1-8B-Instruct with 32 layers, three numeric-answer domains, and English templates.
• Build combined or multi-property detectors. Since HS/AS capture irrelevance and incompleteness while ME captures incoherence, a detector that fuses statistics or explicitly models the hallucination taxonomy may outperform any single statistic.

Target Audience

Researchers and engineers working on LLM reliability, hallucination detection, and mechanistic interpretability, particularly those using hidden-state or attention-map probes. It is also relevant to evaluation practitioners who report cross-domain detection numbers, and to applied teams at organizations deploying foundation models in settings where external ground truth is unavailable at inference time.

Authors’ abstract

Hallucination remains a barrier to deploying generative models in high-consequence applications. This is especially true in cases where external ground truth is not readily available to validate model outputs. This situation has motivated the study of geometric signals in the internal state of an LLM that are predictive of hallucination and require limited external knowledge. Given that there are a range of factors that can lead model output to be called a hallucination (e.g., irrelevance vs incoherence), in this paper we ask what specific properties of a hallucination these geometric statistics actually capture. To assess this, we generate a synthetic dataset which varies distinct properties of output associated with hallucination. This includes output correctness, confidence, relevance, coherence, and completeness. We find that different geometric statistics capture different types of hallucinations. Along the way we show that many existing geometric detection methods have substantial sensitivity to shifts in task domain (e.g., math questions vs. history questions). Motivated by this, we introduce a simple normalization method to mitigate the effect of domain shift on geometric statistics, leading to AUROC gains of +34 points in multi-domain settings.