The Map of Misbelief: Tracing Intrinsic and Extrinsic Hallucinations Through Attention Patterns

AI summary

Overview

Research area: Hallucination detection and uncertainty quantification in large language models, with a focus on attention-based interpretability.

Technical level: Intermediate. Readers should be comfortable with transformer attention maps, token-level probabilities, and standard classification metrics such as AUROC.

Scope: The paper builds a type-aware evaluation framework that separates extrinsic from intrinsic hallucinations, and introduces attention-aggregation variants of the RAUQ uncertainty propagation algorithm, evaluated across six open-source LLMs and several curated QA benchmarks.

What This Paper Is About

LLMs frequently produce fluent but factually wrong or context-contradicting text, and most existing detection methods treat all hallucinations as one undifferentiated failure. This paper separates hallucinations into two categories, extrinsic (outputs unsupported by training knowledge) and intrinsic (outputs that contradict or overstep the supplied input context), and asks whether different detection methods suit different types. The goal is to determine which uncertainty signals work for which hallucination type, and to improve a lightweight attention-based method so that it performs well on both.

Key Contributions

1. New attention aggregation strategies for uncertainty quantification. The authors propose token-level variants (All-Past-Tokens, Input-Tokens) and head-level variants (Mean Across Heads, Attention Rollout with two token spans) on top of the RAUQ algorithm. Some variants improve extrinsic detection and others intrinsic detection, while also being more interpretable than the original attention-guided method.

2. A structured benchmarking protocol that separates hallucination types. The framework evaluates detection using curated benchmarks for extrinsic hallucination (HalluLens: PreciseWikiQA and NonExistentRefusal-MixedEntities) and intrinsic hallucination (SQuAD-v2 unanswerable instances, FaithEval counterfactual and inconsistent subsets), scored with AUROC, AURAC, and PRR.

3. Extensive experiments across six open-source LLMs. The models evaluated are LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Falcon-10B-Instruct, Gemma-2-9B-It, Qwen-2.5-7B-Instruct, and Mistral-Nemo-Instruct-2407, each tested on every dataset.

4. Evidence that sampling-based detectors fail on intrinsic hallucinations. The paper shows that leading sampling-based methods such as Semantic Entropy underperform on intrinsic errors caused by underspecification, motivating attention-based alternatives.

Main Findings

• Attention-based propagation beats probability-only baselines overall. Over the concatenation of all datasets, RAUQ with the proposed variants achieves the highest AUROC. The Rollout, Rollout All, Mean-Heads All, and Mean-Heads Input variants outperform the original RAUQ baseline.

• Semantic Entropy is the strongest detector of extrinsic hallucinations. On PreciseWikiQA and NonExistentRefusal-MixedEntities, Semantic Entropy surpasses both attention-based and probability-based approaches, which the authors attribute to higher response variability when factual knowledge is missing.

• Sampling-based methods fail on intrinsic hallucinations. On SQuAD-v2-unanswerable, FaithEval-counterfactual, and FaithEval-inconsistent, sampling methods including Semantic Entropy perform poorly, which the authors interpret as contradictory inputs producing consistently structured rather than diverse outputs.

• Attention over input tokens works best for intrinsic hallucinations. Variants that aggregate attention over the input or over all past tokens achieve the best intrinsic detection performance.

• A full-dataset table on FaithEval-inconsistent shows large gaps. For Qwen-2.5-7B-Instruct on FaithEval-inconsistent-v1.0, RAUQ Input scores AUROC 0.7222, AURAC 0.4669, PRR 0.4370, with RAUQ All at 0.7222 / 0.4668 / 0.4369, versus Semantic Entropy at 0.4692 / 0.3303 / 0.0025 and EigenScore at 0.4687 / 0.2860 / -0.0253. The average AlignScore on that dataset is 0.388.

• All-Past-Tokens and Input-Tokens variants are nearly identical. The authors attribute this to generally low attention magnitudes and suggest the two could be unified into a single averaging strategy.

• Attention-based methods are an order of magnitude cheaper. Because they require a single pass through the model's attention maps, RAUQ variants reduce computational cost by an order of magnitude compared to Semantic Entropy and EigenScore, which need multiple forward passes.

• Internal uncertainty does not always translate into correct behavior. In histograms of RAUQ Mean Heads scores for Mistral-7B, nonexistent-entity queries score high, answerable queries cluster low, and SQuAD-v2 unanswerable questions fall in between. Answerable examples carry an AlignScore of 0.86, while unanswerable ones still yield an AlignScore of only 0.27, showing a gap between the internal signal and surface behavior.

• Intrinsic detection is less stable. The RAUQ Mean_Heads signal for Mistral becomes less distinguishable when viewed within SQuAD-v2 rather than across datasets, which the authors say contributes to the variance across models.

• Model-level breakdown varies by architecture. Table 2 reports AUROC / AURAC / PRR for every method on each of the six models, with the best AUROC per model highlighted in bold. For LLaMA-3.1-8B-I, for example, EigenScore scores 0.7868 / 0.7817 / 0.5671 and RAUQ All RO scores 0.8016 / 0.7896 / 0.5967.

Methodology in Plain English

The authors start from RAUQ, a recent method that uses attention weights inside a language model as a confidence signal. RAUQ rests on an observation that when a model is about to produce something wrong, the attention from each new token back to the immediately preceding token drops in a small number of "uncertainty-aware" heads. RAUQ picks one such head per layer, uses it to propagate confidence from one generated token to the next, and turns the result into a per-layer uncertainty score.

The paper changes two things about this pipeline. First, instead of looking only at the immediately previous token, they consider other spans: averaging attention over all previously generated tokens, and averaging attention over just the input tokens. The input-token variant is grounded in the intuition that tasks like extractive QA, summarization, and retrieval-augmented generation require the output to be grounded in the source, so weak attention back to the source suggests the model is relying on internal priors.

Second, instead of selecting a single head per layer, they either average across all heads or use attention rollout, which recursively multiplies head-averaged attention matrices across layers (with an identity term to simulate residual connections).

Hyperparameter α, which balances a token's own probability against the propagated signal, was tuned by grid search over [0.1, 0.2, ..., 0.9] to maximize AUROC on the curated benchmarks (HalluLens, SQuAD-v2, FaithEval).

For baselines, they generate 10 samples per prompt using nucleus sampling at temperature 1.0 and one deterministic greedy response at temperature 0.1. Response correctness is judged mainly with AlignScore, except for NonExistentRefusal-MixedEntities, which was manually annotated (score 1 for an explicit expression of not knowing the entity, 0 for fabricated information). Performance is measured with AUROC, AURAC, and PRR. Rather than averaging metrics across datasets, the authors concatenate all instances and compute one set of metrics per method, which they argue avoids over-weighting smaller benchmarks. Dataset sizes are not reported in the provided content.

Why This Matters

Impact on research: The paper argues that lumping all hallucinations together hides a real trade-off. Sampling-based entropy methods excel when the model lacks knowledge, while attention-based methods excel when the input is contradictory or underspecified. It also connects intrinsic hallucination to the concept of underspecification uncertainty, giving a theoretical hook for why input-contradicting errors look different from knowledge-gap errors. The result that a single forward pass can compete with multi-sample methods matters for making detection practical.

Real-world applications:

• Extractive question answering, where the model must answer only from a supplied passage and must refuse when no answer span exists.
• Summarization, where an output that contradicts the source document is a faithfulness failure.
• Retrieval-augmented generation, where grounding the generation in retrieved context is the central requirement.
• Open-domain factual question answering, where the model must recognize when a query concerns an entity it has no knowledge of.

Industry relevance: Detection methods that need only one forward pass are far cheaper to run at scale than methods requiring many sampled generations, which is directly relevant to deployment in safety-critical settings. The paper's framing suggests that production systems could route different hallucination checks depending on whether the risk is factual ignorance or ungrounded generation.

Future Directions

• Testing beyond question answering. The authors state that the effectiveness of their framework in complex reasoning tasks and real-world scenarios remains largely unexplored, and call for examining whether internal representations and attention patterns stay reliable as confidence signals in more intricate tasks.
• Improving the stability of intrinsic detection. Intrinsic performance drops when the uncertainty signal is examined within a dataset rather than across datasets, and this contributes to variance across models, so a more robust intrinsic signal is needed.
• Unifying redundant aggregation strategies. Because All-Past-Tokens and Input-Tokens behave almost identically due to low attention magnitudes, the authors suggest merging them into a single averaging strategy.
• Combining detection strategies by hallucination type. Since no single method dominates both categories, an open question is whether attention-based and sampling-based signals can be combined or routed adaptively based on the type of hallucination risk.

Target Audience

Researchers and practitioners working on LLM reliability, uncertainty quantification, and hallucination detection. It is most useful to readers who already understand transformer attention and evaluation metrics for detection, and to engineers building grounded generation systems who need a cheap, single-pass alternative to multi-sample consistency checks. Readers looking for a precise taxonomy of hallucination types and a benchmark protocol that respects that taxonomy will benefit most.

Authors’ abstract

Large Language Models (LLMs) are increasingly deployed in safety-critical domains, yet remain susceptible to hallucinations. While prior works have proposed confidence representation methods for hallucination detection, most of these approaches rely on computationally expensive sampling strategies and often disregard the distinction between hallucination types. In this work, we introduce a principled evaluation framework that differentiates between extrinsic and intrinsic hallucination categories and evaluates detection performance across a suite of curated benchmarks. In addition, we leverage a recent attention-based uncertainty quantification algorithm and propose novel attention aggregation strategies that improve both interpretability and hallucination detection performance. Our experimental findings reveal that sampling-based methods like Semantic Entropy are effective for detecting extrinsic hallucinations but generally fail on intrinsic ones. In contrast, our method, which aggregates attention over input tokens, is better suited for intrinsic hallucinations. These insights provide new directions for aligning detection strategies with the nature of hallucination and highlight attention as a rich signal for quantifying model uncertainty.