Two Pathways to Truthfulness: On the Intrinsic Encoding of LLM Hallucinations

AI summary

Overview

Research area: Mechanistic interpretability and hallucination detection for large language models (NLP).

Technical level: Intermediate — the paper assumes familiarity with transformer attention, hidden-state probing, and linear classifiers, but its central ideas are explained in accessible terms.

Scope: This paper dissects how LLMs internally encode whether their own generated answers are truthful, identifies two distinct internal information pathways, and turns that finding into two hallucination-detection methods.

What This Paper Is About

Prior work showed that an LLM's hidden representations contain signals that reveal whether its own output is hallucinated, and that a simple linear classifier (a "probe") trained on those representations can detect hallucinations. What was missing was an explanation of where those signals come from. This paper traces the signals to two separate internal routes — one that depends on information flowing from the input question to the answer, and one that draws evidence from the generated answer itself — and then exploits the difference to improve detection.

Key Contributions

1. Mechanism. A systematic investigation into how internal truthfulness signals arise and operate, revealing two distinct information pathways: a Question-Anchored (Q-Anchored) pathway that relies on question-answer information flow, and an Answer-Anchored (A-Anchored) pathway that derives self-contained evidence from the LLM-generated output.

2. Discovery. Large-scale experiments across multiple datasets and model families identify two properties of these mechanisms: (i) an association with LLM knowledge boundaries, and (ii) intrinsic self-awareness of pathway distinctions within the models' internal states.

3. Application. Two pathway-aware detection methods — Mixture-of-Probes (MoP) and Pathway Reweighting (PR) — that exploit the complementary nature of the two mechanisms and are reported to achieve up to a 10% AUC gain over competing approaches.

Main Findings

• Bimodal saliency distribution. Kernel Density Estimation of saliency scores for question-to-answer information flow on TriviaQA and Natural Questions shows a clear bimodal distribution: one peak near zero saliency and another at a substantially higher value. This motivated the two-pathway hypothesis.

• Attention knockout bifurcates predictions. Blocking attention from exact question tokens to subsequent positions (A_l(i, E_Q) = 0 for layers 1..k) causes prediction probabilities to shift substantially for one subset of instances and to remain nearly unchanged for another, across all layers examined. Instances whose prediction flips are labeled Q-Anchored; the rest, A-Anchored. The separation is reported as statistically robust with 95% confidence intervals.

• Token patching confirms the split. Injecting hallucinatory cues by replacing a context sample's exact question tokens with those from a random patch sample produces a significantly higher prediction flip rate for Q-Anchored samples than for A-Anchored samples. Patching random tokens yields much lower flip rates than patching exact question tokens.

• A-Anchored signals survive question removal. When the question is deleted and the probe is fed answer-only hidden states, Q-Anchored instances shift substantially in prediction probability while A-Anchored instances remain largely invariant — evidence that A-Anchored encoding uses self-contained information in the generated answer.

• The pathways track knowledge boundaries. Q-Anchored samples achieve significantly higher answer accuracy and involve more popular entities; A-Anchored samples skew toward less popular, long-tail factual knowledge (entity frequency analysis shown for PopQA). The I-don't-know rate in the appendix follows trends consistent with answer accuracy.

• Models are self-aware of which pathway they use. Probes trained on the models' original internal states predict the pathway. Table 2 reports AUCs: Llama-3-8B 87.80 (PopQA), 75.10 (TriviaQA), 86.31 (HotpotQA), 78.31 (NQ); Llama-3-70B 92.66, 83.91, 87.34, 84.14; Mistral-7B-v0.3 87.64, 85.87, 92.13, 84.83.

• Pathway-aware detection beats baselines. In Table 3, PR scores highest on both models shown: Llama-3-8B 94.01 (PopQA), 83.13 (TriviaQA), 87.81 (HotpotQA), 79.10 (NQ); Mistral-7B-v0.3 93.09, 84.36, 89.03, 79.09. MoP scores 92.11, 81.18, 85.45, 74.64 and 91.66, 83.57, 85.82, 76.87 respectively, against a probing baseline of 88.71, 77.58, 82.23, 70.20 and 87.39, 81.74, 83.19, 73.60.

• Ablations matter. MoP-RandomGate (randomly routing experts) and MoP-VanillaExperts (two vanilla probes as a plain ensemble) both degrade substantially relative to MoP, indicating that pathway specialization and self-awareness gating both contribute.

Methodology in Plain English

The analysis centers on "exact tokens" — the core question tokens (typically the subject and property) and the core answer tokens, identified with semantic frame theory and located using GPT-4o (gpt-4o_2024-11-20), which reached 99.92%, 95.83%, and 96.62% accuracy on exact subject, exact property, and exact answer tokens respectively.

Two interventions do the heavy lifting. First, attention knockout: the researchers zero out attention from the question tokens to everything after them, up to the layer where the probe was trained, then see whether the probe's verdict changes. Second, token patching: they swap a factual instance's exact question tokens for tokens from another example, artificially injecting a hallucinatory cue, and measure how often the probe flips. Together these separate instances that depend on the question from instances that don't. A third test deletes the question entirely and probes the answer alone.

The work covers 12 LLMs — base models (Llama-3.2-1B, Llama-3.2-3B, Llama-3-8B, Llama-3-70B, Mistral-7B-v0.1, Mistral-7B-v0.3), instruction-tuned models (Llama-3.2-3B-Instruct, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.1, Mistral-7B-Instruct-v0.3), and reasoning-oriented models (Qwen3-8B, Qwen3-32B) — on four datasets: PopQA, TriviaQA, HotpotQA, and Natural Questions, with 2,000 training and 2,000 test samples sampled per dataset. Probes are logistic regression models from scikit-learn. The two proposed methods build on these probes: MoP combines a Q-Anchored expert and an A-Anchored expert using the self-awareness probe as a gating coefficient in a convex combination requiring no extra training, while PR adds small learnable scalars that rescale attention from exact answer tokens to exact question tokens, applied only during detection and omitted during generation.

Why This Matters

Impact on research. The paper moves hallucination research past the observation that truthfulness is encoded internally to an account of how it is encoded, distinguishing two pathways with different dependencies. It also connects a representational property (self-awareness of pathway identity) to a mechanistic one.

Real-world applications (implied by the paper's framing):

• Safer deployment of question-answering and assistant systems that need to flag unreliable answers before they reach a user.
• Retrieval-free fact-checking and answer verification pipelines that work from a model's own internal signals rather than external knowledge bases.
• Routing or triage in long-tail, low-resource fact domains, where the paper finds A-Anchored behavior dominates.
• Diagnostic tooling for model developers who need to know whether a failure stems from missing parametric knowledge or from faulty use of provided context.

Industry relevance. The path from detection methods to production systems is direct, but the authors note a constraint: the approach assumes access to internal model representations, which may not be available in strictly black-box settings.

Future Directions

• Adapting the pathway-aware methods to strictly black-box settings where hidden states are unavailable, which the authors explicitly leave for future work and suggest may require additional engineering or alternative approximations.
• Extending the analysis beyond the exact-token configurations used here, and beyond the 12 models studied, to confirm generality.
• Exploiting the self-awareness finding for fine-grained improvements targeted at specific truthfulness encoding mechanisms, which the paper describes as a potential avenue.
• Investigating the pathway distinction in other model families and tasks, since the paper's conclusions rest on four question-answering datasets.

Target Audience

Researchers and engineers working on LLM interpretability, hallucination detection, and trustworthy generation — particularly those already familiar with probing classifiers and attention-level interventions. The paper is also useful to practitioners building answer-verification or reliability layers, though the limitation section makes clear that deployment requires access to internal representations. The appendix details on token selection, activation extraction, and baselines make it a practical reference for anyone reproducing internal-signal detection experiments.

Authors’ abstract

Despite their impressive capabilities, large language models (LLMs) frequently generate hallucinations. Previous work shows that their internal states encode rich signals of truthfulness, yet the origins and mechanisms of these signals remain unclear. In this paper, we demonstrate that truthfulness cues arise from two distinct information pathways: (1) a Question-Anchored pathway that depends on question-answer information flow, and (2) an Answer-Anchored pathway that derives self-contained evidence from the generated answer itself. First, we validate and disentangle these pathways through attention knockout and token patching. Afterwards, we uncover notable and intriguing properties of these two mechanisms. Further experiments reveal that (1) the two mechanisms are closely associated with LLM knowledge boundaries; and (2) internal representations are aware of their distinctions. Finally, building on these insightful findings, two applications are proposed to enhance hallucination detection performance. Overall, our work provides new insight into how LLMs internally encode truthfulness, offering directions for more reliable and self-aware generative systems.