HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs

AI summary

Overview

Research area: Machine learning / large language model reliability, specifically hallucination detection and the theoretical analysis of why LLMs hallucinate.

Technical level: Advanced. The paper builds on Neural Tangent Kernel (NTK) theory, Fréchet differentiability, Céa's lemma, and Freedman's inequality for martingales, and derives eigenvalue-sensitivity bounds. Readers need comfort with kernel methods and concentration inequalities to follow the derivations.

Scope: The paper defines a theoretical decomposition of hallucination risk into data-driven and reasoning-driven components, then introduces HalluGuard, an NTK-based detection score, and benchmarks it across 10 benchmarks, 11 baselines, and 9 LLM backbones.

What This Paper Is About

LLMs fail in two different ways: they can produce content rooted in flawed or incomplete knowledge absorbed during training (data-driven hallucination), or they can go wrong during multi-step decoding even when their knowledge is sound (reasoning-driven hallucination). Existing detectors usually target only one of these and rely on task-specific heuristics, and the paper argues that they also miss how hallucinations emerge and evolve over a generation. The authors build a unified bound that separates the two sources mathematically, then turn that bound into a single score that detects both.

Key Contributions

1. The Hallucination Risk Bound. A theoretical framework that formally decomposes total hallucination risk into a data-driven term (training-time mismatch) and a reasoning-driven term (inference-time instability), showing how the two interact and evolve across decoding steps.

2. An NTK-based detection score, HalluGuard. Defined as det(K) + log σ_max − log κ², combining NTK representational adequacy, rollout amplification, and spectral stability into a computable proxy for the theoretical bound.

3. A large-scale empirical evaluation. Tests across 10 benchmarks, 11 competitive baselines, and 9 popular LLM backbones, reporting state-of-the-art detection performance with the largest margins on reasoning-oriented tasks and smaller models.

4. A demonstration of downstream utility. The score is integrated into beam search at test time, improving accuracy on MATH-500 and Natural over IO prompting, and it is validated on a fine-grained hallucination case study over semantic data (Q5). The model is open-sourced.

Main Findings

• Hallucination type is task-dependent. On the instruction-following Natural benchmark, 88.9% of the 3499 errors are logical missteps (reasoning-driven) and 11.1% are factual inaccuracies (data-driven). On the math-focused MATH-500, of 1985 wrong generations, 1946 (98.1%) are reasoning errors and 19 (1.9%) are factual flaws.

• Data-driven hallucinations scale with mismatch, reasoning-driven ones scale with sequence length. The data-driven term is bounded by a conditioning ratio controlled by pretraining–finetuning mismatch and task signal strength; the reasoning-driven term grows exponentially in sequence length T via a factor α(e^{βT} − 1).

• Each NTK proxy tracks a different failure mode. In the paper's correlation analysis, det(K) correlates most strongly with the data-centric SQuAD (0.84), while log σ_max − log κ² correlates most strongly with the reasoning-oriented MATH-500 (0.88). On TruthfulQA the correlations are 0.61 and 0.67 respectively.

• State-of-the-art detection across task families. HalluGuard reaches 81.76% AUROC and 79.76% AUPRC on MATH-500 (improving on the second-best method by up to 8.3%), 84.59% AUROC and 81.15% AUPRC on RAGTruth (gains up to 7.7%), and 77.05% AUROC and 73.79% AUPRC on TruthfulQA (exceeding the next strongest baseline by as much as 6.2%).

• Biggest gains on smaller backbones. On HaluEval with Llama2-7B, HalluGuard achieves 72.89% AUPRC_r, more than 10% above the second best. Mid-sized models show clear gains (79.01% AUROC_r on GSM8K), and large models improve steadily (83.8% AUROC_r on SQuAD with Llama2-70B).

• Each term follows the ground-truth trend in its own task family. Ablations show the data-driven term tracking the gold curve on SQuAD as hallucination rate varies, and the reasoning-driven term mirroring the monotonic AUROC drop on MATH-500 as reasoning drift increases.

• The score improves test-time reasoning. Integrated into beam search, HalluGuard reaches 81.00% accuracy on MATH-500 with Qwen2.5-Math-7B (about 10% higher than IO Prompt) and 70.96% on Natural with Llama3.1-8B (15.72% above IO Prompt).

• No labels and no runtime cost during inference. The projection layers are self-supervised spectral calibration modules optimized offline with AdamW; the backbone stays frozen and adds zero inference overhead.

Methodology in Plain English

The authors start by defining a continuous "semantic space" where both the correct answer and the model's generated answer can be represented as vectors. Hallucination is then the distance between those two vectors. Using the triangle inequality, that distance splits cleanly into two pieces: how far the model's average output is from the truth (the data-driven part), and how far any single sampled output wanders from that average (the reasoning-driven part).

To bound the first piece, they borrow Céa's lemma and the geometry of the Neural Tangent Kernel. The intuition is that a well-conditioned kernel spectrum — one where the smallest eigenvalue isn't tiny relative to the overall operator — lets the model approximate the true generation closely. Poor conditioning, driven by a mismatch between pretraining and fine-tuning distributions or by weak task signal in the top eigenvalues, produces data-driven hallucinations.

To bound the second piece, they treat autoregressive decoding as a martingale and apply Freedman's inequality. Here the key quantity is how much each decoding step's Jacobian amplifies perturbations; if the per-step amplification compounds multiplicatively, error grows exponentially with the number of steps.

Directly computing step-wise Jacobians for billion-parameter models is intractable, so they derive three tractable proxies that stand in for the theoretical quantities: det(K) (how adequate the representation is), log σ_max (how much rollouts amplify error, where σ_max is the largest spectral norm of the per-step Jacobian), and −log κ² (a penalty for unstable, ill-conditioned spectra, where κ is the condition number). Adding them yields the HalluGuard score. The authors first validate empirically that each proxy correlates with the right task family, then run detection and score-guided decoding experiments.

Why This Matters

This work matters because it reframes hallucination detection from a collection of task-specific heuristics into a single decomposition grounded in learning theory. It argues that data-driven and reasoning-driven hallucinations are not separate problems to be solved with separate tools, and it provides a detector that reports on both simultaneously. Detection results are reported across 9 backbones from GPT-2 (117M) through Llama2-70B, QwQ-32B, Mistral-7B-Instruct, and OPT-6.7B.

Real-world applications (grounded in the domains the paper names):

• Healthcare. The paper's running example is a disease diagnosis that begins as a misclassification and evolves into a distorted recommendation, delaying treatment or risking fatality.
• Law. Cited as a highly regulated high-stakes domain where unfaithful output undermines compliance, interpretability, and safety guarantees.
• Scientific discovery. Also cited as a deployment domain where undetected errors undermine reliability and accountability.
• Test-time reasoning control. Using the score to steer beam search, as the paper does on MATH-500 and Natural, offers a path to catching errors during generation rather than after the fact.

Industry relevance: Because the calibration layers are trained offline and the backbone remains frozen with zero runtime overhead, the method is compatible with serving stacks that cannot afford extra decoding passes. The threshold-free AUROC/AUPRC framing also makes it usable as a monitoring signal rather than a binary gate.

Future Directions

• Extending to fine-grained hallucination beyond benchmarks (Q5). The paper poses this as its fifth research question and reports a case study on semantic data, but the truncated content does not include the Q5 results, so those findings are not reported here.
• Handling the intractable quantities. The bound depends on task- and model-dependent constants (H*, C, k, k_pt) and on step-wise Jacobians; the paper works around this with proxies, and tightening those proxies or estimating the constants more directly is a natural next step.
• Broadening the backbone and task coverage. The current evaluation covers 9 backbones and 10 benchmarks; whether the spectral calibration modules transfer to architectures and modalities not tested is an open question.
• Turning detection into correction. The test-time beam search experiment is the paper's only intervention experiment; whether the same signal can guide repair, not just reranking, remains open.

Target Audience

This paper is written for machine learning researchers working on LLM reliability, uncertainty quantification, and kernel methods, and for practitioners who need a hallucination detector that runs without external references, task-specific heuristics, or extra inference cost. Applied teams in regulated domains such as healthcare, law, and scientific computing will find the deployment properties relevant even if they skip the proofs. Readers wanting only the practical result can focus on the HalluGuard definition, the benchmark tables, and the test-time beam search experiment; the theoretical development in Sections 3.2 and 3.3 requires a stronger mathematical background.

Authors’ abstract

The reliability of Large Language Models (LLMs) in high-stakes domains such as healthcare, law, and scientific discovery is often compromised by hallucinations. These failures typically stem from two sources: data-driven hallucinations and reasoning-driven hallucinations. However, existing detection methods usually address only one source and rely on task-specific heuristics, limiting their generalization to complex scenarios. To overcome these limitations, we introduce the Hallucination Risk Bound, a unified theoretical framework that formally decomposes hallucination risk into data-driven and reasoning-driven components, linked respectively to training-time mismatches and inference-time instabilities. This provides a principled foundation for analyzing how hallucinations emerge and evolve. Building on this foundation, we introduce HalluGuard, an NTK-based score that leverages the induced geometry and captured representations of the NTK to jointly identify data-driven and reasoning-driven hallucinations. We evaluate HalluGuard on 10 diverse benchmarks, 11 competitive baselines, and 9 popular LLM backbones, consistently achieving state-of-the-art performance in detecting diverse forms of LLM hallucinations. We open-source our proposed \model{} model at https://github.com/Susan571/HalluGuard-ICLR2026.