Harnessing Reasoning Trajectories for Hallucination Detection via Answer-agreement Representation Shaping

AI summary

Overview

Research area: Machine learning, specifically hallucination detection for large reasoning models (LRMs) and representation engineering of hidden states.

Technical level: Advanced. The paper assumes familiarity with transformer hidden states, probing classifiers, contrastive objectives, and AUROC evaluation.

Scope in one sentence: The paper introduces Answer-agreement Representation Shaping (ARS), a label-free training procedure that perturbs the model's hidden state at the end of a reasoning trace to build counterfactual answers, then shapes answer embeddings so that answer-agreeing states cluster and answer-disagreeing states separate, improving downstream hallucination detection.

What This Paper Is About

Large reasoning models can produce long, fluent, seemingly well-justified reasoning traces and still end with a wrong answer, and the paper's motivating experiment (Figure 1, Qwen3-8B on TruthfulQA) shows that simply including the reasoning trace can make detection performance mixed or even worse. The core problem is that hallucination is an answer-level property, while traces vary in surface form, so detectors trained on raw trace text or vanilla hidden states may latch onto stylistic artifacts instead of whether the answer is stable and correct. The goal is to distill an answer-centric, detection-friendly signal out of the trajectory without any human hallucination labels and without updating the base model.

Key Contributions

1. A framework for trace-conditioned representation shaping. The authors claim ARS is the first framework to use reasoning traces to shape answer representations for hallucination detection in LRMs, by inducing counterfactual answers through latent perturbations at the end of the reasoning trace.

2. An agreement-driven shaping objective. Rather than editing text (which the paper says requires careful perturbation design and is brittle), ARS relabels latent perturbations by whether the resulting answer agrees with the original, and optimizes a lightweight mapping that pulls agreeing answer embeddings together and pushes disagreeing ones apart.

3. Theoretical and empirical analysis. The paper provides Proposition 4.2, which bounds the supervised-probe hallucination detection error by a term involving agreement separation probability (1 - eta_phi) plus the best achievable error e_alpha from the stability score, together with comprehensive experiments across four datasets and multiple LRM families.

4. Plug-and-play integration with no annotations. The shaped embeddings work with existing embedding-based detectors (supervised probing, CCS, HaloScope, EigenScore), require no ground-truth hallucination labels during training, keep base LRM parameters frozen, and need no sampling at test time.

Main Findings

• Shaping improves intrinsic separability across detectors and datasets. On Qwen3-8B, CCS rises from 66.85 to 86.64 on TruthfulQA and from 59.24 to 88.54 on TriviaQA; HaloScope rises from 57.78 to 71.03 on TruthfulQA and from 55.73 to 67.89 on TriviaQA; supervised probing on MATH-500 rises from 67.03 to 78.17. The trend also holds for DeepSeek-R1-Distill-Llama-8B, where CCS on MATH-500 rises from 54.44 to 86.38.

• Headline gain on TruthfulQA. The paper reports that learned trace-conditioned answer embeddings improve detection performance by 19.79% over vanilla LRM embeddings on TruthfulQA and reach a state-of-the-art detection result of 86.64%.

• Strong comparison against competitive baselines. In Table 1, ARS with CCS reaches 86.64 on TruthfulQA, 88.54 on TriviaQA, 90.37 on GSM8K, and 78.66 on MATH-500 with Qwen3-8B, while general-purpose and LRM-specific baselines remain near the low-70s; TSV, a semi-supervised representation-steering method, is outperformed while ARS uses zero human annotations.

• Transfer across datasets. Training the lightweight mapping on a source dataset and applying it to a target dataset works reliably: learning on GSM8K and testing on TriviaQA yields 87.80% AUROC, close to the 91.62% in-domain probing result on TriviaQA.

• Scaling to larger LRMs. With Qwen3-14B, ARS reaches 77.47 AUROC on TruthfulQA, 4.06% above TSV; on DeepSeek-R1-Distill-Qwen-14B it reaches 78.52 on TruthfulQA and 79.67 on MATH-500.

• Latent intervention at the trace boundary beats text-space interventions. On TruthfulQA with Qwen3-8B, deletion (best over intervention ratios 10% to 90% in steps of 10%) gives CCS 75.25, masking gives 75.50, paraphrasing gives 50.47, while ARS gives 86.64. Paraphrasing performs particularly poorly.

• Perturbation strength matters. Sweeping sigma in {1.00, 1.05, 1.75, 2.45, 2.50}, performance improves at moderate strength (for example sigma = 1.75) and declines as sigma increases further; weak intervention fails to generate informative disagreeing answers, while overly strong intervention produces easier counterfactuals.

• Stable, low-dimensional hyperparameters. Performance improves with output dimension k up to a moderate value then saturates, and M = 6 counterfactual answers per example works well with diminishing returns beyond; defaults are k = 512 and M = 6.

• Layer analysis. ARS improves separability across almost all layers compared to vanilla LRM embeddings, and intermediate layers provide more discriminative signals than early layers.

• Computational practicality. Unlike sampling-based detectors that require multiple forward passes, ARS needs a single model generation at inference, and no perturbations at test time.

Methodology in Plain English

The method has two parts.

First, the researchers pick a fixed control point inside the model: the hidden state at the last token of the reasoning trace, taken from the penultimate layer. This is the moment when the model has absorbed the entire trajectory but has not yet committed to an answer. They add a small random perturbation (for example isotropic Gaussian noise with scale sigma) to this state and let the model continue decoding, producing a counterfactual answer. Each original example gets M such counterfactual answers.

Second, they check whether each counterfactual answer agrees with the original answer under an agreement measure (textual similarity metrics or an LRM judge in practice, requiring no ground-truth labels y). Counterfactuals that agree become positives; those that disagree become negatives. A small linear mapping (a single bias-free projection, trained with Adam at learning rate 1e-4, weight decay 1e-5, cosine learning rate decay, batch size 128) is then trained with a contrastive-style objective using cosine similarity and a temperature tau, so that the mapped original answer embedding sits closer to agreeing embeddings than to disagreeing ones. The base model stays frozen and only this head is trained.

At test time, the mapping is applied to the answer embedding of a single generation, and any embedding-based detector (supervised probing, CCS, HaloScope, EigenScore) can score it and threshold the score into a truthful or hallucinated decision.

Evaluation uses AUROC, with correctness labels from Qwen3-32B and answer agreement judged by the same LRM that produced the trace and answer. Experiments cover TruthfulQA (817 conversational QA pairs), TriviaQA (deduplicated validation split, rc.nocontext subset, 9,960 QA pairs), GSM8K (train split, 7,473 problems), and MATH-500 (500 problems), with 25% of available data reserved for testing, 100 examples for validation, and the remainder for training, using Qwen3-8B/14B and DeepSeek-R1-Distill-Llama-8B / DeepSeek-R1-Distill-Qwen-14B in a zero-shot setting.

Why This Matters

Impact on research. The paper reframes hallucination detection for reasoning models as a representation-shaping problem rather than a scoring-function problem, and shows that a stability signal can be created internally through latent perturbations instead of expensive output-space sampling or human annotation. Proposition 4.2 connects the shaping objective directly to a detection error bound, giving a route to reason about why representation shaping should help.

Real-world applications (bullets):

• Filtering or flagging unreliable answers from AI reasoning assistants before they reach users in question answering and mathematical problem solving.
• Serving as a lightweight confidence gate in multi-step or tool-augmented decision pipelines, where one unsupported intermediate step can propagate to a wrong final answer.
• Routing or escalation: sending only low-stability generations to more expensive verification, since ARS needs a single forward pass rather than many samples.
• Post-deployment monitoring of reasoning systems, where embeddings can be scored without collecting ground-truth labels.

Industry relevance. The training signal requires no human annotations, the base model is not modified, and inference needs only one generation, which suits deployment settings where repeated sampling is too costly. The reported cross-dataset transfer (87.80% AUROC training on GSM8K and testing on TriviaQA) suggests the learned mapping can be reused across domains rather than rebuilt per task.

Future Directions

• Extending beyond the four evaluated tasks. The paper refers to additional experiments on broader reasoning domains in Appendix L and additional sampling strategies in Appendix C, but the main text covers TruthfulQA, TriviaQA, GSM8K, and MATH-500 only.

• Robustness of the agreement signal. The paper notes results remain robust under different agreement measures in Appendix H and reports sensitivity analyses on perturbation design in Appendix K, leaving open how sensitive the framework is to the choice of agreement judge or noise distribution in general.

• Finer-grained choices at the trace boundary. The authors ablate last token versus the last 1% / 2% tokens in Appendix G, suggesting further work on which summary of the trajectory should be perturbed.

• Pushing the theoretical bound. Proposition 4.2 leaves the constant C unspecified in the informal statement and depends on e_alpha, the informativeness of the stability score for truthfulness; tightening either term, or verifying e_alpha across more domains, is an open direction. The qualitative results in Figure 5 are truncated in the provided text, so the full qualitative analysis is not reported here.

Target Audience

Researchers and practitioners working on LLM reliability, uncertainty estimation, and hallucination detection, particularly those studying large reasoning models; representation-engineering and interpretability researchers interested in latent-state interventions; and applied machine learning engineers who need annotation-free, single-pass detectors that plug into existing embedding-based scoring pipelines. Readers without background in transformer hidden states, contrastive objectives, and probing classifiers will find the method section demanding.

Authors’ abstract

Large reasoning models (LRMs) often generate long, seemingly coherent reasoning traces yet still produce incorrect answers, making hallucination detection challenging. Although trajectories contain useful signals, directly using trace text or vanilla hidden states for detection is brittle: traces vary in form and detectors can overfit to superficial patterns rather than answer validity. We introduce Answer-agreement Representation Shaping (ARS), which learns detection-friendly trace-conditioned representations by explicitly encoding answer stability. ARS generates counterfactual answers through small latent interventions, specifically, perturbing the trace-boundary embedding, and labels each perturbation by whether the resulting answer agrees with the original. It then learns representations that bring answer-agreeing states together and separate answer-disagreeing ones, exposing latent instability indicative of hallucination risk. The shaped embeddings are plug-and-play with existing embedding-based detectors and require no human annotations during training. Experiments demonstrate that ARS consistently improves detection and achieves substantial gains over strong baselines. Code is available at: https://github.com/radiolab-ntu/ars_icml2026.