Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders

AI summary

Overview

Research area: Natural Language Processing / Retrieval-Augmented Generation (RAG) faithfulness, mechanistic interpretability, and hallucination detection.

Technical level: Advanced. The paper combines sparse autoencoders (SAEs), mutual-information-based feature selection, generalized additive models (GAMs), and a theoretical analysis of max pooling.

Scope: The paper introduces RAGLens, a lightweight, interpretable detector that flags hallucinations in RAG outputs by probing the internal activations of a frozen LLM with sparse autoencoders, and shows that the same detector can be used to explain and post-hoc mitigate unfaithful generations.

What This Paper Is About

Retrieval-Augmented Generation grounds LLM answers in retrieved passages, but models still contradict or go beyond the evidence they are given. Existing ways to catch this — training big specialized detectors, or asking another LLM to judge faithfulness — either need large annotated datasets or are expensive and opaque. This paper asks whether sparse autoencoders, which break LLM hidden states into interpretable features, can reveal internal signals that reliably flag RAG hallucinations, and whether those same signals can be turned into human-readable explanations and corrections.

Key Contributions

1. Evidence that SAEs capture RAG-specific hallucination features. The authors show that sparse autoencoders disentangle internal activations into features that are specifically triggered during RAG hallucinations, establishing internal LLM representations as a viable basis for detection.

2. RAGLens, a lightweight and interpretable detector. Built on information-based feature selection (mutual information ranking) and additive feature modeling (GAM with bagged gradient boosting), RAGLens outperforms existing detectors in accuracy while remaining small enough to encode only a subset of SAE features.

3. Explanation and mitigation from the same detector. Because the model is additive, predictions decompose into per-feature contributions, which yields both local (instance-specific, token-level) and global (instance-invariant) explanations that can be fed back to the LLM to reduce hallucination.

4. Design-principle analysis of SAE-based detection. Systematic studies of LLM layer choice, feature extractor (SAE vs. Transcoder), pre- vs. post-activation signals, number of selected features, and predictor architecture (LR, GAM, MLP, XGBoost), plus a theorem on why max pooling helps in the sparse-activation regime.

Main Findings

• Detection performance: RAGLens achieves AUC above 80% on both RAGTruth and Dolly with Llama2-7B and Llama2-13B backbones, and consistently outperforms all listed baselines. On RAGTruth with Llama2-7B it reaches AUC 0.8413 / Acc 0.7576 / F1 0.7636, versus 0.7458 / 0.6822 / 0.7190 for the strongest baseline, ReDeEP. On Dolly with Llama2-7B it reaches 0.8764 / 0.7778 / 0.8070, versus ReDeEP's 0.7949 / 0.7373 / 0.7833.

• Cross-model behavior: SAE features do not transfer across LLMs, but a RAGLens detector trained on one LLM's SAE can be applied to text generated by other LLMs. Across RAGTruth, AggreFact, and TofuEval, the SAE-based detector consistently beats each model's own chain-of-thought self-judgment; the paper concludes models "know more than they tell."

• Domain generalization: A detector trained on RAGTruth (AUC 0.8806 on RAGTruth for Llama2-7B) generalizes to AggreFact (0.8019) and TofuEval (0.7637) without retraining, versus 0.4842 / 0.5741 / 0.5562 zero-shot CoT. Detectors trained on AggreFact or TofuEval generalize less well, which the authors attribute to those datasets focusing on single tasks. Across RAGTruth subtasks, a summarization-trained predictor transfers best (0.8191 / 0.8253 / 0.6443 on Summary / QA / Data2txt for Llama2-7B).

• Interpretable features: Two representative Llama3.1-8B SAE features learned on RAGTruth are feature 22790, corresponding to unsupported numeric or time specifics (its GAM shape function rises monotonically with activation), and feature 17721, corresponding to grounded, high-salience tokens. Semantic explanations were distilled by GPT-5 from 24 activation cases per feature.

• Mitigation results: On 450 Llama2-7B outputs from RAGTruth, both instance-level and token-level feedback reduced hallucination rates under every judge, and token-level feedback reduced them further. Reported rates: original 43.78% / 37.78% / 64.44% / 71.11% for Llama3.3-70B / GPT-4o / GPT-o3 / human annotators; with instance-level feedback 42.22% / 36.44% / 60.44% / 62.22%; with token-level feedback 39.11% / 34.22% / 58.88% / 55.56%. A Llama3.1-8B-based detector applied to all 450 examples converted 29 outputs from hallucination to non-hallucination with instance-level feedback and 36 with token-level feedback.

• Layer placement: Across Llama3.2-1B, Llama3-8B, Qwen3-0.6B, and Qwen3-4B, performance peaks around the middle layers for the Summary and QA subtasks of RAGTruth, while Data2txt shows a comparatively flat pattern across layers.

• Extractor and activation point: Pre-activation features consistently beat post-activation features for both SAE and Transcoder on Llama3.2-1B. SAE pre-activation reached Acc/F1 of 0.7810 / 0.7892 on RAGTruth, 0.7308 / 0.7388 on AggreFact, and 0.6865 / 0.6876 on TofuEval. Transcoder and SAE are similar in accuracy, so the paper concludes no clear architectural advantage.

• Predictor choice: GAM consistently outperforms logistic regression and also surpasses MLP and XGBoost, indicating that the overall contribution of SAE features can be captured additively even though individual feature effects are nonlinear.

• Feature budget: Reducing the number of selected features (K') from 1024 to 1 degrades performance, but the decline is much more gradual under mutual-information ranking than under random selection.

• Theory: Theorem 1 shows that in the sparse-activation regime (T × p̄ << 1), the mutual information between the max-pooled activation and the label is approximately a constant times T(Δp)²/p̄, linear in sequence length T and quadratic in the activation-rate gap Δp, with residual O((T p̄)²).

Methodology in Plain English

The researchers take a frozen LLM and run the RAG pipeline, then collect the hidden states at a chosen layer for every generated token. Those hidden states are passed through a sparse autoencoder that converts each one into a long, mostly-zero vector of features — each feature ideally representing one concept. Because the hallucination label applies to the whole answer rather than individual tokens, the token-level activations are collapsed into one vector per example using max pooling, which the paper also justifies theoretically for the sparse setting.

From this pooled vector, they rank each feature by its mutual information with the hallucination label and keep only the top K' features. Those selected features feed a generalized additive model: a logistic-style link function plus one learned univariate shape function per feature, fitted with bagged gradient boosting. This keeps the detector small and makes every prediction a sum of per-feature contributions.

The additive structure produces two kinds of explanations: local ones (which features and which tokens drove a specific verdict, useful for pinpointing fabricated numbers, dates, or entities) and global ones (a stable shape function showing how each feature's magnitude shifts hallucination risk). These explanations are then passed back to the LLM as feedback — either a verdict alone or verdict plus highlighted tokens — so the model can revise its output.

Why This Matters

Research impact: The paper argues it is the first to systematically show that SAE features can detect hallucinations specifically in RAG settings, where the interaction between retrieved evidence and generated text makes detection harder than generic hallucination detection. It also contributes practical design guidance — mid-layer features, pre-activation signals, mutual-information selection, and GAMs over MLPs or XGBoost — and a theoretical account of why max pooling is appropriate for sparse activations. It reframes hallucination detection as a mechanistic-interpretability problem rather than a data-hungry supervised-learning one.

Real-world applications:

• Fact-checking and grounding verification for enterprise RAG assistants built over internal document stores or knowledge bases.
• High-stakes domains such as medical, legal, or financial question answering, where unsupported numbers, dates, and named entities are exactly the failure mode the detector is designed to surface.
• Summarization and data-to-text generation pipelines that must remain faithful to source documents, including dialogue and meeting summarization.
• Post-hoc output repair in deployed systems, where a token-level warning can prompt the model to revise only the ungrounded spans rather than regenerate everything.

Industry relevance: Because RAGLens reuses publicly released SAEs, requires only a small number of features at inference, and avoids calling a large external LLM judge, it points toward cheaper and more transparent guardrails than LLM-as-judge pipelines. Detector weights are tied to a specific model's SAE, but the trained detector can be applied to outputs produced by other models, which fits multi-model production stacks.

Future Directions

• Making SAE features transferable across LLMs, since the paper notes the features themselves do not transfer and detectors must be retrained per backbone SAE.
• Improving generalization for single-task training data: detectors trained on AggreFact and TofuEval transferred much less well than RAGTruth-trained ones, so what makes training data diverse remains an open question.
• Understanding and exploiting task-specific versus shared hallucination signals, given that Summary and QA transfer to each other better than Data2txt does, and that Data2txt shows flat layer-wise performance.
• Extending the explanation-and-mitigation loop beyond the evaluated setup — for example, broader human evaluation, other revision strategies, and whether the gains from token-level feedback hold across more models and tasks.

Target Audience

This paper is most useful for researchers working on LLM faithfulness, hallucination detection, and mechanistic interpretability; NLP engineers building RAG systems who need cheap, transparent verification rather than LLM-as-judge pipelines; and practitioners in factuality-sensitive domains who care about both detection accuracy and knowing why a given output was flagged. Readers should already be comfortable with transformer internals, sparse autoencoders, and additive or statistical modeling; the appendices covering datasets, baselines, proofs, and case studies are aimed at readers who want to reproduce or extend the work.

Authors’ abstract

Retrieval-Augmented Generation (RAG) improves the factuality of large language models (LLMs) by grounding outputs in retrieved evidence, but faithfulness failures, where generations contradict or extend beyond the provided sources, remain a critical challenge. Existing hallucination detection methods for RAG often rely either on large-scale detector training, which requires substantial annotated data, or on querying external LLM judges, which leads to high inference costs. Although some approaches attempt to leverage internal representations of LLMs for hallucination detection, their accuracy remains limited. Motivated by recent advances in mechanistic interpretability, we employ sparse autoencoders (SAEs) to disentangle internal activations, successfully identifying features that are specifically triggered during RAG hallucinations. Building on a systematic pipeline of information-based feature selection and additive feature modeling, we introduce RAGLens, a lightweight hallucination detector that accurately flags unfaithful RAG outputs using LLM internal representations. RAGLens not only achieves superior detection performance compared to existing methods, but also provides interpretable rationales for its decisions, enabling effective post-hoc mitigation of unfaithful RAG. Finally, we justify our design choices and reveal new insights into the distribution of hallucination-related signals within LLMs. The code is available at https://github.com/Teddy-XiongGZ/RAGLens.