Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generation

AI summary

Overview

Research area: Natural Language Processing — retrieval-augmented generation (RAG), hallucination mitigation, and robustness of large language models (LLMs).

Technical level: Advanced. The paper assumes familiarity with transformer hidden states, spectral clustering, graph Laplacians, eigengaps, and Direct Preference Optimization (DPO).

Scope: The paper identifies and mitigates a previously underexplored failure mode in RAG — sensitivity to the order in which retrieved documents are presented — using hidden-state clustering plus preference alignment, evaluated on three QA benchmarks with two retrievers and two backbone LLMs.

What This Paper Is About

RAG systems feed retrieved documents to an LLM to ground its answers, but the authors show that simply reordering the same set of retrieved documents can push the model down entirely different reasoning paths and produce inconsistent or hallucinated answers. They call this permutation-induced hallucination. The paper proposes Stable-RAG, a method that estimates permutation sensitivity by clustering the model's internal reasoning states across document orders and then aligns the model toward consistent, correct outputs.

Key Contributions

1. Characterizing permutation sensitivity. The authors show that RAG systems are highly sensitive to document order even when the same documents (including the gold document) are used, and layer-wise hidden-state clustering reveals that divergent reasoning trajectories emerge in the middle layers and intensify in higher layers.

2. Proposing Stable-RAG. They introduce a model-agnostic method that runs the generator under multiple retrieval orders, clusters the final-layer last-token hidden states, decodes representative states from each cluster, and uses those reasoning results to align hallucinated outputs toward correct answers.

3. Building a preference-data pipeline. They construct four categories of preference tuples (FC, PC, FU, FA) from representative decoding, including a category that encourages abstention with "I don't know" when no gold answer exists in the documents, and train with DPO.

4. Demonstrating broad gains. Across NQ, TriviaQA, and HotpotQA with both DPR and Contriever retrievers, and across LLaMA3-8B-Instruct and Qwen3-8B backbones, Stable-RAG outperforms strong baselines in accuracy and reasoning consistency, with generalization across datasets, retrievers, and Top-K settings.

Main Findings

• Answers vary under permutation even with the gold document first. Using a Top-5 retrieval setting with the gold document included, the authors report that LLM answers vary substantially across permutations of the retrieved set, including when the gold document is fixed in the first position, and even in contexts under 1,000 tokens. Figure 1 reports Perturbation Success Rate (PSR) on the NQ test set over 1,000 randomly sampled instances across LLaMA models.

• Divergence grows with depth. Spectral clustering over document-permuted representations on NQ and HotpotQA shows reasoning trajectories are relatively concentrated in shallow layers, diverge in middle layers, and diverge further in higher layers. Sensitive samples (10+ clusters) show substantially greater divergence than non-sensitive ones (1–2 clusters), localized mainly to higher layers. These measurements use 1,000 randomly sampled instances, 5! = 120 permutations, and are shown for LLaMA3-8B-Instruct on NQ train with DPR and Qwen3-8B on HotpotQA train with Contriever.

• Clustering quality improves with depth. On the NQ train set with DPR, averaged over 10,000 randomly sampled instances, Qwen3-8B reaches F1 77.9 at layer 8, 79.6 at layer 16, 86.6 at layer 24, and 87.6 at layer 36. LLaMA3-8B-Instruct reaches F1 69.3 at layer 8, 81.3 at layer 16, 82.2 at layer 24, and 83.9 at layer 32. The authors therefore use final-layer hidden states for clustering.

• Stable-RAG leads the main benchmark. On LLaMA3-8B-Instruct, Stable-RAG reaches averages of 52.34 SubEM and 52.23 F1 across NQ, TriviaQA, and HotpotQA with Contriever and DPR, versus 48.82 / 49.80 for ATM, 47.50 / 49.11 for RAAT, 47.29 / 48.35 for Pos2Distill, 47.08 / 49.40 for RetRobust, 45.66 / 47.83 for Vanilla RAG, 45.48 / 47.69 for Ms-PoE, 42.13 / 43.26 for Vanilla SFT, and 34.16 / 36.98 for Direct Generation. The exhaustive full-permutation variant (marked with a club symbol) reaches 52.88 / 52.78.

• Gains hold on a second backbone. On Qwen3-8B, Stable-RAG averages 50.27 SubEM and 52.44 F1, versus 49.24 / 51.54 for ATM, 48.45 / 50.93 for Pos2Distill, 48.08 / 50.55 for Vanilla RAG, 47.94 / 50.63 for RAAT, 47.88 / 50.33 for Ms-PoE, 47.47 / 50.66 for RetRobust, 41.18 / 42.66 for Vanilla SFT, and 29.08 / 32.36 for Naive Generation. The exhaustive full-permutation variant reaches 51.10 / 53.40.

• Every preference-data component matters. In the ablation on LLaMA3-8B-Instruct with Contriever (SubEM), the full method scores 48.14 on NQ, 72.05 on TriviaQA, 38.91 on HotpotQA (average 53.03) with an abstention rate of 21.8. Removing the PC component (index a) drops the average to 42.51 (NQ 37.62, TriviaQA 61.37, HotpotQA 28.54). Removing FA (index c) gives 50.87 average (46.73 / 70.14 / 35.75). The variants that remove FU (index b and d) produce abstention rates of 0.0 and 0.5 respectively, with averages of 51.96 and 52.11.

• Order-stability, not DPO alone, drives the gain. Compared with standard DPO using the same base model and optimization strategy, Stable-RAG scores higher SubEM on NQ (48.14 vs 44.76 with Contriever; 52.02 vs 50.88 with DPR), TriviaQA (72.04 vs 68.03; 73.43 vs 71.67), and HotpotQA (38.91 vs 35.96 with Contriever; 29.48 vs 30.43 with DPR).

• Lower PSR across gold-document positions. On NQ test with DPR and LLaMA3-8B-Instruct, Stable-RAG reports the lowest PSR at every position: 28.3 (Pos 1), 54.7 (Pos 2), 67.3 (Pos 3), 72.6 (Pos 4), 73.0 (Pos 5) — lower than Vanilla RAG (50.8, 71.4, 81.9, 85.5, 84.4), Vanilla SFT (47.2, 66.2, 74.8, 80.0, 82.6), RetRobust (35.5, 75.5, 85.3, 88.6, 88.9), ATM (33.7, 64.2, 71.8, 77.4, 77.8), Pos2Distill (29.5, 55.8, 69.4, 72.8, 73.2), and Ms-PoE (31.4, 63.8, 72.1, 73.9, 74.3).

• Smallest drop under shuffling. Under original versus randomly shuffled document order (SubEM, Contriever), Stable-RAG scores 48.14 / 47.23 (drop 0.91) on NQ, 72.05 / 71.76 (drop 0.29) on TriviaQA, and 38.91 / 37.50 (drop 1.41) on HotpotQA — the highest scores in both conditions across all datasets and the smallest absolute drops on NQ and TriviaQA.

• Generalization across datasets, retrievers, and Top-K. Cross-dataset generalization, cross-retriever transfer (train on DPR, evaluate on Contriever), and cross-Top-K robustness experiments, all run on LLaMA3-8B-Instruct, show stable performance and improvements over the best baseline, with the Contriever-trained / DPR-evaluated direction reported in the appendix.

• Modest data requirement. Training-data-size analysis on NQ with Contriever shows performance improving steadily and saturating beyond 15k samples; the authors adopt 15k as default because gains over 20k do not justify added computation, while very limited data (e.g., 1k) drops markedly.

• Internal behavior shifts after alignment. On a random subset of 500 NQ test samples with Contriever, the method reduces clusters for

Authors’ abstract

Retrieval-Augmented Generation (RAG) has become a key paradigm for reducing factual hallucinations in Large Language Models (LLMs), yet little is known about how the order of retrieved documents affects model behavior. We empirically show that under a Top-5 retrieval setting with the gold document included, LLM answers vary substantially across permutations of the retrieved set, even when the gold document is fixed in the first position. This reveals a previously underexplored sensitivity to retrieval permutations. Although existing robust RAG methods focus primarily on enhancing LLM robustness to low-quality retrieval and mitigating positional bias to distribute attention fairly over long contexts, neither approach directly addresses permutation sensitivity. In this paper, we propose Stable-RAG, which exploits permutation sensitivity estimation to mitigate permutation-induced hallucinations. Stable-RAG runs the generator under multiple retrieval orders, clusters hidden states, and decodes from a cluster-center representation that captures the dominant reasoning pattern. It then uses these reasoning results to align hallucinated outputs toward the correct answer, encouraging the model to produce consistent and accurate predictions across document permutations. Experiments on three QA datasets show that Stable-RAG improves answer accuracy, reasoning consistency, and generalization across datasets, retrievers, and input lengths compared with strong baselines.