OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG

AI summary

Overview

Research area: Natural Language Processing / Information Retrieval, specifically retrieval-augmented generation (RAG) and large language model (LLM) decoding.

Technical level: Intermediate. Readers need familiarity with RAG pipelines, attention mechanisms, and standard QA evaluation metrics, but the paper's core idea is presented conceptually.

Scope: The paper proposes OpenDecoder, a training and decoding framework that injects explicit document-quality signals (retriever relevance, LLM-ranker score, and query performance prediction score) directly into an LLM's attention computation so that generation is more robust to noisy retrieved context.

What This Paper Is About

Standard RAG systems append retrieved documents to a prompt and assume the LLM can internally judge which documents are useful; the relevance scores produced by the retriever are essentially discarded at generation time. When retrieved documents are irrelevant or partially relevant, answer quality degrades, and the paper argues that existing workflow-based and fine-tuning-based fixes still rely only on the LLM's internal attention mechanism. OpenDecoder instead reshapes the decoder's token probability distribution using external document-quality scores as explicit indicators, together with robustness training on deliberately noisy document lists.

Key Contributions

1. A new decoding approach. The authors propose OpenDecoder, which modifies LLM decoding in RAG by leveraging relevance signals of the retrieved documents rather than prompt design alone.

2. A training method with three parts. The framework constructs explicit relevance indicators from retrieved documents, teaches the model to use those indicators for answer decoding, and adds robustness training that replaces the original top-k documents with ones of varying relevance levels.

3. Robustness evaluation across noise regimes. Experiments cover five benchmarks under three evaluation settings (Normal, Noisy, Extreme Noisy) with general and multi-hop QA, comparing against Vanilla RAG, Vanilla SFT, RobustRAG, InstructRAG, AstuteRAG, and RbFT.

4. A flexible paradigm. The authors state the approach can be integrated with post-training of LLMs for any purpose and can incorporate any type of external indicator (the paper notes criteria such as relevance, reliability, and authority).

Main Findings

• OpenDecoder leads on average across all three noise settings. With Qwen-2.5-3B-Instruct as the shared backbone, OpenDecoder reaches an average F1 of 34.87 and EM of 32.02 in the Normal setting, 34.16 F1 / 30.81 EM under Noisy, and 27.69 F1 / 23.91 EM under Extreme Noisy. The strongest baseline, RbFT, scores 34.22 F1 / 31.34 EM (Normal), 32.14 F1 / 28.72 EM (Noisy), and 25.53 F1 / 21.90 EM (Extreme Noisy).

• The gap widens as noise increases. Under Extreme Noisy evaluation, Vanilla RAG collapses to 5.58 average F1 and 12.58 average EM, Vanilla SFT reaches 22.77 F1 / 20.70 EM, RobustRAG drops to 2.88 F1 / 4.01 EM, InstructRAG to 9.81 F1 / 9.10 EM, and AstuteRAG to 19.71 F1 / 15.02 EM, while OpenDecoder holds at 27.69 F1 / 23.91 EM.

• Not every dataset shows a win. On NQ in the Normal setting, OpenDecoder records 39.26 F1 versus RbFT's 40.17, and on HotpotQA in Normal both methods report 29.44 F1. Significance markers (t-test at p<0.05 against RbFT and against Vanilla SFT) are attached to most other cells, including all of the Extreme Noisy averages.

• No RAG scores are far lower. With no retrieval at all, average F1 is 18.49; per-dataset F1 is 12.11 (NQ), 30.04 (TriviaQA), 11.07 (PopQA), 16.86 (HotpotQA), and 22.38 (2Wiki). EM is not reported for the No RAG rows.

• Every component of the training framework contributes. Ablation results in the Normal setting show F1 improvements over Vanilla SFT from each mechanism: with guidance alone NQ rises from 33.63 to 37.62, with aggregation to 36.24, with robust training to 38.98, and with the full OpenDecoder to 39.26. The same ordering appears on TriviaQA (50.31 → 55.31 / 55.48 / 55.84 / 56.08) and PopQA (20.46 → 24.33 / 21.59 / 25.14 / 25.95).

• Robust training matters most under extreme noise. For NQ in the Extreme Noisy setting, Vanilla SFT is 19.78, guidance alone gives 21.89, aggregation 21.07, robust training 22.22, and full OpenDecoder 22.50. On 2Wiki the same setting runs 21.76, 22.79, 26.77, 26.52, and 26.99.

• Feature aggregation behaves differently by dataset type. Using the retrieval score alone is described as sufficient for general QA datasets (NQ, TriviaQA, PopQA), where adding more features does not always help and may interfere. For multi-hop datasets (HotpotQA and 2Wiki), aggregating more feature scores gives better performance. The authors also observe that adding the LLM-ranker score improves over Vanilla SFT, which they interpret as evidence that the model cannot implicitly ignore noise through internal processing.

• Simple questions are more noise-sensitive than complex ones. The paper reports that as retrieved-document noise increases, performance drops are more severe on NQ and TriviaQA than on HotpotQA and 2Wiki, and argues that for harder datasets the priority is improving retrieval success before robustness of answer generation.

Methodology in Plain English

A normal RAG system retrieves the top-k documents for a query and pastes them into a prompt. The LLM then decides, through its own attention computation, how much weight each document deserves — but the retriever's own relevance judgment never enters that computation.

OpenDecoder's idea is to feed those judgments back into the decoder. First, the authors build quality indicators for each retrieved document from three sources: the retriever's similarity score, a score from an LLM-based ranker (using the logit of the end-of-sequence token), and a query performance prediction score (using the logit of the token "relevant"). These are aggregated with the retrieval score weighted as dominant and the other two scaled by a constant of 0.5, then normalized to the range [0, 1]. The instruction and the query are given a normalized score of 1.

Second, these normalized scores form a token-level matrix that multiplies the query-key product inside the softmax of the attention computation, so that tokens belonging to documents with low scores contribute less to the generation distribution. In the extreme case where every retrieved document scores very low, the query and instruction dominate, steering the model toward its parametric knowledge instead of the retrieved context.

Third, the model is trained to maximize the probability of the ground-truth answer given the query and retrieved documents. To build noise tolerance, the robustness training reconstructs the document list by keeping the top-5 documents and replacing the rest with partially relevant documents (sampled from ranks 6 to k) and irrelevant documents (sampled from the collection outside the top-k), optionally shuffled in position to counter position bias.

The experimental setup uses the 2018 Wikipedia dump with E5 as the retriever and 10 retrieved documents. Training merges the NQ and HotpotQA training sets for a single epoch, and evaluation uses F1 and Exact Match on the test sets of NQ, TriviaQA, PopQA, HotpotQA, and 2WikiMultiHopQA. In the noisy construction, 5 relevant, 3 partially relevant, and 2 irrelevant documents are used, sampled five times and fixed across all compared methods.

Why This Matters

Impact on research. The paper opens a line of work on treating retrieval-side quality estimates as first-class inputs to decoding, rather than something the LLM must rediscover internally. It reframes the RAG robustness problem as an attention-modulation problem and shows the mechanism is backbone-agnostic in principle and extensible to arbitrary external indicators.

Real-world applications:

• Enterprise or web search assistants that must answer from search results of uneven quality, where some retrieved pages are off-topic.
• Open-domain customer support or documentation QA, where retrievers frequently surface tangentially related articles.
• Multi-hop research assistants that synthesize evidence across several sources, where the paper found extra indicator features help most.
• Deployment scenarios with degraded or adversarial retrieval (the paper's Extreme Noisy setting approximates retrieval failure on difficult queries or domains).

Industry relevance. The work targets the common production situation in which a retrieval pipeline is imperfect and fine-tuning the model alone does not fix it. Because it modifies decoding rather than only the prompt, it offers a path to robustness gains that are stable across noise levels — the property most valued when retrieval quality cannot be guaranteed.

Future Directions

1. Adaptive feature selection. The ablation section states that the most effective features may vary by dataset distribution and concludes a more adaptive feature selection mechanism is needed to enhance generalizability.

2. Extending beyond relevance. The authors position the paradigm as able to incorporate any type of external indicator, naming reliability and authority as examples, which implies open work on what other signals to modulate decoding with.

3. Better retrieval before better robustness for hard datasets. The findings note that on HotpotQA and 2Wiki, retrieval defects are more common and the urgent goal is improving the success rate of retrieving relevant documents.

4. Combining with post-training objectives. The paper claims flexibility to integrate with post-training of LLMs "for any purposes," leaving open how OpenDecoder interacts with other training objectives such as trustworthiness or faithfulness enhancement.

Target Audience

Researchers and engineers working on RAG systems, robust question answering, and LLM decoding. It is most useful to readers who already understand the retrieve-then-read pipeline and want a concrete mechanism for making generation sensitive to retrieval quality; practitioners building retrieval-backed assistants will find the noise settings and baseline comparisons directly applicable, while those new to RAG should first read the background in Section 2.

Authors’ abstract

The development of large language models (LLMs) has achieved superior performance in a range of downstream tasks, including LLM-based retrieval-augmented generation (RAG). The quality of generated content heavily relies on the usefulness of the retrieved information and the capacity of LLMs' internal information processing mechanism to incorporate it in answer generation. It is generally assumed that the retrieved information is relevant to the question. However, the retrieved information may have a variable degree of relevance and usefulness, depending on the question and the document collection. It is important to take into account the relevance of the retrieved information in answer generation. In this paper, we propose OpenDecoder, a new approach that leverages explicit evaluation of the retrieved information as quality indicator features for generation. We aim to build a RAG model that is more robust to varying levels of noisy context. Three types of explicit evaluation information are considered: relevance score, ranking score, and QPP (query performance prediction) score. The experimental results on five benchmark datasets demonstrate the effectiveness and better robustness of OpenDecoder by outperforming various baseline methods. Importantly, this paradigm is flexible to be integrated with the post-training of LLMs for any purposes and incorporated with any type of external indicators.