Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generation

AI summary

Overview

• Research area: Natural Language Processing — retrieval-augmented generation (RAG), reinforcement learning for LLM agents, and chain-of-thought faithfulness evaluation.
• Technical level: Intermediate. Readers should be comfortable with RL fine-tuning concepts (policy, reward, PPO), RAG pipelines, and multi-turn agent trajectories; the paper's faithfulness definitions themselves are explained in plain terms.
• Scope: The paper formalizes three turn-level faithfulness metrics for RL-trained search agents, shows that correctness-based training does not guarantee faithful intermediate reasoning, and proposes VERITAS, a training framework that adds structured faithfulness rewards to the RL loop.

What This Paper Is About

Reinforcement learning has been used to train LLM agents that plan, issue search queries, and reason over retrieved documents (a paradigm the paper calls "agentic search"). These systems are typically rewarded only for whether the final answer is correct, which can produce reasoning traces that look plausible but do not actually connect to the searches performed, the retrieved evidence, or the final answer. The paper defines and measures this problem across three faithfulness dimensions and then introduces a training method that rewards faithful intermediate reasoning rather than correctness alone.

Key Contributions

1. A formal definition and evaluation framework for faithfulness in agentic search, built on three metrics: Think-Search faithfulness (are search queries justified by the preceding thoughts?), Information-Think faithfulness (do thoughts use the retrieved information?), and Think-Answer faithfulness (is the final answer grounded in the preceding thoughts?).
2. A systematic audit of leading RL-based search agents — Search-R1, ReSearch, and DeSA — using official checkpoints, revealing a gap between task performance and reasoning faithfulness.
3. VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a training framework that integrates fine-grained, structured, checklist-style faithfulness rewards into the RL loop (implemented with PPO on top of Search-R1).
4. VERITAS-R1, the resulting model, which improves Information-Think faithfulness by around 14% and Think-Answer faithfulness by around 7.7% over its comparable Search-R1 baseline while also raising task accuracy across seven QA benchmarks.

Main Findings

• Outcome-only training leaves a faithfulness gap. Search-R1-7B-Base PPO scores 0.564 on Information-Think faithfulness on the multi-hop HotpotQA dataset, and the best checkpoint evaluated, ReSearch-7B-Base, reaches only 0.586. The paper states that policies trained against pure outcome-based reward are not fully faithful to retrieved information during thinking.
• Better task performance does not imply better faithfulness. DeSA-7B achieves competitive Exact Match on both NQ and HotpotQA but consistently underperforms Search-R1 and ReSearch in almost all three faithfulness dimensions.
• Think-Search faithfulness is already high in existing agents. Search-R1 scores 0.935 on NQ under LLM-as-a-Judge. This is one reason the authors set the Think-Search reward weight to zero during training.
• A format reward can break reasoning consistency. Search-R1-7B-Base-PPO-Format shows poor Information-Think faithfulness, with traces that jump straight to the next query after retrieved documents, and it continues searching until the maximum number of turns, which hurts Think-Answer faithfulness.
• VERITAS improves faithfulness. In the paper's evaluation, average Information-Think faithfulness across datasets rises from 0.608 for the Search-R1 baseline to 0.697 for the VERITAS Info-Think variant, and to 0.693 for the combined variant. Average Think-Answer faithfulness rises from 0.808 to 0.869 (Info-Think variant), 0.865 (Think-Ans variant), and 0.870 (combined). On general QA datasets the Information-Think average increases from 0.734 for baseline Search-R1 to 0.853 for VERITAS-R1.
• Improving faithfulness also improves accuracy. On multi-hop datasets, average Exact Match improves from 0.361 to 0.380; on general QA datasets, from 0.524 to 0.532. In the main results table, VERITAS-R1 with EM+Info-Think reaches an average of 0.447, marked statistically significant against both Search-R1-7B-Base-PPO and the format variant, compared with 0.431 and 0.434 for those baselines.
• Gains are not uniform. Think-Answer faithfulness improvements are clear on multi-hop datasets but not stable on general QA datasets.
• Rewarding one faithfulness dimension can lift others. VERITAS-R1 improves Think-Search faithfulness on both general QA and multi-hop QA even though Think-Search is not used as a training signal.
• The distilled reward model is reliable. A LoRA-finetuned Qwen2.5-14B-Instruct reward model achieves 0.899 consistency and a Cohen's kappa of 0.797 with Claude-3.7-Sonnet labels. Against 100 human-annotated test examples, the reward model reaches 0.910 consistency (kappa 0.819) versus 0.850 (kappa 0.699) for Claude-3.7-Sonnet.
• Training is stable across seeds. Re-running the Info-Think variant with three independent seeds gives average EM of {0.442, 0.446, 0.445} with a standard deviation of 0.0021.

Methodology in Plain English

1. Define the trajectory schema. An agentic search trajectory is decomposed into tagged blocks: <think> (internal reasoning), <search> (query sent to the retriever), <information> (retrieved documents), and <answer> (final response). Faithfulness is then defined at the transitions between these blocks.
2. Measure each transition with the right tool. Think-Search is checked two ways: an NLI model (google/t5_xxl_true_nli_mixture) treating the think block as premise and the query as hypothesis, plus an LLM-as-a-Judge (Claude Sonnet-4.5) that judges whether the query is well-motivated. Information-Think uses LLM-as-a-Judge only, since it requires judging synthesis of retrieved content. Think-Answer uses NLI entailment plus a substring exact-match (Sub-EM) metric suited to short-form QA.
3. Meta-evaluate the metrics. The authors compute agreement rates and Fleiss's kappa between metric pairs. They find high agreement between the LLM judge and NLI on Think-Search, and moderate inconsistency on Think-Answer, where the NLI model produced false positives on short answers — so Sub-EM is adopted as the automatic Think-Answer metric.
4. Distill an affordable judge. Because LLM-as-a-Judge is too slow and costly for on-policy RL, they sample 27,000 Search-R1 trajectories from NQ and HotpotQA training splits, label them with Claude-3.7-Sonnet, split 24K for training and 3K for evaluation, and LoRA-finetune Qwen2.5-14B-Instruct as a reward model for Information-Think faithfulness.
5. Combine rewards. The training reward sums weighted terms: outcome Exact Match plus Info-Think and Think-Answer faithfulness rewards, with a Think-Search weight set to zero. The chosen weights are w_EM = 1.0, w_info-think = 0.05, and w_think-ans = 0.02.
6. Use a curriculum. Faithfulness rewards are switched off for the first epoch (T1), linearly warmed up over the following half epoch, and applied fully from T2 onward. The authors say applying them immediately improved faithfulness but slightly reduced task performance, which they attribute to over-optimizing for faithfulness.
7. Train and compare. The policy model is Qwen2.5-7B-Base trained with PPO, with learning rate 1e-6, batch size 256, the E5-base-v2 retriever, and the 2018 Wikipedia dump — all matching the original Search-R1 setup. Experiments ran on 32 NVIDIA A100 GPUs, and evaluation covers NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle, with dataset-level paired Wilcoxon signed-rank tests (n = 7, p = 0.05).

Why This Matters

• Impact on research: The paper reframes agentic search evaluation away from answer accuracy alone, provides reusable definitions and meta-evaluated metrics, and shows that process-level faithfulness signals can be turned into a training reward that does not trade off against accuracy. It also provides evidence that a format reward, which is often added to improve task metrics, can severely damage reasoning consistency.
• Real-world applications:
  • Enterprise and web search assistants that must show users why a query was issued and how evidence supports the answer.
  • Customer support and technical troubleshooting bots where a plausible but unsupported answer carries real cost.
  • Multi-hop research or analyst assistants that synthesize several retrieved sources into a single conclusion.
  • Compliance-sensitive domains where the audit trail from question to evidence to answer matters as much as the answer.
• Industry relevance: The reward model is distilled into a 14B parameter model that the authors report is efficient enough for on-policy RL, which matters for teams that cannot afford a large judge model in the training loop. The method is built on the existing Search-R1 framework, so it is relatively easy to adopt, and the metric framework is described as agnostic to model architecture and applicable to any system producing the same structured trajectory.

Future Directions

• Expand beyond the two trained dimensions. Think-Search was excluded from training because existing agents already score highly on it; it remains an open question whether that holds for other architectures or domains.
• Explore alternative reward implementations. The authors explicitly leave richer semantic matching signals for Think-Answer faithfulness, instead of the substring exact-match used here, to future work.
• Improve stability on general QA. The paper reports that Think-Answer faithfulness gains are unstable on general QA datasets and does not resolve why.
• Broaden generalization tests. The framework is described as applicable to ReSearch and DeSA, and to any reasoning-centric system with structured trajectories, but broader cross-architecture and cross-domain validation is not reported in the available content.

Target Audience

Researchers and engineers working on RAG, agentic LLM systems, and reinforcement learning for language models, especially those building or evaluating multi-turn search agents. It is also relevant to evaluation and trustworthiness practitioners who need process-level rather than outcome-level assessments, and to teams deciding how to weight faithfulness against accuracy in a production search agent.

Authors’ abstract

Inspired by the success of reinforcement learning (RL) in Large Language Model (LLM) training for domains like math and code, recent work has begun training LLMs to dynamically plan, query, and reason with search engines as tools -- a paradigm increasingly referred to as agentic search. Although these methods achieve performance improvement across popular short-form QA benchmarks, many prioritize final answer correctness while overlooking the quality of intermediate reasoning steps, which may lead to chain-of-thought unfaithfulness. In this paper, we first introduce a comprehensive evaluation framework for agentic search, covering three distinct faithfulness metrics: Think-Search faithfulness, Information-Think faithfulness, and Think-Answer faithfulness. Our evaluations reveal that canonical agentic search systems trained through Reinforcement Learning from Verifiable Reward (RLVR) using episode-level outcome-based reward -- including Search-R1 and ReSearch -- have significant room for improvement on these faithfulness dimensions. To foster faithful reasoning in agentic search, we introduce VERITAS (Verifying Entailed Reasoning through Intermediate Traceability in Agentic Search), a novel framework that integrates fine-grained turn-level faithfulness rewards into the reinforcement learning process. Our experiments show that models trained with \ours not only significantly improve reasoning faithfulness, but also achieve better task performance compared to baselines trained against episode-level outcome-based reward.