Optimizing Agentic Reasoning with Retrieval via Synthetic Semantic Information Gain Reward

AI summary

Overview

• Research area: Agentic AI, retrieval-augmented reasoning, and reinforcement learning for large reasoning models (LRMs).
• Technical level: Advanced — the paper builds on POMDP formalism, Bayesian belief updating, information theory (entropy, Blackwell dominance), and policy-gradient RL.
• Scope: The paper proposes a theoretically grounded, annotation-free reward signal that scores individual retrieval steps by how much they reduce a model's uncertainty about the correct answer, and trains retrieval policies with it.

What This Paper Is About

When a reasoning model searches the web or queries a knowledge base mid-thought, most training methods only tell it whether the final answer was right. That signal is sparse, delayed, and cannot distinguish a useful search from a useless one. This paper asks how to measure and reward the value of each individual retrieval action, and answers by defining retrieval value as the reduction in the model's uncertainty over the correct answer — computable from the model's own sampled outputs, without human annotation of intermediate steps.

Key Contributions

1. A theoretical reframing of information gain. The authors model agentic retrieval as a POMDP and redefine information gain as the reduction in an uncertainty functional over the agent's belief state. They prove three properties: non-negativity (under ideal Bayesian updates, information gathering never hurts), telescoping additivity (per-step gains sum to global uncertainty reduction), and channel monotonicity (better information sources receive equal or higher gain).

2. A practical, annotation-free reward estimator. The abstract belief state is approximated by a distribution over semantic equivalence classes of sampled answers, where equivalence is decided by bidirectional textual entailment using a pretrained NLI model. This turns the abstract quantity into a computable training reward.

3. A gold-class reward that ties local steps to correctness. Rather than rewarding entropy reduction alone (which can reward confident wrong answers), the practical reward measures how much the probability mass on the correct answer's semantic class increases when retrieved evidence is provided.

4. Empirical validation at multiple scales and settings. InfoReasoner is trained with GRPO on NQ and HotpotQA splits and evaluated on seven QA benchmarks, MATH500, and WebDetective, consistently beating strong retrieval-augmented baselines.

Main Findings

• Consistent gains across seven QA benchmarks. InfoReasoner-3B reaches 34.6% average accuracy versus 30.1% for Search-R1-3B-Instruct and 27.0% for standard RAG. InfoReasoner-7B reaches 39.1%, beating AutoRefine-7B-Base (36.9%) and Search-R1-7B-Instruct (33.7%), with reported average improvements of up to 5.4%.

• Small models punch above their weight. The 3B variant surpasses several 7B baselines, including Search-R1-7B-Instruct, suggesting the reward signal improves parameter efficiency rather than just raw scale.

• Transfers to tool use and long-horizon search. On MATH500 with Python execution, InfoReasoner-7B scores 85.8% versus 84.6% (SimpleTIR-7B) and 80.2% (ZeroTIR-7B). On the WebDetective deep-search benchmark, it lifts Pass@1 from 26.5% to 30.0% over Search-R1-3B-Base and raises knowledge utilization.

• The reward weight has a clear optimum. Ablating the information-gain coefficient shows λ = 0.6 is best; setting λ = 0 (equivalent to Search-R1) drops average accuracy to 29.9%, a 4.7-point gap. Pushing λ to 1.0 or 0.8 degrades performance, meaning information gain should inform but not dominate the reward.

• A distinctive training trajectory. InfoReasoner grows EM score more slowly early on — because it rewards useful retrievals even when the final answer is wrong — then accelerates past Search-R1. Entropy loss spikes during exploration before settling lower, and responses are roughly 30% shorter than Search-R1's.

• Complementary evidence yields synergistic gain. Case-study analysis shows that jointly observing two retrieved documents produces higher information gain than the sum of their individual gains, validating the reward's preference for multi-hop complementary retrieval.

• Twelve samples suffice. Sensitivity analysis over group size shows estimation error decays with a Monte Carlo–like rate and an elbow around M = 12, achieving mean absolute error below 0.02 at roughly one-quarter the compute of a 64-sample oracle.

Methodology in Plain English

The researchers start from an intuition: a good search is one that makes the model more sure of the right answer, and a bad search either leaves it unsure or actively confuses it.

To turn that intuition into a trainable number, they do the following at each retrieval step. First, they ask the model the same question twice — once with only the task framing, and once with the retrieved evidence attached. Second, they sample many candidate answers in each condition. Third, they group those answers into meaning-based buckets: two answers land in the same bucket only if each logically implies the other, as judged by an NLI model. This sidesteps the problem that "Einstein" and "He is Albert Einstein" look different as strings but mean the same thing.

Those buckets give an approximate belief distribution: how much probability mass the model puts on each distinct meaning. The team then computes two things — the overall entropy reduction between the no-evidence and with-evidence conditions, and, more importantly, how much probability mass moves onto the bucket containing the correct answer. The latter becomes the reward for that retrieval step.

That per-step reward is combined with the standard exact-match reward for the final answer, and the whole thing is optimized with Group Relative Policy Optimization, which compares outputs sampled for the same prompt rather than training a separate value network.

A neat side effect: when retrieval returns misleading or contradictory documents and the model's uncertainty rises, the reward goes negative. The authors treat this not as a bug but as a penalty that teaches the policy to avoid confusing searches.

Why This Matters

This work addresses a bottleneck that has limited RL-trained retrieval agents: the absence of dense, principled credit assignment for intermediate search actions. Prior methods either needed human-annotated retrieval demonstrations or relied on sparse final-answer signals and heuristic process supervision. By deriving the reward from information theory and validating it empirically, the paper gives the field a formal justification for step-level retrieval rewards rather than an ad hoc one.

Real-world applications:

• Enterprise and legal research assistants that must decide when to search internal document stores, how many hops to take, and when to stop — with reduced query cost and latency from the shorter, more targeted search traces.

• Scientific and medical literature agents performing multi-hop evidence synthesis, where distinguishing a genuinely informative source from a misleading one is a safety-relevant capability.

• Tool-augmented math and coding agents that interleave reasoning with calculator or interpreter calls, as demonstrated on MATH500.

• Deep web research and fact-checking systems operating over long horizons, as tested on WebDetective, where forgetting and being led astray are measurable failure modes.

Industry relevance: The method requires no manual annotation of retrieval steps, which removes a major cost barrier to scaling. Its demonstrated parameter efficiency — a 3B model outperforming larger baselines — makes high-quality agentic retrieval more deployable on commodity hardware, and the shorter response length directly reduces inference cost.

Future Directions

• Removing the gold-answer dependence. The practical reward uses the correct answer's semantic class, which limits applicability to settings without labeled ground truth. Extending to verifiable-but-unlabeled or intrinsically rewarding domains is a natural next step.

• Scaling beyond 7B parameters. All experiments use 3B and 7B models; whether the dense reward continues to help at frontier scale, or whether its benefit is absorbed by stronger base capabilities, remains untested.

• Richer belief representations. The belief state is approximated by clustering a modest number of samples under an NLI threshold. Finer-grained or calibrated uncertainty estimators, and alternatives to semantic entropy as the uncertainty functional, could sharpen the reward.

• Adversarial and noisy retrieval robustness. The paper notes that negative realized gain arises from poisoned or contradictory context and frames it as a useful penalty, but does not systematically study adversarial retrieval settings.

• Longer-horizon credit assignment. Telescoping additivity justifies per-step rewards in theory, but how well this holds for very long multi-hop trajectories and whether discounting or variance reduction helps at scale is unresolved.

Target Audience

Researchers and graduate students working on LLM agents, retrieval-augmented generation, and reinforcement learning for language models will get the most from the theoretical framing and reward design. Practitioners building production search-and-reason pipelines benefit from the empirical results and the parameter-efficiency and response-length findings. Readers without background in information theory, POMDPs, or policy-gradient methods should expect a steep climb through Sections 2 and 3, though the methodology and findings sections are accessible to a technically literate general audience.

Authors’ abstract

Agentic reasoning enables large reasoning models (LRMs) to dynamically acquire external knowledge, but yet optimizing the retrieval process remains challenging due to the lack of dense, principled reward signals. In this paper, we introduce InfoReasoner, a unified framework that incentivizes effective information seeking via a synthetic semantic information gain reward. Theoretically, we redefine information gain as uncertainty reduction over the model's belief states, establishing guarantees, including non-negativity, telescoping additivity, and channel monotonicity. Practically, to enable scalable optimization without manual retrieval annotations, we propose an output-aware intrinsic estimator that computes information gain directly from the model's output distributions using semantic clustering via bidirectional textual entailment. This intrinsic reward guides the policy to maximize epistemic progress, enabling efficient training via Group Relative Policy Optimization (GRPO). Experiments across seven question-answering benchmarks demonstrate that InfoReasoner consistently outperforms strong retrieval-augmented baselines, achieving up to 5.4% average accuracy improvement. Our work provides a theoretically grounded and scalable path toward agentic reasoning with retrieval. The code is available at https://github.com/dl-m9/InfoReasoner