Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents

AI summary

Paper: "Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents" — Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, Zhenzhe Ying (Venus Team, Ant Group; Individual Author; Renmin University of China). arXiv:2510.14967v2 [cs.CL]. License: CC BY-NC-SA 4.0.

Overview

Research area: Reinforcement learning (RL) for large language model (LLM)-based agents that use search tools across multiple turns of interaction.

Technical level: Intermediate. The paper assumes familiarity with policy-gradient RL, group-relative advantage estimation (GRPO), and multi-turn agent rollouts, but its core idea is described at a conceptual level that a reader with basic machine learning background can follow.

Scope (one sentence): The paper introduces IGPO, a reinforcement learning framework that replaces sparse final-answer-only rewards with dense turn-level "information gain" rewards derived from the model's own belief updates, and evaluates it on seven question-answering benchmarks with search agents.

What This Paper Is About

Search agents built on LLMs are trained with RL to call tools like web search over multiple turns, but existing methods reward the agent only when it produces its final answer. This creates a sparse supervision problem that becomes severe when trajectories are long: the paper reports three consequences — advantage collapse (all rollouts in a group get identical rewards, so no gradient signal remains), a lack of fine-grained credit assignment (the correctness of intermediate turns is hidden), and poor sample efficiency (each rollout yields only one outcome signal). The goal is a reward design that gives dense, reliable, turn-by-turn supervision without needing external reward models or expensive simulation.

Key Contributions

1. An analysis of advantage collapse and the limits of existing process rewards. The authors document how outcome-based optimization produces zero-advantage groups (shown as a proportion of zero-advantage groups during training for GRPO versus IGPO on Qwen2.5-3B/7B-Instruct), and argue that existing process-level rewards rely either on external oracle knowledge/reward models or on high-variance Monte Carlo estimation.

2. IGPO, a policy optimization framework built on turn-level information gain. Each turn is treated as an incremental step in acquiring information about the ground truth, and the turn reward is the marginal increase in the policy's log-probability of producing the correct answer. These intrinsic rewards are combined with outcome rewards to form dense signals.

3. A stable optimization recipe plus a vectorized implementation. Information gain rewards and outcome rewards are z-normalized separately within groups and accumulated into turn-level discounted returns, which replace rollout-level advantages in a GRPO-style clipped surrogate objective. The vectorized implementation appends copies of the ground-truth answer and uses a custom attention mask so all turns are computed in a single forward pass.

4. Extensive experiments across in-domain and out-of-domain benchmarks, showing higher accuracy and better sample efficiency than strong baselines, with the largest gains on a smaller 3B model.

Main Findings

• IGPO achieves the best overall accuracy in the main comparison. Averaged across NQ, TQ, HotpotQA, 2Wiki, Musique, Bamboogle, and PopQA, IGPO scores 60.2, which the paper describes as +6.3 over the best baseline, DeepResearcher (53.9). Per-dataset IGPO scores are NQ 46.4, TQ 80.6, HotpotQA 59.0, 2Wiki 72.1, Musique 32.7, Bamboogle 77.0, and PopQA 53.8.

• Training-based methods beat prompt-based methods. All RL-based baselines, whether outcome-reward or step-reward driven, substantially outperform prompt-based approaches such as CoT (avg 23.4), CoT+RAG (avg 36.4), and Search-o1 (avg 36.4).

• Existing step-reward methods are competitive but unstable. StepSearch-base (avg 46.0), StepSearch-instruct (avg 44.5), ReasoningRAG (avg 42.3), and GiGPO (avg 47.2) occasionally beat outcome-reward methods on individual datasets (e.g., StepSearch on Musique) but lag the strongest outcome-reward method overall.

• IGPO also leads among standard RL algorithms under identical settings. In the second main table, IGPO's average of 60.2 exceeds RLOO (49.7), PPO (51.5), GRPO (51.9), Reinforce++ (47.3), and GSPO (52.0).

• Both reward components are necessary. On Qwen2.5-3B-Instruct, outcome-only (labeled IGPO w/ F1, which reduces to standard GRPO) gives 32.3, information-gain-only gives 34.6, and the combination gives 48.9. On Qwen2.5-7B-Instruct the same three settings give 51.9, 53.5, and 60.2 respectively.

• Information-gain-only training does not collapse. The paper reports that IGPO with only the IG reward reaches performance comparable to or exceeding standard GRPO, which the authors interpret as evidence that the reward is not subject to reward hacking.

• Smaller models benefit most. Compared with standard GRPO, IGPO improves the 3B model by +16.6 points (32.3 → 48.9) and the 7B model by +8.3 points (51.9 → 60.2). The authors attribute the larger 3B gain to more severe advantage collapse in weaker models.

• Faster, more stable training curves. Across all seven datasets, IGPO is reported to steadily outperform its two ablated variants throughout training, converging to higher F1 scores and maintaining stable improvement over steps.

• Larger reduction in ground-truth answer entropy. Measuring entropy change from the initial query (Turn 0) to the last non-answer turn (T-1) during training, the paper reports IGPO consistently achieves a larger entropy reduction than GRPO, indicating intermediate steps move the policy closer to the ground-truth answer distribution.

• Better token efficiency. Measured as performance improvement per token used for gradient updates, performance rises more rapidly under IGPO and the gap over GRPO widens as training progresses.

• Negligible computational overhead. IGPO's info-gain reward computation adds 0.0227s per step, reported as a <0.4% increase for info-gain reward computation and <0.02% end-to-end, giving training speed nearly identical to GRPO. The vectorized implementation is reported to yield an asymptotic speedup of approximately T/3 (e.g., 3× for T = 10).

Methodology in Plain English

The authors start from the observation that a search agent's multi-turn rollout is really a process of accumulating evidence: the first turn states a question, then each turn does reasoning, issues a tool call, and receives documents back, until a final answer turn.

Their core move is to ask, at every turn, a simple question: "How much more likely is the model to produce the correct answer now than it was before this turn?" They measure this by teacher-forcing the ground-truth answer and computing its average log-probability given the rollout prefix up to each turn, then taking the difference between consecutive turns. That difference is the turn-level "information gain" reward. Because it is computed under the model's own probabilities with respect to the known ground truth, it is intrinsic — no separate reward model, no human process labels, no Monte Carlo rollouts are needed. The authors apply a stop-gradient to this reward because of its log-probability origin.

Because the final answer still matters, IGPO keeps the standard outcome reward (word-level F1 against the ground truth, with a negative penalty for malformed output) and combines it with the turn-level rewards. To stabilize learning, information gain rewards and outcome rewards are each z-normalized within the rollout group separately, and then accumulated into a discounted return that assigns credit to each turn while accounting for future turns. The discount factor γ is set to 1.0 with no further tuning.

Optimization then follows the GRPO pattern — a clipped surrogate objective with a KL penalty to a reference model — but the single rollout-level advantage is replaced by the per-turn discounted return. Only decision tokens (reasoning, tool calls, answers) receive gradients; the raw tool responses from the environment are masked out.

For efficiency, rather than running T separate forward passes, the authors append T copies of the ground-truth answer to the end of the trajectory and apply a custom attention mask so each copy can only see its corresponding turn. This computes all turn log-probabilities in one forward pass, and the paper states the resulting overhead is negligible because the ground-truth answer is far shorter than the full reasoning trajectory.

Experimental setup: Backbone model is Qwen2.5-7B-Instruct, trained with the verl framework. In-domain datasets are NQ, TQ, HotpotQA, and 2Wiki; out-of-domain datasets are MusiQue, Bamboogle, and PopQA. The metric is word-level F1. Each training step samples 32 prompts with 16 rollouts per prompt, and the maximum dialogue turns are set to 10. The tool is the google search API, and settings follow DeepResearcher. Reported baseline numbers for other agentic RL methods are copied from their original papers; all RL training methods are run with the same hyperparameter configuration.

Why This Matters

Impact on research. The paper reframes process-level reward design as an intrinsic quantity — the model's own belief update about the ground truth — rather than something borrowed from an external judge or estimated by simulation. If the reported behavior holds up, it offers a simpler alternative in the design space of fine-grained credit assignment for long-horizon agents, and it directly targets the advantage-collapse failure mode that makes group-relative RL unstable on hard multi-turn tasks.

Real-world applications:

• Multi-hop question answering systems that must search the web across several turns to assemble an answer from separate sources.
• Enterprise or domain assistants that call internal retrieval tools iteratively, where intermediate retrieval decisions matter as much as the final response.
• Any tool-using agent with an objectively checkable outcome, where training data is scarce and each gradient update must count.
• Smaller, cheaper deployed models: the paper's largest measured improvement is on the 3B model, which is the regime where serving costs are lowest.

Industry relevance. The work originates from the Venus Team at Ant Group and was supported by the Ant Group Research Intern Program. The reported overhead figures (0.0227s per step; <0.02% end-to-end) and the fact that IGPO uses the same framework (verl) and hyperparameters as GRPO are relevant to practitioners who need accuracy gains without retooling a training stack. The code is released at https://github.com/GuoqingWang1/IGPO.

Future Directions

• Removing the dependence on ground-truth answers. The authors explicitly state this limitation: IGPO relies on the availability of ground-truth answers, which restricts its applicability in open-ended settings. They plan to extend IGPO to broader agentic scenarios beyond search, including tasks without explicit supervision.
• Verifying generality beyond search agents. The evaluation covers seven QA benchmarks with a web search tool; whether the information gain reward transfers to other tool sets and task families is not established in this paper.
• Understanding failure modes and spurious correlations. The paper points to analysis of spurious correlations in Appendix D.4 and details on IGPO's failure modes in Appendix E, but does not state their conclusions in the main text — a natural area for follow-up scrutiny.
• Reward design variants. The authors mention comparisons of different information gain bases (probability versus log-probability) and normalization strategies (joint versus separate) in Appendix D.1, leaving open which design choices are essential versus incidental.

Target Audience

This paper is most useful to RL and LLM-agent researchers and engineers who train tool-using models and are contending with sparse outcome rewards — particularly those already using GRPO-style group-relative optimization and looking for denser credit assignment without adding a reward model or increasing compute. It is also relevant to graduate students and practitioners studying process-level rewards, multi-turn agent training, and sample efficiency, and to applied teams evaluating whether a small-model deployment can be pushed toward the accuracy of a larger one. Readers without background in policy-gradient RL will find the conceptual idea accessible, but the objective functions in Sections 2 and 3 assume some familiarity with actor-critic-free RL objectives and clipping.

Authors’ abstract

Large language model (LLM)-based agents are increasingly trained with reinforcement learning (RL) to enhance their ability to interact with external environments through tool use, particularly in search-based settings that require multi-turn reasoning and knowledge acquisition. However, existing approaches typically rely on outcome-based rewards that are only provided exclusively upon generating the final answer. This reward sparsity becomes particularly problematic in multi-turn settings, where long trajectories exacerbate three critical issues: (i) advantage collapse, where all rollouts receive identical rewards and provide no useful learning signals; (ii) lack of fine-grained credit assignment, where the correctness of intermediate turns is obscured, especially in long-horizon tasks; and (iii) poor sample efficiency, where each rollout yields only a single outcome signal, leading to low data utilization. In this paper, we propose Information Gain-based Policy Optimization (IGPO), a simple yet effective RL framework that provides dense and intrinsic supervision for multi-turn agent training. IGPO models each interaction turn as an incremental process of acquiring information about the ground truth, and defines turn-level rewards as the marginal increase in the policy's probability of producing the correct answer. Unlike prior process-level reward approaches that depend on external reward models or costly Monte Carlo estimation, IGPO derives intrinsic rewards directly from the model's own belief updates. These intrinsic turn-level rewards are combined with outcome-level supervision to form dense reward signals. Extensive experiments on both in-domain and out-of-domain benchmarks demonstrate that IGPO consistently outperforms strong baselines in multi-turn scenarios, achieving higher accuracy and improved data efficiency. Our code is available at https://github.com/GuoqingWang1/IGPO.