KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents

AI summary

Overview

Research area: Reinforcement learning for LLM-based "deep search" agents, specifically the credit-assignment problem in long-horizon, multi-turn search trajectories.

Technical level: Advanced — the paper assumes familiarity with GRPO-style policy optimization, advantage estimation, and PPO clipping.

Scope: A one-page-preview summary of KnowGPO, a reinforcement learning framework that groups trajectory steps by the set of atomic facts they have verified (a "knowledge state") rather than by surface observations, and uses incremental knowledge gain as a step-level process reward.

What This Paper Is About

Deep search agents solve knowledge-intensive questions through many rounds of querying, reading evidence, reformulating, and deciding when to stop. Standard reinforcement learning gives such an agent only one reward at the very end (right or wrong answer), which says nothing about which of its many intermediate search decisions were useful, redundant, or harmful. KnowGPO's goal is to assign credit to individual steps using something externally checkable — the specific facts a step has established — instead of matching raw web pages (as GiGPO does) or watching the model's own confidence (as IGPO does).

Key Contributions

1. Knowledge-space credit assignment. The framework represents each trajectory step by the subset of query-specific atomic knowledge conditions it has satisfied, and groups steps that share the same knowledge state for within-group comparison. This replaces surface-observation matching, which the authors argue is too brittle for deep search.

2. Incremental knowledge gain as a process reward. The number of newly satisfied conditions between consecutive steps serves as a dense, environment-grounded step reward, requiring no learned critic and no reliance on the policy's own answer probabilities.

3. A two-stage construction pipeline for knowledge conditions. Stage 1 derives a reasoning skeleton from the query alone using entity placeholders; Stage 2 grounds those placeholders using the reference answer and a successful reference trajectory obtained via rejection sampling. The authors require the resulting condition set to be atomic, complete, and referentially consistent.

4. Empirical demonstration across three benchmarks. On Xbench (2505 and 2510 splits), BrowseComp-ZH, and the text-only GAIA subset, KnowGPO beats outcome-only and process-supervision baselines by 3.8 to 4.2 points, with an 8B model matching or exceeding most 32B open-source systems.

Main Findings

• Consistent gains over GRPO, GiGPO, and IGPO. On BrowseComp-ZH, KnowGPO reaches 38.8 versus 32.9 for GRPO, 37.3 for GiGPO, and 32.2 for IGPO. On GAIA it reaches 40.2 versus 35.0 for GRPO and GiGPO and 37.9 for IGPO. On Xbench-2505 it reaches 48.0 (GRPO 46.0, GiGPO 44.0, IGPO 46.0) and on Xbench-2510 it reaches 36.0 (GRPO 34.0, GiGPO 35.0, IGPO 34.0). All runs start from the same SFT checkpoint, whose scores are 30.1 on BrowseComp-ZH, 31.1 on GAIA, 39.0 on Xbench-2505, and 29.0 on Xbench-2510.

• KnowGPO is the only method that improves on every split. The paper notes GiGPO degrades by 2.0% on Xbench-2505 relative to GRPO despite winning on BrowseComp-ZH, and IGPO matches GRPO on Xbench-2505 while barely improving on BrowseComp-ZH. KnowGPO's gains over GRPO are reported as 5.9% on BrowseComp-ZH and 5.2% on GAIA, exceeding the strongest step-level baseline by 1.5% and 2.3% on those two benchmarks, with an averaged gain of 8.5% over the SFT initialization.

• 8B scale approaches 32B open-source agents. On BrowseComp-ZH, 38.8 surpasses WebExplorer-8B by 6.8% and WebSailor-32B and DeepDive-32B by 13.3% and 9.1% respectively, trailing DeepMiner-32B-RL (40.1) by only 1.3%. On Xbench-2505, 48.0 outperforms MiroThinker-v1.0-8b (60.6 is its Xbench-2505 score — the paper states a 12.6% improvement over the 8B MiroThinker). A gap to proprietary systems remains: 23.5% to Kimi-K2.5 (62.3) on BrowseComp-ZH and 28.1% to Claude-4-Sonnet (68.3) on GAIA.

• Training curves separate early. All four methods track similarly for the first 16 steps; KnowGPO opens a gap by around step 20 and plateaus near step 32 at roughly 50% validation accuracy on Xbench-2505, while GRPO, GiGPO, and IGPO settle below 46%.

• The mixing coefficient peaks at 0.3. Sweeping ω over {0, 0.3, 0.5} gives Xbench-2505 / Xbench-2510 / BrowseComp-ZH / GAIA scores of 45.0 / 28.0 / 34.0 / 34.0 at ω=0, 48.0 / 36.0 / 38.8 / 40.2 at ω=0.3, and 47.0 / 35.0 / 36.3 / 37.9 at ω=0.5. Notably, ω=0 (grouping but no step reward) still beats GRPO on BrowseComp-ZH and GAIA.

• Both components are individually necessary. Adding incremental knowledge gain to GRPO without grouping improves 1.0% to 2.8% across benchmarks (46.0/34.0/34.6/37.8); adding knowledge-state grouping on top yields a further 1.0% to 4.2%, with the largest gain (+4.2%) on BrowseComp-ZH.

• Incremental gain beats discounted future returns. Replacing incremental gain with the discounted future return G_t inside the same grouping framework drops accuracy by 3.0%, 6.0%, and 3.4% on Xbench-2505, BrowseComp-ZH, and GAIA respectively (45.0/35.0/32.8/36.8).

• Knowledge states group far more steps than observations. On 500 queries from Xbench and BrowseComp with up to 16 rollouts each (5,922 valid trajectories, 55,571 steps), observation-anchored grouping (BGE-large cosine similarity at threshold 0.9) leaves an effective step ratio of 51.8% with a 79.4% singleton ratio; action-anchored reaches 79.4% ESR with 72.0% singletons; knowledge-anchored reaches 97.8% ESR with 19.0% singletons, mean non-singleton group size 21.28, and cross-rollout coverage 4.66. The paper reports a case where two steps with 0.987 retrieval similarity differ sharply in progress (0 of 12 conditions versus 8 of 12).

• Search becomes adaptive, not merely shorter. On queries solved only by KnowGPO, it uses 9.83 turns versus 11.55 for GRPO; on queries solved only by GRPO it uses 10.36; on queries both fail it uses 10.85, the longest of all subsets (GRPO: 11.39, 7.03, 9.51, 11.55 for both-fail, both-pass, GRPO-only, and KnowGPO-only respectively).

• Overhead is negligible. Per-iteration profiling over three training epochs shows trajectory generation at 2,574s, policy update at 583s, probability computation at 310s, and knowledge verification at 356s, with incremental gain computation at 2.73s and group construction at 0.03s. Verification runs asynchronously behind the next rollout, and the two bookkeeping operations account for under 0.08% of the budget.

Methodology in Plain English

For each training question, the system first breaks down "what you need to know" into a checklist of small, single-fact statements — for example, that a certain person was born in a certain city, that the city is in a certain country, and so on. Each checklist item is written with placeholders; a second pass fills in the real entities using the gold answer plus one successful sample trajectory.

During training, the agent runs several complete search attempts per question. After every search step, an LLM verifier (DeepSeek-v3.2 in the experiments) checks which checklist items the newly retrieved evidence has just made verifiable. These are accumulated with a logical OR, so knowledge can only grow — a design choice that also dampens one-off verifier mistakes.

Each step is then labeled by the checklist state it started from. Steps across different attempts that began from the same state are placed in one group, and each step's reward is the number of new checklist items it added. Within the group, this reward is normalized into a step-level advantage, so a step is judged against alternatives taken from the exact same starting point. Separately, the ordinary whole-trajectory right/wrong outcome gives an episode-level advantage. The two are blended with a single weight ω (0.3 works best) and fed into a standard PPO-style clipped objective with a KL penalty to the reference policy.

The key architectural trick is that the design intentionally ignores what the pages looked like. Two attempts that read completely different sources but now know the same facts land in the same group and become directly comparable.

Why This Matters

Impact on research. The paper reframes credit assignment for search agents as a question about epistemic progress rather than state recurrence or model confidence, and provides diagnostic metrics (effective step ratio, singleton ratio, cross-rollout coverage) showing why prior anchors fragment. It also shows that a knowledge-grounded process reward outperforms discounted future returns even when the grouping mechanism is held fixed, which is a useful, separable result for anyone designing dense rewards.

Real-world applications:

• Deep research assistants that must justify multi-hop answers with traceable evidence.
• Fact verification and investigative journalism, where knowing which claims are already established matters.
• Enterprise competitive intelligence and due diligence over heterogeneous web and document sources.
• Multi-turn tool-using agents more broadly, wherever "what has the agent learned so far" can be written as a checklist.

Industry relevance. The 8B-agent-approaching-32B result matters for deployment cost, and the profiling result — verification hidden behind rollout, bookkeeping under 0.08% of the budget — means the method can be added to an existing GRPO pipeline without a meaningful throughput penalty. The dependence on a strong external verifier and a per-query checklist, however, implies a data-preparation and inference cost that production teams would need to absorb.

Future Directions

• Extending beyond discretizable tasks. The authors state KnowGPO assumes progress can be captured by a finite set of verifiable knowledge conditions, which does not fit open-ended agentic tasks; how to relax this is open.
• Reducing verifier dependence. Errors from the LLM verifier can still propagate into step-level advantages despite monotonic accumulation; better verification or noise-robust aggregation is unresolved.
• Removing the reference-trajectory requirement. The construction pipeline needs at least one successful rollout per query to ground entity placeholders, which limits use where successful rollouts are expensive or unreachable.
• Testing scale and initialization transfer. All experiments use Qwen3-8B; whether the effects hold for much smaller or larger backbones, or for reasoning-trained initializations, is explicitly left unverified.

Target Audience

Reinforcement learning researchers and engineers working on tool-using or search-augmented LLM agents who already understand GRPO and advantage estimation and want a drop-in alternative to observation matching or confidence-based process rewards. It is also useful for practitioners building deep-research products who need to know the cost profile and the data prerequisites (checklists plus reference trajectories plus a strong verifier) before adopting it. Readers new to policy optimization will find the problem framing accessible but the method section demanding.

Authors’ abstract

Deep search agents solve knowledge-intensive tasks through long-horizon, multi-turn interactions, where assigning credit to individual search decisions is fundamentally difficult. Outcome-based reinforcement learning provides only a single terminal reward shared across an entire trajectory. Recent process-supervision methods aim to assign credit at finer granularity, but they typically anchor credit to either surface-level trajectory states or model-internal signals, both of which become unreliable in deep search where observations are heterogeneous and retrieved contexts are long and noisy. We propose KnowGPO, a reinforcement learning framework that aligns trajectories by the task-relevant knowledge they have acquired rather than by their surface observations. For each query, KnowGPO decomposes the information required to derive the answer into atomic knowledge conditions, and represents the state of a trajectory at each step by the subset of conditions it has satisfied. Trajectories sharing the same knowledge state are grouped for within-group comparison, and the incremental knowledge gain between consecutive steps serves as an environment-grounded process reward. Experiments on Xbench, BrowseComp-ZH, and GAIA show KnowGPO consistently outperforms outcome-only (GRPO) and process-supervision baselines (GiGPO, IGPO) by 3.8 to 4.2 points, with our 8B model rivaling or exceeding most 32B open-source systems.