BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search

AI summary

Overview

• Research area: Reinforcement learning for large language model agents, specifically agentic search (multi-turn retrieval-augmented reasoning) and model reliability/calibration.
• Technical level: Advanced. The paper assumes familiarity with GRPO, rule-based reward design, ReAct-style agent loops, and retrieval-augmented generation.
• Scope: The paper diagnoses why RL-trained agentic search models almost never say "I DON'T KNOW" (IDK), and proposes BAPO, an RL framework with a group-based boundary-aware reward plus an adaptive reward modulator, evaluated on four multi-hop QA benchmarks with Qwen2.5-Instruct backbones at 3B, 7B, and 14B scale.

What This Paper Is About

RL-based agentic search models (Search-R1, ReSearch, R1-Searcher, Tool-Star) improve accuracy on complex knowledge-intensive questions by letting LLMs plan and call a search tool over multiple turns. However, these models almost never admit "I DON'T KNOW" even when retrieved evidence is insufficient or their reasoning has hit a limit, producing plausible but unreliable answers that users cannot easily verify. The paper's goal is to train agents that recognize their own reasoning boundaries and refuse appropriately, without sacrificing accuracy or learning to use IDK as a shortcut.

Key Contributions

1. Diagnosis of a reliability gap. The authors identify that RL training with only correctness rewards substantially degrades boundary awareness: comparing Qwen2.5-7B-Instruct to its GRPO-trained variant ReSearch-Qwen-7B-Instruct, accuracy rises from 41.25 to 51.25 while the IDK rate collapses from 18.75% to 3.65% and precision moves only from 50.76 to 53.24.
2. BAPO (Boundary-Aware Policy Optimization). A GRPO-based RL framework with two components: a group-based boundary-aware reward that grants 0.5 to an IDK response only when every rollout in the group has a non-positive correctness reward, and an adaptive reward modulator that controls when that reward is active.
3. Adaptive reward modulator. A two-level mechanism that prevents reward hacking: a stage-level switch (reward on only after the IDK proportion falls below a threshold α, and fully applied once validation score plateaus for 5 consecutive steps, with resampling of groups lacking correct answers up to k times) and a sample-level switch (reward deactivated for high-diversity rollouts, defined as having at least G/2 distinct answers, and applied for low-diversity ones).
4. Empirical validation on four benchmarks. Experiments on HotpotQA, MuSiQue, 2WikiMultiHopQA, and Bamboogle with Qwen2.5-Instruct at 3B/7B/14B, using 5,000 training samples, showing reliability gains without compromising average accuracy.

Main Findings

• RL with correctness rewards erodes refusal behavior. Before RL, Qwen2.5-7B-Instruct had accuracy 41.25, precision 50.76, and IDK rate 18.75%; after GRPO training with a correctness reward (ReSearch-Qwen-7B-Instruct), accuracy was 51.25, precision 53.24, and IDK rate 3.65% — the authors interpret the shrinking accuracy–precision gap as weakened boundary awareness.
• Naively rewarding IDK causes reward hacking. Adding a fixed 0.5 reward to IDK responses when a rollout group lacks correct answers prioritized IDK over incorrect answers, and the model learned to default to IDK instead of solving hard problems, impeding accuracy gains compared with the vanilla reward.
• BAPO improves reliability across all four benchmarks. With Qwen2.5-7B-Instruct, BAPO achieved HotpotQA 58.0 acc / 66.6 prec / 65.5 reliability, MuSiQue 29.5 / 38.8 / 36.6, 2WikiMultiHopQA 57.0 / 64.1 / 63.3, and Bamboogle 57.6 / 61.5 / 61.2, which the authors report as a +15.8 average reliability improvement over all compared methods.
• Gains over GRPO come at small accuracy cost. Compared with GRPO (60.0/29.5/59.5/57.6 accuracy on the four benchmarks), BAPO showed average reliability improvements of 9.7%, precision improvements of 11.8%, and accuracy decreases of only 2.2%.
• Sample efficiency versus large-scale agentic search methods. Search-R1 used approximately 90k training samples and ReSearch used 19,938 samples; BAPO-trained models used only 5,000 samples while achieving competitive accuracy and substantially higher reliability.
• Over-conservative reliability baselines collapse. Reliable RFT gained precision but suffered a 27-point accuracy drop relative to the TIR Prompt baseline, hurting overall reliability (for example, HotpotQA reliability 40.2 versus 60.6 for Reliable TIR Prompt). Reliable TIR Prompt improved precision without sacrificing accuracy but could not further improve problem-solving ability.
• Generalization across scales. Relative to Reliable TIR Prompt, BAPO improved reliability by an average of 76.1% at 3B and 27.4% at 14B; relative to GRPO, reliability advantages were 13.9% (3B) and 11.9% (14B). On 3B, BAPO reached 44.8 acc / 52.9 prec / 51.3 reliability versus GRPO's 45.1/45.1/45.1; on 14B, 54.0 / 65.7 / 63.3 versus GRPO's 56.6/56.6/56.6.
• Ablations show both modulator levels matter (3B, averaged over four benchmarks). Full BAPO: 44.8 acc, 52.8 prec, 16.8% IDK, 51.3 reliability. Removing the boundary-aware reward entirely: 30.6 / 62.4 / 53.1% / 44.8. Removing the sample-level modulator: 43.3 / 52.0 / 20.4% / 50.1 (a −1.2 reliability change). Removing both modulators: 37.8 / 56.0 / 35.2% / 49.0.
• Training dynamics separate exploration from boundary awareness. On Qwen2.5-14B-Instruct, during the exploration stage (first 60 steps) the average correctness reward rose from 0.3 to 0.5 while the IDK ratio fell from about 20% to 5%; in the plateau stage correctness growth slowed, the boundary-aware reward rose and stabilized between 0.25 and 0.3, and the IDK ratio recovered to over 25%.
• Refusals are mostly justified. Average rejection success rates were 74.7% (3B), 76.7% (7B), and 76.7% (14B). The rate did not grow with model size, which the authors attribute to the GRPO baseline's overall error rate simultaneously decreasing.
• Hyperparameter sensitivity (3B, averaged). α = 0.0 gave 40.7 acc and 40.7 reliability with a 0.0% IDK rate; α = 0.05 gave 51.3 reliability; α = 0.2 gave 49.2; α = 0.3 gave 50.3. For resampling, k = 0 gave 49.0 reliability, k = 1 gave 49.2, k = 2 gave 51.3, and k = 3 gave 51.2.
• Pass@K saturates. Accuracy improved with increasing K and stabilized after K = 32 on Qwen2.5-7B-Instruct, supporting the assumption that failures reflect problems beyond the reasoning boundary rather than sampling accidents.

Methodology in Plain English

The authors formalize agentic search as a ReAct-style sequential decision process: the model produces interleaved reasoning steps, search actions, and retrieved results using <think>, <search>, <result>, and <answer> tags, ending in a final answer, and is optimized with GRPO (a group-relative RL method) in a cold-start-free setting without preliminary in-domain SFT.

Their core idea is that a question lies outside the model's boundary if none of the G sampled trajectories in a group yields a correct answer. In that case, any trajectory that answers IDK receives an extra reward of 0.5, added to the standard correctness reward (format correctness, and character-level F1 against the ground truth, with −1 for wrong format).

Because a flat IDK bonus invites the model to give up, they gate it. During early training (the exploration stage) the IDK reward is off unless the IDK proportion drops below α; once validation score plateaus for 5 consecutive steps (the plateau stage), the reward turns on fully, and groups with no correct rollout are resampled up to k times until the model either produces IDK or a correct answer. At the sample level, queries whose rollouts are diverse (at least G/2 distinct answers, indicating active exploration) have the IDK reward disabled, while low-diversity queries receive it to sharpen boundary awareness.

Training used 5,000 multi-hop QA samples from HotpotQA and 2WikiMultihopQA, with α set to 5% and k set to 2. Retrieval used FlashRAG with E5-base-v2 as the retriever over December 2018 Wikipedia, retrieving top-5 results, with at most 3 tool invocations. Models were trained with batch size 64 for 2 epochs, rollout size 8, and maximum 8192 tokens. Evaluation used 200 test samples each from HotpotQA, 2WikiMultiHopQA, and MuSiQue, and 125 from Bamboogle, with answers scored by GPT-4 as a judge and reported under accuracy (N_c/N), precision (N_c/(N−N_r)), IDK rate (N_r/N), and a reliability metric defined as (1−ρ_IDK)·prec + ρ_IDK·acc.

Why This Matters

The paper reframes agentic search quality as a joint accuracy-and-honesty problem rather than a pure accuracy race, and it supplies a concrete, implementable reward mechanism for teaching refusal behavior without collapsing the model into an over-cautious IDK machine. It also shows a cheap intervention — 5,000 samples — can outperform methods trained on far larger datasets (90k for Search-R1, 19,938 for ReSearch) on reliability.

Real-world applications:

• Enterprise and medical question answering, where a confident fabricated answer is worse than an explicit admission that the available evidence is insufficient.
• Legal and compliance research assistants, where users need to know when retrieved material does not support a conclusion.
• Customer-facing search and support agents, where surfacing uncertainty lets users seek alternative sources instead of acting on misinformation.
• Multi-hop research tools over private corpora, where the retrieval index may simply not contain the needed fact.

Industry relevance: the work targets the deployment risk of long, hard-to-verify agent trajectories, and its recipe — a group-level reward gate plus staged activation — is a general pattern that can be layered onto existing GRPO-based agent training pipelines with modest extra compute.

Future Directions

• Beyond knowledge-intensive QA. The authors state their evaluation focuses on knowledge-intensive tasks and leaves generalizability to other reasoning problems (for example, mathematical reasoning) to be fully explored.
• Scaling beyond 14B. Due to computational constraints, experiments stop at 14B parameters; behavior at larger model scales is untested.
• Real web search instead of local RAG. The high cost of commercial search APIs meant only a local retrieval setup was used, so the noise and dynamism of live web search — which may create additional reliability challenges — remain unexamined.
• Why rejection success rate plateaus with scale. The finding that rejection success rate does not grow from 3B to 14B (74.7%, 76.7%, 76.7%) because GRPO baseline error drops in parallel suggests an open question about how to keep boundary calibration improving as base capability improves.

Target Audience

Researchers and engineers working on RL post-training, LLM agents, and retrieval-augmented generation who need to build systems that are trustworthy rather than merely accurate. It is also relevant to practitioners deploying agentic search in high-stakes domains, and to readers interested in uncertainty calibration, reward hacking, and exploration–exploitation trade-offs in reinforcement learning from verifiable rewards.

Authors’ abstract

RL-based agentic search enables LLMs to solve complex questions via dynamic planning and external search. While this approach significantly enhances accuracy with agent policies optimized via large-scale reinforcement learning, we identify a critical gap in reliability: these agents fail to recognize their reasoning boundaries and rarely admit ``I DON'T KNOW'' (IDK) even when evidence is insufficient or reasoning reaches its limit. The lack of reliability often leads to plausible but unreliable answers, introducing significant risks in many real-world scenarios. To this end, we propose Boundary-Aware Policy Optimization (BAPO), a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy. BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut. Extensive experiments on four benchmarks demonstrate that BAPO substantially enhances the overall reliability of agentic search.