Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning

AI summary

Authors: Fengran Mo, Yifan Gao, Sha Li, Hansi Zeng, Xin Liu, Zhaoxuan Tan, Xian Li, Jianshu Chen, Dakuo Wang, Meng Jiang arXiv: 2601.13115v2 [cs.CL], 15 Apr 2026 (published 2026-01-19)

Overview

Research area: Natural Language Processing — multi-turn conversational search, retrieval-augmented generation, and reinforcement learning (RL) for agentic large language models.

Technical level: Advanced. The paper assumes familiarity with RL policy optimization (GRPO/PPO), reward shaping, dense retrieval, conversational query rewriting, and mixed-initiative dialogue.

Scope in one sentence: The paper proposes ConvAgent, an RL-trained conversational agent that interleaves reasoning, search, and mixed-initiative actions across dialogue turns using a decomposed three-part reward.

What This Paper Is About

Conversational search systems usually run a static "rewrite, retrieve, generate" pipeline in which each stage is trained separately and needs its own supervision (rewritten queries, relevance labels, gold answers). Deep search agents that jointly optimize retrieval and generation with RL exist, but they target single-turn, self-contained questions and therefore do not handle context-dependent user intent that evolves across turns. This paper builds a conversational agent that learns, through RL with tailored rewards, to rewrite context-dependent queries, use retrieved evidence in its answers, and decide when to clarify or refuse — all within one framework.

Key Contributions

1. An agentic framework for conversational search that jointly optimizes history-conditional query generation, utilization of search results for answer generation, and interactive action decisions through mixed-initiative capabilities.

2. A decomposed RL reward design that adds intermediate components — a search optimization (information gain) reward and a mixed-initiative action (MIA) reward — on top of the outcome reward, enabling exploratory and adaptive behavior that interleaves search and reasoning across turns.

3. Experiments on four conversational search benchmarks (TopiOCQA, INSCIT, QReCC, CORAL), covering in-domain and out-of-domain settings, showing improvements over separated pipeline systems and existing agentic systems.

4. Analyses of why the rewards work, including ablation over reward components, a GRPO-versus-PPO comparison, mixed-initiative action accuracy with downstream impact, correlation between search quality and answer quality, and the effect of conversational context on effectiveness and reasoning-token efficiency.

Main Findings

• Answer generation (Table 1, 7B model): ConvAgent-7b (Qwen2.5-7b) reaches F1 40.3 / EM 23.3 on TopiOCQA, F1 30.1 / LLM-Jud. 58.5 on INSCIT, F1 30.4 / LLM-Jud. 64.6 on QReCC, and F1 28.9 / LLM-Jud. 46.8 on CORAL. The paper states these are the highest F1-scores on TopiOCQA, QReCC, and CORAL, surpassing agentic systems such as AgenticLM, Search-R1, and ChatR1, and also exceeding proprietary LLMs with direct inference (ChatGPT 25.5 TopiOCQA F1; Claude Sonnet-3.5 27.2 TopiOCQA F1). The † marker indicates significant improvements via paired t-test at p < 0.05 over the two comparable agentic systems, Search-R1 and AgenticLM.

• Where ConvAgent is not best: On INSCIT answer generation, UniConv (Mistral-2-7b) reports F1 33.2 versus ConvAgent-7b's 30.1, and on QReCC ChatR1-7b reports F1 31.0 versus ConvAgent-7b's 30.4. The paper describes its model as best or second-best on nearly all datasets and metrics.

• Retrieval performance (Table 2, NDCG@3): ConvAgent-7b scores 29.4 on TopiOCQA, 30.8 on INSCIT, 39.6 on QReCC, and 34.9 on CORAL — the best on all four datasets for both the 3B and 7B sizes. ConvAgent-3b scores 22.7, 24.7, 37.2, and 31.6. Gains are described as more notable on TopiOCQA and INSCIT, where topic switches and mixed-initiative phenomena make retrieval harder.

• Ablation on reward components (Table 3, 7B): The full model is best overall. Removing the information gain (IG) reward drops TopiOCQA to 37.8 F1 / 20.8 EM, INSCIT to 17.6 F1, QReCC to 11.2 F1, and CORAL to 14.8 F1. Removing the MIA reward gives 39.5 TopiOCQA F1, 24.3 INSCIT F1, 30.4 QReCC F1 (above the full model's 27.4), and 24.1 CORAL F1. Removing both rewards causes a sharp decline (37.0, 9.1, 8.6, 3.8 F1 respectively). The paper reports the IG reward contributes more than the MIA reward, and the MIA reward is not always effective on datasets without mixed-initiative phenomena, such as QReCC.

• GRPO outperforms PPO: Replacing GRPO with PPO yields consistently lower results (e.g., TopiOCQA 37.4 F1 / 21.6 EM, INSCIT 29.4 F1, QReCC 24.7 F1, CORAL 20.6 F1), which the authors attribute to more stable training with GRPO.

• Mixed-initiative actions (Table 4): A backbone trained without the action reward almost never takes mixed-initiative actions (No Answer Acc. 1.42, Clarify Acc. 3.67). Training with the reward and separately parsed labels reaches 11.51 No Answer Acc. and 83.19 Clarify Acc., with NDCG@3 improvement of 3.9 and F1 improvement of 5.8. Combining all mixed-initiative content in a single span is worse (6.99 and 78.35, with 0.2 NDCG@3 and 0.8 F1 improvement). Qwen-3 series models show a scaling effect on these metrics (Qwen-3-4b: 24.86 / 80.21; Qwen-3-8b: 20.99 / 82.00; Qwen-3-14b: 16.57 / 83.70; Qwen-3-32b: 2.21 / 81.50) but tend to respond more aggressively without ground-truth evidence.

• Search quality is not the whole story (Figure 2): Search-R1 can reach higher information gain but still answers worse than AgenticLM and ConvAgent, which the authors say shows that without optimization for leveraging retrieved content, agents answer from parametric knowledge. ConvAgent is reported to integrate retrieved information better through the information gain reward.

• Context and efficiency (Figure 3): All models decline as conversations progress on TopiOCQA, with ConvAgent slightly better. On INSCIT, agentic models (ConvAgent and AgenticLM) increase performance in longer conversations, while Search-R1 does not; ConvAgent is reported to use one-third of the reasoning token cost of AgenticLM.

Methodology in Plain English

Setup. Each conversation has multiple turns. At turn n, the system receives the conversation history H_n = {q_i, a_i} for i = 1 to n−1 plus the current query q_n, and must produce an answer a_n grounded in an external passage collection C. The model may also reject unanswerable queries or ask for clarification.

Rewards. The overall trajectory reward is R(τ) = R_outcome + 0.5 × (R_IG + R_MIA).

• Outcome reward: compares the predicted answer to the gold answer with a task metric (Exact Match or F1).
• Information gain (IG) reward: the agent rewrites the context-dependent query into a stand-alone query q'_n, an ad-hoc retriever returns the top-k passages, and the reward measures overlap between those passages and the gold answer. For long gold answers the metric is F1; for short answers it is accuracy, defined as whether the gold answer is contained as a sub-string in the retrieved passages. This avoids needing relevance judgments between queries and gold passages.
• Mixed-initiative action (MIA) reward: the paper states four types of action following Wu et al. (2023) but lists the reaction set as {answer, clarify, no answer}. The reward is 1 if the generated label tokens match the annotated action set for that turn and −0.5 otherwise — a penalty that discourages unnecessary clarifications or refusals.

Optimization. The model is trained with Group Relative Policy Optimization (GRPO), which removes the need for explicit reward and value models; PPO is implemented as an alternative for comparison. The objective maximizes R(τ) while penalizing KL divergence from the reference policy.

Clarification usage. When a clarifying question q^c_n is generated, it is concatenated with the rewritten query as Ret(q'_n ∘ q^c_n) for retrieval, and the clarified question replaces the original query when generating the final answer.

Training and infrastructure. Backbones are Qwen-2.5-3b and Qwen-2.5-7b with an E5 retriever, run on four NVIDIA A100 80GB GPUs. Top-3 retrieved passages are used per turn for all compared systems. TopiOCQA and INSCIT form the training set; evaluation covers in-domain and out-of-domain settings. Hyperparameters: batch size 256, max prompt length 8192 tokens, learning rate 1e-6, 150 total training steps. The MIA reward is applied only on INSCIT, the only dataset with that supervision. Metrics include NDCG@3, the token-/string-based InfoGain overlap measure, F1, Exact Match (on the short-answer TopiOCQA), and LLM-as-a-Judge scoring based on Qwen-3-4B.

Dataset scale (Table 5). TopiOCQA: 205 conversations, 2,514 turns, 25M collection, 12.9 average queries, 9.0 average passages. QReCC: 2,775 conversations, 16,451 turns, 54M collection, 5.3 average queries, 1.6 average passages. CORAL (only the level-d subset, the most challenging with the longest conversations): 200 conversations, 2,153 turns, 0.2M collection, 10.8 average queries, 2.4 average passages. INSCIT: 469 conversations, 2,767 turns, 49M collection, 5.9 average queries, 1.6 average passages.

Baselines. Separated retriever-generator pipelines: ChatQA, UniConv, REFRAG, EvoRAG. Agentic systems: Search-R1, ConvSearch-R1, ChatR1, AgenticLM. Plus an SFT baseline and direct inference from ChatGPT (GPT-3.5) and Claude (Sonnet-3.5). ConvSearch-R1 and ChatR1 need rewritten queries from the original datasets or LLM-generated pseudo data as extra supervised signal. The paper notes it used AI assistants (ChatGPT and Claude) only for testing in its experimental scenarios, not for coding or writing.

Why This Matters

Impact on research. The paper argues that single-turn deep search agents cannot handle user intent that shifts across turns, and that the common "rewrite, retrieve, generate" pipeline optimizes stages separately while missing mixed-initiative decisions. It shows that intermediate rewards based on retrieved-content overlap with gold answers can supervise query rewriting without gold rewritten queries, and that RL training changes behavior qualitatively — it also documents where the approach falls short, such as the declining benefit of the MIA reward on datasets without mixed-initiative phenomena.

Real-world applications:

• Customer-support and e-commerce assistants that must resolve context-dependent follow-up questions and know when to ask a clarifying question instead of guessing.
• Enterprise/domain knowledge assistants where follow-up questions reference earlier turns and answers must be grounded in internal document collections.
• Exploratory research or analyst tools where a user's topic drifts across a session and the system must track the shift rather than decompose a single self-contained question.
• Assistants that must recognize unanswerable or ambiguous requests, where refusing or clarifying is preferable to producing an ungrounded answer.

Industry relevance. The work targets moderate-size open backbones (Qwen-2.5-3b/7b) trained with GRPO on four A100 80GB GPUs and 150 training steps, and reports outperforming proprietary direct-inference baselines on several metrics — which is relevant for teams that need agentic conversational search without relying solely on very large proprietary models. The reported one-third reasoning-token cost relative to AgenticLM speaks to inference economics, while the authors flag additional training and inference compute as a limitation.

Future Directions

1. Efficiency optimization. The authors list the extra computational cost of RL training and inference as an explicit limitation requiring further work.

2. Broader experimental configurations. They note they did not test more types and sizes of LLM backbones and used fixed hyperparameters for reward modeling, suggesting reward settings could be explored further.

3. Better intermediate rewards. The paper frames how to design intermediate rewards that improve use of search results for response generation quality as an open question.

4. Evaluation reliability for long-form answers. The authors observe that LLM-as-a-Judge scores do not always align with F1 scores and call this an open question in evaluating long-form answers; they also note that mixed-initiative datasets beyond INSCIT are not reported, since INSCIT is the only dataset annotated with that supervision.

Target Audience

Researchers and practitioners working on conversational search, conversational question answering, retrieval-augmented generation, and RL post-training of LLM agents. It is most useful to readers who already understand RL policy optimization and dense retrieval metrics, and to engineers building multi-turn assistants that need query rewriting, grounded answering, and clarification-or-refusal behavior in a single trainable model.

Authors’ abstract

Large Language Models (LLMs) have become a popular interface for human-AI interaction, supporting information seeking and task assistance through natural, multi-turn dialogue. To respond to users within multi-turn dialogues, the context-dependent user intent evolves across interactions, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Existing studies usually follow static rewrite, retrieve, and generate pipelines, which optimize different procedures separately and overlook the mixed-initiative action optimization simultaneously. Although the recent developments in deep search agents demonstrate the effectiveness in jointly optimizing retrieval and generation via reasoning, these approaches focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods by surpassing several existing strong baselines.