ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering

AI summary

Overview

• Research area: Natural Language Processing, specifically conversational question answering (CQA), retrieval-augmented generation (RAG), and reinforcement learning (RL) for reasoning agents.
• Technical level: Advanced. The paper assumes familiarity with RL policy optimization (PPO, GRPO, GAE), dense retrieval, and conversational search benchmarks.
• Scope: The paper proposes ChatR1, an RL framework that trains a language model to interleave reasoning, search-tool calls, and answer generation across multi-turn conversations, using a new "intent-aware" reward to reduce reward sparsity.

Authors: Simon Lupart, Mohammad Aliannejadi, Evangelos Kanoulas (University of Amsterdam). arXiv:2510.13312v2 [cs.CL]. Code, models and datasets are released on HuggingFace and GitHub (https://github.com/SimonLupart/ChatR1).

What This Paper Is About

Conversational question answering is hard because a user's intent shifts from turn to turn and individual utterances are underspecified ("one", "wind", comparisons to previously cited countries), so a system must resolve references, reformulate the query, retrieve evidence, and answer. Existing CQA systems mostly use static "rewrite, retrieve, generate" pipelines trained by supervised fine-tuning, and existing RL reasoning agents such as Search-R1 are built for single-turn questions only. ChatR1's goal is to let a model learn, through RL, when and how to search and reason across a multi-turn dialogue rather than following a fixed pipeline.

Key Contributions

1. ChatR1, an RL-based reasoning model for CQA that optimizes multi-turn retrieval and generation end-to-end, learning dynamic behavior instead of a static pipeline, with head-to-head experiments across five CQA datasets.
2. An intent-aware reward that reduces reward sparsity by aligning issued search queries with the user's evolving intent, using human-authored query rewrites as supervision during training only.
3. An ablation showing the intent reward beats alternatives, including a retrieval-based reward built on retrieved passage relevance (hit@3), across multiple reward ratios.
4. Analyses of reasoning behavior, including diversity of reasoning-path lengths, Jaccard overlap between consecutive search queries, retrieval-as-a-tool quality, sensitivity to retrieval system choice, and a comparison of PPO versus GRPO training.

Main Findings

• Intent reward drives most of the gain: ChatR1-3B outperforms the same base model trained without the intent reward by an average of 2.2 F1 points, and paired t-tests (p < 0.05) mark those improvements.
• Query-level rewards beat passage-level rewards: Rewarding F1 between issued queries and gold rewrites outperformed rewarding retrieved passages via hit@3. The authors attribute this to denser signals, disentangling formulation errors from retrieval errors, and annotation gaps in passage relevance labels. The best balance found was a reward ratio of 0.2/1.0 between retrieval and generation rewards.
• Strong generation results on five datasets (F1 / BERTScore): ChatR1-3B scores 29.4 / 80.9 on TopiOCQA, 28.0 / 79.2 on QReCC, 33.2 / 85.5 on INSCIT, 26.0 / 83.1 on MultiDoc2Dial, and 19.2 / 84.0 on FaithDial. ChatR1-7B scores 30.6 / 79.5, 31.0 / 80.7, 32.8 / 85.5, 31.2 / 84.5, and 18.1 / 84.8 respectively.
• Smaller model beats commercial systems: ChatR1-3B outperforms closed-source ChatGPT (GPT-3.5) and Claude despite using far fewer parameters, and matches or exceeds 7B supervised RAG baselines (UniConv, ChatRetriever+Mistral) on TopiOCQA, QReCC, and INSCIT — while ChatR1 uses a 300M retriever versus their 7B retrieval backbones.
• Scaling helps: Moving from 3B to 7B yields an average improvement of 1.4 F1 points and 0.5 BERTScore. However, on FaithDial the 7B variant's F1 (18.1) is below the 3B variant's (19.2).
• Generalization across domains: Trained on QReCC and evaluated on the other four datasets, ChatR1-3B shows minimal loss on MultiDoc2Dial (only 0.2) and generally lower loss than baselines, though slightly higher loss on TopiOCQA (attributed to shorter answers in that dataset). It still surpasses ChatGPT zero-shot on three of the four datasets.
• Metric agreement: LLM-as-judge (gpt-4o-mini) and F1 generally agree in-domain, with a Pearson's r of 0.83. Many zero-shot approaches have low F1 but reasonable LLM-as-judge scores, which the authors read as evidence that fine-tuning teaches answer format and word distribution as well as content.
• Reasoning length varies by dataset: MultiDoc2Dial and QReCC show the longest reasoning traces (MultiDoc2Dial involves multi-hop CQA), FaithDial comparatively shorter, and INSCIT the most dispersed distribution — attributed to its mixed-initiative nature.
• Search queries are not near-duplicates: Jaccard overlap between the first and second queries when more than two are generated is 0.52 (TopiOCQA), 0.25 (QReCC), 0.12 (INSCIT), 0.29 (MultiDoc2Dial), and 0.57 (FaithDial) — low on several datasets, indicating exploration.
• Retrieval-as-a-tool is competitive: ChatR1-3B reaches nDCG@3 / Recall@10 of 24.1 / 43.7 on TopiOCQA and 36.4 / 60.2 on QReCC; ChatR1-7B reaches 26.7 / 46.9 and 37.0 / 61.1. Gold rewrites score 30.6 / 52.5 and 40.6 / 67.2 respectively. ChatR1 has no direct retrieval supervision.
• Generation tracks retrieval quality: Both 3B and 7B improve consistently as retrieval accuracy rises, and the 7B model gains uniformly across retrieval bins; the jump from low to mid retrieval quality is more pronounced on QReCC than TopiOCQA because QReCC's longer web passages require more verification.
• PPO versus GRPO: PPO was more stable across datasets; GRPO models usually collapsed after 100 or 200 steps. GRPO nonetheless produced better final performance on two datasets (TopiOCQA: 32.9 vs 29.4; MultiDoc2Dial: 26.5 vs 26.0), while PPO was better on QReCC (28.0 vs 26.9), INSCIT (33.2 vs 29.2), and FaithDial (19.2 vs 17.0).
• Reward quality matters: On TopiOCQA, using weaker pseudo rewrites as the intent signal degrades everything — T5 rewrites give nDCG@3 22.2, F1 24.7, BERTScore 71.7; Mistral rewrites give 26.0, 27.7, 77.6; GPT-4.1 rewrites give 30.6, 29.4, 80.9.
• Retriever sensitivity at inference: Swapping dense retrieval for BM25 drops F1 by 11.8 points for the 3B model and 7.8 points for the 7B model; adding reranking on top of dense retrieval consistently helps (e.g., ChatR1-7B reaches 32.1 on TopiOCQA and 33.1 on QReCC with dense-rerank).

Methodology in Plain English

The base model is Qwen2.5-3B-Instruct or its 7B variant, trained separately on each of five datasets. At every turn it receives the conversation history and the current user utterance, and produces a trajectory: hidden reasoning inside <think> tokens, zero or more <search> queries (capped at two search calls) whose results are injected back between <information> tokens, and finally an <answer>. Retrieval uses e5-base-v2 (intfloat/e5-base-v2) in zero-shot mode, returning the top-3 passages.

Training uses RL rather than imitation. The reward has two parts: a word-level F1 between the generated answer and the gold answer, given only at the end of the trajectory, plus an intent reward equal to the maximum F1 between any issued search query and the gold human rewrite of the user's question. The rewrites act as supervision only during training and are replaced by GPT-4.1-generated rewrites when a dataset lacks them. Optimization uses PPO with a clipped surrogate objective, token-level loss masking, Generalized Advantage Estimation for the critic, and a KL penalty pulling the policy toward a reference policy. Policy and critic are initialized from the same LLM and fine-tuned independently.

Training configuration: batch size 512, PPO micro batch size 64, max prompt length 3500 tokens, learning rate 1e-6 for the actor, 500 training steps with a checkpoint every 50 steps, run on 4 H100 GPUs. The SFT baseline uses batch size 512, micro batch 16, max input length 1024, max generation length 128, learning rate 1e-4, 500 steps with early stopping on validation loss.

Datasets used (train turns / test turns, corpus, average answer length from the paper's statistics table): TopiOCQA (45,450 / 2,514 turns, 25M corpus, 10.8 / 11.2), QReCC (63,501 / 16,451 turns, 54M corpus, 16.8 / 17.6), INSCIT (1,443 / 2,767 turns, 49M corpus, 30.9 / 40.3), MultiDoc2Dial (18,318 / 3,266 turns, 10K corpus, 20.9 / 20.0), and FaithDial (18,357 / 3,539 turns, 21M corpus, 17.0 / 17.6). INSCIT clarification turns were filtered out, which the paper states reduces the test set from 2,767 to 2,197 turns and yields 1,844 training samples. The QReCC corpus is described as seven times larger than MS MARCO.

Why This Matters

For research, the paper shows that the standard RL recipe used for single-turn search agents does not transfer cleanly to conversation: RL alone gave "limited benefits over SFT," and the fix was reward design rather than more training. The intent-aware reward connects the reward-shaping literature to conversational search, where user intent must be inferred rather than given. It also provides a generalization study across topic shifts, mixed initiative, multi-document grounding, and faithfulness, which is unusual for RL reasoning work.

Real-world applications:

• Multi-turn search assistants of the kind the paper cites (Perplexity.ai, ChatGPT deep research), where systems must decide mid-conversation whether to search again.
• Enterprise and government document Q&A, since MultiDoc2Dial is grounded in multiple governmental documents.
• Customer support and task-oriented assistants, where mixed-initiative exchanges (such as clarification questions in INSCIT) require reading user feedback correctly.
• Query reformulation tooling, where the model learns to resolve pronouns and ellipses into self-contained queries without being explicitly trained to rewrite.

Industry relevance is direct: the paper contrasts its open, inspectable reasoning behavior with closed commercial systems whose internals are unclear, and it reports that a 3B model with a 300M retriever can beat much larger supervised retrieval-generation stacks — a cost-relevant result. The released code builds on the Search-R1 repository and veRL, and the training cost is acknowledged as a limitation.

Future Directions

• Dialogue-level optimization with simulated users and preference-based feedback, rather than per-turn answering.
• Longer conversations: current experiments focus on dialogues of moderate length (10–20 turns), while real interactions are longer and demand stronger memory and context modeling.
• Alternative optimization strategies, such as off-policy methods or curriculum-based training, to improve stability and sample efficiency beyond PPO/GRPO.
• Personalization and user-specific adaptation for more proactive, mixed-initiative dialogue.
• Scaling to larger backbones to test whether stronger emergent reasoning and generalization appear, plus more efficient optimization schedules to reduce training and inference cost.

Target Audience

Researchers and practitioners working on conversational search, retrieval-augmented generation, and RL-trained reasoning agents, especially those already familiar with PPO-style optimization and dense retrieval evaluation. It is also useful for applied teams deciding whether to build static CQA pipelines or train reasoning models end-to-end, and for anyone studying reward design and credit assignment in multi-turn settings. Readers looking for an introductory treatment of RL or conversational IR will find the methodology dense; readers looking for reproducible multi-dataset evidence on reward shaping will find it directly relevant.

Authors’ abstract

We present ChatR1, a reasoning framework based on reinforcement learning (RL) for conversational question answering (CQA). Reasoning plays an important role in CQA, where user intent evolves across dialogue turns, and utterances are often underspecified, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Unlike static `rewrite, retrieve, and generate' pipelines, ChatR1 interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through RL. To address the challenge of sparse and delayed rewards in RL, we propose an intent-aware reward that provides turn-level feedback by aligning retrieval and reasoning with evolving user goals. ChatR1 demonstrates strong performance on both 3B and 7B model backbones, outperforming competitive models on five CQA datasets, measured by different metrics (F1, BERTScore, and LLM-as-judge). We include a diverse set of CQA datasets to cover topic shifts, evolving intents, mixed-initiative dialogues, and multi-document grounding, testing ChatR1's performance from various aspects. Ablation studies confirm the effectiveness of the intent-aware reward. Our analyses further reveal diverse reasoning trajectories and effective use of the search tool. ChatR1 also generalizes robustly across domains, demonstrating that RL-based reasoning enables more flexible and context-aware behavior than static CQA pipelines.