Reinforcement Learning for Long-Horizon Multi-Turn Search Agents

AI summary

Overview

Research area: Natural Language Processing / LLM agents — reinforcement learning for tool-using, multi-turn retrieval agents in the legal domain.

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, LoRA fine-tuning, and policy-gradient RL terminology (GRPO, roll-outs, rewards), but the core argument is accessible.

Scope: A single-benchmark study showing that RL-trained open-weight 14B model can outperform frontier API models on a legal document search task, and that multi-turn horizon length matters both at training and test time.

What This Paper Is About

Prompt-based LLM agents can search documents over multiple turns, but they do not learn from experience. This paper asks whether reinforcement learning, using successful document retrieval as a verifiable reward, can train a smaller open model to search better than much larger frontier models. It also investigates how the number of allowed turns — varied during training and at test time — shapes agent performance on long-horizon search.

Key Contributions

1. Demonstrates that on a legal search benchmark, a 14B RL-trained model outperforms frontier models that are accessible only through APIs.
2. Explores how RL-trained models exploit the multi-turn setting through experiments that restrict the number of turns, both during training and at test-time.
3. Releases a benchmark construction recipe and reward design (partial rewards across four behavioural bands) that makes RL training stable for tool-use agents.
4. Reports that restricting training to very short horizons (2 turns) prevents learning altogether, while the unrestricted model improves steadily.

Main Findings

• RL beats frontier models on this benchmark: The RL-trained Qwen3-14B reached 85% accuracy versus 78% for Gemini 2.5 Pro, 81% for OpenAI o3, 66% for Gemini 2.5 Flash, 53% for base Qwen3-14B, and 33% for naïve RAG with Gemini 2.5 Pro. Average turns were 6.2, 5.3, 7.1, 3.4, 3.7, and 1.0 respectively.
• Tool access alone is not enough: Base Qwen3-14B plateaus at 53% accuracy despite having the same three tools and multi-turn interactions as the RL-trained variant, indicating that tool availability without learned usage is insufficient.
• More turns generally help, up to a point: Under turn restrictions from 1 to 10 turns, all evaluated models improved monotonically with additional turns. The base Qwen3-14B plateaued after 6 turns, while the RL-trained variant and Gemini 2.5 Pro continued improving across all 10 turns.
• Crossover in the low-turn regime: Gemini 2.5 Pro outperforms the RL-trained model when fewer than 5 turns are allowed, but the RL-trained model gains more from longer multi-turn horizons. The authors frame this as a limitation of prompt-based approaches lacking learned exploration strategies.
• Restricting training turns prevents learning: With a 2-turn limit during training, the model showed no meaningful progress, fluctuating around its 10–15% baseline. The unrestricted model improved from roughly 40% to 85% correct document identification over 300 training steps. The authors attribute this to GRPO's need for relative reward comparisons within batches of trajectories — with only 2 turns, too few correct answers are generated to provide positive learning signal.
• Partial rewards were necessary: A four-band reward structure was required to make RL work, ensuring failed trajectories still contribute learning signal and that hallucination is penalized more heavily than admitting uncertainty.

Methodology in Plain English

Dataset. The authors built a legal search benchmark from 5 years of Singapore court judgments. Each judgment was parsed into XML preserving its structural hierarchy, with unique IDs for sections (for example, 2021_SGCA_3:judgement:introduction:p1). Synthetic question-answer pairs came from a three-stage pipeline: extracting patterns from seed queries supplied by practicing lawyers; generating 10 candidate question/answer pairs per document using Gemini 2.5 Pro; and aggressively filtering candidates for realism, difficulty, and variety. The final dataset holds 2,300 questions with ground-truth documents, questions, and answers.

Baseline. A single-turn naïve RAG system runs BM25 keyword search and FAISS semantic search on the raw query, combines the results, and prompts the LLM to answer only from retrieved context — mirroring latency-optimized production RAG with no refinement or follow-up searches.

Agent design. Every agent has three tools: keyword search returning K BM25 results over document paragraphs with section IDs and highlighted snippets; semantic search returning K results by cosine similarity over FAISS-indexed all-MiniLM-L6-v2 embeddings; and a read-document tool that returns the full content of a section by ID. Because IDs are hierarchical (A:B:C), agents can hop to parent sections by truncating the ID (A:B:C → A:B). This produces a two-phase pattern: broad exploration, then targeted reading. The agent loop parses <think>, <tool>, and <answer> sections from each response and continues until an answer is produced.

Training. The base model was Qwen3-14B; Qwen3-0.6B served as a low-cost stand-in during code development. Only LoRA adapter components were trained. The ART library was used with PEFT, unsloth, and trl to train the adapters. For each query, vLLM generated multiple trajectories, from which rewards were computed; YaRN extended vLLM's context to 128k tokens to allow extended multi-turn roll-outs. GRPO then optimized the policy, with group_size set to 6 and 8 groups run per step. LoRA adapters used by vLLM were updated at the end of each step so subsequent roll-outs used the updated policy. Gemini 2.5 Pro served as the reward model, producing binary judgments of roll-out response quality.

Reward bands. Correct answers with proper citations scored [1.0, 2.0], with higher rewards for fewer turns and searches; "I don't know" when evidence was insufficient scored [0.0, 1.0]; incorrect answers scored [-1.0, 0.0] with +0.1 partial credit for each correct document found; and formatting errors preventing tool execution scored [-2.0, -1.0].

Turn-restricted evaluation. To probe multi-turn behaviour, the authors force early termination by prefixing <answer> to the assistant message at turn N, forcing an answer. A 0-turn rollout corresponds to naïve RAG. The flow is: query → response → {reformulate search → response} repeated N times → answer. They also trained a Qwen3-14B variant with a 2-turn limit during training to test generalisation.

Metrics. Thirteen metrics across four categories were tracked: final outcome (answer_correct via LLM judge, sources_correct, returned_i_dont_know, attempted_answer), progress (ever_found_right_doc, ever_read_right_doc), formatting errors (cant_parse_tool_call, bad_tool_call_name, bad_tool_call_args, bad_sources_id), and efficiency (num_turns, num_searches, ran_out_of_turns). Twelve of the thirteen are verifiable without an external judge; only answer_correct requires LLM evaluation, for which Gemini 2.5 Pro gives a True/False classification.

Why This Matters

Impact on research. The paper provides evidence that RL, not just prompting, is what lets search agents compound value across turns — and that this capability is learnable by a 14B open model. The negative result on 2-turn training is equally important: it identifies a concrete failure mode of GRPO-style training when the horizon is too short to generate enough positive examples, which is a practical constraint for anyone designing RL curricula for agents.

Real-world applications:

• Legal research tools that iteratively search and read case judgments to answer practitioner questions with citations.
• Enterprise and regulatory document search, where agents must navigate hierarchical policy or compliance corpora.
• Any RAG deployment where hallucination is costlier than an explicit "I don't know", since the reward structure deliberately encodes that preference.
• Agent training pipelines for new or unfamiliar tool sets and document collections, where RL lets agents discover effective search strategies rather than follow instructions.

Industry relevance. The result challenges the assumption that frontier API models are always the best option for agentic retrieval: a self-hosted, LoRA-tuned 14B model outperformed them on this benchmark. The turn-restriction analysis also gives practitioners a concrete reason to provision longer horizons for agents that will be used on hard search tasks.

Future Directions

• Whether the 2-turn training failure generalizes: does any sufficiently short training horizon prevent GRPO from learning, and is there a minimum viable turn budget?
• Whether learned multi-turn exploration strategies transfer to new tool sets, new document collections, or domains beyond Singapore court judgments.
• Whether the turn-restriction crossover (stronger prompt-based models winning at low turn counts, RL models winning at high ones) holds on other long-horizon interactive tasks.
• Extending the approach beyond a single benchmark of 2,300 questions, and testing how the partial-reward design scales to tasks without a verifiable retrieval signal.

Target Audience

Researchers and engineers working on LLM agents, RL fine-tuning, and retrieval-augmented generation — particularly those building multi-turn search systems where verifiable rewards are available and where turning tool access into competent tool use is the actual bottleneck. The paper is also useful to practitioners deciding between self-hosted RL-tuned models and frontier APIs for document-heavy agentic workloads.

Authors’ abstract

Large Language Model (LLM) agents can leverage multiple turns and tools to solve complex tasks, with prompt-based approaches achieving strong performance. This work demonstrates that Reinforcement Learning (RL) can push capabilities significantly further by learning from experience. Through experiments on a legal document search benchmark, we show that our RL-trained 14 Billion parameter model outperforms frontier class models (85% vs 78% accuracy). In addition, we explore turn-restricted regimes, during training and at test-time, that show these agents achieve better results if allowed to operate over longer multi-turn horizons.