DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research

AI summary

Overview

Research area: Natural Language Processing / agentic AI — specifically reinforcement learning for long-form "deep research" (DR) agents that search, browse, and synthesize attributed reports.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning from human/verifiable feedback, GRPO, LLM-as-a-judge scoring, and tool-using agent loops.

Scope: The paper introduces RLER (Reinforcement Learning with Evolving Rubrics), a training method for open-ended long-form deep research, and uses it to train DR Tulu-8B, an open 8B-parameter agent evaluated on four long-form benchmarks plus a new clinical genetics dataset.

What This Paper Is About

Most open deep research agents are trained with reinforcement learning on short-form question answering tasks, where correctness is easy to verify — but that kind of reward does not extend to realistic long-form research reports, whose quality is under-specified and often requires evidence from outside the model's own knowledge. This paper's goal is to build an openly released model that is trained directly for open-ended, long-form deep research, by automatically constructing and continuously updating the rubric criteria used as the training reward.

Key Contributions

1. RLER (Reinforcement Learning with Evolving Rubrics): a training method in which rubrics are constructed and maintained to co-evolve with the policy model during RL. At each step the method samples several responses and search traces from the model and generates new rubrics that capture and contrast the strengths and weaknesses of those responses, keeping feedback on-policy and discriminative.

2. DR Tulu-8B: described as the first fully open model directly trained for open-ended, long-form deep research, starting from Qwen3-8B and trained with an SFT cold start followed by RLER-based RL using a customized variant of GRPO.

3. GeneticDiseasesQA: a new clinical deep research dataset of 47 expert-curated questions covering 24 pathogenic gene variants, requiring models to search for and synthesize supporting evidence to assess therapeutic eligibility of disease-causing genetic variants.

4. Open infrastructure release: all data, code, and models, plus an extensible deep research library (dr-agent-lib) and an evaluation suite supporting plug-and-play multi-tool search, including infrastructure for asynchronous tool calls and scalable RL over long-horizon tool-use trajectories.

Main Findings

• Outperforms open deep research agents: Across four long-form benchmarks (SQAv2, HealthBench, ResearchQA, DeepResearchBench), DR Tulu-8B (RL) reaches an average of 65.6, exceeding the best prior open baseline (Tongyi Deep Research-30B-A3B, average 50.0) by 15.6 points. The paper reports the improvement over Tongyi DR as 15.6% on average, and states gains of 4.8–41.8 percentage points over the strongest open 8–32B models on the four benchmarks.

• Competitive with proprietary systems: DR Tulu-8B averages 0.7% above OpenAI Deep Research on average (65.6 vs 64.9). It scores 88.3 on SQAv2 and 52.8 on HealthBench, versus 79.6 and 53.8 for OpenAI Deep Research, and 75.7 on ResearchQA versus 79.2 for OpenAI Deep Research. It is reported as matching or exceeding Claude Sonnet Search, Perplexity Sonar (high-reasoning), Perplexity Deep Research, and Gemini Deep Research.

• Much cheaper per query: On SQAv2, OpenAI Deep Research costs about USD 1.8 per query, while DR Tulu-8B is reported as almost three orders of magnitude cheaper at USD 0.0019 (stated elsewhere as $0.0018/query including LM inference via OpenRouter, and $0.00008/query when accounting only for tool APIs). For comparison, Ai2 ScholarQA (Claude Sonnet) costs $1.30/query, Tongyi Deep Research $0.03/query, and WebThinker $0.003/query ($0.015 in report mode). On DeepResearchBench, DR Tulu-8B remains over 10× cheaper than Tongyi DR.

• Short-form QA also improves: Although RL uses only long-form prompts, the RL model improves over the SFT checkpoint on SimpleQA (75.9 vs 75.5), 2Wiki (68.9 vs 66.5), and WebWalker (39.0 vs 31.9), raising the overall average by 3.3 points to 61.3.

• Evolving rubrics matter: Removing evolving rubrics results in up to a 2-point drop in average performance, with the gap widening over training as evolving rubrics capture new knowledge the model explores. Both RLER and static search-augmented rubrics outperform using random rewards instead of rubric-based rewards.

• Search-based rubrics beat closed-book and general rubrics: In a 500-step RL ablation from an intermediate SFT checkpoint (SFTv0.1, average 54.7), initial search-based rubrics give the best overall average (56.7), ahead of closed-book rubrics (55.6) and general rubrics (54.8).

• RLER does not rely on a strong proprietary judge: Replacing GPT-4.1/GPT-4.1-mini with Qwen3-8B as both rubric generator and judge still yields +4.4 average points over the SFT checkpoint (58.3 vs 53.9) after 1000 RL steps, only 1.3 points behind the GPT-judge configuration (+5.7, 59.6).

• RLER gains over the finetuned model: The paper reports gains of 6.4–16.0 points on top of the finetuned model across the four benchmarks.

• Adaptive tool selection: DR Tulu-8B learns to pick tools per task rather than using one hard-coded search tool — on SQAv2 it uses paper search 90% of the time, whereas on DeepResearchBench it relies on web search and browsing about 55% of the time.

• Clinical genetics generalization: On GeneticDiseasesQA, DR Tulu-8B substantially improves over Qwen3-8B across all metrics and outperforms Ai2 ScholarQA on Final Answer correctness; GPT-5- and Gemini-based systems achieve higher Final Answer scores while DR Tulu-8B remains competitive on Evidence Support, Evidence Quality, and Evidence Synthesis. The paper states no other open agents can tackle this task due to their inability to produce reliable, verifiable citations. Numerical values for this comparison appear only in a figure, not in the text.

• Training data mixture matters: In SFT ablations, removing long-form data substantially degrades all long-form benchmarks, while removing short-form data leaves long-form performance largely unchanged but noticeably hurts short-form tasks such as 2Wiki. Long-form benchmarks improve substantially with as little as 5% of the SFT data and largely saturate beyond 50%, while short-form tasks continue to benefit up to the full dataset.

• RL benefits from a stronger SFT cold start and longer training: Starting RL directly from Qwen3 (no SFT cold start) improves over untrained Qwen3-8B but underperforms using even 5% of the SFT mixture as cold-start data; the full SFT mixture does better still. Extending RL was important — some evaluations that initially looked flat (e.g., DRB) improved with longer training. Higher train reward did not necessarily correspond to higher downstream reward.

• Robust to tool errors: The model continues to improve even after extended training with a tool that consistently errors (details in the paper's appendix).

Methodology in Plain English

The core problem is that long-form research reports are hard to grade automatically. Static criteria written in advance cannot anticipate every good (or bad) behavior, and a model grading only from its own memory can miss facts or be gamed.

Evolving rubrics. For each training question, the researchers first run a web search and use an LM to write an initial set of "persistent" rubrics grounded in the retrieved documents. During training, after the policy model produces several rollouts (each including its search context and final answer), a rubric-generating LM sees the question, all the rollouts, and the existing rubric pool, and writes new rubrics of two kinds: positive ones capturing strengths or newly explored relevant knowledge, and negative ones summarizing common undesirable behaviors such as reward hacking (for example, verbatim copying of retrieved content to inflate citation precision).

Scoring. Each rubric is judged by an LM that gives 0, 0.5, or 1 for how well a response satisfies it, and the weighted rubric scores are combined into a single reward. Because the criteria are concrete and itemized, the authors argue the signal is less prone to judge bias than a single holistic score.

Buffer management. To keep the rubric set compact, after every rollout the method scores all responses with the active rubrics, discards rubrics with zero variance (no discriminative value), ranks the rest by standard deviation, and keeps only the top K_max. Three auxiliary rewards — format, search, and citation — encourage correct formatting, effective search, and well-supported citations.

Training recipe. DR Tulu-8B starts from Qwen3-8B. A supervised fine-tuning cold start distills search patterns from a teacher model (GPT-5) into the base model, using 16K trajectories. SFT used one H100 node (8 GPUs) for 5 epochs, totaling 136 GPU hours. RL then uses only long-form prompts — roughly 5K prompts from SearchArena and OpenScholar plus 4K from RaR — with GRPO plus token-level loss, 1-step asynchronous training, tool output masking, sample packing, and asynchronous tool calling. RL ran on 2 H100 nodes (16 GPUs), with the final run using 27,000 GPU hours. GPT-4.1-mini served as the LM judge and GPT-4.1 as the rubric generator.

Tooling. The agent's action space is {think, tool, answer, cite}. Three tools are used at inference: google_search (via the Serper Search API), web_browse (Jina browsing, versus Crawl4AI during training), and paper_search (Semantic Scholar full-text API). Tool usage is capped at 10 calls per rollout with the top 10 snippets retrieved.

Why This Matters

Impact on research. The paper addresses a long-standing barrier in deep research training: data, code, and infrastructure are rarely available together. It shows that reward design, not just model scale or tool access, is the bottleneck for long-form agentic tasks, and demonstrates that an evolving, search-grounded rubric signal can be generated largely automatically rather than written by human experts. It also reports that RLER's gains do not depend on access to a strong proprietary judge model, which makes the recipe reproducible in settings without such models.

Real-world applications:

• Scientific literature synthesis — DR Tulu-8B scores 88.3 on SQAv2 and is described as beating Ai2 ScholarQA (Claude Sonnet) on HealthBench and DeepResearchBench, making open models viable for literature-review-style assistance.
• Healthcare question answering — HealthBench explicitly includes negative rubrics marking harmful responses; DR Tulu-8B scores 52.8 versus 53.8 for the much larger OpenAI Deep Research.
• Clinical genetics — GeneticDiseasesQA targets assessing therapeutic eligibility of disease-causing genetic variants, requiring evidence aggregation from biological databases, research literature, and case reports.
• General-domain research reports — DeepResearchBench covers diverse general-domain topics; here DR Tulu-8B scores 45.4, close to Gemini 3 Pro + Search (46.3) and above OpenAI Deep Research (46.9 recorded with an asterisk) at a fraction of the cost.

Industry relevance. Cost is a first-order concern for deploying research agents. At $0.0018–$0.0019 per query on SQAv2 versus $1.80 for OpenAI Deep Research and $1.30 for Ai2 ScholarQA, DR Tulu-8B shifts deep research from a premium query to something closer to a commodity query. The released dr-agent-lib library, MCP-based tool backend, and asynchronous tool-call infrastructure lower the engineering barrier for teams that want to build or extend their own research agents.

Future Directions

The paper's content does not state an explicit future work section; the following are open questions the work raises.

1. Broader and harder domains. GeneticDiseasesQA (47 questions, 24 variants) is small relative to the four main benchmarks. Extending expert-curated deep research evaluation to other regulated, evidence-heavy fields is a natural next step.

2. Judge/model dependence at the frontier. RLER transfers to an open judge (Qwen3-8B as both generator and judge, +4.4 average points), but still trails the GPT-judge setup by 1.3 points. How far rubric quality can be pushed without any proprietary models remains open.

3. Scaling the policy model. DR Tulu-8B matches or exceeds much larger proprietary systems, which raises the question of how much further the RLER recipe would move a larger open backbone such as the 30B-class models it currently outperforms.

4. Reward ceiling and train/downstream mismatch. The authors observe that higher training reward did not necessarily correspond to higher downstream reward, and that some benchmarks only improved with extended RL (27,000 GPU hours in the final run). Understanding and correcting that mismatch would make RL for long-form agents more predictable.

Target Audience

ML and NLP researchers working on RL for LLM agents, particularly those focused on reward design, LLM-as-a-judge, and long-horizon tool use. Also relevant to practitioners building search-and-synthesis agents who need an open, cost-efficient alternative to proprietary deep research APIs, and to applied researchers in scientific literature synthesis, healthcare QA, and clinical genetics who need models producing verifiable snippet-level citations rather than URL-only links.

Authors’ abstract

Deep research agents perform multi-step research to produce long-form, well-attributed answers. However, most open deep research agents are trained on easily verifiable short-form QA tasks via reinforcement learning with verifiable rewards, which does not extend to realistic long-form tasks. We address this with Reinforcement Learning with Evolving Rubrics (RLER), where rubrics are constructed and maintained to co-evolve with the policy model during training. This allows the rubrics to incorporate newly explored information from search and contrasting model responses, enabling better fact checking and more discriminative on-policy feedback. Using RLER, we develop Deep Research Tulu (DR Tulu-8B), the first fully open model that is directly trained for open-ended, long-form deep research. Across four long-form deep research benchmarks in science, healthcare, and general domains, DR Tulu substantially outperforms existing open deep research agents (by 15.6% over Tongyi DR on average) and matches or exceeds proprietary deep research agents (by 0.7% over OpenAI DR on average), while being significantly smaller and cheaper per query (1000x cheaper than OpenAI DR per query).