HuatuoGPT-3: RL-Only Domain Adaptation from Base Models

AI summary

Overview

Research area: Natural Language Processing — reinforcement learning (RL) post-training and domain adaptation of large language models, applied to the medical domain.

Technical level: Advanced. The paper derives gradient magnitudes for mixed-policy RL objectives, defines two named failure modes, and introduces a modified policy-optimization objective with a probability floor, gradient rescaling, and a reward-based retention rule. Familiarity with GRPO, PPO-style clipping, advantages, and on-policy versus off-policy training is assumed.

Scope: The paper diagnoses why standard mixed-policy RL underperforms for domain adaptation, proposes the One-stage Policy Optimization (OnePO) method, and validates it on medical domain adaptation, scaling it into the open-source HuatuoGPT-3 series (9B and 27B). Authors are affiliated with The Chinese University of Hong Kong, Shenzhen; Shenzhen Research Institute of Big Data; Shenzhen Loop Area Institute; and National Health Data Institute, Shenzhen. The paper is dated 05 Oct 2026 (arXiv:2610.05966v1 [cs.CL]).

What This Paper Is About

Domain adaptation usually follows an "SFT then RL" pipeline, where supervised fine-tuning gives an easy cold start but may reduce output diversity and constrain later exploration, while adding multi-stage optimization complexity and risking forgetting. The authors ask whether a pretrained base model can instead be adapted through RL alone, and find that both pure on-policy RL (which has a cold-start problem) and standard mixed-policy RL (which mixes in teacher outputs) fall short in two specific ways. OnePO is their answer: treat teacher outputs as transient guidance that is strengthened early and retired once the policy surpasses it.

Key Contributions

1. Diagnosis of existing RL paradigms. The authors analyze the standard routes for domain adaptation and identify two failure modes in standard mixed-policy RL: Gradient Starvation (most informative teacher-output tokens contribute only weak gradient signals) and Teacher-Distribution Anchoring (stale teacher outputs keep pulling the policy toward the teacher distribution even after stronger on-policy strategies emerge). They validate this diagnosis through controlled pilot studies.

2. A new RL-only paradigm: OnePO. OnePO is a one-stage RL framework that bypasses SFT and uses teacher outputs as transient guidance, via two mechanisms: Adaptive Objective Evolution (a corrected teacher-output objective with a probability floor and gradient rescaling) and Teacher Retirement (retain teacher outputs only while they exceed the on-policy frontier).

3. Validation in the medical domain. With 20K training samples, OnePO reaches 67.2 on HealthBench (Total), outperforming SFT+RL and pure RL by 2.7 and 7.4 points respectively, and surpassing SFT+RL on all open-ended and closed-ended benchmarks tested under the same teacher source.

4. Scaling to the HuatuoGPT-3 open-source model series. Applying the same recipe to larger data and model sizes yields HuatuoGPT-3-9B and HuatuoGPT-3-27B, with the 27B variant reaching 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, exceeding frontier baselines such as GPT-6 Astra. Models and code are released at the FreedomIntelligence/HuatuoGPT-3 repository.

Main Findings

• Gradient starvation is a concrete optimization barrier. In standard ratio-based mixed-policy surrogates with a constant denominator, the per-token gradient scales with the current-policy probability π_θ^(i,t), so it vanishes exactly for the unfamiliar, low-probability teacher tokens that need the most learning. Access to teacher outputs alone is therefore insufficient: in the AHA-Medicine pilot, pure RL fails (it never observes the synthetic fact) and mixed-policy RL stays near zero, while SFT+RL reaches the 0.5 imitation reward but plateaus there.

• Teacher outputs anchor the policy late in training. In the anchoring pilot, standard mixed-policy RL (which retains teacher outputs throughout) improves early but later saturates, whereas dynamically retiring teacher outputs achieves substantially better performance. Since the two settings differ only in whether teacher outputs are progressively removed, persistent mixing anchors optimization to the teacher distribution.

• OnePO beats SFT+RL and pure RL under identical data, algorithm and hyperparameters. With teacher DeepSeek-V3.2 (thinking), OnePO scores 67.2/44.5 on HealthBench Total/Hard versus 64.5/40.7 for SFT+RL (+2.7/+3.8) and 59.8/25.2 for pure RL. With teacher GPT-5 Chat, OnePO reaches 65.4/39.1 versus 63.6/37.3 for SFT+RL.

• Teacher choice matters, and richer reasoning traces help. DeepSeek-V3.2 (thinking) as teacher yields stronger results than GPT-5 Chat (no explicit reasoning trace): OnePO reaches 67.2 versus 65.4 on HealthBench Total.

• All three mechanisms are necessary. Ablating OnePO on HealthBench Total/Hard: full OnePO 65.4/39.1; without the probability floor (c = 0) 49.1/16.7; without gradient rescaling 57.3/30.7; without Teacher Retirement 52.7/18.6. The no-retirement variant even falls below pure RL (59.8) and lands close to the teacher GPT-5 Chat itself (51.9).

• Mix-type matters for rewards. Verifiable-only training performs strongly on closed-ended tasks but only slightly improves open-ended quality; rubric-only training shows the opposite pattern. Mixing both rewards gives the most balanced performance and even surpasses the rubric-only setting on open-ended tasks.

• Stricter retirement thresholds are better. A clear ordering emerges: Max > Mean > Min > None, where looser thresholds cause earlier and more severe plateaus. Under both strong and weak teacher settings the retirement ratio approaches 100% late in training, with the stronger teacher phasing out more slowly.

• Robust to teacher quality. With the weak teacher Qwen3-8B (thinking), OnePO still scores 66.0 on HealthBench and 64.2 on Medbullets (gains of +43.9 and +33.5 over Qwen3-8B-Base), versus 67.2 and 65.2 with the strong teacher.

• Cross-domain gains. With Qwen3-4B-Base, OnePO outperforms SFT+RL and pure RL in writing (74.6 on WB, 47.8 on CW-v3) and law (51.9 on LexEval, 67.4 on LawBench).

• General capability is preserved. OnePO reaches 54.2 on IFEval and 92.3 on GSM8K, versus 48.8/89.7 for SFT+RL. It obtains the highest GSM8K score across all training paradigms compared.

• Scaled models surpass frontier baselines on HealthBench Professional. HuatuoGPT-3-9B scores 68.3 and HuatuoGPT-3-27B scores 71.4, exceeding GPT-6 Astra (62.5) and GPT-5.2 (52.8). On HealthBench Total, HuatuoGPT-3-27B reaches 70.1 versus 57.8 for GPT-6 Astra; on Medbullets (5-option) GPT-6 Astra scores 88.9 versus 87.3 for HuatuoGPT-3-27B, and on MMLU-Pro (Med) Gemini 3.1 Pro scores 72.0 on MedXpertQA (Text) versus 48.1 for HuatuoGPT-3-27B.

Methodology in Plain English

The team starts from a pretrained base model and skips supervised fine-tuning entirely, training only with reinforcement learning. For each question they sample a group of answers from the current policy and also keep one answer produced by an external teacher model, then score all of them with a reward function.

Two corrections make teacher answers useful. First, Adaptive Objective Evolution addresses gradient starvation: because a teacher's answer contains words the current model considers very unlikely, the standard RL update gives those words almost no learning signal. OnePO replaces the denominator of the importance ratio with a floor value (c = 0.1) and multiplies by a rescaling factor, so that low-probability teacher tokens with positive advantage receive a strong, SFT-level learning signal. The two corrections cancel their auxiliary constants, leaving an advantage-weighted token-wise log-likelihood gradient. This signal is selective rather than applied to every teacher token, and it switches off automatically once a token's probability rises above the floor, at which point the objective reduces to standard GRPO with no manual schedule.

Second, Teacher Retirement decides when to throw teacher answers away. A teacher output is retained only if its reward is strictly greater than the best on-policy reward in the same group; if retained it replaces a randomly selected on-policy output, keeping the group size constant. Early on, teachers usually win and provide guidance; as the policy improves, fewer teacher outputs qualify and training returns to purely on-policy exploration, preventing anchoring.

The training data is 20K samples: 10K closed-ended high-difficulty medical multiple-choice questions from MedQA and MedMCQA scored by verifiable exact match, and 10K open-ended prompts generated by GPT-5 Chat from PMC-OA case reports with multiple scoring rubrics, scored by an LLM judge following the HealthBench protocol. To reduce API cost, an 8B grader is trained on GPT-4.1-annotated samples. The 20K mixed data are trained jointly in a single RL pipeline. OnePO and the GRPO baseline share hyperparameters: rollout size 8, batch size 128, mini-batch size 16, learning rate 2 × 10^-6, DAPO techniques with clipping interval [0.2, 0.28], and no KL loss or KL reward penalty. Controlled experiments use Qwen3-8B-Base on a single node of 8 × H200 GPUs.

For the HuatuoGPT-3 series, the open-ended subset grows from 10K to 20K with enriched rubrics, 20K rubric tasks from RubricHub (10K medical, 10K cross-domain) are added, closed-ended multiple-choice questions scale to 30K, and training runs for approximately 600 RL steps. HuatuoGPT-3-9B comes from Qwen3.5-9B and HuatuoGPT-3-27B from Qwen3.8-27B.

Evaluation covers open-ended response quality (HealthBench Total and Hard, and HealthBench Professional) and closed-ended medical knowledge and reasoning (Medbullets 5-option, MMLU-Pro medical subset, MedXpertQA Text, scored by exact match), plus IFEval and GSM8K for general capability retention. HealthBench Professional is scored with its official grader (GPT-5.4 with low reasoning effort) and length adjustment, while HealthBench Total/Hard use GPT-4.1 without length adjustment.

Why This Matters

This paper argues that the widely used "fine-tune first, then reinforce" recipe is not the only viable path to domain adaptation, and that a one-stage RL-only route can work if teacher guidance is designed as temporary rather than permanent. For research, the main contribution is diagnostic: naming and demonstrating Gradient Starvation and Teacher-Distribution Anchoring gives a concrete explanation for why mixed-policy RL plateaus, and OnePO's corrections offer a mechanism that removes itself automatically rather than requiring a hand-tuned schedule. The result that abstaining from SFT preserves general reasoning (highest GSM8K among the compared paradigms) is also notable for the ongoing debate about capability loss during specialization.

Real-world applications:

• Medical question answering and clinical decision support. Knowledge-heavy closed-ended tasks (Medbullets, MMLU-Pro medical, MedXpertQA Text) and open-ended clinical responses are both addressed in one pipeline.
• Clinician-facing response generation. HealthBench Professional targets real-world clinician tasks and applies length adjustment to reduce bias toward longer answers, making it a more realistic proxy for deployed assistants.
• Other high-stakes specialized domains. The cross-domain results on writing and law suggest the recipe is not medicine-specific.
• Domain adaptation under limited data. The controlled experiments use only 20K training samples, which matters where curated expert data is expensive.

Industry relevance: the paper is a recipe for turning an existing pretrained base model into a domain expert with a single training stage and no SFT data pipeline, which reduces stage count, objective shifts, and hyperparameter surface. The released HuatuoGPT-3-9B and 27B models, and the fact that OnePO remains robust when the teacher is a much weaker model (Qwen3-8B (thinking)), lower the barrier for organizations without access to the strongest teacher models.

Future Directions

• Reduce dependence on teacher availability and quality. OnePO is robust to a weaker teacher (66.0 versus 67.2 on HealthBench), but the paper notes the stronger teacher phases out more slowly; how far teacher quality can degrade before OnePO loses its advantage is not reported.
• Broaden cross-domain validation. The writing and law experiments are described as preliminary and use Qwen3-4B-Base; scaling OnePO beyond medicine to additional domains and model sizes remains open.
• Extend scaling behavior. The paper reports approximately 600 RL steps for the HuatuoGPT-3 series and up to 27B parameters; how OnePO behaves at larger scales, longer training, or with many teacher outputs per prompt (the setup uses M ≤ 1) is not reported.
• Sensitivity of the probability floor. The choice c = 0.1 is used and its selection is deferred to an appendix; systematic study of how the floor interacts with model size, token probability distributions, and reward type is left open, as is comparison against alternative ways of restoring gradient magnitude.

Target Audience

This paper is most useful to RL and post-training researchers working on LLM alignment and domain adaptation, particularly those already familiar with GRPO and DAPO-style training who want a principled alternative to SFT+RL. It also serves applied teams building specialized domain models (medical, legal, financial) who need to decide between multi-stage pipelines and single-stage RL, engineers interested in how teacher distillation can be made temporary rather than persistent, and medical AI practitioners evaluating open-source medical LLMs against frontier baselines. Readers seeking a beginner-level introduction to reinforcement learning will find the derivations heavy going; readers seeking deployment details such as serving cost, latency, or safety auditing will not find them reported here.

Authors’ abstract

Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation. However, pure on-policy RL suffers from a cold-start problem, while mixed-policy RL still falls short: informative tokens in teacher outputs are learned too slowly in early training, and stale teacher outputs can hinder later improvement. We identify these two failure modes as Gradient Starvation and Teacher-Distribution Anchoring. To address them, we propose One-stage Policy Optimization (OnePO), which treats teacher outputs as transient guidance for policy improvement. OnePO combines Adaptive Objective Evolution to strengthen learning on informative low-probability teacher tokens and Teacher Retirement to discard teacher outputs once the current policy can surpass them. On medical adaptation, OnePO achieves 67.2 on HealthBench (Total) with only 20K training samples, outperforming SFT+RL and pure RL by 2.7 and 7.4 points, respectively. We further scale OnePO to produce HuatuoGPT-3, an open-source medical LLM series whose 27B variant reaches 70.1 on HealthBench (Total) and 71.4 on HealthBench Professional, surpassing frontier models such as GPT-6 Astra. Models and code are available at https://github.com/FreedomIntelligence/HuatuoGPT-3.