RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

AI summary

Authors: Hongli Zhou, Hui Huang, Wei Liu, Chenglong Wang, Xingyuan Bu, Lvyuan Han, Fuhai Song, Muyun Yang (corresponding author), Wenhao Jiang, Hailong Cao, Tiejun Zhao Affiliations: Faculty of Computing, Harbin Institute of Technology; School of Computer Science and Engineering, Northeastern University; M-A-P arXiv: 2601.14032v2 [cs.CL]; published 2026-01-20 Code: https://github.com/Joe-Hall-Lee/RM-Distiller

Overview

Research area: Natural Language Processing — reinforcement learning from human feedback (RLHF), reward modeling, and knowledge distillation from large language models.

Technical level: Intermediate. The paper assumes familiarity with Bradley-Terry reward modeling, preference pairs, and RLHF policy optimization (PPO/GRPO/DAPO), though the core ideas are explained conceptually.

Scope: The paper introduces RM-Distiller, a framework that distills the refinement, scoring, and generation capabilities of a generative teacher LLM into a small discriminative reward model, and evaluates it on reward-model benchmarks, downstream RLHF alignment, domain adaptation, bias robustness, and data efficiency.

What This Paper Is About

Training reward models (RMs) for RLHF normally requires large-scale, high-quality human preference annotations, which are expensive and hard to obtain. A common workaround is to let a powerful generative LLM label response pairs as chosen/rejected and train an RM on those binary labels — but this treats the teacher as a mere binary classifier and discards most of its knowledge. RM-Distiller asks how to more effectively distill a generative LLM into an RM by systematically exploiting three separate teacher capabilities: refining responses, scoring responses, and generating text.

Key Contributions

1. First systematic study of RM distillation. The authors frame reward-model distillation from generative LLMs as an under-explored problem and present what they describe as the first systematic research on how to distill generative LLMs into discriminative RMs.
2. A multifaceted distillation strategy (RM-Distiller). The framework exploits three teacher capabilities — refinement, scoring, and generation — to provide richer supervision than binary preference labels alone.
3. Two new training techniques plus a regularization term. Contrastive Refinement synthesizes highly correlated response pairs; Margin-Aware Regression with Self-Calibrated Scoring aligns the RM's reward difference with teacher-assigned score margins; Generative Regularization anchors the student backbone in the teacher's output distribution.
4. Broad empirical validation. Experiments show gains on RewardBench and RM-Bench, on downstream RLHF alignment across three RL algorithms, in a new language domain (Arabic), on bias-generalization benchmarks, and in a low-data (1,000-instruction) setting.

Main Findings

• Reward-benchmark gains over traditional distillation. With Qwen2.5-3B-Instruct as the student and GPT-4o as the teacher in the unlabeled scenario, RM-Distiller reaches RewardBench average 79.5 and RM-Bench average 61.1, versus 70.3 and 55.9 for the vanilla BT Classifier. With Qwen3-14B as teacher, it reaches 80.2 and 62.7, versus 71.8 and 56.7 for BT Classifier.

• Gains hold in the labeled scenario. With GPT-4o as teacher and ground-truth-consistent pairs, RM-Distiller scores RewardBench average 83.6 and RM-Bench average 65.9, versus 73.8 and 63.2 for BT Classifier. With Qwen3-14B as teacher, it scores 84.5 and 66.4, versus 79.9 and 66.0 for Margin BT.

• Each component adds accuracy. The ablation study shows a steady increase as modules are added. For GPT-4o as teacher: BT Classifier 70.3 / 55.9 (RewardBench / RM-Bench averages), + Margin-Aware Regression 72.7 / 57.3, + Contrastive Refinement 76.8 / 58.8, + Generative Regularization 79.5 / 61.1. For Qwen3-14B: 71.8 / 56.7, then 75.7 / 59.5, then 78.7 / 61.5, then 80.2 / 62.7.

• Self-Calibrated Scoring beats plain pairwise scoring. With Qwen3-14B as teacher, pairwise scoring gives 75.8 / 60.7, while Self-Calibrated Scoring gives 78.7 / 61.5.

• Better downstream RLHF alignment. Using Llama-3-8B as the policy (SFT cold-start on UltraChat), RM-Distiller-based rewards improve AlpacaEval 2.0 length-controlled scores over a BT Classifier reward on both prompt sets and all three algorithms. On HH-RLHF prompts: BT Classifier gives PPO 14.32, GRPO 17.89, DAPO 18.25; RM-Distiller gives PPO 16.90, GRPO 20.10, DAPO 20.52 (SFT baseline: 8.86). On ShareGPT prompts: BT Classifier gives PPO 13.80, GRPO 17.92, DAPO 17.45; RM-Distiller gives PPO 16.24, GRPO 19.85, DAPO 20.15.

• Complex instruction following also improves under RL. On CFBench, the strongest RM-Distiller configuration (DAPO, HH-RLHF prompts) reaches 50.09 on CSR and 63.31 on ISR, versus 48.59 and 62.48 for the BT Classifier under DAPO.

• Fast domain adaptation is a headline result. In an Arabic-domain experiment (10,000 sampled instructions, 1,000-sample test set filtered by GPT-5), RM-Distiller with Qwen2.5-3B-Instruct and Qwen3-14B reaches 85.4 accuracy, versus 72.2 for BT-Qwen2.5-3B-Instruct, and surpasses larger generalist RMs including Skywork-Reward-V2-8B (81.3, 40,000K samples as reported), Skywork-Reward-8B-v0.2 (75.9, 80K), URM-LLaMA-3.1-8B (76.7, 100K), Llama-3-OffsetBias-8B (83.2, 70K), and Tulu-3-8B-RM-RB2 (83.2, 350K).

• Greater robustness to bias and out-of-distribution tasks. On EvalBiasBench, the BT Classifier collapses to 30.0 with a GPT-4o teacher and 47.5 with a Qwen3-14B teacher, while RM-Distiller reaches 57.5 and 63.8 respectively. On IFBench, RM-Distiller scores 57.9 and 57.7, exceeding the BT Classifier (50.2 and 52.5) and even the teacher's direct prompting scores (GPT-4o 50.8; Qwen3-14B 54.2).

• Strong data efficiency. With only 1,000 training instructions, the BT Classifier drops to 50.3 / 48.1 (GPT-4o teacher) and 64.6 / 51.9 (Qwen3-14B teacher), while RM-Distiller holds at 76.1 / 59.9 and 76.4 / 60.2.

• Cheap training overhead. RM training runs on 2 NVIDIA A800 GPUs and requires approximately 3 GPU hours; the RL experiments use 16 H20 NVIDIA GPUs.

• Hyperparameter sensitivity. For the generative-regularization weight α, RewardBench/RM-Bench averages are 76.8 / 58.8 at α=0.0, 78.2 / 60.5 at α=0.1, 79.5 / 61.1 at α=0.2, and 76.0 / 58.8 at α=0.3. For β: 78.9 / 60.4 at β=0.0, 78.2 / 62.2 at β=0.1, and 80.2 / 62.7 at β=0.2 (the β=0.3 row is cut off in the provided content).

• Comparison against external RMs. RM-Distiller surpasses several reference models reported in Table 1, including Eurus-RM-7B, GRAM-LLaMA3.2-3B-RewardModel, JudgeLRM-7B, and LLM-as-a-Judge results from Claude-3.5-Sonnet-20240620, Llama-3.1-70B-Instruct, GPT-4o, and Qwen3-14B.

Methodology in Plain English

The pipeline has four stages.

1. Build a starting set of preference pairs. For each instruction, the authors pick several candidate models with different families and sizes and collect one response from each into a response pool. From that pool they randomly draw two responses and ask the teacher LLM which is better, giving a chosen response and a rejected response plus a binary label. This is exactly what standard RM distillation does.

2. Refine the rejected response (refinement capability). Instead of stopping there, the authors ask the teacher to diagnose why the rejected response was rejected and then to rewrite it with minimal edits so it reaches the teacher's own answer quality while keeping the original structure and phrasing. The result is a pair whose two members differ almost only in the attribute that determines quality. A worked example is given: "You can use L[0] to access the last element of a list" becomes "You can use L[-1] to access the last element of a list," with the teacher's reference answer noting that Python negative indexing uses -1 for the last item. This forces the RM to learn subtle quality differences rather than superficial cues like length or formatting.

3. Score the responses and learn the margin (scoring capability). The teacher assigns scalar quality scores to responses rather than just a binary label. The student is trained so that the difference between its rewards for the chosen and rejected responses matches the difference between the teacher's scores, using a squared-error margin loss. To make the scores more reliable, the authors use Self-Calibrated Scoring: the teacher is shown the score it previously gave to the rejected response as a fixed anchor and then scores the refined response on the same scale, with the refinement dialogue reused as context. Pairs are then filtered, keeping only those that clear both a minimum score-margin threshold and a minimum edit-distance threshold.

4. Keep the backbone fluent (generation capability). Because RM training is purely discriminative, it can erode the student's underlying language ability. The authors therefore retain the student's language-model head and train it against the teacher's outputs using a combination of negative log-likelihood and KL-divergence distillation, weighted by coefficients α and β. If the teacher is closed-source or the vocabularies do not match, β is set to 0. The final objective is a weighted sum: (1 − α − β) times the margin loss plus the regularization loss. The final training set merges the refined pairs with the originally sampled pairs.

Experimental setup. Training uses a 10,000-sample subset drawn from Skywork-Preference-v0.2. Two scenarios are tested: unlabeled (only instructions are used, responses come from candidate models such as Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Gemma-2-9b-it, and Vicuna-7b-v1.5) and labeled (original response pairs are kept, filtered to cases where the teacher agrees with the ground-truth label). The student is Qwen2.5-3B-Instruct; teachers are GPT-4o and Qwen3-14B in non-thinking mode. Evaluation uses RewardBench and RM-Bench; the RL stage uses Llama-3-8B with PPO, GRPO, and DAPO, judged by GPT-4o-0806.

Why This Matters

Impact on research. The paper argues that the widely used practice of treating a teacher LLM as a binary preference annotator leaves substantial performance on the table. It shows that a small (3B) discriminative RM can be distilled to beat larger open-source RMs and approach or exceed much larger generative judges on RewardBench and RM-Bench, while retaining the cheap inference of a discriminative model. It also opens a template — decomposition of a teacher into refinement, scoring, and generation capabilities — that could generalize to other distillation targets.

Real-world applications:

• Building domain-specific reward models quickly for niche areas (the paper demonstrates this in Arabic) where generalist RMs underperform and annotation is scarce.
• Cheaper, lower-latency reward signals for production RLHF pipelines, since a 3B discriminative model replaces repeated calls to a large generative judge.
• Improving instruction-following and safety-oriented alignment for open policy models, where the paper shows gains on AlpacaEval 2.0, FollowBench, and CFBench.
• Reliable evaluation/ranking in low-resource data settings, where the paper shows strong results from only 1,000 instructions.

Industry relevance. The method avoids additional inference overhead at deployment, trains in roughly 3 GPU hours on 2 A800 GPUs, and works with both closed-source teachers (GPT-4o, where β is set to 0) and open-source teachers (Qwen3-14B, where full KL distillation applies). The bias and out-of-distribution results (EvalBiasBench, IFBench) speak directly to reward hacking concerns that matter when RMs gate real policy updates.

Future Directions

1. Combining human and LLM feedback. The conclusion states that alignment can draw on both human and AI feedback, and that future work will try to distill preference signals from human-LLM collaboration to further improve alignment efficacy.
2. Extending capability decomposition beyond the three capabilities studied. The paper focuses on refinement, scoring, and generation; whether other teacher abilities (for example long-form reasoning or tool use) can be distilled into RMs is left open.
3. Scaling and broader generalization studies. The reported experiments use a 10,000-sample Skywork-Preference-v0.2 subset and one student backbone (Qwen2.5-3B-Instruct); behavior at larger data scales and different student sizes is not reported in the provided content.
4. Understanding when Self-Calibrated Scoring and edit-distance filtering pay off. The thresholds τ_s and τ_e are introduced, but the provided content (which is truncated, including part of the β-weight ablation and the appendices) does not report a sensitivity analysis for them.

Target Audience

Researchers and engineers working on RLHF, reward modeling, and LLM alignment; practitioners who want cheaper reward signals than large generative judges; and graduate students interested in knowledge distillation from LLMs. Readers should already be comfortable with preference-pair training, Bradley-Terry objectives, and the basics of policy-optimization algorithms such as PPO, GRPO, and DAPO to get the most from the experiments section.

Authors’ abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. Due to the difficulty of obtaining high-quality human preference annotations, distilling preferences from generative LLMs has emerged as a standard practice. However, existing approaches predominantly treat teacher models as simple binary annotators, failing to fully exploit the rich knowledge and capabilities for RM distillation. To address this, we propose RM-Distiller, a framework designed to systematically exploit the multifaceted capabilities of teacher LLMs: (1) Refinement capability, which synthesizes highly correlated response pairs to create fine-grained and contrastive signals. (2) Scoring capability, which guides the RM in capturing precise preference strength via a margin-aware optimization objective. (3) Generation capability, which incorporates the teacher's generative distribution to regularize the RM to preserve its fundamental linguistic knowledge. Extensive experiments demonstrate that RM-Distiller significantly outperforms traditional distillation methods both on RM benchmarks and reinforcement learning-based alignment, proving that exploiting multifaceted teacher capabilities is critical for effective reward modeling. To the best of our knowledge, this is the first systematic research on RM distillation from generative LLMs.