Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

AI summary

Overview

Research area: Large language model alignment and fine-tuning, specifically learning from demonstration data without human preference annotations (bridging supervised fine-tuning and reinforcement learning from human feedback).

Technical level: Advanced. The paper assumes familiarity with Markov decision processes, behavior cloning, KL-regularized RL objectives, proximal policy optimization (PPO), DPO, and inverse reinforcement learning.

Scope: The paper proposes and empirically evaluates Self-Rewarding PPO (SRPPO), a two-stage fine-tuning method that derives an implicit reward from the log ratio of an SFT policy to its pretrained base model, then refines the model with PPO using that reward.

What This Paper Is About

Supervised fine-tuning (SFT) is the standard way to align language models to demonstrations, but because it is an off-policy method equivalent to behavior cloning, it overfits to the training distribution and generalizes poorly out of domain, especially when high-quality data is limited. The authors ask whether on-policy RL training techniques (like those used in the preference-learning stage of RLHF) can be applied using demonstration data alone, without any human preference labels. Their answer is SRPPO, which reuses the SFT model itself as the source of a reward signal.

Key Contributions

1. A "coherent reward" derived from the SFT policy. The reward for a prompt-response pair is the log ratio between the SFT policy and the pretrained base policy, log p_SFT(y|x) / p_PT(y|x), inspired by coherent soft imitation learning. The pretrained policy acts as a baseline and the SFT policy as a target, defining an alignment direction for further training.

2. Self-Rewarding PPO (SRPPO), a two-stage framework. Stage one performs standard SFT on demonstrations; stage two applies PPO to the SFT policy using the coherent reward as an implicit, self-generated reward with no reward model training or inverse reinforcement learning.

3. Use of additional prompts beyond the demonstrations. Unlike SPIN, which is restricted to demonstration prompts because it has no explicit reward, SRPPO can sample extra prompts from the task distribution during the PPO stage, since the coherent reward evaluates any prompt-response pair.

4. Empirical evaluation across three data-overlap regimes. The authors test minimum overlap, medium overlap, and diminished overlap between SFT data and PPO prompts on Mistral-7B and LLAMA3-8B, comparing against SFT, extended SFT, SPIN, and PPO with an independent preference reward model.

Main Findings

• SRPPO beats SFT on overall averages under minimum overlap. For Mistral-7B, SRPPO scores 32.43 average versus 29.96 for SFT and 21.81 for the pretrain baseline; SRPPO leads on IFEval (47.60 / 41.37 loose/strict accuracy), GSM8k (46.93 EM), and AlpacaEval (12.47 / 13.23 length-controlled win rate / win rate).

• The gains hold for LLAMA3-8B as well. SRPPO scores 31.17 average versus 27.74 for SFT and 24.50 for the pretrain baseline, with IFEval at 41.49 / 37.41 and GSM8k EM at 51.10 (above the pretrain baseline's 50.11).

• Extended SFT overfits and hurts out-of-domain tasks. Training SFT for more epochs (e.g., 6 versus 2) improved IFEval and GPQA in some settings but reduced math reasoning; for Mistral-7B, SFT (Extended) fell to 29.04 GSM8k EM and 24.22 average, below plain SFT.

• An independent preference reward model gives only marginal gains. PPO with the Fsfairx-LLAMA3-RM reward model reached 31.06 / 30.22 on IFEval and 48.90 GSM8k EM for LLAMA3-8B, below SRPPO's 41.49 / 37.41 and 51.10; GPQA and AlpacaEval results for that baseline are reported as "-" (not reported).

• SRPPO outperforms SPIN despite SPIN using more data. SPIN's first-iteration DPO used the full 350k examples from TULU-v2-mix, yet it reached 28.29 average on Mistral-7B versus SRPPO's 32.43.

• Medium overlap: subset SFT improves chat but damages math, and SRPPO recovers it. Fine-tuning on a 9k GPT-4-annotated UltraFeedback subset raised Mistral-7B SFT to 22.63 / 16.21 AlpacaEval but dropped GSM8k to 35.18; SRPPO recovered to 41.39, which the paper describes as a 4.06% EM improvement, and pushed IFEval to 49.40 / 44.00 and the average to 33.33.

• Diminished overlap: the advantage persists. After adding 40k TULU-v2-mix examples to reduce prompt overlap, SRPPO still beat SFT on Mistral-7B across the board (28.26 versus 26.37 average; 48.56 / 44.72 IFEval; 30.55 GSM8k; 15.59 / 9.01 AlpacaEval).

• Reward quality depends on the base model and SFT quality. The authors observe the coherent reward generalizes poorly if the pretrained model has weak out-of-distribution generalization (they cite small models such as Phi-2) or if SFT data is low quality or already overfit.

• A token-wise reward variant was tried and rejected. Making the coherent reward token-level introduced length degeneration that complicated training; it is discussed in Appendix E and left to future work.

Methodology in Plain English

The method has two sequential phases.

Phase 1 — Supervised fine-tuning. Start from a pretrained base model and fine-tune it on a demonstration dataset of prompt-response pairs using the ordinary maximum-likelihood objective. This produces the "SFT policy."

Phase 2 — Self-rewarding reinforcement learning. Compute a reward for any prompt-response pair as the log ratio of the SFT model's probability of that response to the pretrained model's probability of the same response. This number is positive when the SFT model likes the response more than the base model did, and it effectively encodes the direction in which SFT moved the model. Rather than training a separate reward model, the researchers simply reuse these two frozen copies of the model. Then they run PPO, where the model generates its own responses on-policy, the coherent reward scores them (assigned at the end-of-sequence token for the whole response), and the policy is updated with the standard clipped surrogate objective, a critic initialized from the SFT policy, and a KL penalty toward the reference.

Experiments use Mistral-7B and LLAMA3-8B, SFT on TULU-v2-mix (326k examples from 11 sources) and PPO prompts from UltraFeedback (64k examples). Three overlap regimes are tested: minimum (SFT only on TULU-v2-mix), medium (an added 9k GPT-4-annotated UltraFeedback subset), and diminished (the same subset plus 40k extra TULU-v2-mix examples). SFT used batch size 128 and 2 epochs; PPO used a rollout buffer of 1024, batch size 128, KL coefficient 0.2 or 0.5, clip coefficient 0.2, critic learning rate 9e-6 with 35 rollout buffers of warmup, and actor learning rates chosen from {5e-8, 2e-8, 1e-8}. Evaluation uses lm-evaluation-harness on IFEval, GSM8k (5-shot exact match), GPQA (few-shot and few-shot chain-of-thought), and AlpacaEval, with code built on Huggingface Transformers and OpenRLHF and hardware on NVIDIA A100 GPUs.

Why This Matters

Impact on research. The paper offers a way to get on-policy RL benefits in alignment without preference data, reward-model training, or bi-level optimization. It reframes the SFT checkpoint itself as a reward source, which is a notably lightweight alternative to inverse reinforcement learning approaches and to DPO-based self-play methods like SPIN. It also provides evidence for the long-held intuition that prolonged SFT overfits while on-policy training generalizes better.

Real-world applications.

• Fine-tuning domain assistants (legal, medical, customer support) where a small set of expert-written responses exists but pairwise preference labels do not.
• Adapting models to tasks where prompts are plentiful but high-quality answers are scarce, since SRPPO can train on extra prompts gathered cheaply.
• Using the coherent reward as an evaluator for filtering or scoring SFT data, which the authors suggest as an additional use.
• Improving instruction following, math reasoning, and conversational quality in deployed models without a separate, costly human-annotation pipeline.

Industry relevance. Removing the need for preference annotation and a separately trained reward model cuts both cost and engineering complexity, and the method is compatible with existing PPO infrastructure such as OpenRLHF and with alternative policy-optimization algorithms including REINFORCE, RLOO, GRPO, and VinePPO.

Future Directions

• Substituting PPO with other on-policy algorithms (REINFORCE, RLOO, GRPO, VinePPO) to see whether the coherent reward transfers as cleanly.
• Revisiting the token-wise version of the coherent reward, which the authors found caused length degeneration and left unresolved.
• Understanding and mitigating the failure modes: the reward's usefulness depends on the pretrained base model's generalization ability and on the quality of the SFT data, both of which are currently described as empirical observations rather than solved problems.
• Using the coherent reward for SFT data filtering and response-quality evaluation, reducing human involvement in the alignment loop.

Target Audience

Machine learning researchers and engineers working on LLM post-training and alignment who already understand SFT, PPO, and preference learning. It is most useful to practitioners who need to align models from a limited set of demonstrations without access to preference annotations, and to researchers interested in implicit reward design and the tradeoffs between off-policy imitation and on-policy reinforcement learning.

Authors’ abstract

Supervised fine-tuning (SFT) has emerged as a crucial method for aligning large language models (LLMs) with human-annotated demonstrations. However, SFT, being an off-policy approach similar to behavior cloning, often struggles with overfitting and poor out-of-domain generalization, especially in limited-data scenarios. To address these limitations, we propose Self-Rewarding PPO, a novel fine-tuning method that leverages on-policy techniques to enhance generalization performance. Our approach combines the strengths of SFT and proximal policy optimization (PPO) to achieve more effective alignment from demonstration data. At its core is a reward function designed as the log policy ratio between the SFT model and the pretrained base model. This function serves as an implicit reward signal, using the pretrained policy as a baseline and the SFT policy as a target. By doing so, it enables on-policy fine-tuning without relying on human preference annotations. The integration of this self-rewarding mechanism with PPO addresses key limitations of SFT, improving generalization, data efficiency, and robustness. Our empirical evaluation across a range of natural language processing tasks demonstrates that Self-Rewarding PPO consistently outperforms traditional SFT methods. The results highlight the effectiveness of our approach in aligning LLMs using demonstration data, particularly in scenarios where high-quality annotated data is scarce.