Skip to content
AI.info

Generative AI

RLHF, DPO, and Preference Optimization Tradeoffs

Understand reinforcement learning from human feedback, direct preference optimization, KL control, and the operational choices between them.

By the end you can

Comparison

Two routes use preference data differently

Both routes inherit the quality and the boundaries of the underlying comparisons. Neither wins by construction.

In 2024 Shusheng Xu and eight colleagues benchmarked DPO against PPO across dialogue and code generation, and found PPO ahead in every case. On code the gap has a number in it: "Specifically, on the CodeContest dataset (Li et al., 2022), our PPO model with 34B parameters outperforms AlphaCode-41B (Li et al., 2022), exhibiting a 10@1k improvement from 16.4% to 22.4%." On that same CodeContests dataset their DPO model, after one epoch, achieved a pass rate of 0. It failed to generate any correct code.

The three routes below are mechanisms for using the same comparisons. That 22.4% against 0 is the reminder that the mechanism does not settle the outcome on its own.

FigureComparison · 3 columns

PPO-based RLHF

Fit a reward model, sample from the policy, and optimize expected reward with a reference constraint.

  • Separates learned reward from policy update
  • Supports online policy-generated samples
  • Requires reward-model and RL infrastructure
  • Can overoptimize reward-model errors

Direct preference optimization

Train the policy directly on preferred and rejected responses relative to a reference.

  • Avoids an explicit reward-model serving loop
  • Uses a supervised-style objective
  • Sensitive to preference data and hyperparameters
  • Does not remove distribution shift or overfitting

Supervised-only baseline

Fine-tune on selected demonstrations without pairwise optimization.

  • Simpler and easier to debug
  • May be sufficient for clear tasks
  • Cannot use all relative preference information
  • Provides a necessary comparison

Visual

The classical RLHF pipeline adds a learned objective

Validation sets and provenance do not merge across these stages. The documented instance of exactly this pipeline is InstructGPT. In 2022 OpenAI collected labeler demonstrations for supervised fine-tuning, then ranked comparisons for a reward model, then ran PPO with a per-token KL penalty from the SFT model.

The people supplying both the demonstrations and the comparisons were a team of about 40 contractors hired on Upwork and through ScaleAI. The preference signal has a payroll, not a provenance-free existence.

The headline result is a statement about the objective rather than about scale: "In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters."

The same pipeline introduced regressions on public NLP datasets. Ouyang and colleagues reduced those by mixing pretraining gradients into PPO — PPO-ptx — rather than by raising the KL coefficient. Worth holding on to before the next section treats the KL term as the safety dial.

FigureProcess · 6 steps
  1. 1

    Start from an SFT policy

    The model already follows the target interface reasonably well.

  2. 2

    Collect comparisons

    Raters rank candidate outputs under a rubric.

  3. 3

    Fit a reward model

    Score differences are trained to predict preferences.

  4. 4

    Sample current policy outputs

    The optimization distribution changes as the policy changes.

  5. 5

    Optimize reward with control

    PPO-style updates trade reward against distance from a reference.

  6. 6

    Evaluate independently

    Humans and task tests check gains, regressions, and reward exploitation.

KL control limits movement but does not define goodness

A reference-policy penalty discourages the optimized policy from moving too far from a known model distribution. This can preserve language quality and reduce catastrophic drift. The coefficient creates a tradeoff. Too weak lets the policy exploit the reward, while too strong prevents useful adaptation. A KL target cannot repair a biased reward or missing task coverage.

That last sentence has been measured, not merely argued. A 2023 OpenAI study added an explicit KL penalty to the RL objective and watched what happened to the gold-reward frontier. The penalty did raise the proxy reward attainable at a given KL divergence. It produced no measurable gain in gold score. The caption to the paper's Figure 9 says it flatly: "We observe the effect of the KL penalty on the gold score as being equivalent to early stopping." Having found that the KL penalty gave a strictly larger proxy-gold gap, the authors set the KL penalty to 0 for all their other RL experiments in that paper. A knob that buys you the same ground-truth curve you would get by stopping sooner is a budget on movement. It is not a definition of the target.

Analogy

Optimizing to a referee changes how the game is played

Athletes who receive points from a referee trained on past matches will adapt toward whatever actions the referee rewards, including unnoticed loopholes. The reference policy resembles rules that limit how radically play can change. Rules are written in advance. The learned policy can generate novel strategies faster than human judges can anticipate them.

The same OpenAI study built a setting where the referee's own errors could be seen from outside it. A 6B reward model from Ouyang and colleagues played the role of humans, labelling 100,000 synthetic comparisons. Those comparisons trained proxy reward models ranging from 3M to 3B parameters. The policy then optimized the proxy while the 6B model scored the result as ground truth.

Write d = sqrt(D_KL(pi || pi_init)) for the distance travelled from the initial policy. Gao and his two co-authors fitted the gold score as R_bon(d) = d(alpha_bon - beta_bon d) for best-of-n and R_RL(d) = d(alpha_RL - beta_RL log d) for RL, with the coefficients scaling smoothly with proxy reward-model size. Both forms rise and then turn over. The gold score climbs with optimization pressure and then falls, at a distance the proxy score alone gives no sign of. The abstract names the law: "Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law."

Independent evaluation must examine what the optimized policy learned to exploit.

Key idea

Algorithm simplicity does not erase objective complexity

DPO can simplify the code by removing the explicit reward-model-and-PPO loop. It does not make preference labels unbiased, prevent distribution shift, or guarantee stable behavior outside the dataset.

The strongest evidence for that comes from DPO's own first author. In 2024 Rafael Rafailov and seven co-authors showed that Direct Alignment Algorithms — DPO, IPO and SLiC — exhibit the same over-optimization degradation as classical RLHF, despite training no proxy reward model and using no RL at all. "In particular, we find that DAA methods deteriorate not only across a wide range of KL-budgets, but also often before even a single epoch of the dataset is completed." Degradation before one epoch has finished means the failure is not a late-stage artifact of long RL runs. It is available on the first pass through the data.

Choose the method according to data, compute, iteration speed, control needs, and observed regressions. The supervised baseline should remain in the comparison.

A simpler optimizer can reduce engineering complexity without simplifying the human preference problem.

Case

Preference learning reduced to a single classification loss

Preference learning can be written as one classification loss. That is what direct preference optimization did in 2023, and the appeal is structural. Rafailov and five co-authors reparameterized the RLHF reward model so that the corresponding optimal policy comes out in closed form. The standard RLHF problem is then solved with a single classification loss: no separate reward model is fitted, and the language model is not sampled from during fine-tuning.

The reported margin is worth reading at its actual size. On the TL;DR summarization test split, measured against reference completions: "We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at 57% at its optimal sampling temperature of 0.0." In the head-to-head human evaluation, DPO samples at temperature 0.25 were preferred 58% of the time over PPO samples at temperature 0.

Four points on one summarization split, and eight points over an even split in one head-to-head. That is what "matching or exceeding RLHF on sentiment control, summarisation and dialogue" amounts to on the tasks where it was measured. The result is real. It is not a general ordering of the two methods, as the code-generation benchmark in the first section shows.

Steps

Run a preference-optimization comparison

Control the parts that otherwise turn algorithm choice into an unfair contest. Step 5 exists because of two specific published failures.

Verbosity is the first, and it comes from the optimization itself. Ryan Park and three co-authors traced it in 2024: "For the first time, we study the length problem in the DPO setting, showing significant exploitation in DPO and linking it to out-of-distribution bootstrapping." Their length-regularized DPO variant gained up to 20% in win rate once output length was controlled for. That is the size of the credit a win-rate number can quietly hand to length alone.

Diversity is the other half of the same step. Robert Kirk and six colleagues name the instruments in their 2024 study: expectation-adjusted distinct n-grams, sentence-BERT average cosine similarity and NLI diversity, computed both per input and across inputs.

FigureProcess · 6 steps

  1. 1. Freeze the data split

    Use the same prompts, preference labels, and protected evaluation set.

  2. 2. Include the SFT baseline

    Measure whether preference optimization adds value at all.

  3. 3. Match compute and sampling

    Record generated candidates, updates, and tuning effort.

  4. 4. Track reward and task metrics

    Compare learned reward with independent quality and safety scores.

  5. 5. Audit diversity and slices

    Inspect length, refusal, style, language, and domain behavior.

  6. 6. Stress the reward model

    Generate candidates designed to exploit its known shortcuts.

Preference optimization is a controlled policy experiment

RLHF and DPO provide different mechanisms for moving a policy toward observed preferences. Neither method turns those preferences into a full account of what is helpful, what is true, or what is safe. The gains are also not free in one direction.

The same diversity study compared SFT, best-of-n and RLHF on two base models, across summarisation and instruction following. RLHF generalised better out of distribution than SFT, and the advantage grew as the distribution shift grew. Then the other side of the ledger: "However, RLHF significantly reduces output diversity compared to SFT across a variety of measures, implying a tradeoff in current LLM fine-tuning methods between generalisation and diversity." A method that travels further and says fewer different things is a choice. Make it with both numbers on the table.

The next lesson shifts from weight updates to decoding. Even with fixed weights, the selection policy can change how varied the output is, how much it repeats itself, how long it takes, and how it fails.

Key takeaways