Diverse Preference Learning for Capabilities and Alignment

AI summary

Overview

Research area: LLM alignment and preference learning, with a social choice theory framing.

Technical level: Intermediate (requires familiarity with RLHF, DPO, KL divergence, and basic probability).

One-sentence scope: The paper diagnoses KL-regularization as the cause of diversity loss in RLHF/DPO and proposes Soft Preference Learning (SPL), which decouples entropy from cross-entropy to control diversity independently, improving diversity, best-of-N problem solving, and calibration.

What This Paper Is About

Alignment algorithms like RLHF and DPO make LLM outputs less diverse, causing the model to over-represent majority preferences and collapse toward a narrow set of responses. The authors trace this behavior to the KL-divergence regularizer, which simultaneously controls bias toward a reference policy and output diversity, and they argue these two roles should be separated. They propose Soft Preference Learning (SPL), a modification that splits the KL penalty into distinct entropy and cross-entropy terms so diversity can be tuned without sacrificing quality.

Key Contributions

1. The paper identifies KL-regularization as the source of diversity loss in aligned language models and connects it to a social choice analysis, proving that the KL term heavily biases the model toward majority-preferred outputs.
2. It proposes Soft Preference Learning (SPL), which decouples the entropy and cross-entropy terms in the KL penalty to allow independent control of generation diversity, with a proof that SPL can achieve proportional representation of population preferences when α = 1.
3. It demonstrates empirically that SPL improves output diversity in chat domains, best-of-N accuracy on difficult math problems, and logit calibration on common multiple-choice benchmarks.
4. It shows SPL performs global (sequence-level) temperature scaling and is a Pareto improvement over standard token-level temperature scaling.

Main Findings

• KL term causes mode collapse: Under Proposition 3.1, the optimal RLHF/DPO policy is π(y) ∝ π_ref(y) p^(1/β). With typical empirical values of β in [0.01, 0.1], the KL penalty exponentiates population preferences to the 10th or 100th power. The paper's example: if 80% of the population prefers y and 20% prefers y′, then with β = 0.1 the optimal policy generates y with 99.9999% probability and y′ with 0.0001% probability.

• SPL separates diversity from reference bias: Under Proposition 3.2, the optimal SPL policy is π(y) ∝ π_ref(y)^(β/α) p^(1/α). Standard RLHF and DPO are special cases of SPL with α = β. As α → 1, the policy becomes less focused on majority preferences, and at α = 1 SPL recovers proportional representation.

• SPL is a proper scoring rule at α = 1: Corollary 3.1 states that SPL with entropy bonus α = 1 is a proper scoring rule weighted by a reference policy prior, meaning the distribution over outputs can match the population preference distribution they were trained on.

• Global rather than token-level temperature: The ratio α/β acts as a global "temperature" applied to the entire sequence. Unlike token-level temperature scaling, global temperature scaling preserves the relative probability ordering of sequences, so majority-preferred sequences remain the most likely outputs.

• Diversity-quality Pareto improvement (chat): Using LoRA finetuning of Mistral-7B-Instruct-v0.2 on HH-RLHF for 5,000 steps, SPL Pareto-dominates DPO with standard temperature scaling across all nine metrics and outperforms all sampling methods on six. Sweeps covered token-level temperature [1, 1.5]; min-p with p_base = 0.1 and temperature [1.3, 4]; top-p with p = 0.9 and temperature [1.1, 1.7]; top-k with k = 180 and temperature [1.1, 2.5]; and SPL global temperature [1, 11], all with β = 0.1.

• Diversity metrics used: Three diversity metrics were computed over 16 generated responses for each of 500 inputs on a held-out test split of HH-RLHF: expected cosine similarity between embeddings, a logical disagreement metric, and a content diversity metric. The latter two follow the design of human diversity questionnaires in Tevet & Berant (2021), with pools of 4 responses rated by gpt-4o-mini on a scale of 1-5.

• Quality metrics: Arena-Hard (500 queries, judged by gpt-4o-mini-2024-07-18, win-rate against gpt-4-0314), an average reward from a reward model trained on HH-RLHF over 500 held-out inputs with 16 responses each, and reference cross-entropy over the HH-RLHF test set.

• Dataset noise rationale: The paper notes a 37% cross-rater disagreement rate within the HH-RLHF preference dataset, and suggests some overweighting of majority preferences can act as denoising when variation is random error.

• Best-of-N problem solving: Using a Mistral-7B base model SFT-trained on UltraChat and LoRA finetuned for one epoch following the Zephyr recipe with β = 0.1 on Ultrafeedback-200k, SPL improved best-of-N accuracy on hard math problems. On GSM8K-Hard at best-of-128, SPL outperformed standard DPO by 10% and temperature-scaled DPO by 4%. SPL needed 84 samples to match DPO at best-of-128 (34% savings) and 106 samples to match temperature-scaled DPO (17% savings). On easy and medium questions, standard DPO performed best.

• Best-of-N setup details: Evaluation used GSM8K and MATH with few-shot chain-of-thought, 128 completions on a random split of 200 problems from each dataset. MATH difficulties correspond to Level 1, Level 3, and Level 5; GSM8K groups problems as easy if they take 4 or fewer samples to solve, medium if 5-64 samples, and hard if more than 64 samples.

• Calibration: Models trained with higher global temperatures consistently displayed lower calibration error on TruthfulQA and MMLU (57 subjects), using Expected Calibration Error and Brier Score alongside accuracy. The DPO model (global temperature of 1) was less calibrated than the base model, while SPL models with global temperatures slightly greater than 1 consistently attained both higher accuracy and lower calibration error than their DPO counterparts.

• Story-generation illustration: Figure 1 shows example stories generated by DPO, DPO with temperature scaling (t = 1.4), and SPL (global temperature α/β = 2) over 100 generated stories. DPO responses reuse the same doctor name, gender, and family relationship; temperature scaling adds diversity but produces many non-word tokens; SPL increases diversity with significantly less degradation.

Methodology in Plain English

The authors begin with a theoretical analysis. They model a population with conflicting preferences over two possible model completions and derive the closed-form optimal policy of RLHF and DPO. They show that the KL regularization parameter β appears as an exponent on the population preference probability, which sharply amplifies majority preferences when β is small (as it typically is in practice).

They then propose SPL, which replaces the single KL penalty with two separate terms: an entropy bonus (weighted by α) that encourages diversity, and a cross-entropy penalty (weighted by β) that keeps the model close to the reference policy. They derive a DPO-style objective for SPL that skips the reinforcement learning step, and they show the resulting optimal policy has the form π(y) ∝ π_ref(y)^(β/α) p^(1/α). Setting α = β recovers standard RLHF/DPO; setting α = 1 yields proportional representation.

Empirically, they finetune language models with LoRA and compare SPL against DPO combined with token-level temperature scaling, min-p, top-p, and top-k sampling. They sweep the entropy bonus (equivalently, global temperature) for SPL and the sampling temperature for baselines, then plot diversity-quality Pareto curves. They then test best-of-N problem solving on grade-school and competition math datasets, and finally measure logit calibration on two multiple-choice benchmarks.

Why This Matters

Impact on research. The paper reframes a widely used alignment ingredient (the KL penalty) as the mechanism behind mode collapse, and offers a principled decomposition into entropy and cross-entropy terms. It connects alignment to social choice theory and provides proofs about proportional representation, giving the diversity-alignment literature a formal target rather than a purely empirical one. It also positions SPL as a Pareto improvement over temperature scaling, which is currently a standard inference-time workaround.

Real-world applications.

• Chat assistants that must represent a range of societal viewpoints rather than a single dominant narrative, for example on policy or ethics questions.
• Inference-time scaling systems such as best-of-N sampling and tree search for mathematical or reasoning problems, where diverse strategies are needed to find correct answers.
• Multiple-choice question answering where well-calibrated confidence matters, such as factual question answering and knowledge benchmarks.
• Content and story generation, where lexical and structural repetition is a visible quality problem.

Industry relevance. The method is an offline supervised objective with a DPO-style form, meaning it can be applied without the complexity of multi-agent reinforcement learning setups that some prior diversity-aware methods require. The paper reports sample savings (34% relative to DPO, 17% relative to temperature-scaled DPO on GSM8K-Hard at best-of-128), which is directly relevant to inference compute budgets.

Future Directions

1. Exploring alternative, semantically grounded diversity metrics, such as metrics constructed from embeddings or LLM judges, that could be integrated directly into preference learning algorithms.
2. Studying the tradeoff between diversity and quality at large α values more carefully, including the role of dataset noise and whether lower temperatures act as a denoising majority-vote-like prior.
3. Extending the analysis beyond the Bradley-Terry/Plackett-Luce assumptions to preference distributions with intransitivity, which the paper notes cannot be perfectly fit by Bradley-Terry models.
4. Applying and testing SPL in additional domains beyond chat, math, and multiple-choice calibration, given that prior entropy-regularization work by Xiao et al. (2024) did not include generative experiments.

Target Audience

This paper is most useful for alignment and LLM training researchers, particularly those working on preference learning objectives (RLHF, DPO), diversity and mode collapse, or calibration. It also benefits social choice and AI governance researchers interested in how alignment algorithms represent minority versus majority preferences, and applied ML engineers who need to trade off output diversity against quality at inference time.

Authors’ abstract

The ability of LLMs to represent diverse perspectives is critical as they increasingly impact society. However, recent studies reveal that alignment algorithms such as RLHF and DPO significantly reduce the diversity of LLM outputs. Not only do aligned LLMs generate text with repetitive structure and word choice, they also approach problems in more uniform ways, and their responses reflect a narrower range of societal perspectives. We attribute this problem to the KL divergence regularizer employed in preference learning algorithms. This causes the model to systematically overweight majority opinions and sacrifice diversity in its outputs. To address this, we propose Soft Preference Learning, which decouples the entropy and cross-entropy terms in the KL penalty - allowing for fine-grained control over LLM generation diversity. From a capabilities perspective, LLMs trained using Soft Preference Learning attain higher accuracy on difficult repeated sampling tasks and produce outputs with greater semantic and lexical diversity. From an alignment perspective, they are capable of representing a wider range of societal viewpoints and display improved logit calibration. Notably, Soft Preference Learning resembles, but is a Pareto improvement over, standard temperature scaling.