When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets

AI summary

Overview

Research area: Data-centric alignment of large language models, specifically the curation and comparison of preference optimization datasets used for Direct Preference Optimization (DPO).

Technical level: Advanced. The paper assumes familiarity with LLM post-training, DPO, reward models, and benchmark evaluation, though its central argument (that data quality and composition drive alignment performance) is readable at a conceptual level.

Scope: A systematic, sample-level comparison of five open-source DPO corpora across eight models and 14 benchmarks, culminating in a curated mixture called UltraMix that is smaller than the strongest individual dataset yet scores higher.

What This Paper Is About

Frontier LLMs are aligned using large preference datasets that their developers do not release, while the open-source community has produced several DPO corpora whose contents are poorly documented and rarely compared under identical training conditions. This paper asks how these open datasets actually differ in sample quality, task coverage, and preference-order coherence, and whether those differences can be exploited to build a smaller but better mixture. The goal is to move preference data curation from guesswork toward measurable, reward-informed recipes.

Key Contributions

1. Comparative evaluation. The first systematic cross-analysis of five open DPO datasets — TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs — spanning general-purpose and domain-specific tasks, with preference fine-tuning run on eight different models of various scales and evaluated on 12 benchmarks from the Open LLM Leaderboards plus HumanEval and HumanEval+ code generation tasks, holding all training parameters constant for fair comparison.

2. Sample-level annotations and analysis. Every preference pair is annotated with the Magpie framework for task category (12 classes), query difficulty, input quality, safety, and language, plus a novel "preference reward" computed with the FsfairX reward model to independently validate whether chosen completions are genuinely better than rejected ones.

3. A new DPO mixture. UltraMix is built by selectively drawing high-quality samples from all five corpora, removing noisy and redundant samples. The final UltraMix-190k is 30% smaller than TuluDPO and exceeds its performance across key benchmarks while improving task diversity.

4. Public release. Annotated versions of all five datasets plus UltraMix are released on Hugging Face to support data-centric preference optimization research.

Main Findings

• TuluDPO is the strongest individual dataset. For both Llama-3.1-8B-TuluSFT and Qwen-2.5-7B-TuluSFT, TuluDPO beats the other four corpora on the overall average and on both Open LLM Leaderboard aggregates (Llama overall 53.96 versus 52.09 for ORPO, 51.39 for UltraFeedback, 50.44 for HelpSteer, 50.16 for Code-Preference-Pairs, and 50.09 for SFT; Qwen overall 59.56 versus 58.52, 58.39, 57.02, 58.11, and 56.55 respectively).

• Dataset strengths are task-specific. ORPO generally outperforms UltraFeedback on most code and particularly math tasks, but for Qwen, UltraFeedback surpasses ORPO on instruction following and reasoning. HelpSteer falls behind both. Code-Preference-Pairs shows noticeable gains primarily on code benchmarks but falls short on math, reasoning, and instruction following.

• Information seeking dominates every corpus. Information seeking accounts for 51% of HelpSteer and 49% of UltraFeedback, with TuluDPO at 38% and ORPO at 37%. Math is second, highest in ORPO at 29% and 17% in TuluDPO. Conversational types such as editing, role playing, brainstorming, and planning are underrepresented at only 2–6% each.

• Misaligned preference orders are common. Only 70–80% of samples in TuluDPO, ORPO, UltraFeedback, and HelpSteer have a chosen completion that the independent reward model actually scores higher than the rejected one, suggesting some preference decisions are arbitrary or based on near-identical completions. Code-Preference-Pairs appears less affected, possibly because missing features or syntax errors make preference signals more salient.

• Prompt quality correlates with preference reward. Average reward assigned to chosen completions rises with input quality across most datasets, which the authors describe as the first empirical evidence that poorly written instructions may lead to subpar preference alignment. Code-Preference-Pairs is an outlier for "very poor" inputs, where contradictory or incomplete prompts still yield functional answers through fallback assumptions.

• HelpSteer's pairs are weakly separated. Its reward-difference histogram centers close to zero, consistent with preference pairs built from similarly rated completions. More comprehensive datasets like TuluDPO, ORPO, and UltraFeedback show broader reward differences reaching the positive tail.

• A quality-only filter is insufficient. UltraMix-170k (170k samples, 37% smaller than TuluDPO) improved TruthfulQA for Llama from 56.78% to 61.45% and MATH for Qwen from 43.13% to 47.55%, but underperformed TuluDPO on code and IFEval. A task distribution analysis showed it underrepresented information seeking and reasoning by 20% and 13% relative to TuluDPO.

• Progressive boosting works. Adding 17,000 math and code samples produced UltraMix-187k; adding a further 3,000 instruction-following samples produced UltraMix-190k, which is 30% smaller than TuluDPO. For Llama, UltraMix-190k reaches 69.05% on HumanEval, 48.08% on HumanEval+, 81.13% on IFEval, 82.48% on GSM8K, and 23.56% on MATH, with an overall average of 56.04 versus TuluDPO's 53.96. For Qwen, it reaches 82.27% on HumanEval, 63.05% on HumanEval+, 79.88% on IFEval, 82.70% on GSM8K, 49.55% on MATH, and an overall average of 62.05 versus TuluDPO's 59.56.

• Results generalize across architectures and scales. Testing six additional open SFT-tuned models (Apertus-8B-SFT, OLMo-2-7B-SFT, SmolLM-3-3B-SFT, Instella-3B-SFT, SmolLM-2-1.7B-SFT, OLMo-2-1B-SFT), UltraMix-190k leads on overall average in every case — for example Apertus-8B-SFT at 49.60 versus TuluDPO's 47.66, and OLMo-2-1B-SFT at 38.74 versus 37.63.

• Filters must be combined. Granular ablations show that quality-based, task-based, or reward-based filters alone are insufficient; UltraMix's gains stem from the principled combination of those signals.

Methodology in Plain English

The researchers first fix the training pipeline so that only the dataset changes. Using Open-Instruct, they run DPO on two SFT-tuned base models (Llama-3.1-8B-TuluSFT and Qwen-2.5-7B-TuluSFT, both SFT-tuned with TuluSFT) and evaluate on 12 Open LLM Leaderboard tasks plus HumanEval and HumanEval+.

They then take all five datasets apart at the sample level. The Magpie annotation pipeline uses Llama-3.3-70B-Instruct as a judge to label each prompt's task category, difficulty, input quality, safety, and language. Separately, a reward model called FsfairX (based on Llama-3-8B-Instruct) scores both the chosen and rejected completion of every pair, producing a "preference reward" that is independent of whatever labels the dataset originally shipped with. A comparison with an alternative reward model is included to confirm the choice is not over-fitted to one scorer.

With these labels, they design filters: keep prompts rated "excellent" or "good," drop "very easy" prompts, and require the chosen reward to exceed the rejected reward. Samples above the 25th percentile of chosen reward are kept from TuluDPO, ORPO, UltraFeedback, and HelpSteer, with a stricter 80th percentile applied to Code-Preference-Pairs to avoid over-representing code. Deduplication removes overlap, notably between TuluDPO and UltraFeedback.

The first mixture is then diagnosed per task category, and the recipe is refined twice: a top-up of math and code samples, then a top-up of information seeking and reasoning samples drawn from above the 70th reward percentile while relaxing the input-quality requirement to "average." Each stage is re-trained and re-evaluated to confirm the gain.

Why This Matters

Impact on research. The paper supplies an evaluation protocol that holds training fixed, which prior DPO dataset comparisons lacked, and it introduces preference reward as a scalable, model-based way to audit preference ordering without human annotations. Its finding that a substantial share of open preference pairs are misaligned challenges the assumption that released chosen/rejected pairs are trustworthy, and it gives dataset designers a concrete, transferable curation recipe.

Real-world applications:

• Smaller, cheaper alignment runs. A mixture that is 30% smaller than TuluDPO while scoring higher reduces the compute and cost of preference fine-tuning for teams with limited budgets.
• Better open-weight models. Labs building models on open data can adopt UltraMix or its recipe to raise benchmark scores without licensing proprietary preference data.
• Domain-targeted dataset design. The demonstration that boosting specific task categories (math, code, instruction following) moves the corresponding benchmarks supports task-aware curation for specialized assistants.
• Data quality auditing. The reward-based ordering check can be reused to vet any preference dataset before purchase, license, or training.

Industry relevance. Preference data is a competitive bottleneck for alignment, and proprietary corpora are rarely shared. This work shows that careful, measured curation of public corpora can close part of the gap, making data selection — rather than data volume — the operative lever.

Future Directions

• Extending the annotation and curation pipeline beyond the five studied corpora, since the authors state the method is generalizable to any DPO dataset.
• Applying reward-based curation to preference data used by other alignment methods, given that curation strategies developed for SFT did not transfer directly to DPO.
• Determining how to set filtering thresholds and task-boost proportions in advance, rather than through the iterative re-training and per-category diagnosis used here.
• Investigating whether the 70–80% preference-order alignment rates in TuluDPO, ORPO, UltraFeedback, and HelpSteer reflect genuinely ambiguous pairs or correctable annotation errors, and whether targeted relabeling could recover further performance.

Target Audience

Researchers and practitioners working on LLM post-training and alignment, especially those building or selecting preference datasets for DPO. It is also valuable for dataset curators, evaluation engineers at organizations deciding which public corpora to train on, and anyone studying data-centric approaches to model training. Readers should be comfortable with reward models, benchmark evaluation, and the DPO objective, though the central conclusions about quality filtering and preference coherence remain accessible without that background.

Authors’ abstract

Aligning large language models (LLMs) is a central objective of post-training, often achieved through reward modeling and reinforcement learning methods. Among these, direct preference optimization (DPO) has emerged as a widely adopted technique that fine-tunes LLMs on preferred completions over less favorable ones. While most frontier LLMs do not disclose their curated preference pairs, the broader LLM community has released several open-source DPO datasets, including TuluDPO, ORPO, UltraFeedback, HelpSteer, and Code-Preference-Pairs. However, systematic comparisons remain scarce, largely due to the high computational cost and the lack of rich quality annotations, making it difficult to understand how preferences were selected, which task types they span, and how well they reflect human judgment on a per-sample level. In this work, we present the first comprehensive, data-centric analysis of popular open-source DPO corpora. We leverage the Magpie framework to annotate each sample for task category, input quality, and preference reward, a reward-model-based signal that validates the preference order without relying on human annotations. This enables a scalable, fine-grained inspection of preference quality across datasets, revealing structural and qualitative discrepancies in reward margins. Building on these insights, we systematically curate a new DPO mixture, UltraMix, that draws selectively from all five corpora while removing noisy or redundant samples. UltraMix is 30% smaller than the best-performing individual dataset yet exceeds its performance across key benchmarks. We publicly release all annotations, metadata, and our curated mixture to facilitate future research in data-centric preference optimization.