When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF

AI summary

Overview

• Research area: Large language model alignment, specifically reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under noisy human preference labels.
• Technical level: Advanced. The paper combines probabilistic modeling of label noise, gradient analysis of DPO-style losses, consistency and convergence theorems, and empirical evaluation on billion-parameter language models.
• Scope: The paper proposes Flipping-Aware Direct Preference Optimization (FA-DPO), an instance-dependent robust loss and iterative training procedure that corrects for preference flipping in offline human preference datasets.

What This Paper Is About

Preference datasets used to align LLMs are often corrupted because annotators flip labels — the response they mark as preferred is actually the worse one — and this corruption happens at different rates for different samples rather than at one fixed rate. Existing robust RLHF methods assume flipping occurs at a constant rate across the dataset, which the authors argue does not reflect reality. The paper's goal is to model flipping probability as a function of each individual sample and use that estimate to correct the DPO training loss so the learned policy matches what clean data would have produced.

Key Contributions

1. A probabilistic model for instance-dependent preference flipping. The paper formalizes annotation as two stages — labeling according to a true human intention model (the Bradley-Terry model), followed by an instance-dependent contamination step — and derives a flipping-aware loss in which the corrupted preference probability is (1 - ε)p + ε(1 - p), with ε varying per sample.
2. A feature-based flipping estimation module. The flipping probability is modeled as a logistic function of a permutation-equivariant feature map built from response length, perplexity, and reward margin, which the paper states are features validated in prior natural language processing work.
3. An iterative optimization algorithm. The method jointly optimizes the flipping model and the LLM, with a warmup phase for the flipping model, and is described as compatible with both original RLHF and DPO algorithms.
4. Theoretical guarantees. The paper proves consistency of the learned preference probability (the minimizer on corrupted data equals the minimizer on clean data) and Q-linear convergence of the flipping model parameters under boundedness and feature coverage assumptions.

Main Findings

• Preference flipping is costly: The paper cites Gao et al. (2024), reporting that a 10% increase in preference flipping ratios can produce a 30% decrease in alignment performance measured by win rate.
• FA-DPO is the most robust method across noise levels: On UltraFeedback with Pythia-1B, prediction accuracy for FA-DPO moves from 73.05 at 0% flipping to 70.77 at 40% flipping, while DPO falls from 68.22 to 51.87 and rDPO falls from 70.13 to 47.67.
• Gains grow with noise: The accuracy improvement over the runner-up baseline rises from +2.30 at 0% to +16.70 at 40% on UltraFeedback, and from +0.37 to +36.83 on HH_Golden.
• HH_Golden degrades more sharply under noise: The paper attributes this to a larger chosen-rejected gap in that dataset, and notes all methods achieve much higher accuracy on clean HH_Golden than on UltraFeedback.
• Benefits transfer to larger models: With LLama-3.1-8B at 40% flipping on UltraFeedback, FA-DPO reaches 78.87 accuracy and 68.50 win rate, versus 64.96 and 56.10 for DPO, an improvement of +12.28 and +12.40. With Mistral-7B at 40%, FA-DPO reaches 78.49 accuracy and 59.00 win rate, an improvement of +14.13 and +14.00.
• The learned flipping model tracks the true noise: Figure 1(a) shows predicted flipping probabilities align closely with the actual flipping distribution; Figure 1(b) shows clear separation between flipped and non-flipped samples; Figure 1(c) plots the relationship between predicted flipping and length-based features.
• Warmup matters: Under a 20% flipping ratio on HH_Golden, training without warmup with 20 noise and 50 policy steps yields 77.00 accuracy and 53.20 win rate, whereas the same configuration with warmup yields 98.40 accuracy and 82.90 win rate.
• Gradient behavior differs from prior robust losses: For low flipping probabilities (ε < 0.5) the FA-DPO weight increases with model confidence; at ε = 0.5 the weight is zero, filtering out ambiguous samples; when ε > 0.5 the gradient direction reverses toward the opposite preference, which the paper describes as self-correction absent in cDPO and rDPO.
• Computational cost is comparable to DPO: The paper states that training is limited to a single epoch, and that the perplexity and reward margin features come directly from the policy's forward pass log-likelihoods with no extra computation overhead.

Methodology in Plain English

The authors treat a noisy preference label as the result of two separate events: what a human actually intended (governed by the standard Bradley-Terry model) and whether an external factor caused the annotation to flip. They write the observed probability of a preference as a mixture of the true probability and its opposite, weighted by a per-sample flipping probability, and then correct the DPO loss accordingly.

To estimate that per-sample flipping probability, they train a small logistic model over features describing each preference triplet: the average and absolute difference of response lengths, the average and absolute difference of the policy's log-likelihoods (perplexity-style features), and the average and absolute difference of the implicit DPO reward (margin). The features are built so that swapping the order of the two responses leaves them unchanged. The logistic model and the LLM are then optimized alternately, starting with a warmup phase in which the flipping model is trained first before the two are updated in turns.

For experiments, they simulate instance-dependent flipping by initializing a flipping model, setting a threshold of τ = 0.8 so that samples with a high enough flipping probability are actually flipped, and training the model's parameters so that the fraction of flipped samples matches a target ratio η. They evaluate on UltraFeedback and Anthropic's HH_Golden, with Pythia-1B, LLama-3.1-8B, and Mistral-7B, and use prediction accuracy and win rate against the SFT model as metrics, with DeepSeek-V3 as the evaluator for Pythia-1B and GPT-4o for the larger models.

Why This Matters

• Impact on research: The work moves robust RLHF beyond the assumption of a single fixed flipping rate, reframing annotation noise as a sample-level property and providing consistency and convergence guarantees for the correction. It also gives a direct gradient-level comparison with cDPO and rDPO, showing that its correction is a reparametrization of the DPO weight rather than an additive adjustment.
• Real-world applications:
  • Cleaning preference datasets collected through crowdsourcing, where annotator attention and cognitive load vary from sample to sample.
  • Defending alignment pipelines against deliberate or malicious alteration of annotations.
  • Reducing the cost of collecting clean preference data by making partially corrupted datasets usable.
  • Making alignment training more reliable when chosen and rejected responses come from noticeably different distributions.
• Industry relevance: Because FA-DPO is designed to work with existing RLHF and DPO pipelines and the paper reports computational cost comparable to or lower than standard DPO, the method is positioned as a drop-in robustness improvement for organizations that train on large offline preference datasets.

Future Directions

• Relaxing the reliance on a correct reward or policy model in the convergence theorem, which requires p_φ = p* or p_θ = p*.
• Testing whether the feature set generalizes beyond length, perplexity, and reward margin, and whether other feature families improve flipping estimation.
• Extending the framework beyond pairwise preference flipping, for example to other forms of annotation corruption in RLHF pipelines.
• Investigating performance when the flipping mechanism is not well approximated by the assumed logistic form, since the paper itself notes the practical mismatch between estimation and real-world mechanisms, limiting the simulated flipping model to length-based features when η > 0.

Target Audience

Researchers and practitioners working on LLM alignment, robust machine learning, and learning with noisy labels, particularly those familiar with RLHF, DPO, and the Bradley-Terry model. Readers need a background in probabilistic modeling and preference optimization to follow the derivations, though the plain-language framing of flipping as a post-transition after human intention is accessible to a broader technical audience.

Authors’ abstract

Quality of datasets plays an important role in large language model (LLM) alignment. In collecting human feedback, however, preference flipping is ubiquitous and causes corruption in data annotation; the issue necessitates the alignment algorithms with improved robustness against potential flipped pairs. To this end, this paper introduces a Flipping-Aware Direct Preference Optimization (FA-DPO) algorithm tailored to preference flipping from a reinforcement learning with human feedback (RLHF) perspective. We dissect the inherent human intention model and the preference flipping mechanism introduced by external factors as two distinct stages; in the latter, we introduce an instance-dependent flipping probability on the basis of the Bradley-Terry (BT) model. Further, by leveraging features relevant to preference annotation, we capture uncertainty in judgments and model preference flipping patterns. In practice, we design a simple yet efficient iterative optimization algorithm compatible with the original RLHF and DPO algorithms. In our experiments, we investigate the instance-dependent preference flipping model under multiple circumstances for evaluation of our proposed method, as well as other baseline methods.