An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

AI summary

Overview

Research area: Natural Language Processing — post-training of large language models, specifically preference-based alignment (DPO, KTO, ORPO, PPO/RLHF, GRPO) and domain adaptation.

Technical level: Intermediate. The paper assumes familiarity with preference optimization objectives and domain shift, but its structure — three testbeds, seven training configurations, two base models — is empirical and readable for anyone who knows the basics of LLM fine-tuning.

Scope (one sentence): A systematic empirical comparison of five alignment objectives and several source-to-target adaptation strategies across summarization, question-answering helpfulness, and safety, measuring both generalization to the target domain and the diversity of the resulting model outputs.

What This Paper Is About

Preference tuning aligns language models to human judgments of quality, helpfulness, or safety, but prior work has shown it can degrade performance outside the domain it was trained on. What has not been systematically explored is whether adaptation strategies — such as supervised fine-tuning on target-domain data or pseudo-labeling target-domain prompts with a larger teacher model — can mitigate that degradation, and what they cost. The authors run a controlled study across five alignment objectives plus SFT, two base models, and three source-to-target testbeds to find out which choices actually help and which trade-offs they introduce.

Key Contributions

1. A controlled comparison of five preference optimization objectives — DPO, KTO, ORPO, PPO (RLHF), and GRPO — alongside SFT, under domain shift, evaluated on summarization, question-answering helpfulness, and safety.
2. Evidence that the adaptation strategy often matters more than the alignment objective, with pseudo-labeling proving the strongest strategy and producing its largest boost where the domain shift is widest.
3. An analysis of a generalization–diversity trade-off, showing that pseudo-labeling substantially reduces domain-shift degradation while inducing mode collapse in syntactic, semantic, and logical output diversity.
4. A set of diagnostic analyses isolating domain relevance from teacher quality, testing data efficiency of pseudo-labeling, measuring sensitivity of the LLM judge to epistemic mismatch, and examining the effect of SFT ordering.

Main Findings

• Pseudo-labeling dominates summarization transfer. Pseudo-labeled SFT achieves the highest overall target win rate for Llama-3.1-8B at 83.37, and lifts OLMo-3-7B DPO to 72.26, above all non-synthetic methods. Without pseudo-labeling, adaptation hinges on the SFT stage.

• Base models show a large domain difficulty differential on summarization. Llama-3.1-8B scores 44.97 on TL;DR but only 15.96 on CNN/DM (gap 29.01), while OLMo-3-7B shows a small gap (2.64) but a lower baseline performance.

• SFT reduces the summarization gap when source and target data are mixed. Source-only SFT for Llama-3.1-8B reaches a 36.07 target win rate, a +20.11 gain over base, but still trails its source win rate by 23.50. Mix-SFT narrows the gap to 4.25, a 19.25 gain over source-only SFT. However, OLMo-3-7B DPO on SFT with D_(S+T) initialization yields 87.78 on source with a 20.88 gap, showing source specialization can be amplified in a model-dependent way.

• Offline alignment over-specializes to the source domain. On OLMo-3-7B, DPO with target SFT reaches 91.00 source win rate yet suffers a 30.60 gap. On Llama-3.1-8B, source-only DPO reaches 89.87 source with a 31.78 target gap, 10 times larger than GRPO's 3.79. ORPO (gap 25.66) and direct KTO (gap 38.00) also show large gaps.

• GRPO prevents domain over-specialization. It obtains a 62.57 source win rate, +18.27 over PPO, while keeping the generalization gap at 3.79. With target initialization, GRPO remains stable (gap -2.99).

• PPO underperforms in-domain on summarization. For Llama-3.1-8B, PPO reaches 59.69 on target versus 44.30 on source, surpassing its own source performance. The intermediate SFT model achieved 59.57 on source, but PPO's online updates cause this to regress to 44.30. The authors attribute this partially to source forgetting from PPO's training instability rather than learned domain-general competence.

• QA helpfulness is largely invariant to domain shift. Base models already show near-zero gaps (-6.78 for Llama-3.1-8B), and gaps cluster near zero across configurations, with DPO using SFT D_(S+T) initialization yielding 0.60. While summarization gaps span up to 47%, QA win rates stay within a narrow ±8 band. Pseudo-labeling still lifts the QA ceiling on Llama-3.1-8B (72–76 target vs ~60 for non-synthetic). Online RL methods favor the target domain in QA (GRPO gaps -6.66 to -7.75; PPO -8.24).

• Safety shows moderate domain sensitivity. Base models score below 50% (Llama-3.1-8B 38.40/32.90; OLMo-3-7B 35.94/48.22). Source SFT provides marginal improvement (39–42 target), confirming cybercrime refusal does not transfer to violence prompts. GRPO with a pseudo-labeled reward model reaches 99.80/95.60, but with a source reward model remains at 43.90/39.50.

• KTO peaks at safety without pseudo-labeling. Among non-pseudo-labeled methods, direct KTO achieves the highest target safety on OLMo-3-7B (76.49%; source 74.52%), substantially outperforming direct DPO on the same model (58.12% target; 55.20% source). The authors suggest this may reflect KTO's use of unpaired desirable/undesirable labels.

• Pseudo-labeling achieves near-perfect safety but can be undone. On Llama-3.1-8B, all pseudo-labeled objectives except PPO achieve over 95% target safety and over 99% on source. Yet DPO on OLMo-3-7B erases the pseudo-labeled safety floor: SFT on D_T^synth achieves 99.78/95.64, but adding DPO drops scores to 74.52/63.14. This does not occur with KTO or ORPO on OLMo-3-7B (all >95%).

• Alignment reduces diversity, and pseudo-labeling causes mode collapse. Source-only SFT maintains the highest semantic diversity (0.46), while DPO and ORPO drop to 0.24 and 0.32. Under pseudo-labeling, semantic diversity falls to 0.07–0.08 across offline objectives and EAD falls from 0.86 to 0.51. Online RL preserves more diversity under pseudo-labeling: PPO 0.22 and GRPO 0.25 versus 0.07–0.08, with GRPO slightly outperforming DPO by 0.11.

• Logical diversity is reduced by pseudo-labeling. High NLI scores (>1.0) indicate contradictions while lower scores indicate consistency. Pseudo-labeling reduces NLI to 0.88, whereas SFT-Mix and SFT-Target maintain 1.05–1.10. The authors note that for summarization, lower diversity here is desirable as it suggests consistent factual retrieval, but it comes at the cost of syntactic and semantic variety.

• Domain relevance, not teacher quality, drives the pseudo-labeling benefit. Applying the same procedure to the source domain improves target win rate by +8.08 (SFT) and +4.79 (DPO), but the resulting gaps (28.26 SFT, 27.64 DPO) are similar to training on the original source data (23.50 and 31.78). Target-domain pseudo-labeling reduces these to 12.33 (SFT) and 17.29 (DPO). For DPO, target synthetic outperforms source synthetic by +15.62 points (78.50 vs 62.88).

• LLM judges are insensitive to epistemic mismatch. A domain-aware judge penalizes wrong-domain style. Source-only models lose 3–4 points, while pseudo-labeled DPO improves by +3.48. The near-zero QA gaps therefore partly reflect judge insensitivity. The authors characterize AskEngineers → AskCulinary as a near domain shift where surface format is retained but epistemic style and pragmatic persona change.

• Pseudo-labeling is highly data-efficient. Reducing pseudo-labeled data by 90% (a 10% subset) causes only negligible drops. For SFT, KTO, and ORPO, the 10% subset slightly outperforms the full dataset on target performance.

• SFT ordering matters. For Llama-3.1-8B, SFT on D_T before D_S achieves a target win rate of 56.40, compared to 35.22 when reversed. SFT D_T → DPO D_S yields 56.82; inserting an intermediate source SFT step (SFT D_T → SFT D_S → DPO D_S) increases this to 65.56.

• Structural mode collapse is quantifiable. The pseudo-labeled model maps topically unrelated CNN/DM articles onto a single template ("The article discusses… The author argues/advises/describes… The article also recommends"), while the source-only TL;DR DPO model preserves distinct editorial voices. By exact matching, this frame of discourse markers appears in ~10% of pseudo-labeled summaries against only ~0.2% of source-trained outputs.

Methodology in Plain English

The setup mimics a realistic constraint: you have human preference annotations for one domain (the source) but only unlabeled prompts for the domain you actually care about (the target). The goal is a policy that performs well on the target.

The authors run two axes in combination. The alignment objective axis covers supervised fine-tuning plus five preference methods: DPO and KTO and ORPO (offline, reward-model-free or reference-free), and PPO and GRPO (online reinforcement learning). The adaptation strategy axis controls what data each stage sees: source only, target only, a source-plus-target mixture, or target data that has been pseudo-labeled.

Pseudo-labeling works by taking every prompt in the unlabeled target corpus, generating candidate responses with a larger teacher model, and constructing preference pairs where the teacher-generated response is preferred and the dataset's original response is dispreferred. That synthetic set is then reformatted per objective — pairs for DPO, ORPO, and reward models; binary labels for KTO; prompt-and-preferred-response for SFT.

Evaluation is threefold. Generalization is measured as the win rate of an LLM-as-a-judge (GPT-5-nano) preferring the model output over a human reference, with the generalization gap defined as source win rate minus target win rate. Safety is measured as a safety score, defined as one minus the violation rate from the OpenAI Moderation API. Diversity is measured across N = 500 prompts with K = 16 sampled generations at temperature T = 1.0, using syntactic diversity (expectation-adjusted distinct n-grams for n = 1, …, 5 with a length-bias correction), semantic diversity (one minus average pairwise SBERT cosine similarity), and logical diversity via a natural language inference model.

The judge itself was validated against human annotations on QA helpfulness using 60 items and 3 annotators per domain; human–judge agreement (62–80%) is comparable to inter-annotator agreement (69–77%).

Why This Matters

Impact on research. The paper reframes preference tuning under domain shift as a two-part problem — adaptation and alignment — and shows empirically that the commonly varied part (the objective) is often less decisive than the less-discussed part (the adaptation data). It also introduces a concrete, measurable generalization–diversity trade-off and a diagnostic (epistemic mismatch) showing that standard LLM-as-a-judge win rates can over-reward structural coherence while missing domain-specific style, which has direct implications for how alignment work is evaluated.

Real-world applications:

• Summarization systems for new verticals (e.g., news, legal, or medical) where only unlabeled text exists — pseudo-labeling provides a lightweight route to target-domain priors.
• Domain-specific assistants like the AskEngineers → AskCulinary case, where a model can sound correct while violating a community's expected expertise and pragmatic conventions.
• Safety alignment across harm categories — adapting refusal behavior from one category (cybercrime) to another (violence and physical harm) without target annotations.
• Synthetic-data pipelines at reduced cost — the finding that 10% of the synthetic set matches or beats the full set dramatically lowers teacher-generation expense.

Industry relevance. Any team performing post-training on a domain where labeled preferences are scarce faces the choice this paper studies. The results argue for investing in target-domain pseudo-labeling rather than tuning the objective function, warn that some objective/model combinations (notably DPO on OLMo-3-7B safety) can overwrite prior safety training, and quantify the deployment risk of mode collapse in user-facing generative products where output variety matters.

Future Directions

1. Diversity-aware adaptation. Pseudo-labeling maximizes win rates but collapses output variety. Objectives that retain target-domain generalization while preserving syntactic and semantic range remain an open problem, particularly for creativity-oriented tasks.
2. Better judges and evaluation protocols. The domain-aware judge revealed that near-zero QA gaps partly reflect judge insensitivity to epistemic mismatch. Scaling domain-aware evaluation, or defining richer persona- and pragmatics-aware criteria, is a natural follow-up.
3. A principled account of the offline–online gap. The paper attributes offline over-specialization to fixed source preferences versus moving rollout distributions with a KL anchor, but does not test this mechanism directly. Controlled experiments could confirm the explanation.
4. Making the safety overwrite predictable. DPO erasing the pseudo-labeled safety floor on OLMo-3-7B but not on Llama-3.1-8B, and not with KTO or ORPO, is an unexplained model-dependent effect worth isolating.

Target Audience

This paper is most useful to researchers and engineers working on LLM post-training, preference optimization, and domain adaptation, especially those who must align models to a target domain without labeled preference data. It also serves evaluation researchers concerned with LLM-as-a-judge reliability, and safety practitioners interested in whether refusal behavior transfers across harm categories. Readers need a working familiarity with objectives such as DPO, PPO, and GRPO, and with the distinction between SFT and preference optimization, to get full value from the results tables.

Authors’ abstract

Preference tuning aligns base language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference tuning degrades performance and reduces helpfulness outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization, question-answering helpfulness, and safety alignment tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling substantially reduce domain-shift degradation but induce mode collapse, revealing a generalization-diversity trade-off.