Language Chain in Alignment: Cross-lingual Ranking Preference Optimization

AI summary

Overview

Research area: Natural Language Processing, specifically cross-lingual alignment and preference optimization for Large Language Models.

Technical level: Advanced. The paper assumes familiarity with Direct Preference Optimization (DPO), reinforcement learning from human feedback, and Learning-to-Rank objectives such as LambdaLoss and nDCG.

Scope in one sentence: The paper introduces CRPO, a training framework that replaces single-language binary preference comparisons with a four-candidate hierarchical ranking structure spanning English and a target language.

What This Paper Is About

Alignment data for LLMs is overwhelmingly English-centric, so models tuned on it behave inconsistently in other languages, sometimes answering a non-English prompt in English or producing lower-quality responses. This paper asks whether a model's already-internalized English preference knowledge can be used as a reference signal to align a target language, rather than collecting expensive new human annotations. The proposed answer, Cross-lingual Ranking Preference Optimization (CRPO), reframes alignment as ranking four parallel responses instead of comparing two.

Key Contributions

1. The CRPO framework, which integrates ranking principles into cross-lingual preference alignment so that the model optimizes a hierarchical priority between language and quality rather than a flat binary comparison.

2. A distribution-level analysis of reward and log-probability shifts showing that CRPO steers the target-language preference manifold more effectively than existing methods, as measured on a held-out test set.

3. Evidence that the cross-lingual hierarchy design itself drives the gains, demonstrated by showing that removing hierarchical cues (assigning uniform weights to all response pairs) causes a notable performance drop and triggers English bias.

4. A study of weighting schemes, showing that three LambdaLoss-derived schemes (LambdaRank, nDCG2, nDCG2++) all outperform both SFT+DPO and a uniform weighting, with nDCG2 used as the primary scheme for main experiments.

Main Findings

• Consistent AlpacaEval gains across five languages. CRPO achieves the best length-controlled win rate (LC) for all three models in essentially every language tested. For Llama-3-8B, CRPO reaches LC of 62.45 (Chinese), 67.97 (Indonesian), 68.45 (Korean), 61.84 (Swahili) and 53.86 (Bengali), versus SFT+DPO at 60.24, 59.24, 64.97, 46.23 and 34.77 respectively.

• Large gains in low-resource settings. CRPO attains a Swahili raw win rate (WR) of 62.17 with Llama-3-8B, while SFT+DPO reaches 46.95 and CLO reaches 44.93. The paper attributes this to using well-aligned English pairs as a logical anchor.

• Baselines can degrade in low-resource languages. The paper reports that Llama-3-8B suffers a collapse compared to SFT in Bengali, with notably low WR, and that CLO's performance drop indicates that enforcing language-consistent responses through binary cross-lingual hierarchies is insufficient.

• Knowledge benchmarks also improve. MMMLU for Llama-3-8B Indonesian rises to 45.94 with CRPO versus 41.69 for SFT+DPO and 36.41 for CLO, a gap the paper describes as exceeding the strongest baseline by over 4 points. Korean Belebele for Llama-3-8B reaches 68.66 (SFT+DPO 57.55, CLO 59.44).

• Reward margins and chosen-response likelihood both increase. On a held-out test set, CRPO shifts the reward-difference distribution positively and raises the log-likelihood of chosen responses across all languages, whereas other methods increase the reward margin while their log-likelihood distributions remain largely congruent with SFT.

• External reward model confirms absolute quality gains. Using Skywork-Reward-V2-Qwen3-8B, CRPO scores highest in nearly all configurations. The largest cited escalation is Mistral-7B Indonesian, where SFT+DPO scores -0.037 and CRPO reaches 0.805.

• Hierarchy matters more than the specific weighting. All three weighting schemes outperform SFT+DPO and the uniform scheme, while uniform weighting causes a notable drop attributed to loss of alignment between languages and resulting English bias.

• Gain values were manually tuned. Standard exponential gains (7, 3, 1, 0) yielded suboptimal win rates; the proposed configuration (9, 7, 5, 4) keeps intra-language quality gaps larger than the language gap. A deliberately inverted configuration (9, 5, 7, 4) performed slightly lower in most languages, except Swahili.

Methodology in Plain English

Data. The researchers sampled 3,000 instances from UltraFeedback and translated them into five target languages (Chinese, Indonesian, Korean, Swahili, Bengali) using gpt-5-chat, producing preference pairs that are fully parallel to the original English pairs. The dataset, including English pairs, was split 90% training and 10% test.

Training signal. For each instance, four responses are assembled: the chosen and rejected responses in the target language, and the chosen and rejected responses in English. These are ranked in a fixed hierarchy, with the target-language chosen response on top, followed by the English chosen response, the target-language rejected response, and the English rejected response. This ordering means a high-quality English answer outranks a low-quality target-language answer, which supplies a quality signal independent of language while still pushing the model to answer in the target language.

Objective. Rather than the binary log-sigmoid comparison used in DPO, CRPO borrows from LambdaLoss, weighting each pair of responses by how much swapping their order would hurt the ranking metric. Pair weights are computed from gain and rank-discount terms. The final loss blends a ranking loss with a negative log-likelihood term on the target-language chosen response, balanced by a hyperparameter alpha set to 0.2 for CRPO and SFT+DPO and 0.5 for CLO, with beta set to 0.1.

Models and training. Three models were used: Llama-2-7B, Llama-3-8B, and Mistral-7B-v0.1. Training used a learning rate of 8e-6, two epochs, a maximum sequence length of 3072, a linear schedule with 10% warmup, and four NVIDIA A100 GPUs.

Evaluation. Conversational ability was measured with the multilingual version of AlpacaEval (805 questions from 5 datasets), judged by gpt-5-chat with WR and LC reported. Knowledge and comprehension were measured with MMMLU (zero-shot) and Belebele (one-shot). Internal dynamics were analyzed via reward-difference and log-probability distributions, and absolute quality via the external Skywork-Reward-V2-Qwen3-8B reward model.

Baselines. SFT+DPO (standard alignment using in-language pairs) and CLO, a DPO variant that treats the target-language response as chosen and the English response as rejected, and restricts the NLL loss to target-language responses.

Why This Matters

Impact on research. The paper reframes cross-lingual alignment as a ranking problem rather than an isolated binary comparison, and shows that a model's English preference knowledge can serve as a structural anchor for other languages without new human annotation. It also provides an intrinsic analysis linking reward margins to chosen-response likelihood, arguing that maximizing reward margins alone can suppress rejected responses rather than promote desirable ones.

Real-world applications:

• Deploying assistant models in languages such as Bengali, Swahili or Indonesian where high-quality native preference data is scarce.
• Reducing language confusion in customer-facing systems that must reply in the user's language.
• Improving multilingual question answering and reading comprehension in knowledge-intensive domains.
• Using an English-annotated preference corpus as a reference to bootstrap alignment for a new target language.

Industry relevance. The approach is built directly on DPO, needing no explicit reward model or reinforcement learning pipeline, which keeps it compatible with existing alignment toolchains. The trade-off is that each training step processes four response candidates instead of two, which the authors acknowledge as a higher computational cost during training.

Future Directions

• Dynamic gain adaptation. The conclusion states that the authors plan to investigate adapting the gain structure based on linguistic similarity and training stage.

• Reducing ranking-space complexity. The paper notes that mitigating the optimization complexity inherent in expansive ranking spaces would help extend robust alignment to a wider array of language pairs.

• Closing the compute gap. The limitations section identifies the four-candidate-per-step structure as inherently more resource-demanding, and leaves the question of whether this overhead can be reduced open.

• Evaluation beyond general capability. The authors note their benchmarks may not capture language-specific characteristics or cultural contexts, leaving evaluation of cultural nuance and localized content as an open problem.

Target Audience

Researchers and engineers working on LLM alignment, multilingual NLP, and preference optimization, particularly those who already understand DPO and want a technical account of how ranking-based objectives can be extended across languages. The paper is also relevant to practitioners who need to improve model quality in lower-resource languages without commissioning new human preference annotations.

Authors’ abstract

The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.