CLewR: Curriculum Learning with Restarts for Machine Translation Preference Learning

AI summary

Overview

Research area: Natural Language Processing, specifically preference optimization for multilingual machine translation, combined with curriculum learning.

Technical level: Intermediate. The paper assumes familiarity with preference optimization objectives such as DPO, CPO and ARPO, and with automatic MT evaluation metrics (BLEU, COMET, METEOR, TER, chrF).

Scope: The paper proposes and evaluates a data-ordering strategy for preference-based fine-tuning of large language models for translation, testing it across three general LLM families, one MT-adapted model, three preference optimization algorithms, six Romance languages and Chinese.

What This Paper Is About

Preference optimization methods such as DPO, CPO and ARPO have been used to improve machine translation quality in LLMs, but almost all of this work ignores the order in which training samples are presented. The authors introduce CLewR (curriculum learning with restarts), which sorts preference triplets from easy to hard using translation metrics and repeats that easy-to-hard ordering at every epoch, with the explicit goal of preventing the model from forgetting the easy examples it saw early in training. The paper tests whether this ordering trick produces consistent gains across different models and different preference optimization algorithms.

Key Contributions

1. The authors propose CLewR, a data-level curriculum learning strategy for machine translation preference optimization in which an easy-to-hard ordering of triplets is restarted at every epoch to avoid catastrophic forgetting.
2. While earlier work applied curriculum learning to DPO (CurriDPO), this paper extends curriculum learning to newer preference optimization algorithms, namely CPO and ARPO, and also introduces a variant called CLewR-z that derives the curriculum score from the ARPO distance z.
3. The authors introduce enhanced ARPO variants that modify the distance function z to combine the original policy-based distance with distances derived from MT metrics (BLEU and COMET), yielding ARPO-z' variants V1 through V9.
4. They report consistent improvements across multiple model families (Gemma2, Llama3.1, Qwen2.5, and GemmaX2) and preference optimization algorithms (DPOP, CPO, ARPO), and release code publicly at a linked GitHub repository.

Main Findings

• CLewR improves advanced preference optimization methods: On the six-language Romance group, applying CLewR improved CPO and ARPO across the tested general LLMs. For Gemma2-9B, CPO went from 33.53 to 36.24 BLEU for en→xx, and ARPO went from 35.37 to 36.63 BLEU. For Qwen2.5-7B, CPO went from 27.68 to 30.05 BLEU, and ARPO from 30.41 to 31.56. For LLaMA3.1-8B, CPO went from 1.47 to 3.97 BLEU and ARPO from 33.50 to 34.49.
• CLewR beats CurriDPO on COMET: Across all three general LLMs, DPOP+CLewR outperformed DPOP+CurriDPO in COMET, with statistically significant improvements on 2 of the 3 models (Gemma2 and Qwen2.5), measured by paired bootstrap resampling with 10,000 samples at a 95% confidence interval.
• CLewR did not boost standalone DPOP: The authors hypothesize this is because the DPOP margin depends on both the policy and the reference model, and the typically positive reference margin offsets part of the policy margin, weakening the optimization signal relative to methods that rely only on the policy margin.
• Enhanced ARPO variants give further gains: ARPO-z'-V1 and ARPO-z'-V2 improved results beyond the other preference methods, and ARPO-z'-V2 obtained the best performance on average, which the authors attribute to incorporating external semantic signals from MT metrics into the z' distance.
• MT-adapted models benefit less: On GemmaX2-9B, gains from preference optimization and curriculum learning were less pronounced than for the general LLMs, and the strongest overall performance for GemmaX2 came from DPOP (38.23 BLEU and 89.01 COMET for en→xx). New GemmaX2 variations still surpassed both the original GemmaX2 and X-ALMA.
• Generalization to Chinese: For Gemma2-9B on English↔Chinese, CLewR-z improved both ARPO-z' variants on both directions and both metrics. For example, ARPO-z'-V1 went from 41.34 to 42.46 BLEU for en→zh, and from 27.66 to 29.49 BLEU for zh→en.
• Overall statistical picture: In the configuration-level analysis, 13 out of 20 configurations showed statistically significant improvements, 4 showed no significant change, and 3 resulted in performance degradation. The largest increases were obtained by applying CLewR over CPO, while CLewR over DPO degraded performance.
• Complementary metrics agree: CLewR generally improved lexical diversity (MATTR), chrF, and reduced edit distance (TER) for the more advanced preference optimization methods, and the GPT5.1 LLM-as-a-judge scores were mostly consistent with BLEU and COMET.
• Ablation findings: In the English-to-Romanian ablation, the best ARPO variants typically involved z_COMET, z_BLEU helped when averaged with z_θ and z_COMET, and smaller values of η1, η2 and η3 led to better performance.

Methodology in Plain English

The authors take an existing set of preference triplets: a source sentence, a preferred translation, and a rejected translation. For each triplet they compute how similar the chosen and rejected translations are, using the average of three normalized translation metrics (BLEU, COMET-22 and METEOR), each rescaled to a 0-1 range. This average becomes a curriculum score. The triplets are then sorted by this score so that the easiest pairs come first, and training proceeds through all of them in that fixed order without any random shuffling. Crucially, when the next epoch begins, the ordering restarts from the easiest examples rather than continuing or shuffling, which is the "restarts" part of the name and the mechanism intended to counter catastrophic forgetting.

The same procedure is applied on top of whatever preference optimization loss the researcher is using: DPOP, CPO or ARPO. For ARPO specifically, the authors also derive the curriculum score from the ARPO distance z, giving a variant called CLewR-z, and they build an enhanced ARPO whose distance mixes the original policy-based distance with BLEU- and COMET-derived distances, weighting each with scalar hyperparameters.

Training used LoRA adapters of rank 64 on top of the base models, with a learning rate of 5·10⁻⁵, a warmup ratio of 0.1, mini-batch size 4, and the Adam optimizer. Gemma2, Qwen2.5 and Llama3.1 were each trained for three epochs; GemmaX2, being already adapted for MT, was trained for one epoch. Experiments used the Flores-200 benchmark and X-ALMA preference data (81K train and 7K validation triplets). Evaluation covers translation to and from English for Catalan, Galician, Italian, Portuguese, Romanian and Spanish, with GemmaX2 restricted to Italian, Portuguese and Spanish because it was not adapted for MT on the other three languages. Chinese was used to test generalization beyond Romance languages. Each setup was run three times, with the checkpoint having the best validation performance selected per run, and average scores plus standard deviations reported.

Why This Matters

Impact on research. The paper shows that data ordering, a factor largely ignored in the preference optimization literature for MT, produces measurable and often statistically significant gains on top of strong algorithms. It also argues that curriculum learning is more naturally compatible with some preference objectives (CPO, ARPO) than others (DPOP), which is a useful negative result rather than just a success story. By combining policy-derived distances with metric-derived distances, it also proposes a way to inject MT evaluation signals directly into the preference objective.

Real-world applications.

• Improving translation quality of open-weight multilingual LLMs used for document, website or customer-support localization.
• Reducing post-editing effort for professional translators, since CLewR reduced TER for the more advanced preference optimization methods.
• Fine-tuning models for low-resource or under-served language pairs where a model's zero-shot translation is poor, as seen with Qwen2.5-7B and LLaMA3.1-8B before tuning.
• Cross-lingual deployment where a single model must handle both directions (into and out of English) for many languages.

Industry relevance. The method is a training-schedule change rather than a new model architecture, so it can be layered onto existing preference optimization pipelines with the same data and the same loss functions. The use of LoRA adapters (rank 64) and a small number of epochs suggests relatively modest compute requirements, which matters for teams fine-tuning models such as Gemma2, Qwen2.5 or Llama3.1.

Future Directions

• Extending CLewR to other NLP tasks beyond machine translation, which the authors explicitly name as future work.
• Testing on more language groups and more model families to strengthen generalization, since the current study is limited to the six Romance languages defined in X-ALMA, with generalization demonstrated only on Chinese.
• Investigating why CLewR helps CPO and ARPO but not standalone DPOP, given the authors' hypothesis about the reference-model margin offsetting the policy margin.
• Addressing bias: the authors note that CLewR, like generic preference optimization algorithms, does not specifically address training data and model-specific biases, and that additional debiasing techniques would be needed.
• Exploring further z' combinations, since only a subset was examined in the main paper and in the English-to-Romanian ablation.

Target Audience

Researchers and practitioners working on machine translation with large language models, particularly those already using or evaluating preference optimization methods such as DPO, CPO and ARPO. It is also relevant to anyone studying curriculum learning and training-data ordering, and to engineering teams doing preference fine-tuning of open-weight multilingual models who want a low-cost change to their training loop. Readers without background in preference optimization or MT evaluation metrics will need to consult the cited primary sources for DPO, CPO and ARPO before the method details fully land.

Authors’ abstract

Large language models (LLMs) have demonstrated competitive performance in zero-shot multilingual machine translation (MT). Some follow-up works further improved MT performance via preference optimization, but they leave a key aspect largely underexplored: the order in which data samples are given during training. We address this topic by integrating curriculum learning into various state-of-the-art preference optimization algorithms to boost MT performance. We introduce a novel curriculum learning strategy with restarts (CLewR), which reiterates easy-to-hard curriculum multiple times during training to effectively mitigate the catastrophic forgetting of easy examples. We demonstrate consistent gains across several model families (Gemma2, Qwen2.5, Llama3.1) and preference optimization techniques. We publicly release our code at https://github.com/alexandra-dragomir/CLewR.