Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

AI summary

Overview

Research area: Natural language processing; post-training and preference alignment of large language models for lower-resource languages.

Technical level: Intermediate. The core idea is easy to grasp, but the training objective involves policy-gradient reinforcement learning and KL-divergence regularisation.

Scope: A case study on Norwegian Bokmål showing that an on-policy reinforcement-learning post-training method, judged by a disfluent reward model, produces more fluent language models than supervised finetuning on machine-translated data — without any instruction-tuning data in the target language.

What This Paper Is About

Instruction-tuning and preference optimisation work well for English and Chinese, which have large human-written instruction datasets and fluent instruction-tuned models that can generate synthetic data. Lower-resource languages have neither, so the standard workaround is to machine-translate English datasets — but translation introduces translationese, which makes the resulting models disfluent. The paper asks how to build a fluent, preference-aligned model for a lower-resource language when no instruction dataset exists in that language, and proposes an on-policy method that never trains on translated responses.

Key Contributions

1. An on-policy reinforcement-learning post-training method for lower-resource languages that maintains fluency and requires no instruction dataset in the target language — the policy is trained only on responses it samples itself.
2. A native-speaker evaluation with five native Norwegian speakers showing that on-policy training produces more fluent models than supervised finetuning on machine-translated data (preferred in 67.5% of pairwise comparisons).
3. A demonstration that fluent aligned models can be bootstrapped using disfluent judges: the policy outperforms its own reward model on fluency, so post-training is possible for languages that have no fluent instruction-tuned model available.
4. Ablations showing the importance of avoiding any exposure to translated responses, even in small quantities, during training; plus an analysis of how the choice of judge model, translation model and SFT length affect outcomes.

Main Findings

• On-policy RL is preferred over translated SFT by native speakers. In pairwise fluency judgements, the on-policy RL model won 67.5% against translated SFT. Average win-rates were 79.7 for on-policy RL, 60.0 for translated SFT and 10.3 for Mistral Nemo. On-policy RL beat Mistral Nemo 91.8% of the time and translated SFT beat Mistral Nemo 87.5% of the time. About one third of comparisons between the two Norwegian-trained models were judged "equal".

• The policy is more fluent than its judge. The judge used in the main experiment was Mistral Nemo 12B, whose own fluency score was 67.0, while the policy trained with its rewards reached 92.2. Across eight judges (three Mistral models, three Qwen models, two Llama models), the Pearson correlation between judge fluency and trained-policy fluency was only 0.067 — policies were fluent regardless of how (dis)fluent the judge was.

• Automated fluency scoring tracks the human ranking. A Bradley-Terry fluency scorer agreed with the annotators' preferred ranking in 85.5% of non-tie cases, slightly above the 83.2% rate at which annotators agreed with the consensus. It scored on-policy RL at 2.47 (92.2%), translated SFT at 1.94 (85.7%) and Mistral Nemo at 0.76 (65.3%).

• Fluency stays stable throughout on-policy training. Measured every 25 training steps, the fluency score held around 93% after converging in the first 50 steps. An initial score of 87.5% was traced to the model answering in English (a carryover from the English SFT stage) rather than to disfluent Norwegian. In contrast, SFT on translated Norwegian caused a clear decline in fluency.

• More SFT data, and translated SFT data, hurt fluency. Starting RL from checkpoints trained for 1, 2 and 4 epochs on the English dataset yielded fluency scores of 94.2, 93.2 and 92.8 respectively. Starting from a checkpoint trained for 1 epoch on the translated dataset yielded 91.0.

• Translation quality matters but does not remove the problem. Fluency of models trained on data translated by different systems: Tower-Plus (72.7B) 85.7, MADLAD-400 (10.7B) 82.4, NLLB-200 (3.3B) 75.5, Seed-X (7.5B) 73.4, OPUS Eng-Gem (0.1B) 68.2. The largest model, Tower-Plus, gave the most fluent result.

Methodology in Plain English

The method has three stages, with one guiding rule: the model is never trained on unnatural text.

1. Pretraining on the target language. A base model learns the language from native text. For Norwegian the authors build on NorMistral 11B, a continually pretrained Norwegian base model (Samuel et al., 2025). This stage is not studied in the paper.

2. Short supervised finetuning on English. To teach the model the chat format and how to respond to prompts, it is trained for one epoch on the 1,000 curated English prompt-response pairs from LIMA — only 31 training steps. The brevity is deliberate, so the model does not forget its Norwegian.

3. On-policy alignment on the target language. The model generates its own responses to translated prompts from the No Robots dataset, and a judge model grades those responses (it can also see the gold English response). Training uses a Reinforce-style policy-gradient loss on these self-generated responses, with advantages normalised by the mean and standard deviation over a group of sampled responses, and with KL-divergence regularisation toward the reference policy. Each step takes 128 prompts and samples 8 responses per prompt. The authors use a Rao-Blackwellized estimate of the KL term (Amini et al., 2025) that uses the full next-token distribution, and normalise the log-likelihood by total response length to counter length bias (Rastogi et al., 2025). Training is synchronously parallelised by postponing the sampled-policy update, which makes samples off-policy by only three steps.

For evaluation, five native Norwegian speakers compared 300 response pairs (100 seed prompts from the Norwegian mimir-instruct dataset, three model pairings per prompt) in a randomised A/B/equal format, spending roughly 15–20 hours each. Responses were generated with nucleus sampling at temperature 0.5, top_k 64 and top_p 0.9.

Why This Matters

The paper argues that post-training for lower-resource languages should move away from machine-translated instruction data because it bakes translationese into the model. It shows a route to aligned, fluent models for such languages that needs neither native-written instruction data nor an existing fluent instruction-tuned model in the target language — only a judge that understands the language well enough to compare responses. This connects to work on weak-to-strong generalisation (Burns et al., 2024; Charikar et al., 2024) and to RLAIF findings that judges can evaluate quality they cannot themselves generate (Lee et al., 2024; Zheng et al., 2023), and it echoes concurrent work by Kunz (2026) on machine-translated Swedish.

Real-world applications:

• Building chat assistants and instruction-following models for languages that lack native instruction datasets.
• Localisation and content pipelines where machine-translated training data degrades output quality over time.
• Adapting existing monolingual or multilingual base models to a new target language without collecting new annotation.
• Supporting native-speaker-driven evaluation as the quality signal for model releases in under-served languages.

Industry relevance: organisations that already own a pretrained model for a specific language can align it using their own infrastructure and a multilingual judge, rather than paying for translated instruction data or relying on a large English-centric model. The paper also releases the annotation dataset, giving others a reusable benchmark for Norwegian fluency.

Future Directions

• Extending the case study beyond Norwegian Bokmål to other lower-resource languages that also lack fluent instruction-tuned judges, to test how far the "disfluent judge, fluent policy" effect generalises.
• Determining how much target-language understanding a judge needs before the reward signal degrades: the paper tests judges down to Qwen 2.5 14B (fluency 39.0), which still produced a policy at 93.1, but does not probe the lower bound.
• Investigating what the choice of judge does affect, since the paper finds it changes other response qualities but not fluency.
• Clarifying whether the fluency advantage of on-policy training persists under more training compute, different prompt domains, and settings where no reference English gold response is available to the judge.

Target Audience

Researchers and engineers working on multilingual and lower-resource NLP, preference optimisation and RLHF/RLAIF, and post-training pipelines. It is also relevant to teams productising language models in languages without native instruction data, and to anyone interested in how evaluation quality and generation quality can diverge.

Authors’ abstract

We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.