How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

AI summary

Overview

Research area: Multilingual safety alignment and red-teaming data generation for large language models (Natural Language Processing / AI safety).

Technical level: Intermediate — readers should be comfortable with fine-tuning (SFT, DPO), LLM-as-a-judge evaluation, and standard safety metrics such as attack success rate and over-refusal.

Scope: This paper introduces a template-based, fully automated pipeline that turns a small set of English harmful prompts into a large multilingual corpus (MSafeTemp) of harmful, benign, and culturally localized prompts, and uses it both to expose multilingual safety gaps in open models and to train models that are safer outside English.

What This Paper Is About

Safety alignment for large language models is heavily English-centric, so a harmful request that a model reliably refuses in English may be answered once it is translated into another language, and even more so when it is adapted to local cultural referents. Collecting that kind of data at scale is hard: native-speaker red-teaming is slow and costly, machine translation keeps English entities, and asking aligned models to translate or paraphrase harmful content often just gets a refusal. The authors solve this by abstracting a harmful prompt into a template — the harmful words are replaced by typed placeholder slots — which is then translated and refilled with generic or culturally local entities, producing harmful variants, benign minimal pairs, and localized variants at scale.

Key Contributions

1. A template-based, culturally grounded multilingual safety-data pipeline that turns a small seed of English harmful prompts into fluent, harm-preserving, culturally authentic variants across 60 languages, with matched benign minimal pairs and jailbreak wrappers, each validated by a dual-stage LLM judge. The pipeline also releases automatically generated cultural cards for all languages so others can reproduce and extend the data.

2. MSafeTemp, a large-scale grounded multilingual safety corpus — a provenance-tracked collection of more than 1 million validated prompts across 44 languages, split by source seed so that no descendant of a test seed enters training, plus a smaller culturally localized harmful subset, a benign subset, and held-out red-teaming and over-refusal evaluation sets.

3. A multilingual safety evaluation showing that English-only evaluation misjudges model safety: translated attacks that fail in English succeed, and culturally localized variants expose further failures that translation alone misses.

4. Training ablations (SFT followed by DPO) demonstrating that the corpus improves attack resistance and reduces over-refusal in multilingual settings, including in languages not present in the training set.

Main Findings

• English-only evaluation is misleading. Olmo and both Apertus models break more often after translation in nearly every language on both tracks — Olmo on HarmBench goes from 0% to 13.7% attack success rate. Gemma shows the effect only on the WildGuard track, while Llama-3.3-70B-Instruct is most vulnerable in English (13.0% Orig. vs 5.3% TR and 2.6% PH on HarmBench).

• Translated variants break models 2.4–4.4× as often on the held-out track. On the 500 held-out WildGuard seeds, translated variants break Olmo, Gemma, and both Apertus models 2.4–4.4 times as often as the original English seeds. Qwen3.5-27B and GPT-OSS-20B stay near or below 1% on every arm, but GPT-OSS-20B pays with the highest over-refusal in the panel (refusing 32–45% of benign pairs).

• Localization reaches failures translation misses. With wording held fixed, localized and generic fills break models about equally often, yet they agree on fewer than half of the behaviors they break (Jaccard 0.40–0.45), and roughly 45% of arm-specific failures occur only with local entities. Localization raises ASR for Apertus-8B in every language shown and lowers it in most languages for the other models; across languages, PH lowers ASR for 5 of 7 leading models on HarmBench and 6 of 7 on WildGuard, but raises it for Apertus-8B and the instruction-tuned Olmo checkpoint.

• Low attack success is not the same as good alignment. Strongly refusing models over-refuse (GPT-OSS-20B refuses 32–45% of benign pairs), more permissive models remain easier to attack outside English, and only a few, such as Qwen3.5-27B on HarmBench, reach low ASR without a large refusal cost. Parameter scale alone does not remove this tradeoff.

• Paraphrased templates matter, and so do individual slot fills. On WildGuard, paraphrases break all seven models more often than the main template (up to +3.7%); on HarmBench they do not. Templates broken by only some of their fills produce 60–100% of all unsafe answers, and scoring a single fill per template misses roughly half of the broken templates.

• Jailbreak wrappers help unevenly. On 200 matched template families wrapped in ten distinct PyRIT jailbreaks, wrappers raise average per-attempt ASR for Apertus and Llama but show no clear gain over PH for Olmo or Qwen; the higher cumulative rate from trying several wrappers reflects extra attempts, not higher per-attempt effectiveness.

• Training narrows the gap and transfers to unseen languages. SFT sharply reduces multilingual attack success across model families and checkpoints — for example, Apertus-8B-2509 base HarmBench ASR falls from 14.2% to 3.3% after SFT and 1.5% after DPO. Gains appear in languages absent from the training set, close to those in seen languages, indicating cross-lingual transfer rather than memorization.

• SFT raises over-refusal; DPO mostly recovers it. For Apertus, SFT raises multilingual over-refusal on benign pairs only slightly (1.6% → 13.5% on HarmBench for the base checkpoint) and DPO brings it back down (7.7%), ending below the instruction-tuned model. For Olmo, which starts from very low over-refusal, the rise is substantial (2.7% → 28.4% after SFT, 19.8% after DPO).

• Capability cost is concentrated in math. SFT costs some general capability — modest on MMLU but larger on GSM8K (Apertus-8B-2509 base: 60.1 MMLU and 49.9 GSM8K, versus 59.1 and 45.2 after SFT). The authors attribute this to the aggressive nature of SFT training and stronger weight editing.

• Human validation supports quality but qualifies the benign labels. In an IRB-approved study with safety experts who are native or fluent speakers (15 unique ratings, 11 languages, 600 judgments), prompts were rated natural at 4.43/5 and culturally fitting at 4.46/5. Raters endorsed 91% of harmful labels but only 65% of benign ones, often because a softened or fictional control still reads as harmful, so over-refusal on these benign pairs may overstate true over-refusal.

• The corpus is diverse, not collapsed onto a few fills. Harmful slots receive over 2.8M distinct entities (distinct-to-total ratio around 0.7), benign slots are more varied still (0.94), no entity appears in more than 1% of a language's harmful fills, and diversity does not differ significantly between high- and low-resource languages.

Methodology in Plain English

The core trick is to never hand a language model a complete harmful request. Instead, a generator model takes an English harmful prompt and extracts the single words or short phrases that make it harmful, replacing them with typed placeholder tags named after their intent (for example <illicit_substance>). What remains is an abstract template that is no longer itself harmful, so it can be translated and paraphrased without tripping the model's own safety training.

Each template passes a first LLM judge that checks it is valid and self-contained. The generator then translates the surrounding text while keeping the placeholder tags in English, and a second judge checks meaning preservation, slot integrity, translation quality, and target-language consistency — with a patching step that edits only the flagged spans rather than retranslating everything, for at most three rounds. Every template also gets two validated paraphrases, so each seed and language has three templates.

Three branches then fill the slots:

• TR (harmful, translated): generic harmful fills — five per template, up to 15 variants per seed and language — judged for harmfulness, language quality, coherence, and preservation of the original framing.
• OR (benign minimal pairs): the same template filled with safe values, producing questions phrased like harmful ones but with ordinary intent. These are split into direct controls (safe to answer fully) and bounded controls (a good answer must still withhold transferable harmful detail). Over-refusal is measured only on direct; bounded prompts are used only as training signal.
• PH (culturally localized): for non-English languages, a per-language cultural card lists locally relevant groups, organizations, events, concepts and institutions. These are used as slot values where they fit, and the generator may inflect and rephrase around the entity for natural grammar; results pass the same final judge as TR.

Judges make a separate Boolean decision per criterion, which are combined in code — a fix for an observed polarity-inversion failure where a judge's reasoning called a variant harmful but its overall verdict said otherwise. An LLM grammar check was dropped because it produced mostly false flags and sometimes introduced errors; grammar is instead constrained during generation. Generation, generation-judging, evaluation-judging, and cultural-card verification use different model families.

Data sources. 200 HarmBench standard behaviors (contextual and copyright categories dropped) form a held-out evaluation track. 2,500 WildGuardMix harmful prompts from the wildguardtrain split form the training track, expanded into 44 languages, with 2,000 seeds and their descendants as training and the remaining 500 as a held-out test split. HarmBench expands to a 216,486-prompt benchmark in 60 languages; the WildGuard seeds are enriched into a 1.8M validated set in 44 languages.

Models used. Gemma-4-31B-IT for generation, GPT-OSS-120B as the generation judge, GPT-OSS-Safeguard-120B for taxonomy and response judging, and Gemini Flash 3.6 to verify cultural cards.

Evaluation and training. GPT-OSS-Safeguard-120B judges greedy responses of each target model; ASR is the share of harmful prompts answered with unsafe compliance, and OR is the share of benign direct pairs refused. For training, models are fine-tuned for one epoch on a mix of refusals and benign answers (generated by Gemma-4-31B-IT, verified by GPT-OSS-120B) combined with general instruction data, with loss only on assistant tokens, then DPO is applied on preference pairs sampled from the fine-tuned model itself. The same recipe is applied to base and instruction-tuned checkpoints of Apertus-8B-2509, Apertus-v1.5-8B, Llama-3.1-8B and Olmo-3-7B.

Why This Matters

Impact on research. The paper argues that safety should be treated less as a general model-capability axis and more as a combination of model, language, and culture axes. It provides a reproducible, automated alternative to costly native-speaker red-teaming, and it demonstrates that English refusal rates are not a sufficient proxy for multilingual safety — a methodological point that affects how alignment work is benchmarked.

Real-world applications:

• Safer assistants for speakers of low-resource languages, who the paper notes are the least protected and constitute an easy attack vector for misuse.
• Pre-deployment safety auditing of multilingual models in 44–60 languages with held-out harmful and matched benign test sets.
• Over-refusal measurement and tuning, so that safety gains do not come at the cost of refusing ordinary, harmless requests in non-English languages.
• Language- or region-focused alignment projects that can reuse the open pipeline and cultural cards to scale data for a particular community.

Industry relevance. The corpus is released under a permissive license with a content warning gating access on Hugging Face, and the code is public. The training ablations give practitioners a concrete recipe — SFT on the corpus followed by DPO on the model's own samples — and quantify the safety/helpfulness/capability tradeoffs, including the caveat that around 20% of OR-Bench samples are judged harmful and should be refused, so perfect scores on that benchmark are not optimal.

Future Directions

• Stronger validation of automated judging. The pipeline relies heavily on LLM judges. Two judges from another model family preserved the direction of headline results on a 5,000-response subsample, but a full ablation over judge models remains future work.
• Larger human evaluation. The human study covers only a subset of languages with one or two respondents each; broader human validation of naturalness and cultural fit is left open.
• Re-labeling localized variants. Harm labels are assigned to the English seed and inherited by all descendants, but a localized entity such as an extremist group or a regional slur can make a variant more severe than its inherited label suggests.
• Beyond single-turn SFT/DPO. The prompts are single-turn and follow fixed templates; extending to multi-turn and iterative attacks (for instance using localized variants as seeds for automated iterative attackers) and to RL-based alignment methods remains future work.

Target Audience

Researchers and engineers working on LLM safety alignment, red-teaming, and multilingual evaluation; dataset builders interested in scalable synthetic data generation; and teams deploying models to non-English or low-resource language markets who need practical guidance on the safety/over-refusal/capability tradeoff. It is also useful for policy-oriented readers who want evidence that English-centric safety evaluation understates risk for other language communities.

Authors’ abstract

Safety alignment for Large Language Models is heavily focused on English with limited multilingual representation, particularly damaging low-resource language performance. Related open training datasets are often limited in size or do not follow region-localized harmful topics to broaden the scope of alignment beyond cultures. We propose a scalable template-based augmentation pipeline to produce massive multilingual refusal and benign corpora, extracting safe templates from harmful prompts, translating them and then filling in the slots with culturally authentic entities, both harmful and benign. As a result, we release MSafeTemp, a validated corpus spanning 60 languages useful for multilingual safety alignment and evaluation. Using the dataset, we show that English-only evaluation misjudges multilingual safety: for several models, attacks that fail in English succeed once translated, while culturally localized variants expose further failures that translation alone misses. We also confirm that augmenting safety training data through templates allows for higher diversity and quality of alignment, resulting in improved safety performance beyond English. MSafeTemp and the respective pipeline are aimed at facilitating research and development of safer and more responsible models for less-represented cultures.