Unlocking LLM Safeguards for Low-Resource Languages via Reasoning and Alignment with Minimal Training Data

AI summary

Overview

Research area: Natural Language Processing — safety and content moderation with large language models, specifically multilingual LLM safeguards for low-resource languages.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning (SFT), reinforcement learning for LLMs, group relative policy optimization (GRPO), direct preference optimization (DPO), and KL-divergence regularization.

Scope: The paper proposes ConsistentGuard, a three-stage training pipeline (SFT cold start, GRPO reasoning training, and Constrained Alignment Optimization) that builds a multilingual safeguard from Qwen2.5-3B using only 1,000 seed training samples, and evaluates it on three safety benchmarks extended to six languages.

What This Paper Is About

Existing LLM safeguard systems are mostly classifier-style models such as Llama Guard and ShieldGemma: they output a label without explaining why, and their accuracy drops sharply on low-resource languages such as Bengali. Newer reasoning-based safeguards like GuardReasoner produce explanations, but are trained on a single mainstream language, so their reasoning is inconsistent across languages. This paper's goal is to build a safeguard that reasons transparently, transfers that reasoning from high-resource to low-resource languages, and does so with very little training data.

Key Contributions

1. A reasoning-based multilingual safeguard training framework that improves explainability, effectiveness, and cross-lingual generalization for low-resource languages.
2. Constrained Alignment Optimization (CAO), a novel RL-based alignment algorithm that targets cross-lingual reasoning inconsistency to reduce the performance gap caused by language imbalance.
3. An evaluation on three datasets across six languages, with analysis supporting the method's working mechanism, effectiveness, and robustness.
4. Release of reasoning-based multilingual safeguard training code plus extensions of three existing English safety benchmarks to six languages to support future research.

Main Findings

• Strong results from a tiny data budget: Using only 1,000 training samples and a 3B-parameter model, ConsistentGuard achieved second-place rankings on most languages, while baselines such as GuardReasoner used substantially larger datasets — 127,600 samples — and some baselines used over 100,000 samples.
• OpenAI Moderation results: ConsistentGuard (3B) scored 78.94 (en), 76.46 (fr), 76.83 (zh-cn), 77.50 (jp), 72.10 (bn), and 73.26 (hi), compared with GuardReasoner (3B) at 74.87, 77.67, 76.68, 77.12, 70.52, and 72.08, and Llama Guard 3 (8B) at 79.69, 79.90, 78.06, 77.71, 74.64, and 78.63.
• ToxicChat results: ConsistentGuard (3B) scored 84.26 (en), 82.39 (fr), 82.32 (zh-cn), 81.22 (jp), 73.55 (bn), and 73.79 (hi), versus GuardReasoner (3B) at 84.23, 84.60, 84.46, 84.44, 73.85, and 78.47.
• SimpleSafetyTests results: On this benchmark of 100 simple positive test cases, ConsistentGuard (3B) scored 97.96 (en), 96.91 (fr), 91.30 (zh-cn), 92.47 (jp), 90.11 (bn), and 89.50 (hi), compared with GuardReasoner (3B) at 98.48, 97.96, 97.44, 95.29, 91.89, and 97.44.
• Reasoning training improves generalization: Comparing reasoning and non-reasoning models showed reasoning training significantly improves performance in all languages. On SimpleSafetyTests, the SFT (3B) model scored 69.28 on Bengali and 80.24 on Hindi, while the R1-GRPO (3B) model reached 86.36 and 87.01, and the full method reached 89.50 and 89.50.
• CAO beats DPO for alignment: Alignment ablations showed CAO raised performance in most languages while DPO failed. On OpenAI Moderation, CAO improved Japanese (76.40 without alignment to 77.50), Bengali (71.15 to 72.10), and Hindi (71.98 to 73.26), whereas DPO improved English (77.40 to 78.48) and French (77.67 to 77.52) but reduced Chinese (77.45 to 72.14) and Hindi (71.98 to 70.82). On SimpleSafetyTests, CAO improved English (96.91 to 97.96) and French (95.83 to 96.91) while DPO reduced Chinese (92.47 to 89.50).
• LLaMA models hold up better in Bengali and Hindi: Llama Guard and GuardReasoner showed smaller cross-lingual performance drops on Bengali and Hindi, reflecting stronger pretraining on those languages.
• Reasoning widens the language gap: The authors find that model reasoning can enlarge the performance gap between languages — motivating the alignment stage.
• Limited alignment gains from RL: The paper notes that RL-based alignment is more effective than DPO but still relies on a large parallel corpus, which explains the limited improvement observed.
• Explainability comes with an evaluation gap: Because the approach is generative, each safeguard judgment includes a detailed explanation of which rules the conversation violates and why, but the paper states that evaluating explanation quality is difficult due to the lack of ground truth.

Methodology in Plain English

The framework has three stages, all built on Qwen2.5-3B.

Stage 1 — Cold start. The authors manually designed a three-step solution plan for the safeguard task: understanding, rule matching, and judging. They then used DeepSeek V3 671B as a teacher model to generate step-by-step reasoning traces conditioned on inputs and their ground-truth labels, and distilled this into the 3B base model with supervised fine-tuning. The purpose is to give the small model initial task-specific reasoning ability before reinforcement learning, since GRPO's self-evolution characteristic limits small models.

Stage 2 — Reasoning training with GRPO. Free-form long reasoning is impractical for a safeguard because it hurts classification efficiency, so the authors added two rewards built from simple functions on top of the standard format and correctness rewards. The reward is:

r = sin(L / (2 · L_best) · π) + [sin((p − 2)/2 · π) + 1]

where L is the length of the model's reasoning, L_best is a predefined optimal reasoning length set to 512 in the experiments, and p quantifies the repetition rate of trigrams within the reasoning process. The first term keeps reasoning length stable; the second discourages the model from gaming the length reward by reducing diversity.

Stage 3 — Cross-lingual alignment with CAO. All English seed data was translated into five other languages using Google Translate. The GRPO-trained model was sampled multiple times per language, and outputs were split into a successful set and a failure set. For each failure, the authors found a successful case in another language and built a four-part training sample: the failure input as q, the failure output as the rejected sequence p_l, the successful output as the chosen sequence p_w, and the full successful sequence q_a ⊕ p_a as an anchor. The objective is L = L_CAO + L_c, where L_CAO is a DPO-style log-sigmoid term over the chosen and rejected sequences with hyperparameter β, and L_c is a KL-divergence term that regularizes the anchor sample so alignment does not distort the model's representation of it.

Data and evaluation. Training data was drawn by mixing four English-only open-source safety datasets — Aegis, BeaverTails, ToxicChat, and WildGuard — and randomly sampling 1,000 instances from a total of 127,600. Evaluation used three benchmarks (OpenAI Moderation, ToxicChat, SimpleSafetyTests) extended to French, Chinese, Japanese, Bengali, and Hindi via Google Translate, giving six languages (en, fr, zh-cn, jp, bn, hi). The primary metric was macro-F1. For reliable multilingual benchmarking, 10% of each benchmark was manually checked by professional translators. All experiments ran on two NVIDIA A100 40G GPUs.

Why This Matters

Impact on research. The work tests whether reinforcement-learning alignment — not just SFT or DPO — can transfer safety reasoning across languages, and it shows that a controllable reasoning chain plus cross-lingual alignment can close part of the high-resource/low-resource gap with a small model and 1,000 samples. It also contributes translated benchmark extensions, which is often the bottleneck for multilingual safety research.

Real-world applications:

• Content moderation platforms that need consistent harmful-content detection across many languages, not only English.
• Chatbot and customer-service deployments in markets where Bengali, Hindi, Japanese, or Chinese are primary languages.
• Safety layers in agentic systems that must both block harmful prompts and produce an auditable rationale for each decision.
• Compliance and trust-and-safety workflows where an explanation of which rule was violated is required for review or appeals.

Industry relevance. The paper's central selling point is cost: a 3B model trained on 1,000 samples reaches results comparable to or better than larger baselines trained on over 100,000 samples, which matters for organizations that cannot afford large-scale multilingual annotation. The interpretability requirement and the released code and benchmark extensions also fit directly into production trust-and-safety pipelines.

Future Directions

• Extending evaluation beyond the six languages tested, since generalization to other low-resource languages remains untested.
• Testing the framework on larger parameter models or different architectures, because validation was limited to a 3B-parameter model.
• Broadening the training data beyond the small, domain-specific seed set to check robustness in wider contexts.
• Moving beyond classification accuracy toward human preference or long-context evaluations, and developing a way to measure explanation quality, which the paper says is difficult without ground truth.
• Improving alignment efficiency, since the paper notes that RL-based alignment still depends on a large parallel corpus.

Target Audience

Researchers and engineers working on LLM safety, multilingual NLP, and content moderation who are already comfortable with fine-tuning and reinforcement-learning alignment methods. It is most useful for practitioners who need safety coverage in low-resource languages under tight data and compute budgets, and for those interested in reasoning-based safeguards that provide explanations alongside classifications.

Authors’ abstract

Recent advances in LLMs have enhanced AI capabilities, but also increased the risk posed by malicious requests, highlighting the need for effective LLM safeguards to detect such queries. Existing approaches largely rely on classifier-based methods that lack interpretability and perform poorly on low-resource languages. To address these limitations, we propose ConsistentGuard, a novel reasoning-based multilingual safeguard, which enhances explainability via reasoning and boosts knowledge transfer between languages through alignment. With only 1,000 training samples, our method demonstrates superior performance on three datasets across six languages, outperforming larger models trained with significantly more data, and exhibits strong interpretability and generalization ability. We also contribute a multilingual benchmark extension and release our codes to support future research.