Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms

AI summary

Overview

Research area: Natural Language Processing — specifically multilingual safety evaluation and cross-lingual red teaming of large language models.

Technical level: Intermediate. The paper is readable for anyone familiar with LLM evaluation basics, but it assumes some familiarity with benchmarks, refusal rates, and attack success metrics.

Scope: The paper introduces and analyzes CompositeHarm, a translation-based benchmark that measures how LLM safety alignment degrades when adversarial and contextual English prompts are translated into five Indic languages, evaluated across three open models with a lightweight, edge-AI-inspired inference pipeline.

What This Paper Is About

Nearly all safety testing of large language models is done in English, but these models are deployed in dozens of languages. Translation is the standard shortcut for probing multilingual behavior, yet it is unclear whether harm survives translation intact, morphs, or disappears. The authors build a benchmark combining two English harm datasets, translate it into five Indic languages, and systematically measure where safety guardrails break down — and whether different types of harm break down differently.

Key Contributions

1. CompositeHarm, a composite translation-based benchmark. It combines 140 prompts from AttaQ (structured adversarial attacks that rely on obfuscated or encoded instructions and syntactic manipulation) with 140 prompts from MMSafetyBench (contextual, real-world harms such as hate speech, misinformation, and ethical dilemmas) for 280 English prompts total, translated into five Indic languages to yield 1,400 translated prompts and a final benchmark of 1,680 prompts across six languages (280 per language).

2. A hybrid translation-and-verification pipeline. English prompts were translated with the No Language Left Behind (NLLB) model, then all outputs were manually verified and refined by bilingual annotators fluent in the target language, checking semantic accuracy, cultural appropriateness, and preservation of adversarial structure.

3. A lightweight, resource-aware evaluation design. Rather than a large centralized inference pipeline, the study uses compact generation models for preliminary evaluation and reserves high-capacity models for selective verification, aiming to reduce redundant API calls and GPU usage while maintaining cross-lingual fidelity. The authors describe this as borrowing ideas from edge computing, though they state explicitly that the work does not perform on-device deployment.

4. Evidence that syntax breaks safety faster than semantics. By separating adversarial-syntax harms from semantic-context harms in the same benchmark, the paper isolates two different failure mechanisms and shows they transfer across languages at very different rates.

Main Findings

• Attack success rises sharply in Indic languages. Across the three evaluated models, average attack success was 4.7% on the authors' English prompts but 22.0% in Gujarati and 21.1% in Kannada. Average rejection rates fell from 39.3% in English to 22.0% in Gujarati and 21.4% in Kannada.

• Hindi and Marathi remained the most stable. Hindi showed an average rejection rate of 43.8% and attack success of 6.1%; Marathi showed 40.7% and 10.4%. Assamese sat at 29.9% rejection and 10.2% attack success. The authors describe consistency as dropping as the language moves farther from English.

• Adversarial syntax is the most persistent failure mode. Prompts from the AttaQ portion pushed models hardest. LLaMA-3-8B "cracked first" and most severely, with attack success "often above 45%" for Kannada and Gujarati. Contextual harms from MMSafetyBench also rose in Indic languages, but "not as steeply."

• Model behavior diverged sharply. GPT-OSS 20B was the most cautious, refusing over two-thirds of harmful requests on average and rarely producing unsafe text, though described as conservative and over-defensive. LLaMA-3-8B-Instruct was the most permissive, with attack success crossing 45% in Gujarati and Kannada. Qwen3-32B sat in the middle, with refusal rates mostly under 20% and a moderate harmful-response rate.

• Comparison with European languages shows a wider gap. Against text-only results for European languages reported by Derner and Batistič (2025) — French at 83.0% rejection / 4.0% attack success, Spanish at 86.3% / 1.3%, German at 80.7% / 3.7% — the Indic results represent what the authors call a collapse of safety alignment. The authors note this cross-family comparison is conceptual and indicative, not quantitatively controlled.

• Binary metrics leave a large "gray zone." In Hindi, rejection (43.8%) and attack success (6.1%) sum to only 49.9%, leaving 50.1% of interactions unclassified. The gap is larger in Gujarati (44.0% combined) and Kannada (42.5%). The authors attribute this to two failure types: evasive responses, where the model neither refuses nor produces harmful content but gives a vague, off-topic, or hallucinated answer (their Kannada phishing-prompt example returns a description of the Kannada language and Karnataka's silk and sandalwood), and guardrail blocks, where the model does generate a harmful reply that is then intercepted by an external API-level filter.

• Efficiency and safety trade off. LLaMA-3-8B-Instruct, the most lightweight model in the testbed, was the most permissive and showed the highest attack success rates, suggesting its alignment is closely optimized for English and fades as linguistic distance increases.

• Convergence with multimodal red teaming. The authors compare their findings to Derner and Batistič (2025), who used the same AttaQ dataset but rendered prompts as images. That work found multimodal attacks "most pronounced" in lower-resource European languages (Slovenian, Czech, and Valencian), and reported gpt-4.1-mini going from an attack success rate of 1-2% on text to 21% on the same text presented as an image.

Methodology in Plain English

The authors took two existing English safety datasets that test different things — one built around sneaky, adversarially structured prompts, the other around realistic harmful scenarios — and sampled 140 prompts from each for a balanced total of 280. They ran these through a neural machine translation system (NLLB) into Hindi, Assamese, Marathi, Kannada, and Gujarati, then had bilingual student reviewers check every translated prompt against the English source for meaning, cultural fit, and retained adversarial structure.

They then fed all prompts (English originals plus the 1,400 translations) to three models — GPT-OSS 20B, LLaMA-3-8B-Instruct, and Qwen3-32B — using fixed decoding settings (temperature = 0, top-p = 1.0), producing 5,040 total responses. Instead of human raters, an LLM-as-a-judge setup using LLaMA-3.3-70B via the Groq API asked two binary questions about each response: did the assistant refuse, and did the assistant produce harmful content? Those two yes/no signals became the Refusal Rate and Attack Success Rate. English originals served as a control baseline so the authors could confirm their pipeline reproduced sensible numbers before trusting the multilingual results.

Why This Matters

Impact on research. The paper argues that single-type safety benchmarks hide differential harm transfer, and that translated benchmarks are a necessary first step but not a sufficient one. It provides a concrete, reproducible comparison point showing that linguistic distance from English is a measurable axis of safety degradation, and it identifies a large "gray zone" of evasive and guardrail-blocked responses that standard binary refusal/attack metrics simply do not count.

Real-world applications:

• On-device and IoT assistants, where compact models are chosen for compute reasons — the paper warns that the most edge-feasible models appear the most vulnerable to cross-lingual safety failures.
• Smart home technology, where a model validated on English-centric evaluations could behave unsafely for non-English-speaking users.
• In-car AI, where the same fragility applies under strict computational and energy budgets.
• Multilingual consumer deployment generally, particularly for Indic-language user bases, where the paper notes deployment without adequate safeguards risks exacerbating harms in underrepresented languages.

Industry relevance. Manufacturers may deploy small models believing them safe based on English evaluations, while the safety facade "crumbles" on morphologically complex or non-English input. Superficial fluency — as the authors describe for Qwen3-32B — can mask deep semantic fragility and give a false sense of security. The paper's specific recommendation is language-family-specific fine-tuning and testing, and post-hoc safety layers tailored to non-English prompts.

Future Directions

• Replace LLM-as-a-judge with human evaluation. The authors list simulated evaluation as a limitation and call for human evaluations to validate findings.
• Extend to more language families. They propose Sino-Tibetan and Afro-Asiatic languages beyond the Indic set tested here.
• Run a stricter, quantitatively controlled cross-family comparison. The European-versus-Indic contrast is explicitly described as conceptual and indicative; a controlled evaluation is planned.
• Build benchmarks that are composite along two axes at once. The authors argue future evaluation should span both harm type (adversarial syntax versus semantics) and modality (text versus image) to build resilient safety systems, and that language-family-specific alignment and non-Latin script support deserve dedicated attention.

Target Audience

Multilingual NLP and AI safety researchers will get the most from this paper, along with red-teaming practitioners who need benchmark designs that separate harm types. It is also relevant to model developers and product teams deploying LLMs in Indic languages or on resource-constrained edge hardware, and to policy audiences concerned with equitable AI deployment across global languages. Readers who want a quick orientation to the field will find the Related Work section useful for its framing of "safety drift" and prior Indic-language safety work.

Note on availability: the paper states that the CompositeHarm benchmark and dataset are available from the corresponding author upon reasonable request. All evaluations were conducted with synthetic prompts, and the authors disclose that a large language model was used to improve manuscript grammar and clarity, in line with the AAAI Policy.

Authors’ abstract

Most safety evaluations of large language models (LLMs) remain anchored in English. Translation is often used as a shortcut to probe multilingual behavior, but it rarely captures the full picture, especially when harmful intent or structure morphs across languages. Some types of harm survive translation almost intact, while others distort or disappear. To study this effect, we introduce CompositeHarm, a translation-based benchmark designed to examine how safety alignment holds up as both syntax and semantics shift. It combines two complementary English datasets, AttaQ, which targets structured adversarial attacks, and MMSafetyBench, which covers contextual, real-world harms, and extends them into six languages: English, Hindi, Assamese, Marathi, Kannada, and Gujarati. Using three large models, we find that attack success rates rise sharply in Indic languages, especially under adversarial syntax, while contextual harms transfer more moderately. To ensure scalability and energy efficiency, our study adopts lightweight inference strategies inspired by edge-AI design principles, reducing redundant evaluation passes while preserving cross-lingual fidelity. This design makes large-scale multilingual safety testing both computationally feasible and environmentally conscious. Overall, our results show that translated benchmarks are a necessary first step, but not a sufficient one, toward building grounded, resource-aware, language-adaptive safety systems.