BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts

AI summary

Overview

Research area: Natural Language Processing — LLM safety, multilingual/low-resource NLP, guardrail classifiers, code-mixing and transliteration robustness.

Technical level: Intermediate. The concepts are accessible to anyone familiar with LLM safety filtering, but the corpus construction pipeline, paired statistical testing, and evaluation metrics assume some NLP background.

Scope: The paper builds a paired 10,245-prompt Bangla safety corpus in three written forms, benchmarks six LLMs and three guard models on it, and releases BanglaShield, a 237M-parameter encoder that classifies prompt severity far more accurately and consistently across scripts than any evaluated LLM.

What This Paper Is About

Safety classifiers are typically trained and tested on standard-script text, but Bangla speakers routinely write in native script, in Latin-script romanization, and mixed with English, Hindi, or Urdu. The same harmful request can therefore appear in several written forms with identical meaning. The paper asks whether a harmful prompt caught in Bangla script is also caught when romanized or code-mixed, and whether a compact purpose-built classifier can make graded (four-level) safety judgments that stay stable across these forms.

Key Contributions

1. A paired, graded Bangla safety corpus. The authors consolidate 10,245 prompts from six sources (LinguaSafe 3,409; BanglaEcom 2,881; XSafety 2,599; BanglaSafe 541; IndicSafe 500; MultiJail 315) into a four-level taxonomy (Safe 3,490; Offensive 1,569; Harmful 2,693; Severe 2,493) using deterministic, source-aware mapping rules, and render each prompt in three aligned forms that share labels and a group-aware 70:10:20 split (train 7,172; validation 1,024; test 2,049).

2. A paired benchmark of LLM safety across written forms. Six LLMs (Gemma 4 31B, Llama 4 Maverick 400B, Command A+, GPT-4.1 mini, Claude Haiku 4.5, and the Bangla-specialized TigerLLM 9B) are evaluated on 36,882 prompt classifications and 24,318 judged responses, and three guard models (Llama Guard 3 8B, PolyGuard-Qwen 7B, IndicGuard 4B) on prompt classification, with paired bootstrap confidence intervals and McNemar tests.

3. BanglaShield, a compact script-robust classifier. A fine-tuned MuRIL-base encoder (237M parameters, trained on 21,516 prompts across the three forms) that leads on every written form and is far more stable across forms than LLMs.

4. Public release of corpus, splits, judge outputs, and model to support script-robust safety research.

Main Findings

• LLMs under-detect risk even in native script. LLM classifiers reach only 33.0–43.9 macro-F1 on Bangla-script prompts; 47.0% of unsafe native-script prompts are labelled Safe (the unsafe-as-safe rate, USR), and 70.9% of unsafe prompts receive a lower severity label than the gold one. Only 12.3% of Severe prompts are classified correctly, while 47.5% become Harmful and 34.1% become Safe. Errors are overwhelmingly one-directional: only 0.8% of Safe prompts are misclassified as unsafe.

• Romanization makes things substantially worse. Mean macro-F1 drops 7.8 points versus Bangla script and USR rises to 63.4%. The drop is statistically significant for five of six models. Command A+ is the most affected, losing 19.9 macro-F1 points and labelling 87.8% of romanized unsafe prompts Safe. Severe-class F1 falls from 20.4 to 11.1 (a 46% loss). Code-mixing hurts less (−2.4 points on average), and Gemma 4 is the only model with no significant change under either form.

• Response safety degrades in parallel. The mean appropriate-response rate (ARR) falls from 78.3% on Bangla script to 77.0% on code-mixed and 68.0% on romanized prompts — a 10.3-point drop. GPT-4.1 mini on Severe prompts falls from 68.3% to 51.9% under romanization. Between 5% and 22% of responses to Harmful and Severe prompts provide material or direct assistance. Classification macro-F1 and ARR are strongly correlated across the 18 model–form cells (r = 0.84).

• Self-classification is a weak filter. Using the model's own four-way prediction as an input filter reduces harmful compliance by only about five points (roughly 15% relative), leaving models compliant with about 30% of Harmful and Severe prompts (mean HCR 35.8 → 30.2 on script, 34.0 → 29.1 on romanized). The same under-estimation that hurts classification lets unsafe prompts through, while the filter also blocks 5–29% of Offensive prompts that deserve an answer.

• Dedicated guard models do not close the gap. Three multilingual guards pass 30.3–57.3% of unsafe native-script prompts; Llama Guard 3 passes more than half in every form. PolyGuard's USR roughly doubles under romanization (32.7 → 60.4) and IndicGuard's rises from 30.3 to 62.9. PolyGuard and IndicGuard lower mean HCR to 16.5–22.6% but block up to 63% of Offensive prompts.

• BanglaShield leads by a wide margin. It reaches 87.8 macro-F1 on native script, 88.3 on code-mixed, and 84.4 on romanized prompts, with USR at or below 1.0% in every form — outperforming the best LLM on each form by 42.8–46.1 points with roughly 130× fewer parameters than Gemma 4 31B. Its macro-F1 changes by +0.4 under code-mixing and −3.5 under romanization, versus mean LLM changes of −2.4 and −7.8.

• LLM predictions are unstable item by item. When only the written form changes, LLMs flip their label on 9–31% of items, and correct-to-wrong flips outnumber wrong-to-correct flips for every LLM and form. Only 34.5–45.9% of items are classified correctly in all three forms, versus 81.7% for BanglaShield. BanglaShield is the only system whose code-mixed flips favour the correct label (2.3% wrong-to-correct vs 1.8% correct-to-wrong).

• No single written form is sufficient for training. A Bangla-script-trained variant scores 87.2/76.3/39.4 macro-F1 across script/code-mixed/romanized, a code-mixed variant 81.2/85.9/42.9, and a romanized variant 44.7/34.6/80.2. Only pooled training (87.8/88.3/84.4) is strong on every form.

• Culturally grounded harms are hardest. On BanglaSafe (written natively from Bangladeshi court records and news reports), LLMs classify only 9.0% of native-script items correctly on average and respond appropriately to only 53.9%; BanglaShield reaches 84.3%. Benign BanglaEcom prompts are handled almost perfectly by everyone (99.8% LLM vs 100.0% BanglaShield), so overall degradation comes almost entirely from unsafe sources.

• The effect comes from form, not from a particular embedded language. Code-mixing lowers mean LLM macro-F1 by similar margins for English (38.9 → 36.3), Hindi (38.0 → 35.8), and Urdu (37.0 → 35.2), and the accuracy gap to Bangla script stays between 1.5 and 2.0 points across substitution-rate bins.

Methodology in Plain English

The authors start by pooling Bangla data from six existing resources and forcing them onto one shared four-level risk scale (Safe, Offensive, Harmful, Severe), using fixed rules that translate each source's own category definitions rather than re-annotating items by hand. Because most safety datasets contain few benign examples, they convert 2,881 negative product reviews from BanglaEcomReviewCorpus into first-person customer-support requests using deterministic category templates — these carry negative sentiment but ask for no harmful help, so they serve as Safe controls.

They then create three versions of every prompt. The Bangla-script version is the original text. The romanized version is produced by an LLM transliteration model (bodhan-ai/indic-translate) instructed to change only the script, keeping words and word order. The code-mixed version is built by translating the whole prompt with NLLB-200, aligning words to the original with SimAlign, and then swapping a randomly sampled 35–50% of eligible words for their English (60%), Hindi (25%), or Urdu (15%) counterparts. Crucially, all three versions of a prompt inherit the same label and the same data split, so any behavioural difference is attributable to written form alone; a shared group_id keeps near-duplicates and variants from leaking across splits.

For the benchmark, six LLMs are asked to assign one of the four labels to each test prompt under JSON-constrained output, and separately to answer the 1,351 unsafe test prompts in each form. Those responses are scored by two independent LLM judges on compliance and label-appropriate behaviour, with a third model adjudicating disagreements (judges agree on 81.9% of responses, Cohen's κ = 0.56). Three off-the-shelf guard models are run on the prompt alone. Metrics include accuracy, macro-F1, USR (share of unsafe prompts waved through as Safe), ARR (share of responses that are appropriate for the gold label), and HCR (share of harmful prompts actually complied with), with 1,000 paired bootstrap resamples and McNemar's test for significance.

Finally, they fine-tune a MuRIL-base encoder on the union of all three training forms and select the checkpoint with the best pooled validation macro-F1.

Why This Matters

The paper reframes multilingual safety failure as partly a script problem rather than purely a language problem: a filter that looks adequate on native-script Bangla silently fails on the romanized text that many speakers actually type. It also shows that the cheapest common mitigation — asking the model to police its own input — recovers only a small fraction of the gap, and that existing multilingual guards are not a substitute for in-language, in-script training data.

Real-world applications:

• Social media and messaging moderation for Bangla-language content, where users routinely romanize to avoid script-based filters or simply out of habit.
• Chatbot and assistant deployment in South Asia, where a prompt-level guard needs to hold up across the several ways the same request can be typed.
• Content-safety review pipelines at platforms serving Bangladeshi and Bengali diaspora users, including code-mixed English/Hindi/Urdu inputs.
• Low-resource language safety auditing, as a template for building paired, multi-script benchmarks in other languages with multiple written conventions.

Industry relevance is direct: BanglaShield delivers a 237M-parameter guard that beats 31B–400B general models on this task, which is a practical deployment profile for latency- and cost-sensitive moderation systems. The finding that a model's own safety judgment is a weak filter argues against self-guarding as a cost-saving shortcut in production.

Future Directions

• Replace synthetic transformations with naturally occurring text. The romanized and code-mixed forms are generated automatically and not manually validated; collecting real romanized and naturally code-switched Bangla prompts would test whether the observed degradation holds for authentic writing.
• Move from deterministic label mapping to human annotation. Labels currently come from each source's category definitions rather than per-item annotation, and the benign controls are not manually validated; item-level annotation would sharpen the graded taxonomy and its boundary cases.
• Close the residual romanization gap. BanglaShield still loses 3.5 macro-F1 going from native script to romanized text and drops 6.2% of items from correct to wrong, so script-invariant representations remain an open modelling problem.
• Design guards that filter harmful prompts without over-blocking Offensive ones. Dedicated guards cut harmful compliance but block up to 63% of Offensive prompts that warrant an answer, so the precision/recall trade-off for graded, non-binary guards is unresolved.
• Extend the paired multi-script design to other languages with competing written conventions, and test whether pooled multi-form training is necessary there too.

Target Audience

NLP safety researchers and practitioners building guardrails or moderation systems for low-resource and multilingual settings; engineers deploying LLM applications for Bangla-speaking or South Asian markets; dataset curators interested in paired, controlled benchmark construction; and policy or trust-and-safety teams who need evidence on how script and code-mixing choices affect whether harmful requests are caught.

Authors’ abstract

Safety classifiers for large language models (LLMs) are typically built and evaluated on standard-script text, yet Bangla, like many lower-resource languages, is routinely written in native script, in Latin-script romanization, and code-mixed with English, Hindi, or Urdu. We introduce BanglaShield, a resource and model for graded prompt-safety classification across these written forms. We consolidate 10,245 Bangla prompts from six sources under a four-level taxonomy (Safe, Offensive, Harmful, Severe) and render each prompt in three aligned forms that share labels and a group-aware split, enabling strictly paired comparisons. Benchmarking six contemporary LLMs, including a Bangla-specialized one, on 36,882 classification queries and 24,318 judged responses, we find that LLM classifiers label 47.0% of unsafe native-script prompts as Safe, rising to 63.4% under romanization, while response-level safety accuracy drops by 10.3 points. Three multilingual guard models also pass 30.3-57.3% of unsafe native-script prompts. Using a model's own classification as an input filter reduces harmful compliance by only about five points. In contrast, BanglaShield, a compact fine-tuned encoder, reaches 87.8 macro-F1 on native-script and 84.4 on romanized prompts, outperforming the best LLM on each form by 42.8-46.1 points with ~130x fewer parameters than Gemma 4 31B. We release the corpus, splits, judgments, and model.