destroR: A Benchmark and Adversarial-Training Defense for Bangla Transfer Models under Meaning-Preserving Attacks

AI summary

Overview

Research area: Adversarial robustness of transformer-based transfer learning models for Bangla natural language processing, specifically sentiment classification.

Technical level: Intermediate — the paper assumes familiarity with transfer learning, adversarial attacks on text, and standard robustness metrics such as attack success rate, but it is framed around a clear pipeline rather than novel architecture design.

Scope: The paper introduces destroR, a unified pipeline that supplies meaning-preserving Bangla attack recipes, benchmarks five multilingual and Bangla-specific transfer models against five attacks across four datasets, and hardens every model with adversarial training.

What This Paper Is About

Transformer-based transfer models are now the default approach for Bangla sentiment classification, but how brittle they are to deliberately crafted inputs has barely been studied. Existing work has not combined a Bangla-specific attack suite with a defense that demonstrably restores robustness. This paper fills that gap by building both sides of the equation: attacks that rewrite Bangla sentences into fluent, semantically faithful variants, and adversarial training that reduces how often those attacks still succeed.

Key Contributions

1. Three meaning-preserving Bangla attack recipes. A paraphrase attack, a back-translation attack, and a one-hot word-swap attack. Unlike noise-injection attacks, these regenerate fluent, semantically faithful sentences, so the model is pushed toward prediction perplexity rather than being confused by corrupted input.

2. A robustness benchmark for Bangla transfer models. Five models (BanglaBERT, BanglishBERT, XLM-RoBERTa, MuRIL, and IndicBERTv2) are evaluated across four datasets against five attacks — the three recipes plus two strong word-substitution baselines, TextFooler and BAE — all under an identical protocol.

3. Adversarial training as a defense, applied to every model. Each model is hardened through adversarial training, and the paper reports a full robustness matrix rather than a single defended system.

4. Released artifacts for reproducibility. All models, adversarial data, and code are made publicly available.

Main Findings

• Word-substitution baselines outperform the semantically constrained recipes. TextFooler and BAE are described as more potent than the paper's paraphrase, back-translation, and word-swap attacks, with BAE reaching a 54.2% attack success rate.

• Adversarial training on the union of all attack families works broadly. Training on the combined attack set lowers residual attack success for every attack tested, rather than only for the attack used during training.

• MuRIL is stronger than the Bangla-dedicated models. Contrary to expectation, the Indic-multilingual MuRIL backbone is markedly more robust than the specifically Bangla-focused models.

• Prior work left a documented gap. The abstract states that no earlier study paired a Bangla attack suite with a defense that measurably recovers robustness; per-model and per-dataset breakdowns beyond the aggregate claims above are not detailed in the abstract.

Methodology in Plain English

The authors treat robustness as a two-stage engineering problem. First, they design attacks that keep Bangla sentences grammatical and semantically equivalent to the original while changing the wording — for example, rephrasing a sentence, translating it out to another language and back, or swapping individual words. The point is to test whether a classifier is relying on superficial word cues rather than meaning.

Second, they set up a controlled comparison: the same five pretrained transformer models, the same four sentiment datasets, and the same five attacks run identically on every combination, so differences reflect the models rather than the test conditions. The three new recipes are placed alongside two well-established word-substitution attackers as reference points.

Third, they retrain every model on adversarial examples drawn from all attack families together — a union-based adversarial training regime — and then re-run the full attack matrix to see how much of the original vulnerability survives. Reporting the complete matrix, rather than a single favorable number, is the core of the evaluation.

Why This Matters

Impact on research. Robustness evaluation for high-resource languages such as English is comparatively mature; Bangla and similar low-resource languages lag behind. This work provides a reusable attack suite, a common evaluation protocol, and a defense baseline, giving future Bangla robustness research something to compare against instead of ad hoc attack implementations.

Real-world applications:

• Content moderation and abusive-comment filtering on Bangla social media platforms, where users can rephrase harmful text specifically to evade automated filters.
• Sentiment and opinion analytics for Bangla-language customer reviews, where paraphrased or lightly reworded feedback should not flip a model's prediction.
• Misinformation and claim detection pipelines in Bangla news and messaging apps, where meaning-preserving rewrites are a natural evasion strategy.
• Any deployed Bangla text classification API, where robustness gaps translate directly into inconsistent behavior on legitimate-looking inputs.

Industry relevance. Organizations deploying Bangla-language NLP in South Asia face a choice between a Bangla-dedicated model that looks like the natural fit and a multilingual backbone. The finding that the multilingual MuRIL model is more robust than Bangla-specific models is a concrete, testable argument for reconsidering that default choice when reliability matters more than narrow benchmark accuracy. The adversarial training recipe also offers a straightforward hardening step that can be applied to an existing model rather than requiring a redesign.

Future Directions

• Extend the pipeline beyond sentiment classification to other Bangla tasks such as named entity recognition, hate speech detection, and question answering, where meaning-preserving attacks may behave differently.
• Investigate why multilingual backbones resist attacks better than language-dedicated ones, and whether that robustness can be transferred into Bangla-specific models through pretraining or fine-tuning choices.
• Test adaptive attacks against the adversarially trained models, since defenses tuned on a known attack union can invite attacks designed to circumvent exactly that training distribution.
• Evaluate meaning preservation with human annotators, since the recipes are described as semantically faithful by construction, and human judgments would quantify how faithful the perturbed sentences actually remain to native readers.
• Weigh the cost of adversarial training, including any accuracy trade-off on clean inputs — a dimension the abstract does not report.

Target Audience

Researchers and graduate students working on adversarial NLP, robustness evaluation, or low-resource language processing, particularly those focused on South Asian languages. NLP engineers and ML practitioners preparing to deploy Bangla text classifiers in production will also benefit from the benchmark results and the model-comparison finding, as will dataset and benchmark builders looking for a template for constructing attack-and-defense evaluations in under-resourced languages.

Authors’ abstract

Transformer-based transfer models now dominate Bangla sentiment classification, yet their adversarial robustness remains largely unexamined, and no prior study pairs a Bangla attack suite with a defense that measurably recovers robustness. We address this gap with destroR, a unified pipeline for evaluating and hardening Bangla text classifiers. First, we introduce three meaning-preserving Bangla attack recipes a paraphrase attack, a back-translation attack, and a one-hot word-swap attack that perturb inputs while regenerating fluent, semantically faithful sentences, inducing model prediction perplexity rather than input noise. Second, we construct a robustness benchmark that evaluates five transfer models (BanglaBERT, BanglishBERT, XLM-RoBERTa, MuRIL, and IndicBERTv2) across four datasets against five attacks, placing our recipes against two strong word-substitution baselines, TextFooler and BAE, under an identical protocol. Third, we harden every model through adversarial training and report a full robustness matrix. Our analysis yields three findings: word-substitution baselines are more potent than semantically constrained recipes (BAE reaches a 54.2% attack success rate); adversarial training on the union of all attack families lowers residual attack success for every attack; and, contrary to expectation, the Indic-multilingual MuRIL backbone is markedly more robust than the Bangla-dedicated models. All models, adversarial data, and code are released for full reproducibility.