X-MuTeST: A Multilingual Benchmark for Explainable Hate Speech Detection and A Novel LLM-consulted Explanation Framework

AI summary

Overview

• Research area: Natural Language Processing, specifically explainable hate speech detection in multilingual (Hindi, Telugu, English) settings.
• Technical level: Intermediate. The paper assumes familiarity with transformer encoders, attention, and explainability metrics, but its two-stage training scheme and n-gram scoring procedure are described step by step.
• Scope in one sentence: The paper releases human word-level rationale annotations for Hindi, Telugu, and English hate speech datasets, and proposes X-MuTeST, a two-stage training framework that fuses human rationales, an n-gram-based saliency method called X-MuTeST, and LLaMA-3.1 rationales into a final explanation. Accepted at AAAI 2026 (preprint, arXiv:2601.03194v1 [cs.CL], 06 Jan 2026), by authors from IIT Indore, IIIT Dharwad, Arizona State University, and IIT Mandi.

What This Paper Is About

Hate speech detection systems increasingly need to explain why they flag a text, not just produce a label, but model-generated rationales often disagree with how humans actually perceive offensiveness, especially in under-resourced Indic languages where cultural and societal cues are missed by LLMs. Existing rationale resources are almost entirely English, with HateXplain (Mathew et al. 2021) being one of the few exceptions, and no comparable resource existed for Hindi or Telugu. The paper's goal is to provide word-level human rationales for three languages and to build a training and explanation framework that improves both classification accuracy and the alignment of explanations with human judgment.

Key Contributions

1. Benchmark human-annotated rationales for hate speech detection in Hindi and Telugu alongside English: token-level rationale annotations for 6,004 Hindi, 4,492 Telugu, and 6,334 English samples.
2. A hybrid explainability method that combines LLM-based explanations with traditional transformer attention improvement, using a novel n-gram-based scoring formulation to identify salient tokens.
3. The X-MuTeST framework, a two-stage training procedure (Stage-1 guided by human rationales, Stage-2 guided by model-derived n-gram attention masks) designed to balance classification and explainability.
4. Empirical validation across three languages plus generalizability experiments on HateXplain and HateBRXplain, evaluated with both performance metrics (accuracy, F1, macro-F1) and explainability metrics (Token-F1, IOU-F1, Comprehensiveness, Sufficiency).

Main Findings

• Telugu: X-MuTeST with LLM achieves the best results across all metrics — accuracy 0.8881, F1 0.8762, macro-F1 0.8849 — surpassing the next-best baseline (Muril-Rationale-XMuTeST) by 1.41% in accuracy and 1.29% in macro-F1. It reaches Token-F1 0.6231 and IOU-F1 0.3189 (improvements of 4.77% and 2.76% over Muril-Rationale-XMuTeST) and a sufficiency score of 0.0448 versus 0.0747 for that baseline.
• Hindi: X-MuTeST with LLM reaches accuracy 0.8745 and Token-F1 0.4344, gaining 3.88% in accuracy and 2.27% in Token-F1 over the next-best model. Sufficiency improves to 0.4768 versus 0.5042.
• English: X-MuTeST with LLM scores accuracy 0.8604, F1 0.8827, macro-F1 0.8513, Token-F1 0.5125, comprehensiveness 0.8433, and the lowest sufficiency score of 0.0507. Mistral-24.07 records the highest IOU-F1 among LLMs at 0.3176, while GPT-4o reaches Token-F1 0.4613 with lower classification accuracy of 0.7802.
• LLMs lag on classification: On Telugu, GPT-4o scores 0.6484 accuracy, LLaMA-3.1 scores 0.6362, and Mistral-24.07 scores 0.6402 — the paper attributes this largely to inaccurate transliteration from Latin to Telugu script. On Hindi, LLaMA-3.1 shows high Token-F1 (0.4263) but its classification drops sharply by 35.73%.
• N-gram explanations beat LIME: In every encoder/configuration pairing in the three main tables, switching from LIME to the X-MuTeST explainability method improves plausibility and faithfulness scores without changing classification performance (e.g. Muril-XMuTeST on Telugu: Token-F1 0.5583 and IOU-F1 0.2919 versus Muril-LIME's 0.5515 and 0.2840).
• Human rationales help training: Models trained with rationale integration (e.g. Muril-Rationale-LIME, Muril-Rationale-XMuTeST) consistently outperform their non-rationale counterparts on both classification and explainability.
• Stage-2 sensitivity: Classification peaks at α = 0.6 for Hindi and English and at α = 0.7 for Telugu; performance drops at the boundary conditions where all weight goes to either attention loss or classification loss.
• Ablation: Removing either of the two training stages degrades performance, supporting the value of the dual-stage design.
• Generalizability: On HateXplain, X-MuTeST attains IOU-F1 of 0.314 and Token-F1 of 0.587, surpassing the best reported results in that work by 9.2% and 8.1% respectively, with a 5.6% improvement in comprehensiveness. On HateBRXplain, X-MuTeST with LLM gains 6.9% in Token-F1 and 2.3% in sufficiency over the strongest baseline.

Methodology in Plain English

The researchers gathered hate speech data in three languages: Hindi and English from the HASOC 2020 and 2021 contests (Twitter samples, binary HATE/NOT HATE labels, with 15% held out for testing), and Telugu from the HOLD-Telugu task at DravidianLangTech 2024 (YouTube comments, transliterated from Latin to Telugu script using IndicXlit). Five annotators were hired with three annotators assigned per language; each post was annotated by three people, who marked hate-indicating tokens with a 1 and all other tokens with 0.

Because initial agreement was only moderate to substantial (61–80%), the team ran an iterative annotation loop with extra guidance sessions and re-annotation, which raised agreement to overall Kappa scores of 81.00 (Telugu), 83.15 (Hindi), and 85.10 (English). Annotators received 150 hours of free GPU access valid for one year as compensation.

The framework trains in two stages. In Stage-1 (three epochs), the model's attention is pushed toward the human-marked tokens using an attention alignment loss combined with the normal classification loss, weighted so that α = 0.3 on the attention term. In Stage-2, the human labels are replaced with attention targets generated by the paper's own n-gram explainability method; the top k tokens are selected (k = 5 if the sequence has at least 10 tokens, otherwise half the sequence length, rounded up), and α is raised to 0.6 for Telugu and English and 0.7 for Hindi (the sensitivity analysis section instead reports best results at α = 0.6 for Hindi and English and 0.7 for Telugu).

The n-gram explainability method works by measuring how much the model's prediction changes when overlapping unigrams, bigrams, and trigrams are removed. Each token's importance is the weighted sum of these logit drops (weights 0.5, 0.3, and 0.2 for unigram, bigram, and trigram), normalized across tokens. The final explanation is the union of this token set and the rationale set produced by LLaMA-3.1, with an agreement score computed as the intersection over the union of the two sets. LLaMA-3.1 was chosen because it showed higher token-level alignment with human rationales than other LLMs for Telugu and Hindi. MURIL is used as the encoder for Telugu and English, and XLMR for Hindi, each selected as the best-performing encoder for that language after fine-tuning.

Why This Matters

Hate speech moderation is a high-stakes task where a prediction without a justification is hard to trust, audit, or appeal. This paper matters because it (a) releases rationale resources for Hindi and Telugu where none previously existed, and (b) shows that explanation quality can be improved deliberately during training rather than bolted on afterward, while also improving classification accuracy.

Real-world applications:

• Content moderation platforms that need to show moderators or users which specific words triggered a hate speech flag, in Hindi, Telugu, or English.
• Appeal and transparency workflows, where a platform must explain a takedown decision in a language the affected user actually speaks.
• Regulatory and compliance reporting, where automated decisions increasingly require documented, human-checkable reasoning.
• Cross-lingual safety tooling for low-resource languages that mainstream LLM pipelines handle poorly, as the paper's LLM baselines demonstrate on Telugu.

Industry relevance: The finding that general-purpose LLMs underperform on Telugu classification and rationale alignment suggests that off-the-shelf LLM moderation is not sufficient for Indic-language platforms, and that purpose-trained encoders with explicit explanation supervision can be both more accurate and more explainable. The released annotations also give teams a direct evaluation target for their own explainability pipelines.

Future Directions

• Extending to more languages and scripts. The conclusion explicitly calls for expanding the framework to handle more languages, and the Telugu transliteration issue suggests script handling deserves dedicated attention.
• Improving or replacing the LLM consultation component. LLaMA-3.1 was chosen empirically over GPT-4o and Mistral-24.07, so the paper leaves open how newer or differently prompted LLMs would affect the union-based explanation.
• Resolving the α settings and studying their role more systematically. The paper reports different optimal α assignments in the methodology versus the sensitivity analysis, and the boundary-condition behaviour it observes invites a broader study of the attention/classification trade-off.
• Broadening generalizability testing. The two additional benchmarks

Authors’ abstract

Hate speech detection on social media faces challenges in both accuracy and explainability, especially for underexplored Indic languages. We propose a novel explainability-guided training framework, X-MuTeST (eXplainable Multilingual haTe Speech deTection), for hate speech detection that combines high-level semantic reasoning from large language models (LLMs) with traditional attention-enhancing techniques. We extend this research to Hindi and Telugu alongside English by providing benchmark human-annotated rationales for each word to justify the assigned class label. The X-MuTeST explainability method computes the difference between the prediction probabilities of the original text and those of unigrams, bigrams, and trigrams. Final explanations are computed as the union between LLM explanations and X-MuTeST explanations. We show that leveraging human rationales during training enhances both classification performance and explainability. Moreover, combining human rationales with our explainability method to refine the model attention yields further improvements. We evaluate explainability using Plausibility metrics such as Token-F1 and IOU-F1 and Faithfulness metrics such as Comprehensiveness and Sufficiency. By focusing on under-resourced languages, our work advances hate speech detection across diverse linguistic contexts. Our dataset includes token-level rationale annotations for 6,004 Hindi, 4,492 Telugu, and 6,334 English samples. Data and code are available on https://github.com/ziarehman30/X-MuTeST