BanglaRhet: Benchmarking Classical and Transformer Models for Rhetorical and Persuasion Detection in Bangla Political Speech

AI summary

Overview

Research area: Natural language processing (NLP) for a low-resource language — specifically rhetorical and persuasion technique detection in Bangla political speech, using transformer models benchmarked against classical baselines.

Technical level: Intermediate. The paper assumes familiarity with transformer fine-tuning, TF-IDF baselines, macro-F1, and class imbalance, but its framing and analysis are accessible to readers with general machine learning background.

Scope: A single benchmark study that introduces a manually annotated Bangla political speech corpus and evaluates four pretrained transformer models against two tuned classical baselines on two fine-grained single-label classification tasks.

What This Paper Is About

Political speech works through rhetorical style (how something is said) and persuasive intent (what the statement is trying to accomplish), but Bangla NLP research has mostly stopped at sentiment and stance, which cannot tell a metaphor-driven appeal apart from a direct call to action. The authors build BanglaRhet, a manually annotated corpus of 30,289 Bangla political speech segments, and use it to benchmark transformer models on two classification tasks: identifying a segment's dominant rhetorical technique and its dominant persuasion technique. The goal is to establish the first reproducible baseline results for technique-level rhetorical and persuasion analysis in Bangla.

Key Contributions

1. A benchmark framework for fine-grained rhetorical and persuasion technique detection in Bangla political discourse, formulated as two independent single-label classification tasks over a nine-label rhetorical taxonomy and a seven-label persuasion taxonomy.
2. A manually annotated corpus of 30,289 Bangla political speech segments, drawn from 15,032 sources, which the authors describe as the first resource of its kind for Bangla.
3. An empirical comparison of Bangla-specific and multilingual transformers against tuned classical baselines, showing that every transformer model outperforms TF-IDF with Logistic Regression and TF-IDF with linear SVM, and that Bangla-specific pretraining (BanglaBERT) performs best.
4. Class-level and robustness analysis, including confusion matrices, per-class F1 scores, an ablation on the controlled downsampling procedure, and a source-grouped re-evaluation to test whether segment-level splitting inflates the results.

Main Findings

• BanglaBERT sets the benchmark: It achieves 65.40% macro-F1 for rhetorical technique detection and 66.46% macro-F1 for persuasion technique detection, the highest of all evaluated models. Its accuracies are 0.6460 and 0.6619 respectively.
• Transformers clearly beat classical baselines: BanglaBERT exceeds the best tuned classical baseline by 19.2 macro-F1 points on rhetorical detection and 13.8 points on persuasion detection. The tuned baselines were TF-IDF + Logistic Regression (0.4587 and 0.5270 macro-F1) and TF-IDF + SVM (0.4624 and 0.5114 macro-F1).
• Model ranking on both tasks: BanglaBERT > XLM-RoBERTa-Base > SahajBERT > Bangla-BERT-Base > classical baselines. XLM-RoBERTa-Base reaches 0.6110 rhetorical and 0.6405 persuasion macro-F1; SahajBERT reaches 0.5974 and 0.6227; Bangla-BERT-Base reaches 0.5285 and 0.5720.
• Controlled downsampling helps every model: Removing the ten most overrepresented rhetorical–persuasion co-occurrence pairs improved macro-F1 across all four transformers on both tasks. The largest single gain was BanglaBERT on rhetorical detection (0.6214 to 0.6540, a gain of 3.26 points).
• Results are not badly inflated by source leakage: Under the original segment-level split, 61.23% of test segments and 59.35% of validation segments shared a source article with training data. Under a leakage-free source-grouped split, BanglaBERT's macro-F1 fell only 1.02 points (rhetorical, to 0.6438) and 1.08 points (persuasion, to 0.6538).
• Errors cluster around semantic overlap: In the rhetorical task, 46 Emotional Language instances were predicted as Repetition, with additional confusion between Repetition and Contrast, and between Exaggeration and Emotional Language. In the persuasion task, Moral Appeal and Emotional Appeal were frequently predicted as Blame Assignment.
• Per-class F1 varies by conceptual clarity: Strongest rhetorical classes were None (0.794), Rhetorical Question (0.752), and Anecdote (0.666); weakest were Emotional Language (0.551), Contrast (0.590), and Exaggeration (0.597). Strongest persuasion classes were None (0.840) and Unity Call (0.813); weakest were Blame Assignment (0.564), Moral Appeal (0.576), and Logical Appeal (0.586).
• Comparable to the closest English benchmark: Against SemEval-2020 Task 11, where the best of 32 systems reached 62.1% micro-F1 on 14 propaganda techniques for English news, the authors' 65.40% and 66.46% macro-F1 are broadly in line or ahead.
• A source of imbalance quantified: Blame Assignment, Call to Action, and Unity Call together account for 67.2% of all segments; Contrast co-occurs with Blame Assignment 2,923 times versus at most 824 times with any other persuasion label.

Methodology in Plain English

The authors assembled the corpus from publicly available Bangla political news and speech articles. They started with 1,890 links from the existing Motamot dataset and added links from Prothom Alo and Manab Zamin covering 2014–2025, successfully processing 15,032 articles into usable segments. A segment qualified only if it contained a direct quotation or reported statement attributed to a political figure or institution; where a speaker could not be identified the segment was kept and marked unknown_speaker. Journalism, background context, and commentary from non-political actors were excluded except where kept for the None class. Passages longer than the 128-token input limit were condensed, with both authors checking that rhetorical and persuasive content survived.

Two authors annotated all 30,289 segments. They first jointly annotated a pilot sample to align label boundaries on conceptually similar pairs (such as Emotional Language versus Emotional Appeal) and continued until disagreement stabilized. Each segment received one dominant label per task — the one most central to the segment's overall effect — with disagreements settled by discussion. Agreement was measured on a random 500-segment subset before consensus resolution: Cohen's kappa of 0.67 for rhetorical technique and 0.71 for persuasion technique.

Before experiments, the authors made two data decisions. They excluded the rare Parallelism class (137 instances, more than 15 times rarer than the next smallest class, Metaphor at 2,128, leaving only about 14 test instances) rather than oversampling or merging it. They also manually inspected the rhetorical–persuasion co-occurrence table and removed the ten most overrepresented pairs (for example, 2,000 Contrast–Blame Assignment segments and 1,571 Emotional Language–Blame Assignment segments) because the imbalance was pairwise, not just per-class, and uniform downsampling could not correct it. This left a processed benchmark of 22,565 samples.

They split the processed data 80%–10%–10% into train, validation, and test, stratified on rhetorical labels, with persuasion labels encoded separately over the same splits. All segments were tokenized with each model's own tokenizer and padded or truncated to 128 tokens. Four transformers were fine-tuned with a standard Hugging Face sequence classification head: Bangla-BERT-Base, SahajBERT, BanglaBERT, and XLM-RoBERTa-Base. For comparison, they trained TF-IDF with Logistic Regression and TF-IDF with a linear SVM, tuning regularization strength, vocabulary size, and n-gram range by grid search on validation. All models trained for ten epochs with AdamW, a learning rate of 2×10⁻⁵, batch size 16, class-weighted cross-entropy loss, and a single fixed random seed (42) used across splitting, downsampling, and training. Checkpoints were selected on validation macro-F1, and models were evaluated on accuracy and macro-F1, with macro-F1 emphasized given the imbalance.

Why This Matters

Impact on research. Bangla political NLP has almost entirely operated at the sentiment or stance level. This paper moves the field to technique-level analysis, provides a reproducible baseline, and shows quantitatively that bag-of-words representations are insufficient for the task. It also distinguishes rhetorical form from communicative function in a way that the closest prior resource, English propaganda detection, does not.

Real-world applications:

• Media monitoring: flagging which rhetorical and persuasion techniques dominate coverage of political speech, rather than only whether coverage is positive or negative.
• Political communication research: letting social scientists quantitatively compare how different speakers or periods deploy blame, unity calls, moral appeals, or emotional appeals.
• Content moderation and platform transparency: identifying persuasion-heavy or emotionally charged political content at scale in Bangla.
• Civic media literacy: surfacing technique labels to help readers recognize, for example, when an emotional or moral framing is being given the surface form of a direct accusation.

Industry relevance. Any organization processing Bangla-language political content at scale — news aggregators, social platforms, media analytics vendors, and political consultancies operating in Bangladeshi and other Bangla-speaking markets — gets a validated starting point. The paper also reports practical engineering findings: Bangla-specific pretraining (BanglaBERT) beats multilingual pretraining (XLM-RoBERTa-Base) on this task, and controlled co-occurrence downsampling yields measurable gains for every model tested.

Future Directions

• Multi-label modeling. Both tasks currently assign a single dominant label, but political statements frequently combine strategies. The authors identify multi-label approaches as an important direction.
• Formalizing the downsampling procedure. The co-occurrence reductions were applied through manual inspection rather than a parameterized, automated rule; the authors call for future work to formalize this step.
• Improving minority-class performance. Moderate macro-F1 and low per-class scores for overlapping categories point to a need for better minority-class and overlap-aware modeling.
• Broadening the data. The corpus is bounded to 2014–2025 and mainly Bangladeshi political news sources, so performance may vary across time periods, speakers, and Bangla-speaking regions. The authors also list source-level evaluation and context-aware modeling as open directions.

Target Audience

Researchers working on Bangla or other low-resource NLP will get the most from this paper, particularly those interested in political text classification, propaganda and persuasion detection, and resource creation. It is also relevant to computational social scientists and political communication scholars who need a technique-level rather than sentiment-level view of political discourse, to NLP practitioners building Bangla-language content analysis pipelines, and to benchmark-oriented researchers interested in how class imbalance and source leakage affect fine-grained classification results.

Authors’ abstract

Political discourse often uses rhetorical and persuasive language to frame narratives, influence public opinion, and mobilize audiences. While Bangla natural language processing has made progress in sentiment analysis and opinion mining, systematic benchmarking of transformer models for fine-grained rhetorical and persuasion technique detection in Bangla political speech remains largely underexplored. This paper presents a benchmark study of transformer-based models for detecting rhetorical form and persuasive intent in Bangla political discourse. Using BanglaRhet, a manually annotated corpus of 30,289 Bangla political speech segments collected from publicly available political news sources, we formulate two supervised single-label classification tasks: rhetorical technique detection (contrast, repetition, exaggeration, metaphor, rhetorical questions) and persuasion technique detection (blame assignment, call to action, unity call, moral, emotional, and logical appeals). We evaluate four transformer-based models, BanglaBERT, BanglaBERT-Base, SahajBERT, and XLM-RoBERTa-Base, against classical TF-IDF baselines. BanglaBERT achieves the highest performance, with 65.40% macro-F1 for rhetorical technique detection and 66.46% for persuasion technique detection, outperforming the best tuned classical baseline by 19.2 and 13.8 macro-F1 points, respectively. Class-level analysis indicates that errors are mainly associated with semantic overlap among labels, figurative language, and class imbalance. The results provide initial benchmark baselines for Bangla rhetorical and persuasion-aware political discourse analysis and highlight the need for context-aware and multi-label modeling.