Multiclass Hate Speech Detection with RoBERTa-OTA: Integrating Transformer Attention and Graph Convolutional Networks

AI summary

Overview

• Research area: Natural Language Processing, specifically fine-grained multiclass hate speech detection combining transformer models with graph neural networks and structured ontology knowledge.
• Technical level: Advanced. The paper assumes familiarity with transformer architectures (RoBERTa/BERT), scaled dot-product attention, Graph Convolutional Networks, and standard training and evaluation protocols.
• Scope: A single study comparing a RoBERTa baseline against a proposed RoBERTa-OTA architecture on a 39,747-sample, 5-class demographic hate speech dataset, evaluated with 5-fold cross-validation plus a robustness analysis under text perturbations.

What This Paper Is About

Automated hate speech detection is usually framed as a binary problem (hateful vs. not hateful), and even the newer multiclass systems rely only on patterns learned from training data, without any explicit structured knowledge about how demographic hate categories relate to one another. The authors propose RoBERTa-OTA, which pairs RoBERTa's contextual text embeddings with a small hand-specified ontological graph of the five target categories, processed by a Graph Convolutional Network. The goal is to classify hate speech into Age, Ethnicity, Gender, Religion, and Other Hate categories while improving accuracy on the hardest, most implicitly-worded categories without adding much computational cost.

Key Contributions

1. An ontology-guided dual-stream architecture (RoBERTa-OTA) that processes text through RoBERTa-base plus scaled dot-product attention, and in parallel processes a five-node, six-feature-per-node hate speech ontology graph through an enhanced 3-layer GCN, then concatenates the two representations (768-dimensional text features with 32-dimensional ontology features, giving an 800-dimensional combined vector).
2. A defined ontological knowledge representation for hate speech categories, where each of the five demographic categories is encoded by six semantic dimensions (demographic targeting, cultural identity targeting, gender-related characteristics, religious/belief targeting, linguistic complexity, and targeting diversity) grounded in the authors' own linguistic analysis of the 39,747-sample dataset.
3. State-of-the-art multiclass results on the SOSNet-derived dataset, reporting 96.04% accuracy and 96.06% weighted F1 for RoBERTa-OTA, compared to 95.02% accuracy and 95.04% F1 for the RoBERTa baseline and 94.38% accuracy / 94.44% F1 for the previous best approach, SOSNet with BOW+XGBoost.
4. A combined efficiency and robustness analysis, quantifying the overhead of the added components (0.33% parameter increase, +10.3% FLOPs, +19.2% GPU memory, +12.2% training time per fold) alongside a six-perturbation robustness study on 3,000 sampled tweets.

Main Findings

• Overall accuracy and F1: RoBERTa-OTA reaches 96.04% accuracy and 96.06% weighted F1, versus 95.02% accuracy and 95.04% F1 for the RoBERTa baseline — a 1.02 percentage point improvement in both. Precision rises from 95.12% to 96.09%, and recall from 95.02% to 96.04%.
• Comparison to prior state of the art: Against SOSNet's best reported result on the same dataset (BOW+XGBoost at 94.38% accuracy, 94.44% F1), RoBERTa-OTA improves by 1.66 percentage points in accuracy and 1.62 percentage points in F1. The paper also cites SBERT+SOSNet at 92.70% accuracy and 92.58% F1.
• Gains concentrate in the hardest categories: Gender-based hate speech improves from 90.70% to 93.06% F1 (+2.36 percentage points) and Other Hate from 88.94% to 91.32% (+2.38 percentage points). By contrast, the high-performing categories move only marginally: Age F1 goes from 98.63% to 98.75%, Ethnicity from 98.41% to 98.44%, and Religion from 98.28% to 98.62%.
• One category's accuracy slightly decreases: Ethnicity accuracy falls from 98.10% to 97.84% (and recall from 98.10% to 97.84%) under RoBERTa-OTA, even though its F1 and precision rise.
• Linguistic heterogeneity justifies fine-grained models: Religion hate speech has the highest linguistic complexity in the dataset (average 198.0 characters, 39.6 tokens, 10.1 nouns), Age averages 173.5 characters and 37.0 tokens, Ethnicity 139.3 characters and 30.4 tokens, Gender 136.4 characters and 29.3 tokens, and Other Hate is shortest at 85.7 characters and 18.4 tokens while showing the highest URL rate (0.18 per sample).
• Categories are lexically separable: Pairwise Jensen-Shannon divergence between class unigram language models ranges from 0.449 to 0.483 (mean 0.466), with Other Hate and Religion maximally divergent (0.483) and Age and Gender minimally divergent (0.449).
• Small architectural cost: Parameter count rises from 124.65M to 125.06M (0.33% overhead), model size from 475.5 MB to 477.1 MB (+1.6 MB, 0.34%), FLOPs from 31.9 to 35.2 billion (+10.3%), peak GPU memory from 2.6 GB to 3.1 GB (+19.2%), and training time from 24.5 to 27.5 minutes per fold (+12.2%).
• Faster convergence despite higher per-epoch cost: RoBERTa-OTA required 29 total epochs across 5-fold cross-validation versus 31 for the baseline, which the authors attribute to ontology-guided learning.
• Robustness advantages under noise: On 3,000 sampled tweets, at 15% character deletion the baseline's F1 drops to 76.17% (a 21.32 point decrease) while RoBERTa-OTA holds 79.64% (18.89 point decrease), a 3.47 point advantage. At 15% character insertion, the gap is 4.62 points (81.20% vs. 85.82% F1). Under 30% abbreviation injection, RoBERTa-OTA keeps 98.36% F1 (a 0.17 point drop from clean text) versus 96.84% F1 for the baseline (0.65 point drop). Slang substitution produced minimal degradation for both models, with RoBERTa-OTA staying at 98.53% F1 at all three intensity levels.
• Clean-text robustness baseline: Under no perturbation, the baseline scores 97.50% accuracy and 97.49% F1 while RoBERTa-OTA scores 98.53% accuracy and 98.53% F1 (+1.03 and +1.04 points).

Methodology in Plain English

The authors frame the task as five-way classification: given a short piece of text, decide whether it targets Age, Ethnicity, Gender, Religion, or Other Hate. They build two comparable systems so the only difference is their proposed addition.

The baseline is RoBERTa-base (124.6M parameters) with an enhanced classification head that reduces RoBERTa's 768-dimensional text representation down to five class scores.

RoBERTa-OTA runs two streams in parallel. The text stream passes the input through RoBERTa-base, adds scaled dot-product attention layers over the 768-dimensional representations, and mean-pools the result into a single 768-dimensional vector. The ontology stream starts from a hand-built graph: five nodes, one per hate category, each described by six numbers such as whether the category involves religious targeting or how linguistically complex it is. The graph is fully connected, so every category links to every other. A three-layer Graph Convolutional Network shrinks these node features from 6 to 64 to 64 to 32 dimensions, and mean pooling produces a 32-dimensional ontology vector. The two vectors are concatenated into an 800-dimensional representation, which passes through three linear layers (800→400→200→5) with batch normalization, layer normalization, and decreasing dropout (0.3, 0.2, 0.1).

Both models use PyTorch on two NVIDIA Ampere A100 GPUs (6912 CUDA cores, 40GB RAM each), the Transformers library, and PyTorch Geometric. Training uses the AdamW optimizer with learning rate 1e-5, betas (0.9, 0.999), weight decay 1e-5, batch size 16, a maximum of 20 epochs, early stopping with patience of 3 monitoring validation F1, and cross-entropy loss with label smoothing (α = 0.1). Text is tokenized with RoBERTa's tokenizer at a maximum sequence length of 128. Evaluation uses 5-fold stratified cross-validation, reporting accuracy and weighted F1 along with precision and recall. Finally, the authors stress-test both models by injecting six kinds of social-media-style noise (character deletion, substitution, and insertion at 5/10/15%; word deletion at 10/20/30%; abbreviation injection and slang substitution at 10/20/30%) into 3,000 sampled tweets.

Why This Matters

Impact on research. The paper argues that multiclass demographic hate speech detection is under-explored relative to binary detection, and that injecting explicit structured knowledge — rather than relying purely on learned representations — is a productive direction. It also shows that improvements from such knowledge concentrate on implicitly-worded categories (Gender, Other Hate), which is where standard transformer models are weakest, and it reports the full efficiency and robustness picture rather than accuracy alone.

Real-world applications.

• Social media content moderation pipelines that must route reports to category-specific review policies rather than a single hate/not-hate flag.
• Trust-and-safety dashboards that track abuse trends by targeted demographic group over time.
• Community platforms and forums needing category-aware filtering of user-generated posts.
• Robustness-sensitive deployments where user text is noisy through typos, abbreviations, and slang, which the perturbation study directly addresses.

Industry relevance. The paper emphasizes deployment feasibility: the added ontology branch costs only 0.33% more parameters and 1.6 MB more model size, and the authors argue that in a production system processing millions of messages daily, a 2.36 percentage point improvement in gender-based hate speech detection translates to thousands of additional correctly identified harmful posts. The faster convergence (29 vs. 31 epochs) partially offsets the 12.2% per-fold training time increase.

Future Directions

• Multilingual and cross-cultural evaluation. The authors explicitly state that cross-linguistic evaluation remains future work and plan to test RoBERTa-OTA on multilingual datasets to assess effectiveness across diverse cultural and linguistic contexts.
• Efficiency optimization. They aim to reduce computational overhead without compromising accuracy, which is relevant given the 10.3% FLOPs and 19.2% memory increases reported.
• Advanced knowledge integration. The strong results motivate further investigation into richer knowledge-integration techniques, especially for capturing evolving patterns in online hate speech as language shifts.
• Scaling and external validation. All results come from one 39,747-sample dataset derived from the SOSNet framework and one ontology hand-crafted by the authors from that same dataset's linguistic analysis; whether the same gains hold on independently constructed datasets and ontologies is an open question the paper does not resolve.

Target Audience

Researchers and graduate students working on abusive language detection, computational social science, or graph-augmented NLP who want a concrete example of combining transformer embeddings with a knowledge graph. It is also relevant to applied machine learning engineers and trust-and-safety practitioners evaluating whether a small accuracy gain on hard categories justifies added architectural complexity — though readers without a background in transformers and graph neural networks will find the methodology section demanding.

Authors’ abstract

Multiclass hate speech detection across demographic categories remains computationally challenging due to implicit targeting strategies and linguistic variability in social media content. Existing approaches rely solely on learned representations from training data, without explicitly incorporating structured ontological frameworks that can enhance classification through formal domain knowledge integration. We propose RoBERTa-OTA, which introduces ontology-guided attention mechanisms that process textual features alongside structured knowledge representations through enhanced Graph Convolutional Networks. The architecture combines RoBERTa embeddings with scaled attention layers and graph neural networks to integrate contextual language understanding with domain-specific semantic knowledge. Evaluation across 39,747 balanced samples using 5-fold cross-validation demonstrates significant performance gains over baseline RoBERTa implementations and existing state-of-the-art methods. RoBERTa-OTA achieves 96.04\% accuracy compared to 95.02\% for standard RoBERTa, with substantial improvements for challenging categories: gender-based hate speech detection improves by 2.36 percentage points while other hate speech categories improve by 2.38 percentage points. The enhanced architecture maintains computational efficiency with only 0.33\% parameter overhead, providing practical advantages for large-scale content moderation applications requiring fine-grained demographic hate speech classification.