Reasoning-Aware Multimodal Fusion for Hateful Video Detection

AI summary

Overview

Research area: Computer vision and multimodal machine learning for online safety, specifically hateful video detection.

Technical level: Intermediate to Advanced. The paper assumes familiarity with transformer architectures, cross-attention, vision-language models (VLMs), and standard multimodal benchmark evaluation.

Scope: The paper introduces Reasoning-Aware Multimodal Fusion (RAMF), a framework that combines structured VLM-generated reasoning with a two-stage multimodal fusion pipeline (Local-Global Context Fusion and Semantic Cross Attention) to classify videos as hateful or non-hateful.

What This Paper Is About

Automatically detecting hateful content in videos is hard because harmful intent is often expressed through subtle combinations of what is shown, what is said, and how it is said, spread unevenly across a video's timeline. Existing systems either fuse video frames, audio and transcript in a shallow way, or bolt on VLM-generated reasoning text without distinguishing it from ordinary modality features or checking whether that reasoning is trustworthy. RAMF addresses both weaknesses at once: it generates three complementary reasoning texts from a VLM under contrasting assumptions, and it fuses them with the underlying modalities in a structured, two-stage process.

Key Contributions

1. Contrastive reasoning. A structured three-stage VLM pipeline that produces (i) an objective description of the video content, (ii) a hate-assumed interpretation, and (iii) a non-hate-assumed interpretation. This supplies complementary semantic perspectives instead of a single narrative, and the paper argues it is more reliable than Chain-of-Thought (CoT) reasoning, which prior work has shown may not faithfully reflect a model's actual reasoning.
2. Local-Global Context Fusion (LGCF). A module that runs two parallel channels over each modality's embedding sequence: a Local Temporal Channel using 1D convolution and max pooling over time, and a Global Temporal Channel using adaptive average pooling. A learned sigmoid gate combines the two into a compact per-modality representation. The motivation is that hate cues can erupt briefly or be dispersed across an entire video, and that LSTM-based approaches have weak local modelling capability.
3. Semantic Cross Attention (SCA). An attention mechanism with Cross-Head Convolution (CHC), which applies a shared 3×3 2D convolution across attention heads to capture high-order cross-modal correlations, and Structural Mixing Convolution (SMC), which splits heads into even- and odd-indexed groups, concatenates them, and mixes them with grouped 1D convolution. Unlike Multi-Token Attention (MTA), a single shared convolution is applied to all heads rather than independent per-head kernels.
4. A two-stage hierarchical fusion and evaluation protocol. Objective-grounded inputs {T, A, V, T_O} are fused first into Y1; the contrastive signals {T_H, T_N} then refine this into Y2 for classification. The authors also re-collected and re-partitioned both datasets into a five-fold cross-validation setup with mutually exclusive test sets, rather than fixing one split and varying random seeds.

Main Findings

• Headline improvements: On two real-world hateful video datasets, RAMF improves over state-of-the-art methods by 3% in Macro-F1 and 7% in hate class recall, with the best result in every dataset and feature combination tested.
• HateMM results (BERT/MFCC/ViT features): RAMF reaches 83.7 Macro-F1, 84.3 accuracy, 78.6 hate-class precision, 83.7 hate-class recall, versus 82.3 / 82.9 / 78.0 / 80.2 for the authors' own MF (Multimodal Fusion) configuration without VLM inference, and 81.0 Macro-F1 for the prior state-of-the-art MoRE.
• HateMM results (BERT/CLAP/CLIP-style T2·A2·V2 features): RAMF reaches 85.1 Macro-F1, 85.6 accuracy, 79.8 precision, 85.5 recall, versus 83.2 / 83.7 / 78.3 / 82.6 for MF.
• Other datasets: On MultiHateClip (Chinese) RAMF reaches 69.3 Macro-F1 and on MultiHateClip (English) 64.1 Macro-F1 with T1·A1·V1 features; with T2·A2·V2 features it reaches 70.9 and 71.7 respectively.
• Fusion module value without reasoning: The MF row (no VLM inference) still outperforms previous fusion methods, which the authors present as evidence that LGCF and SCA are effective on their own.
• Ablation — removing SCA drops Macro-F1 from 82.96 to 80.28 (↓2.91) on the MF baseline; removing LGCF drops it to 80.74 (↓2.45); removing the MLP drops it to 80.83 (↓2.24). Replacing LGCF with an LSTM drops it to 77.96 (↓5.50).
• Ablation — attention variants: Standard attention gives 79.07 Macro-F1 (↓4.43) and cross-attention 78.89 (↓1.84), both below MTA at 81.94 (↓1.14) and simple concatenation at 81.94 (↓1.21). Removing CHC costs 2.20 Macro-F1; removing SMC costs 1.09.
• Ablation — reasoning design: Replacing contrastive reasoning with CoT (MF-CoT) yields 83.24 Macro-F1 (↓1.14). Removing the hate/non-hate assumptions costs 2.06 Macro-F1, which the paper describes as a decline of over 2%. Removing objective descriptions costs 0.61. Processing everything in a single SCA layer instead of two costs 1.80.
• Robustness to VLM choice: Swapping Qwen 2.5-VL-32B (RAMF 1: 84.26 Macro-F1, 83.75 accuracy) for LLaMA 4-17B (RAMF 2: 84.35 Macro-F1, 83.62 accuracy) changes results only marginally. The wider VLM comparison on HateMM reports Gemini-2.5-Flash at 83.38 Macro-F1 / 84.17 accuracy, GPT-5-mini at 83.96 / 84.72, Qwen2.5-VL-7B at 81.81 / 82.69, and Qwen3-VL-2B at 82.59 accuracy (its Macro-F1 is not reported in the table).
• Constraint noted: The 16-sampled-frame Qwen 2.5-VL-32B configuration was chosen "to accommodate hardware constraints."
• Data size deviation: The experiments use 1,083 HateMM videos and 959/964 videos from the Chinese/English MultiHateClip subsets; the authors state these differ from the 1,000 videos per subset reported by Wang et al. because some original videos were removed from Bilibili and YouTube.

Methodology in Plain English

The system treats a video as three input streams — spoken words transcribed by OpenAI's Whisper model, audio, and video frames — plus three extra text streams generated by a vision-language model.

Generating the reasoning. The VLM is shown 16 sampled frames per video and prompted three times. First it writes a neutral, factual description of what is happening. Then it is asked to assume the content is hateful and explain what would support that reading. Finally it is asked to assume the content is not hateful and offer alternative explanations such as satire, art, or personal conflict. The paper's argument is that forcing both interpretations reduces hallucination, provides a self-correcting second opinion, and grounds the output in visual evidence rather than a single unverifiable story.

Encoding. The transcript and the three reasoning texts are encoded with BERT, multilingual BERT, or the HateXplain (HXP) model into 768-dimensional embeddings, padded or truncated to a fixed length of 100, then passed through small MLPs. Audio is resampled to 16kHz for 40-dimensional MFCC features and to 48kHz for 512-dimensional CLAP embeddings. Video frames are processed by ViT or ViViT into 768-dimensional embeddings and by CLIP into 512-dimensional embeddings.

Fusing within each modality. LGCF takes each modality's sequence and sends it down two paths: a convolution-plus-max-pooling path that highlights short, salient moments, and an average-pooling path that summarises the whole video. A learned gate decides, element by element, how much of each to keep. The result is one compact vector per modality.

Fusing across modalities. SCA stacks the modality vectors and applies attention where the attention scores themselves are convolved: a shared 3×3 convolution lets heads exchange information, and an even/odd head-splitting step mixes heads that are far apart in the stack. This happens twice — first over {text, audio, video, objective description} to build a grounded representation, then over {that representation, hate-assumed reasoning, non-hate-assumed reasoning} to produce the final representation, which is pooled and passed through an MLP to output the hateful/non-hateful decision.

Evaluation protocol. Both datasets are split 70%/10%/20% for train/validation/test inside each of 5 folds, with strictly non-overlapping test sets across folds. Models use the Adam optimiser with a learning rate of 10⁻⁴ and cross-entropy loss. An average of 100 frames per video is sampled for HateMM and 32 for MultiHateClip. Reported metrics are Macro-F1, accuracy, and hate-class precision, recall and F1. For MultiHateClip's binary task, hate and offensive labels are merged into a single hate label.

Why This Matters

Impact on research. The paper challenges two habits in multimodal hate-speech work: treating VLM-generated reasoning as just another text feature, and trusting a single reasoning narrative. It offers an alternative — deliberately generating and contrasting opposing interpretations, then fusing them hierarchically — and supplies ablations showing that CoT substitution, removal of the contrastive assumptions, and collapsing the two-stage fusion each hurt performance. It also argues that CoT lacks fidelity to a model's true decision process, which matters for a domain where transparency is a legal requirement, and it strengthens evaluation practice by moving to non-overlapping five-fold test sets instead of repeated runs on one fixed split.

Real-world applications:

• Platform content moderation pipelines that need to flag hateful videos for review rather than rely on user reports.
• Human moderator assistance, where the objective description plus hate-assumed and non-hate-assumed analyses give reviewers a structured rationale rather than a bare score.
• Regulatory and compliance workflows that require explainable, auditable algorithmic decisions on content takedowns.
• Trust-and-safety research infrastructure built on the publicly released code and data at the linked GitHub repository.

Industry relevance. The framework is designed to slot into the standard three-modality pipeline already used in industry (frames, audio, transcript), and the paper reports that gains hold under two different feature configurations and two different VLMs. The ablation showing that a single shared convolution across attention heads avoids the parameter increase of per-head kernels and reduces inference time speaks directly to production cost concerns. Recall on the hate class — where the largest reported gain, 7%, appears — is typically the metric platforms weight most heavily, since misses are more damaging than false alarms.

Future Directions

• Temporal localisation. The paper observes that hate cues may erupt briefly or be dispersed across a whole video, yet the output is a single video-level binary label. Predicting when in the video the hate occurs is an open question the framework does not address.
• Language and platform coverage. Evaluation is on English HateMM and Chinese/English MultiHateClip subsets, with some videos lost to removals from Bilibili and YouTube. Whether the contrastive reasoning prompts transfer to other languages and cultures is not tested.
• Reasoning reliability at scale. The paper validates robustness across several VLMs (Gemini-2.5-Flash, GPT-5-mini, Qwen2.5-VL-32B, LLaMA-4-17B, Qwen2.5-VL-7B, Qwen3-VL-2B) and notes a 32B model was used partly due to hardware constraints, but the quality of the generated reasoning itself and how to detect flawed reasoning paths remain open.
• Beyond binary classification and shared attention. Determining how far the shared-head convolution in SCA generalises to other multimodal tasks, and whether finer-grained label spaces than the merged binary hate/not-hate setup could be supported, are natural next steps.

Target Audience

Researchers and graduate students working on multimodal machine learning, video understanding, or online content moderation; trust-and-safety engineers and applied scientists building hate-speech detection systems; and policy or platforms-focused readers interested in explainability and transparency requirements for automated content decisions. Readers should be comfortable with attention mechanisms and multimodal fusion terminology. The paper carries a disclaimer that it contains sensitive content that may be disturbing to some readers.

Authors’ abstract

Hate speech in online videos is posing an increasingly serious threat to digital platforms, especially as video content becomes increasingly multimodal and context-dependent. Existing methods often struggle to effectively fuse the complex semantic relationships between modalities and lack the ability to understand nuanced hateful content. To address these issues, we propose an innovative Reasoning-Aware Multimodal Fusion (RAMF) framework. To tackle the first challenge, we design Local-Global Context Fusion (LGCF) to capture both local salient cues and global temporal structures, and propose Semantic Cross Attention (SCA) to enable fine-grained multimodal semantic interaction. To tackle the second challenge, we introduce adversarial reasoning-a structured three-stage process where a vision-language model generates (i) objective descriptions, (ii) hate-assumed inferences, and (iii) non-hate-assumed inferences-providing complementary semantic perspectives that enrich the model's contextual understanding of nuanced hateful intent. Evaluations on two real-world hateful video datasets demonstrate that our method achieves robust generalisation performance, improving upon state-of-the-art methods by 3% and 7% in Macro-F1 and hate class recall, respectively. The source codes and data required to reproduce our results are available at https://github.com/Multimodal-Intelligence-Lab-MIL/RAMF.