MigrationNarrate: A Dataset for Detection of Migration Narratives in YouTube Videos

AI summary

Overview

• Research area: Narrative detection in online video, combining natural language processing, computational social science, and migration discourse analysis.
• Technical level: Intermediate. The paper assumes familiarity with classification benchmarks, transformer encoders, and LLM prompting/fine-tuning.
• Scope: The paper introduces MigrationNarrate, the first dataset for detecting UK migration narratives in YouTube video transcripts, together with baseline benchmarks and an error analysis.

Note on metadata: the paper is listed on arXiv under the Computer Vision category (arXiv:2608.20984v1), but its methods and experiments operate exclusively on video transcripts, with multimodal signals left to future work.

What This Paper Is About

Narrative detection has been studied on text from platforms such as X, Facebook, and news articles, but migration narratives have been largely unstudied because no dedicated annotated dataset existed. At the same time, public discourse has shifted toward video platforms, where narratives travel through multimodal signals and reach very large audiences. The paper's goal is to fill both gaps by building and benchmarking a labelled dataset of migration narratives in YouTube video transcripts, using a two-level taxonomy of 12 super-narratives and 53 narrative labels (Section 3.1 and the annotation procedure both state 53 narrative labels, although Table 1 in the paper lists the MigrationNarrate narrative count as 12/51).

Key Contributions

1. MigrationNarrate dataset: The first dataset for detecting UK-related migration narratives in video transcripts, consisting of 1,115 YouTube video transcripts annotated with migration super-narratives and narrative labels, plus 4,425 automatically filtered, unlabelled migration-related videos made available to support weak and semi-supervised learning.
2. Benchmarks: Evaluation of a pre-trained encoder model (RoBERTa-large) and both open-source LLMs (Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-3-12b-it) and closed-source LLMs (GPT-4o, GPT-5.4) under zero-shot, few-shot, and fine-tuning settings.
3. Error analysis: A manual inspection of 60 development-set transcripts where the top open-source models misclassified narratives, producing a taxonomy of five recurring error types.
4. Annotation protocol and taxonomy operationalisation: Use of the European Commission Joint Research Centre (JRC) migration narrative taxonomy, with GPT-5.1-generated narrative definitions reviewed and validated by the authors, and a hierarchical annotation interface that showed only super-narratives first.

Main Findings

• Encoders versus LLMs: The fine-tuned RoBERTa-large model reached a Macro-F1 of 0.446 for super-narratives, close to GPT-4o, slightly surpassing it in the few-shot setting (RoBERTa 0.446 vs. GPT-4o few-shot 0.440), and above all open-source LLMs in zero-shot and few-shot settings, but below GPT-5.4 (Macro-F1 0.480).
• Fine-tuning changes the ranking: After fine-tuning, Llama achieved the best overall super-narrative Macro-F1 of 0.525 and Gemma reached 0.502, both surpassing the closed-source best results (GPT-4o 0.440, GPT-5.4 0.480). Fine-tuned Qwen remained below RoBERTa with a Macro-F1 of 0.434.
• Fine-grained detection favours closed-source models: For narrative-level classification, GPT-4o and GPT-5.4 achieved the best Macro-F1 scores of 0.350 and 0.324, while the strongest open-source models, Llama and Gemma, reached 0.304 and 0.282 even after fine-tuning.
• Prompting helps, mostly: Few-shot prompting improved over zero-shot for almost all models, with Llama the only exception, where performance slightly declined (super-narrative Macro-F1 dropped from 0.226 zero-shot to 0.115 few-shot).
• Task difficulty is high: Best reported narrative-level Macro-F1 in the main results table is 0.350, showing that fine-grained narrative detection remains challenging.
• Class imbalance: The most prevalent super-narrative is "Immigration burdens the economy & welfare narrative" with 190 videos; the most frequent individual narrative is "Political rivals' policy failed/ is failing/ will fail" with 65 videos; 106 videos are labelled None.
• Annotation agreement is modest: Cohen's kappa before adjudication was 0.45 at super-narrative level and 0.30 at narrative level, described as moderate and fair respectively.
• Error patterns: In the 60-example error analysis, the dominant error type was Surface-Narrative Misalignment (49.7%), followed by Fine-Grained Label Confusion (26.67%), Multi-Narrative Overlap (12.28%), Failure to Detect Counter-Narratives (10%), and Weak Narrative Signal (5%).

Methodology in Plain English

The authors first collected potentially migration-related YouTube videos through two routes: a curated set of channels and playlists (25 playlists and 21 unique channels) covering institutional media, political actors, and non-institutional commentators, and a search-based route using 77 search phrases (15 general, such as "Channel crossings UK", and 62 narrative-oriented, such as "immigration housing crisis UK").

They then narrowed the pool in stages. Metadata filtering kept only videos of three minutes or less published between 1 January 2024 and 30 September 2025, downloaded with yt-dlp and transcribed with faster-whisper, leaving 20,992 videos. Semantic filtering used E5-Large embeddings to compare each transcript against ten immigration-related queries, keeping transcripts whose maximum cosine similarity was 0.8 or higher, leaving 5,540 videos. GPT-5.1 was then used to pre-annotate the dominant super-narrative and narrative with a confidence score; videos with confidence of at least 0.8 were eligible, and up to 40 videos per narrative were sampled with a round-robin strategy balancing publication year and channel, yielding 1,115 videos.

Human annotation was done by six international relations and politics students, compensated at 17.58 GBP per hour, using the Teamware tool. A pilot study with two students produced a gold set of 30 videos. Annotators had to reach Cohen's kappa of at least 0.45 on super-narratives and 0.35 on narratives to join the main phase. Data was split into 11 batches, each annotated by two annotators plus an adjudicator.

For experiments, the data was split by stratified sampling into 774 train, 117 development, and 224 test videos. RoBERTa-large was fine-tuned separately for super-narratives and narratives; the three open-source LLMs were tested zero-shot, few-shot, and with QLoRA fine-tuning; GPT-4o and GPT-5.4 were tested zero-shot and few-shot. All LLMs used a two-step hierarchical prompt: first identify the super-narrative, then the corresponding narrative, mirroring the human annotation interface.

Why This Matters

• Research impact: It opens a new domain, migration narratives in video transcripts, where no public annotated dataset previously existed, and provides the first benchmarks and an error taxonomy for that task.
• Methodological insight: The results show that fine-tuning open-source LLMs can beat closed-source zero-shot and few-shot prompting at the coarse super-narrative level, but not at fine-grained narrative level, which informs model selection for similar classification tasks.
• Assessment of task limits: The error analysis shows where models fail, such as relying on salient keywords instead of the dominant argument, mistaking counter-narratives for the narrative being rejected, and over-predicting narratives when the signal is weak.

Real-world applications:

• Monitoring migration-related content on video platforms at scale, where the paper notes YouTube reaches over 2.83 billion users worldwide as of March 2026, with approximately 340 million daily users and more than 1.8 million hours of videos uploaded daily.
• Supporting media and fact-checking organisations that need to track recurring migration framings across many channels.
• Enabling policy-oriented analysis of how migration is framed in public communication, building on the JRC taxonomy the dataset uses.
• Providing a reusable resource for training weak- or semi-supervised narrative classifiers using the 4,425 unlabelled filtered videos.

Industry relevance: the benchmark comparison of encoder models against open- and closed-source LLMs at different cost points is directly applicable to content moderation, media analytics, and platform trust-and-safety pipelines that must classify short-form video content. The hierarchical prompting setup is also relevant for any fine-grained, high-label-count classification problem.

Future Directions

• Multimodal modelling: The paper states that its approach uses transcripts only and does not account for video, audio, or user interactions, and identifies incorporating visual content and audio cues as an important direction.
• Clearer label boundaries and multi-label annotation: The error analysis attributes Fine-Grained Label Confusion and Multi-Narrative Overlap to ambiguity in the taxonomy, suggesting clearer label boundaries and support for multi-label annotation of videos containing several strong narratives.
• Larger and less biased collection: The limitations section notes that manually curated channels, playlists, and taxonomy-driven search phrases may introduce selection bias, and that the dataset is annotated by only two annotators with relatively low agreement, resulting in potential label noise.
• Expanding the labelled data: The paper notes that the released filtered corpus can support weak and semi-supervised learning and can be manually annotated to expand the dataset.

Target Audience

Researchers in narrative detection, computational social science, and migration and media studies will benefit most, particularly those working on dataset construction for fine-grained, multi-class labelling tasks. It is also useful for NLP practitioners comparing encoder models with open- and closed-source LLMs under zero-shot, few-shot, and fine-tuning settings, and for analysts and platform teams interested in scalable detection of migration-related narratives in video content.

Authors’ abstract

Narratives are central to how social communication is framed, making their detection critical for understanding and analysing public discourse. Prior work has explored narrative detection and extraction across diverse domains; however, migration narratives remain significantly understudied, primarily due to the absence of dedicated annotated datasets. Furthermore, public communication has recently shifted towards video-centric platforms, where narratives are conveyed through multimodal signals and consumed at scale. Despite this shift, narratives in videos remain largely unexplored. To bridge these gaps, we introduce MigrationNarrate, the first multimodal dataset for detection of migration narratives in the UK, consisting of 1,115 YouTube video transcripts annotated using a two-level taxonomy of 12 migration super-narratives and 53 narrative labels. This paper details the dataset design, collection, and annotations; together with benchmark results using a combination of pre-trained encoder models and both open- and closed-source Large Language Models. Finally, a thorough error analysis offers insights for future work.