GENDIS: Gender-Related Discourse and Stance in Multilingual YouTube Comments from India, Kenya, and Nigeria

AI summary

Overview

• Research area: Natural Language Processing and computational social science, specifically multilingual detection of gender-related speech and classification of its evaluative stance on social media.
• Technical level: Intermediate. The paper is readable with basic knowledge of classification metrics (F1) and language models, though the benchmarking setup, annotation-agreement statistics, and transfer experiments require some machine-learning background.
• Scope: The paper introduces Gendis, a two-level annotation framework and human-annotated evaluation set of 9,600 YouTube comments from India, Kenya, and Nigeria, plus a benchmark of toxicity, supervised, and zero-shot LLM systems on both levels of that framework.

What This Paper Is About

Most NLP work on gender and online speech is framed around harm, such as sexism, misogyny, or toxicity detection, which lumps together all non-harmful content and hides the difference between comments unrelated to gender and comments that mention gender neutrally or positively. Existing datasets are also concentrated in high-resource languages and Global North settings, even though gendered meanings are expressed through locally specific slang, code-switching, and norms. The paper builds a locally grounded, multilingual dataset and framework that first asks whether gender is referenced at all, and then asks whether the gender-related content is negative, positive, or neutral.

Key Contributions

1. A two-level annotation framework that separates whether gender is referenced from how the gender-related content is evaluated (negative, positive, neutral, or unsure), allowing harmful gender-related speech to be distinguished from both gender-unrelated and non-harmful gender-related content.
2. Gendis, comprising locally reviewed multilingual keyword lists and a human-annotated set of 9,600 comments from country-associated YouTube channel corpora in India, Kenya, and Nigeria, together with disaggregated annotations and analyses of annotator disagreement.
3. A benchmark of a toxicity baseline, zero-shot large language models, and lightly supervised classifiers at both levels, showing that generic toxicity detection is poorly suited to the task and that small amounts of country-specific supervision are highly valuable.

Main Findings

• Gender-related speech is common, and mostly not negative: In the uniform random stratum, 23.8–35.0% of comments are gender-related (23.8% in Kenya, 95% CI 21.7–26.0; 30.3% in India, 28.1–32.7; 35.0% in Nigeria, 32.6–37.4). The abstract reports that among those with a resolved stance, 82% are neutral or positive; the descriptive statistics for the 1,226 of 4,500 random comments with a resolved Level 2 majority report 67% neutral, 18% negative, and 16% positive.
• Detection is easier than stance: Across every model, macro-F1 is far higher on Level 1 (is the comment gender-related) than on Level 2 (stance). SetFit, the strongest model at Level 2, reaches 0.83 at Level 1 but 0.67 at Level 2. Neutral and positive speech are the hardest categories, and positive comments are most often misread as negative rather than recognized as affirming.
• Light supervision rivals frontier zero-shot models: SetFit matches the strongest zero-shot large language model at both levels while running locally at a fraction of the cost, improving over gpt-5-nano at Level 2 by eleven points (0.67 versus 0.56) and over the open decoders by more than twenty-five points. Only the frontier gpt-5.4 comes close at Level 2 (0.64), and it is a large proprietary model billed per call.
• Generic toxicity detection fails: The toxicity baseline is weakest overall (Level 1 macro-F1 0.46, Level 2 macro 0.31, end-to-end 0.26), never predicts the positive class, and on the random stratum collapses to 0.47 while recovering only 7% of gender-related comments.
• Open decoders struggle specifically on neutral speech: Gemma-2-9B and Llama-3.1-8B are roughly competitive on Level 1 (0.79 and 0.76 versus 0.81 for gpt-5-nano) but fall well behind at Level 2 (0.41 each versus 0.56), collapsing on the neutral class (per-class F1 0.17 and 0.28).
• Cross-country transfer is the bottleneck: Leave-one-country-out training is far worse than within-country or pooled training (0.75 at Level 1, 0.47 at Level 2) and steepest when India is held out (Level 1 0.69), even though it uses more data. This points to country-specific distribution shift rather than training-set size.
• Performance drops for local languages and code-mixed text: For gpt-5-nano, Level 1 macro-F1 is 0.87 on English and falls to 0.72–0.76 elsewhere, and Level 2 falls from 0.62 to 0.36. SetFit is the most robust on out-of-list content (Level 2 0.61 versus 0.36 for gpt-5-nano).
• Annotation disagreement varies sharply by country: Krippendorff's alpha at Level 1 is .80 in India, .79 in Kenya, and .54 in Nigeria; at Level 2 it is .83 in India, .72 in Kenya, and .49 in Nigeria; for the combined final label it is .77, .72, and .47. Nigeria accounts for 306 of the 362 gender-related comments with no Level 2 majority (18.9% of its 1,623 gender-related comments), compared with 14 in India (1.0% of 1,470) and 42 in Kenya (3.1% of 1,350).
• The datasets are strongly multilingual and code-mixed: Code-mixing affects 32% of Indian comments, 40% of Kenyan comments, and 30% of Nigerian comments; English appears in 58%, 67%, and 82% respectively, alongside Hindi (46% of Indian comments), Swahili (51% of Kenyan comments), and Nigerian Pidgin (18% of Nigerian comments); 6 to 12% of comments are in languages outside each country's list.
• Negative gender-related speech exceeds platform-wide hate-speech rates: Gender-negative speech runs from 2.5% of all comments in India to 9.2% in Nigeria in the keyword-independent sample, compared with 0.7% on average (0.2% Kenya, 0.3% Nigeria, 1.7% India) reported by Tonneau et al. (2025) for a representative day of Twitter.

Methodology in Plain English

The authors first built a large YouTube corpus. Using a paid Social Blade subscription, they collected the 5,000 channels with the highest subscriber counts for each of India, Kenya, and Nigeria, then pulled publicly available video metadata and comments through YouTube's Data API v3 between May 2025 and February 2026, producing roughly 11 million videos and 1.7 billion comments (approximately 1.6 billion for India, 85 million for Kenya, and 80 million for Nigeria).

To find likely gender-related comments, they built country-specific multilingual keyword lists. Seed terms came from prior gender and harm datasets and annotation schemes, were expanded in November 2025 using Grok 4, and were then reviewed by three country-context reviewers per country. The final lists contain 1,530 entries (778 for India, 386 for Nigeria, 366 for Kenya); 776 of these were observed in the collected comments (498 in India, 137 in Kenya, 141 in Nigeria).

For each country they sampled 3,200 unique comments in three strata: 1,500 drawn via the keyword lists, 1,500 drawn uniformly at random with no keyword condition, and 200 drawn from comments matching no keyword. The random stratum supports unbiased prevalence estimates; the keyword-free stratum describes a subpopulation and is excluded from prevalence estimation.

Nine annotators, three per country, each labeled every comment in their country independently, seeing only the comment text with no surrounding context. Level 1 asked whether gender is referenced explicitly or implicitly; Level 2 asked whether the gender-related content is negative (including benevolent sexism and restrictive norms), positive, neutral, or unsure. Stance is defined as how a comment treats gender rather than its surface tone, so benevolent sexism counts as negative and negative labels take precedence when positive and negative content co-occur. Labels were aggregated by majority, with comments lacking a two-of-three Level 2 agreement retained as "no Level 2 majority."

The benchmark compares seven systems on the same 9,600 majority labels: an off-the-shelf multilingual toxicity classifier mapped onto the labels, two supervised classifiers (XLM-R and SetFit, trained with out-of-fold predictions across within-country, pooled cross-validation, and leave-one-country-out regimes), and four zero-shot LLMs (open: Llama-3.1-8B and Gemma-2-9B; closed: gpt-5-nano and gpt-5.4), prompted with fixed label definitions and no prompt search. Macro-F1 is the headline metric, reported for Level 1, Level 2 (conditioned on gold gender-related comments), and an end-to-end four-way score.

Why This Matters

The paper argues that harm-centered operationalizations hide most gender-related speech and make it impossible to say how often gender is invoked and how it is framed. By providing a denominator for gender-related discourse independent of harm and by grounding the resource in three Global South contexts with locally reviewed keyword lists and locally informed annotation, it offers a way to measure gender-related speech across its full evaluative range rather than only its harmful pole. The transfer results also carry a practical message: gendered meaning is local, and locally collected supervision can beat much larger zero-shot models at lower cost.

Real-world applications:

• Content moderation and platform safety: Distinguishing neutral or supportive gender-related discussion from harmful content could reduce over-removal of legitimate speech, while the finding that gender-negative speech reaches 9.2% of comments in Nigeria speaks to how much moderation capacity is needed.
• Gender equality and anti-GBV program monitoring: The framework gives development and policy organizations a measurable indicator of when gender is invoked online and whether it is framed negatively, positively, or neutrally, including tracking of equality advocacy and counterspeech.
• Model and dataset development for under-resourced languages: The multilingual, code-mixed keyword lists and annotated data support building NLP tools for Hindi, Swahili, Sheng, Nigerian Pidgin, Yoruba, Hausa, Igbo, and their mixed forms.
• Auditing existing toxicity tools: The toxicity baseline's failure to recover neutral or positive gender-related speech is direct evidence for practitioners relying on generic toxicity classifiers that these tools measure only part of the relevant phenomenon.

Industry relevance is mainly for platform trust-and-safety teams, developers choosing between small supervised models and costly frontier APIs, and organizations building localized language technology for African and South Asian markets. The cost argument is central: a contrastive classifier trained on country-specific labeled data matched or beat frontier zero-shot models while running locally.

Future Directions

• Improving stance classification, especially positive and neutral speech: Every model found these classes hardest, and positive comments that condemn violence or femicide share vocabulary with the negative pole, leaving the largest open room for improvement.
• Understanding and mitigating cross-context transfer failure: Leave-one-country-out training underperforms within-country training despite using more data, so the field needs to determine what makes gendered meaning non-transferable and how much locally collected data is needed per new context.
• Extending coverage beyond three countries and one platform: Transfer to other regions, languages, and platforms is untested, and the authors state that measuring gender-related speech across settings will require locally grounded data in each new context.
• Handling contested and ambiguous cases better: 362 comments had a gender-related Level 1 majority but no Level 2 majority, and the benchmark excludes unresolved cases, under-representing precisely where measurement is hardest; larger and more demographically varied annotator panels could also allow disagreement to be attributed to specific annotator characteristics, which the current panels were too small to support.

Target Audience

NLP researchers working on multilingual, code-mixed, or socially situated text classification; computational social scientists and gender studies researchers studying online discourse; platform content-moderation and trust-and-safety practitioners operating in South Asian and African markets; and development organizations or policymakers who need measurable indicators of gender-related online speech. Readers most likely to benefit are those interested in dataset construction with locally grounded annotation, in benchmarks comparing lightly supervised models against zero-shot LLMs, and in why gender-related speech cannot be reduced to toxicity.

Authors’ abstract

Research on gender-related online content is often operationalized through harm-centered tasks that collapse non-harmful gender-related and unrelated content, while existing datasets remain concentrated in high-resource languages and Global North settings. To bridge these gaps, we introduce GENDIS, a dataset of 9,600 multilingual, human-annotated YouTube comments from India, Kenya, and Nigeria (3,200 per country), annotated using a two-level framework that first identifies whether gender is referenced and then classifies gender-related content as negative, positive, or neutral. This separation provides a denominator for gender-related discourse independent of harm, enabling both prevalence estimation and characterization of its evaluative range. In the uniform random stratum, 23.8-35.0% of comments are gender-related, and among those with a resolved stance, 82% are neutral or positive. Across models, gender-reference detection is substantially easier than stance classification, especially for positive and neutral speech. Lightly supervised classifiers match frontier zero-shot large language models at substantially lower cost and outperform generic toxicity detection, while cross-country transfer remains limited. GENDIS provides a locally grounded resource for measuring when gender is invoked online and how that reference is framed.