Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

AI summary

Overview

Research area: Natural Language Processing — subjective/controversial content classification, annotator disagreement modeling, multi-agent LLM systems, and preference optimization (DPO, GRPO).

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, Direct Preference Optimization, reinforcement learning from human feedback, LoRA adapters, and clustering evaluation.

Scope: The paper introduces MAP-PO (Multi-Agent Perspectivist Preference Optimization), a framework that clusters annotators by labeling behavior and trains one LLM agent per behavioral cluster to preserve disagreeing perspectives in sexism detection, evaluated across four combinations of two languages and two backbone models.

What This Paper Is About

Sexism detection is inherently subjective: the six annotators who label each tweet in the EXIST 2024 dataset frequently split 4–2 or 3–3 on whether a text is sexist, and standard NLP pipelines discard that disagreement by collapsing it into a majority vote. This paper argues that disagreement is signal rather than noise and builds a multi-agent system in which each agent is trained to reproduce the labeling behavior of a distinct group of annotators, so that the perspectives are preserved rather than averaged away. The central practical question the paper asks is how to train such agents so that each one stays faithful to its own group instead of drifting into caricature.

Key Contributions

1. Behavioral rather than demographic perspectives. The authors show that clustering annotators by three labeling-behavior features (YES rate, agreement rate with the annotator majority, and label entropy) produces more useful agent identities than demographics, since chi-squared tests find no significant association between cluster membership and gender, age, ethnicity, education, or country.

2. A multi-agent preference-optimization recipe for perspectivist NLP. They adapt the Mars-PO framework to fine-tune one LLM agent per annotator perspective for sexism detection, defining each agent's preferred output as its own cluster's majority label and its rejected output as the opposite label on texts where clusters disagree.

3. A systematic comparison of DPO- and GRPO-based objectives with individual and team rewards across a 2 × 2 grid of two backbones (gpt-4.1-mini and Qwen3-8B) and two languages (English and Spanish).

4. Identification and repair of agent polarization. They demonstrate that training agents only on their own cluster's preferences drives each agent outside the annotator distribution it is meant to represent, and that any shared team-level signal — implicit in data mixing, explicit in preference pairs, or explicit in the reward — restores calibration in every setting tested.

Main Findings

• Cluster-specific fine-tuning is necessary. Without fine-tuning, the three agents behave almost identically. In the GPT-EN setting the zero-shot ensemble agrees on 96.8% of texts and reaches only 64.7 Team F1; on the Qwen backbone the three agents predict identically on every test text in both languages (0.0% disagreement). Persona prompts recover some diversity (79.2% agreement) but reach only 68.3 Team F1.

• Individual-only DPO polarizes the agents. On GPT-EN, DPO drives the Cluster 1 agent to 1.5% YES (target 21.5%) and the Cluster 3 agent to 95.3% (target 63.5%), with disagreement rising to 93.9% and calibration error to 18.7%; Team Accuracy drops 3.0 points and Overall F1 falls to 71.2. This pattern replicates in all four settings, with Cluster 1 and Cluster 3 pinned at 0.000 and 0.997 (GPT-ES) and 0.000 and 1.000 (both Qwen settings), disagreement saturating at 94–100%, calibration error rising to 18.7–25.1 pp, and Avg Cluster F1 collapsing to 53–61 under DPO against 75–80 under SFT.

• A team anchor restores calibration in every setting. Mars-PO, which adds shared preference pairs on unanimous-agreement texts, fixes the overshoot completely: Team Accuracy 89.3%, disagreement 34.5%, calibration error 4.4%, with Cluster 1 returning to 22.8% YES and Cluster 3 to 55.0%. GRPO with a team-weighted reward pulls agents back, with mean calibration error of 2.0 pp on Qwen-EN, 8.0 on Qwen-ES, and 5.0 on GPT-ES, and disagreement returning to the 33–46% range.

• Best headline result. GRPO at α = 0.17 in GPT-EN reaches 89.9 Team F1 and 90.3% team accuracy (95% bootstrap CI [86.6, 93.3]), ahead of every baseline; persona prompts, the strongest baseline, sit at [69.1, 79.5], separated by CI non-overlap.

• A stronger KL constraint does not substitute for a team signal. Sweeping β from 0.1 to 0.5 cuts disagreement by 21.4 pp, but even the best value (72.5%) leaves the agents disagreeing on almost three of four texts, whereas Mars-PO reaches 34.5%.

• Small team weights suffice; the design axis is presence, not strength. The α sweep over {0, 0.17, 0.33, 0.5} spans only 1.7 pp in Team Accuracy, with α = 0.17 exceeding α = 0 by 0.4 pp. The disagreement curve is non-monotone: 42.7% at α = 0, 43.6% at 0.17, 36.3% at 0.33, and 50.6% at equal weight. DPO-only versus Mars-PO differs by 2.7 pp Team Accuracy and 59 pp disagreement.

• Method ranking depends on the backbone. GRPO leads SFT by 0.8 pp on GPT-EN (89.9 vs. 89.1), the three stages sit within 0.9 pp of each other on GPT-ES, and SFT stays ahead of GRPO on the smaller Qwen backbone (84.7 vs. 80.9 English; 81.0 vs. 80.6 Spanish). The authors read this as a capacity effect, since rejection-sampling GRPO retrains on the model's own filtered outputs.

• Agents behave like their clusters, except under DPO-only. After SFT every Qwen-EN agent's YES rate lies within 0.2 standard deviations of its cluster's annotator mean (0.1 for GPT-EN); GRPO keeps |z| ≤ 0.54; DPO-only agents fall outside their cluster distributions with z-scores from −4.3 to +5.1 across the four settings.

• Demographics do not explain clusters. Chi-squared tests are non-significant for every demographic variable and Cramér's V is at most 0.13 (ethnicity), with gender and age below 0.07; gender is balanced overall (174/174 M/F).

• Label balancing hurts. An ablation on Qwen-EN shows label-balanced SFT drops Team F1 from 84.7 to 83.4 and raises mean calibration error from 1.7 to 10.9 pp, because balancing erases the YES-rate differences that define the clusters.

• English and Spanish differ in measurable ways. The Spanish test set has a YES majority (58.2%, versus 39.3% English); after SFT the mean YES-rate error on Qwen-ES is 5.4 pp against 1.7 on Qwen-EN, and after GRPO 8.0 against 2.0. Spanish trails English by 3.7 to 7.3 Team F1 at zero-shot and after SFT in both backbones, but the gap falls to 0.3 on GPT after DPO and reverses on Qwen, where Spanish leads by 2.4. The authors note the two subsets share only 13 annotators and differ in texts and class balance as well as language.

Methodology in Plain English

The work proceeds in three stages.

1. Discovering perspectives. The authors take the EXIST 2024 dataset — 3,260 English tweets labeled by 348 annotators and 3,660 Spanish tweets labeled by 390 annotators, with exactly 6 annotators per tweet — and describe each annotator with three numbers: how often they say YES, how often they agree with the majority of the six annotators, and how balanced their YES/NO decisions are. They standardize these features and run K-Means separately per language. For English they select three clusters by silhouette score (0.43); for Spanish the silhouette score technically prefers two clusters (0.465 vs. 0.349 at three), but the authors choose three because the two-cluster solution merges annotators spanning 7% to 89% YES rate into one group, whereas three clusters reduce the mean deviation between an annotator's YES rate and its cluster mean from 0.099 to 0.059 and keeps every cluster at no less than 17% of annotators.

2. Training one agent per cluster. Each agent is fine-tuned in two stages. Phase 1 is supervised fine-tuning on its own cluster's majority labels, mixed with 30% shared positive examples (texts where all present clusters agree) and 20% team-alignment examples labeled with the annotator majority. Phase 2 is either DPO on preference pairs built from cluster-disagreement texts, or GRPO approximated through rejection sampling: sample K = 8 predictions per text, score them with a convex reward combining an individual term (did the agent match its own cluster?) and a team term (did the agent majority match the annotator majority?), keep the samples scoring at least ½, and re-fine-tune.

3. Evaluating two axes at once. Because a method could cheat by being faithful to clusters but useless as a team, or accurate as a team but unfaithful to clusters, the authors report Cluster F1 (per agent, against its cluster majority), Team F1 (agent majority against annotator majority), and Overall F1, the harmonic mean of the two, which cannot be inflated by trading one axis for the other. Data is split by annotator groups to prevent leakage.

Why This Matters

Impact on research. The paper challenges the default practice of collapsing annotator disagreement into a majority vote, and it provides a concrete demonstration that a multi-agent system can preserve perspectives without sacrificing team-level accuracy. The polarization finding is the more transferable result: a purely individual objective, even with a strong KL constraint, pushes models into caricature, and some shared anchor is required. That applies well beyond sexism detection to any task where the "ground truth" depends on who you ask.

Real-world applications.

• Content moderation triage, where agent disagreement can route genuinely contested posts to human reviewers instead of a single confident label.
• Annotation quality control, where training agents to mimic behavioral clusters can flag texts with predictably split human responses.
• Dataset design, where behavioral clustering offers a cheaper alternative to collecting and using demographic metadata about annotators.
• Model auditing for subjective tasks, where a set of perspective-faithful agents can expose how a single-label system systematically suppresses minority readings.

Industry relevance. Platforms that deploy automated moderation or recommendation systems must decide whose standards to encode. MAP-PO's design keeps the choice explicit — three named behavioral clusters rather than one averaged label — and its disagreement rate is offered as a confidence signal. The authors emphasize, however, that the agents inherit the EXIST annotator pool's composition (two languages, roughly 75% of annotators identifying as White/Caucasian), so deploying on a different population requires fresh validation.

Future Directions

• Transfer beyond EXIST 2024. The paper tests a single dataset and a single task, binary sexism detection in two languages from the same shared task. Whether behavioral clustering works on other subjective tasks, or on corpora without a perfect block design, remains open.

• Richer behavioral descriptors. Each annotator is summarized by only three features, and the authors note that label entropy is a deterministic function of the YES rate, so the space is effectively two-dimensional. Per-topic or temporal labeling patterns could separate perspectives that these three features merge.

• Handling sparse cluster representation. For Cluster 1 and Cluster 3, many texts have only one annotator from that cluster, so the "cluster majority" is a single person's opinion; the all-clusters-present evaluation subset is correspondingly small (47 non-tied texts in English, 111 in Spanish), giving wide confidence intervals.

• Closing the Qwen gap. GRPO trails SFT on the smaller backbone, which the authors attribute to noisier self-generated training signal. Making the team-reward stage pay off at smaller model scales is an open problem.

Target Audience

This paper is most useful to NLP researchers working on subjective or controversial content classification, annotator disagreement, and perspectivist modeling, as well as to researchers building multi-agent LLM systems and applying preference-optimization methods such as DPO and GRPO. It will also interest practitioners in content moderation and annotation-pipeline design who need to decide whether to average away disagreement or preserve it, and readers of prior work on the EXIST shared tasks who want to see disagreement modeled rather than discarded.

Authors’ abstract

When people label text for sexism, they often disagree, and not because some of them are wrong: they genuinely perceive sexism differently. Most NLP systems discard this disagreement by collapsing it into a majority vote. We propose the Multi-Agent Perspectivist Preference Optimization (MAP-PO) framework to keep these different perspectives. On the EXIST 2024 dataset of labeled English and Spanish tweets, we first cluster annotators by their labeling behavior rather than their demographic attributes. We then fine-tune one Large Language Model agent per cluster to reproduce that cluster's annotation behavior, and coordinate the agents with preference optimization that combines individual and team-level rewards. We evaluate MAP-PO in four settings defined by two languages and two backbone language models, asking whether each agent reproduces the annotations of its own cluster and whether the agents together reproduce the majority label. Two findings hold in all four settings. First, without fine-tuning the agents behave almost identically, so cluster-specific training is necessary. Second, we show that training each agent only on the labels of its own cluster pushes the agents far beyond the clusters they should represent, while adding a shared team-level training signal consistently keeps each agent calibrated to its cluster.