Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations

AI summary

Overview

Research area: Computational social science at the intersection of Large Language Model (LLM) agent-based modeling, online content moderation, and toxicity research.

Technical level: Advanced. The paper assumes familiarity with agent-based modeling, LLM prompting, toxicity classifiers, and distributional/statistical comparison measures, though its core argument is accessible.

One-sentence scope: The paper introduces COSMOS, an LLM-powered simulator of Online Social Network (OSN) conversations that runs paired factual and counterfactual simulations to measure how much content moderation strategies actually change toxic behavior.

What This Paper Is About

Evaluating whether online moderation actually works is hard: private platforms restrict API access, toxic behavior is sparse in real data, and field observation gives researchers no control over confounding variables. The authors argue that generating empirical evidence through simulation, rather than collecting it from the real world, minimizes cost while maximizing experimental control. To that end they build COSMOS, a simulator in which LLM-based agents with demographic and psychological profiles post and comment in an OSN-like environment, with a parallel counterfactual simulation where moderation is applied and everything else is held constant.

Key Contributions

1. A counterfactual moderation simulator (COSMOS). The system runs two parallel simulations per experiment — a factual feed where agents act freely and a counterfactual feed replicating the same behavior with only moderation interventions added — enabling direct measurement of moderation's effects.
2. A novel use of agent memory modules as the interface for ex ante moderation. Memory modules carry moderation messages that persist and influence an agent's future counterfactual outputs, while a separate sensory module carries environmental context.
3. An empirical use case comparing moderation strategies. The authors implement One-Size-Fits-All (OSFA), Personalized Moderation Interventions in Neutral, Empathizing and Prescriptive variants (PMI-N, PMI-E, PMI-P), and BAN with tolerance values e ∈ {1, 2, 4, 8}.
4. Custom evaluation measures for moderation. They define mass divergence ΔM, quantile divergence Δq, and the Content Loss Ratio (CLR = 1 − |V̂| / |V|), assessed for significance via the Mann-Whitney U-test.

Main Findings

• Simulated toxic behavior is psychologically believable and consistent. Aggregating factual data from all simulations, the authors find significant Spearman correlations (p-value < 0.01) between toxicity and Big Five traits that mirror correlations in real data (PANDORA): agreeableness (real ρ = −0.18, simulated ρ = −0.32), conscientiousness (real ρ = −0.11, simulated ρ = −0.16), and neuroticism (real ρ = 0.04, simulated ρ = 0.07). Agent-level consistency shows real average σ = 0.17 versus simulated average σ = 0.20.

• Toxicity propagates across threads. Parent and children node toxicity correlates significantly in the factual feeds (ρ = +0.39, p-value = 0.0). In the sub-population run with the top-5 toxic agents and the least toxic one, agents (including the anomalous one) significantly increased their toxicity relative to their full-population behavior.

• Personalized moderation is the more effective ex ante strategy. PMI-N brought significant reductions in most runs (1, 3, 4, 5), with ΔM values of −0.09***, −0.08*, −0.11*, −0.11*. OSFA and PMI-P yielded lower reductions on average. A t-SNE projection of BERT-encoded ex ante messages shows PMI-N exploring wider regions of the semantic space.

• Low-tolerance bans produce deplatforming effects rather than redirection. BAN-e delivered large negative ΔM proportionally to e (e.g., −0.54*** to −0.52** at e = 1), but by removing agents rather than changing their behavior. At e = 1 the CLR was 0.45 ± 0.04, including a fraction 0.27 ± 0.03 of "healthy" contents with toxicity below THR. The macro-average recall for identifying content worth losing was 0.55 at e = 1, 0.60 at e = 2, and 0.58 at e = 4.

• Moderation outcomes depend on psychological traits. When ΔM is computed over subsets of agents sharing a trait, all moderation strategies follow similar trends, but only PMIs and BAN-e with e ≤ 4 produce significant divergences. Moderation successfully targets prototypical toxic agents (low agreeableness, high neuroticism, low conscientiousness).

• Moderation mostly affects extreme toxicity. Quantile divergence Δq averaged across runs shows effects concentrated at q ≥ 0.8, with varying effects in the 0.6 ≤ q ≤ 0.8 range. BAN-e with e ≤ 4 shows a bimodal trend, with significant reductions also for milder toxic behavior.

Methodology in Plain English

The environment is a news feed represented as a directed graph, plus a counterfactual twin of that feed. Each LLM-backed agent has a profile module (demographics and Big Five personality traits), a sensory module that reads the environment, and a memory module that holds moderation messages.

At each of n timestamps, agents are shuffled and each samples an action from a fixed probability distribution: post, comment, or do nothing. Posting picks a random topic; commenting selects a node via a temperature-scaled softmax (τ = 3) that favors recent nodes, with agents forbidden from replying to their own nodes or replying twice to the same node. The LLM then generates a factual post or comment, and the same prompt — filled with the counterfactual sensory and memory modules — generates the counterfactual output, both conditioned on a common seed to limit stochastic decoding effects.

A toxicity detector scores the counterfactual output against a threshold (THR = 0.6). If it exceeds the threshold, the violation counter increments and moderation activates: BAN blocks the agent from future generations once violations exceed tolerance e; OSFA writes a default message into counterfactual memory; PMI writes a message generated by instructing the LLM to impersonate a moderator. Because downstream agents read the counterfactual feed, moderation effects can cascade to nodes with no memory of moderation at all.

Profiles were built by combining demographic attributes (age, gender, race, income, education, sex orientation, political leaning) from the General Social Survey with psychological traits from PANDORA (15M comments from 10k Reddit users), using stratified sampling over discretized Big Five scores to produce 25 profiles, enriched with 5 outliers detected via Isolation Forest. The LLM is an uncensored version of SOLAR-10B, selected for its lowest perplexity on a sample of ground-truth OSN data (PANDORA) versus other tested LLMs; the toxicity detector is Google's Perspective API, returning a score from 0 to 1. Prompt templates were selected by comparing generated toxicity distributions with the ground-truth PANDORA distribution: cal_tox was chosen for the lowest Kullback-Leibler divergence (KL_no_tox = 1.37, KL_yes_tox = 0.57, KL_cal_tox = 0.07). Decoding used top-k = 50, temperature 0.8, nucleus sampling p = 1.0. Simulations used n = 50 timestamps and action probabilities post: 0.5, comment: 0.5, do_nothing: 0; the sub-population run used n = 250.

Why This Matters

Impact on research: The paper positions simulation as a complement to — not a replacement for — field observation. COSMOS offers a controllable testbed for hypotheses in the social sciences (for example, about toxicity contagion or the role of personality in moderation response) at a fraction of the cost of collecting and curating real platform data, and the authors state it is the first simulator of its kind, so no established benchmarks, direct competitors, or baseline methods exist for comparison.

Real-world applications:

• Testing automated moderation systems before deployment, including comparing message-based (ex ante) nudges against banning (ex post).
• Designing personalized warning or intervention messages targeted to a user's socio-psychological profile.
• Exploring the trade-off between toxicity reduction and content loss when setting ban tolerances.
• Studying how moderation effects propagate through conversation threads to users who were never moderated themselves.

Industry relevance: Platforms weighing moderation policies face exactly the trade-off the paper quantifies — BAN at e = 1 produced the biggest toxicity reduction but removed 0.45 ± 0.04 of content, of which 0.27 ± 0.03 was not toxic. The simulated ban classifier's recall for content "worth losing" (0.55 at e = 1) resembles a random classifier, which is a concrete argument against naive low-tolerance banning. The authors explicitly encourage mindful industrial use, noting that automated moderation is still in its infancy and full replacement of human moderators remains controversial.

Future Directions

• Improve content validation and reduce hallucinations. The authors applied 2-means clustering on BERT representations of generated posts and comments and found one cluster of redundant hallucinations accounting for approximately 7% of content. They call for more LLM tuning and reliable validation.
• Add subjective, human-based realism evaluation. The paper acknowledges that human assessments — ideally with psychologists or sociologists — are needed, including checking alignment between simulated and real-world responses to moderation interventions.
• Extend the simulator beyond conversations. Planned extensions include followings and likes, which would enable more advanced recommendation systems based on social connections and agent preferences, and potentially the emergence of homophily and polarization that could influence moderation outcomes.
• Improve scalability and address bias. The authors plan client-server architectures and more efficient models to scale toward real-sized OSN populations, and leave the question of LLM societal bias — plus the unclear extent of self-preference bias in role-playing settings — to future work.

Target Audience

Researchers in computational social science and agent-based modeling; NLP and AI researchers working on LLM-based social simulation; trust-and-safety practitioners and platform policy analysts interested in quantifying moderation trade-offs; and social scientists studying online toxicity, contagion, and personality effects, who can use the open-source code (https://github.com/gfidone/COSMOS) as a testbed.

Authors’ abstract

Online Social Networks (OSNs) widely adopt content moderation to mitigate the spread of abusive and toxic discourse. Nonetheless, the real effectiveness of moderation interventions remains unclear due to the high cost of data collection and limited experimental control. The latest developments in Natural Language Processing pave the way for a new evaluation approach. Large Language Models (LLMs) can be successfully leveraged to enhance Agent-Based Modeling and simulate human-like social behavior with unprecedented degree of believability. Yet, existing tools do not support simulation-based evaluation of moderation strategies. We fill this gap by designing a LLM-powered simulator of OSN conversations enabling a parallel, counterfactual simulation where toxic behavior is influenced by moderation interventions, keeping all else equal. We conduct extensive experiments, unveiling the psychological realism of OSN agents, the emergence of social contagion phenomena and the superior effectiveness of personalized moderation strategies.