Truth with a Twist: The Rhetoric of Persuasion in Professional vs. Community-Authored Fact-Checks

AI summary

Overview

Research area: Natural Language Processing / computational social science — specifically the rhetoric of fact-checking, persuasion-technique detection, and crowdsourced content moderation (Community Notes).

Technical level: Intermediate. The paper uses standard non-parametric statistics (Spearman correlation, Mann–Whitney U, Cliff's delta, one-sided proportion z-tests) and a pretrained multilingual persuasion-technique classifier, but all measures are explained plainly.

Scope: A large-scale comparison of persuasion techniques in crowd-authored Community Notes versus two professionally authored debunk corpora, plus an analysis of how the crowd rates the helpfulness of persuasive notes.

What This Paper Is About

Researchers and commentators have argued that volunteer-written Community Notes on X may be less objective than professional fact-checks, relying more on emotionally charged or manipulative wording. No study had systematically tested this assumption. The paper measures how much persuasive rhetoric appears in crowd-written versus professional debunks, and whether the crowd's own rating mechanism rewards or punishes that rhetoric.

Key Contributions

1. First large-scale comparison of persuasion techniques in crowd-written versus professionally written debunks, using Community Notes (2,016,841 debunks, extracted August 10, 2025), EUvsDisinfo (8,522 English debunks), and the Database of Known Fakes (DBKF; 681,346 debunks).
2. Direct test of the prior hypothesis that community debunks are more persuasive than professional ones — the paper finds no evidence for it at the aggregate level.
3. Per-technique rhetorical profiling, showing which specific techniques are more or less prevalent in CNs relative to each professional corpus, and tying those differences to institutional/editorial context.
4. Evidence on rater behaviour, quantifying the correlation between a note's persuasiveness and its helpfulness ratings, and identifying which individual techniques crowd raters penalise versus reward.

Main Findings

• No aggregate difference in persuasiveness: Mann–Whitney U tests comparing the average number of persuasion techniques per debunk between CNs and both EUvsDisinfo and DBKF were insignificant, with p-value = 1.00 for both comparisons. The authors state there is no evidence that CNs use a greater number of persuasion techniques than professional fact-checks.

• Persuasiveness correlates positively, but negligibly, with helpfulness: The Spearman correlation between the number of persuasion techniques in a note and its average helpfulness rating was statistically significant but extremely small (ρ ≈ 0.039, p-value < 10⁻⁴), and in the positive direction — contrary to the authors' hypothesis, more persuasive notes were rated slightly more helpful on average.

• Techniques with the largest positive association with helpfulness (Cliff's δ): Consequential Oversimplification (δ = 0.255), Causal Oversimplification (δ = 0.115), Appeal to Hypocrisy (δ = 0.114), and Appeal to Time (δ = 0.1053).

• Techniques with the strongest negative association with helpfulness: Flag Waving (δ = −0.142), Guilt by Association (δ = −0.088), and Straw Man (δ = −0.068).

• Most techniques have negligible practical effect: The remaining techniques showed |δ| < 0.05 despite highly significant p-values driven by the large sample size. Two techniques — Appeal to Authority (p = 0.064, δ = −0.005) and False Dilemma / No Choice (p = 0.227, δ = 0.005) — were not statistically significant with negligible effect sizes.

• CNs versus DBKF: Only a small set of techniques was significantly more prevalent in CNs — Conversation Killer (z = 45.36), Repetition (z = 137.26), False Dilemma-No Choice (z = 22.85), and Whataboutism (z = 6.98), all with p < 0.001. For most other techniques, prevalence was lower in CNs, contrary to the initial hypothesis.

• CNs versus EUvsDisinfo: A larger set of techniques was significantly more prevalent in CNs — Appeal to Hypocrisy (z = 16.04), Appeal to Popularity (z = 5.18), Appeal to Values (z = 12.85), Consequential Oversimplification (z = 6.22), Conversation Killer (z = 19.08), Doubt (z = 3.97, p = 0.00004), Obfuscation-Vagueness-Confusion (z = 12.04), Questioning the Reputation (z = 15.41), Red Herring (z = 4.84), Straw Man (z = 2.14, p = 0.016), and Whataboutism (z = 4.86), all with p < 0.001 unless stated. The authors attribute this to the stricter format and domain specificity of EUvsDisinfo debunks.

• Crowd raters penalise specific rhetoric: Notes using Flag Waving, Guilt by Association, or Straw Man received lower helpfulness ratings, even though the cumulative number of techniques in a note was not penalised. The authors suggest the Straw Man reaction may be a case of the "Note Not Needed" (NNN) flagging phenomenon, whereby users label notes as irrelevant to the post.

Methodology in Plain English

The researchers gathered three corpora of debunks: the full daily Community Notes set from X, EUvsDisinfo's pro-Kremlin-disinformation debunks, and the Database of Known Fakes. For CNs they removed notes that classify a post as "Not Misinformation", so that the comparison matched the professional datasets, which only publish debunks of misinformation claims.

They ran every debunk through a state-of-the-art multilingual persuasion-technique classifier (Wu et al., 2023), keeping only predictions above a 0.8 confidence threshold. To keep the computation consistent, all texts were truncated to a maximum of 512 tokens. The full technique list and definitions are in the paper's supplementary material.

To answer the first research question — whether the crowd can identify and penalise persuasive language — they mapped each Community Note rating (Not Helpful = 1, Somewhat Helpful = 2, Helpful = 3), averaged ratings per note, and measured the relationship between a note's number of persuasion techniques and its average helpfulness using the Spearman correlation coefficient. For each individual technique they compared helpfulness distributions between notes that used it and notes that did not, using the Mann–Whitney U test, and reported the direction and magnitude of the effect via Cliff's delta.

To answer the second question — whether crowd debunks are more persuasive — they first compared the average number of techniques per note across datasets with the Mann–Whitney U test. They then ran a separate test for each technique, stating directional hypotheses that CNs would have higher prevalence than DBKF (H₁) and than EUvsDisinfo (H₂), against a null of equal or lower prevalence, using one-sided proportions z-tests. The code and supplementary material are openly available on GitHub.

Why This Matters

Impact on research. The paper directly tests and challenges a common assumption — that crowdsourced fact-checks are more subjective and rhetorically manipulative than professional ones — and finds no aggregate support for it. It reframes the debate away from a simple "crowd versus professional" dichotomy toward institutional context: EUvsDisinfo operates in a narrow geopolitical niche under stringent editorial oversight, while DBKF spans a far broader thematic and organisational space, and these differences are visible in the rhetoric itself.

Real-world applications:

• Content-moderation platform design: Insights into which rhetorical techniques create the most cross-ideological friction can inform Community Notes guidelines and contributor training.
• Rater interface and flagging systems: The finding that raters penalise Flag Waving, Guilt by Association, and Straw Man connects directly to the "Note Not Needed" flagging phenomenon, suggesting where interfaces could surface those signals explicitly.
• Fact-checking organisation strategy: Professional outlets can see how their editorial norms and topical specialization shape the rhetorical profile of their output relative to crowdsourced alternatives.
• Persuasion-detection tooling: The per-technique prevalence tables provide a benchmark of which techniques actually appear in real-world debunk corpora across 56–72 languages.

Industry relevance. The work speaks to trust-and-safety teams at social platforms, fact-checking organisations, and builders of multilingual misinformation-detection tooling — anyone who needs to reason about whether community-driven and professional moderation convey comparable rhetorical discipline.

Future Directions

• Distinguishing institutional context from authorship: Because CNs behave differently against DBKF and EUvsDisinfo, future work could disentangle how much rhetorical variation comes from crowd-versus-professional authorship versus organizational aims, editorial norms, and the type of misinformation being countered.
• Why the cumulative effect is invisible to raters: Contributors penalise specific techniques but not the overall accumulation of persuasion techniques. Understanding this gap could improve note-quality signalling.
• The Straw Man and NNN connection: The authors propose that penalising Straw Man may be a particular case of "Note Not Needed" flagging; this link is offered as an interpretation and invites direct investigation.
• Language and sample coverage: CNs span 72 languages and DBKF 56, while EUvsDisinfo is English-only. The paper reports distributions in supplementary material, and per-language analyses are a natural extension.

Target Audience

This paper is most valuable to researchers in NLP, computational social science, and misinformation studies; trust-and-safety and content-moderation practitioners at social platforms, especially those working with Community Notes or comparable crowdsourced systems; fact-checking professionals and journalism researchers interested in editorial norms and rhetorical style; and graduate students looking for a clear methodological template combining large-scale persuasion detection with non-parametric statistical analysis.

Authors’ abstract

This study presents the first large-scale comparison of persuasion techniques present in crowd- versus professionally-written debunks. Using extensive datasets from Community Notes (CNs), EUvsDisinfo, and the Database of Known Fakes (DBKF), we quantify the prevalence and types of persuasion techniques across these fact-checking ecosystems. Contrary to prior hypothesis that community-produced debunks rely more heavily on subjective or persuasive wording, we find no evidence that CNs contain a higher average number of persuasion techniques than professional fact-checks. We additionally identify systematic rhetorical differences between CNs and professional debunking efforts, reflecting differences in institutional norms and topical coverage. Finally, we examine how the crowd evaluates persuasive language in CNs and show that, although notes with more persuasive elements receive slightly higher overall helpfulness ratings, crowd raters are effective at penalising the use of particular problematic rhetorical means