LLM-Based Adversarial Persuasion Attacks on Fact-Checking Systems

AI summary

Overview

Research area: Natural Language Processing, specifically adversarial robustness of automated fact-checking (AFC) systems and computational analysis of persuasion/propaganda techniques.

Technical level: Intermediate. The paper's concepts (claim verification, evidence retrieval, attack success rate) are explained clearly, but familiarity with fact-checking benchmarks and adversarial evaluation helps.

Scope: The paper introduces "persuasion injection attacks," an adversarial framework in which a generative LLM rewrites claims using persuasion techniques to degrade both the veracity classification and evidence retrieval stages of automated fact-checking pipelines, evaluated on the FEVER and FEVEROUS benchmarks.

What This Paper Is About

Automated fact-checking systems are used to detect false claims, but they can be attacked by small changes to the claims they process. Prior adversarial work has focused on surface noise (typos, synonyms) or semantic edits, but nobody had tested whether the rhetorical tricks used in real disinformation campaigns, such as manipulation, simplification, and distraction, could be weaponised against these systems.

This paper asks whether an LLM can rephrase a claim using persuasion techniques while keeping its original truth label intact, and whether that rephrasing makes AFC systems fail. The attack is tested separately on the veracity classifier (given gold evidence, and with the claim alone) and on the evidence retriever (sparse BM25 and dense Contriever), so the two failure modes can be told apart.

Key Contributions

1. Persuasion injection attacks: a new class of adversarial attack that exploits persuasion techniques, rather than noise or semantic edits, to induce failures in AFC systems.
2. A decoupled evaluation strategy that separates evidence retrieval from veracity classification, so the attack's effect on each stage can be measured independently.
3. Identification of which persuasion techniques are most damaging, with Manipulative Wording techniques (notably Obfuscation) found to degrade retrieval and classification simultaneously, leading to complete pipeline failure.
4. Open release of the attack code (github.com/joaoaleite/persuasion-injection-attack) to support future work on strengthening AFC systems.

Main Findings

• Persuasion beats prior attacks by more than double. The paper reports that persuasion injection reduces accuracy more than twice as much as previously studied attacks such as synonym substitutions and character perturbations.

• Baselines cause only modest damage. On FEVER, baseline adversarial attacks reduce claim-only accuracy by at most 9 points and evidence-based accuracy by at most 10 points; on FEVEROUS, accuracy drops stay below 4 points in most settings.

• Blind persuasion causes large drops. Under a Blind attacker (random technique), claim-only accuracy drops by up to 13.2 points on FEVER and 9.9 points on FEVEROUS, while gold-evidence accuracy falls by up to 17.9 points on FEVER and 16.8 points on FEVEROUS.

• Oracle persuasion collapses performance. Under an Oracle attacker (worst-case variant per claim), claim-only accuracy falls to at most 0.043 on FEVER and 0.044 on FEVEROUS. Even with gold evidence, accuracy remains between 0.127 and 0.164 on FEVER and between 0.216 and 0.354 on FEVEROUS.

• All three verifiers are similarly vulnerable. Average accuracy drops under Blind attack are RoBERTa −0.136, Llama-3-8B −0.133 and Mistral-7B −0.133; under Oracle attack, RoBERTa −0.706, Llama-3-8B −0.666 and Mistral-7B −0.719.

• Evidence helps against evasion, but not much against a smart attacker. Blind Evasion ASR is above 30% in claim-only models, dropping to 7% (FEVER) and 4% (FEVEROUS) with gold evidence; the Oracle attacker still forces evasion at 75.9% on FEVER and 43.4% on FEVEROUS, over a 10x increase relative to Blind.

• Sabotage is worse with evidence. Blind Sabotage ASR rises from below 22% in claim-only models to above 44% for gold-evidence models; Oracle Sabotage ASR stays near 100% in both settings.

• Obfuscation is the single most damaging technique. It is an outlier in both degradation types, with an average Evasion ASR of 27% and |ΔRecall@5| of 0.54 on BM25 and 0.44 on Contriever. It works by replacing concrete entities with vague generalisations (for example "50 days" becomes "approximately one and a half months", "Thanksgiving" becomes "holiday season"), which denies the retriever matching keywords and introduces ambiguity for the classifier.

• Effective techniques span three categories. Techniques passing the evasion threshold (Evasion ASR ≥ 0.05 with gold evidence in both datasets) include Obfuscation and Repetition (Manipulative Wording), Slogan (Call), and Flag Waving and Appeal to Popularity (Justification). Attack on Reputation and Distraction techniques collapse to near-zero evasion rates when gold evidence is available.

• Retrieval is degraded too. The strongest lexical baseline (Character Noise) drops BM25 Recall@5 by 0.106 points on FEVER and 0.024 on FEVEROUS versus baseline. Blind persuasion degrades BM25 Recall@5 by 0.174 on FEVER and 0.159 on FEVEROUS, and Oracle persuasion collapses Recall@k to near-zero for all k in {3, 5, 7, 10}.

• The effect transfers to dense retrieval. Blind persuasion degrades Contriever Recall@5 by 0.080 on FEVER and 0.121 on FEVEROUS, and the Oracle attacker collapses Recall@5 to at most 0.090 across all four dataset–retriever combinations. Technique rankings correlate between retrievers (Spearman ρ=0.825 on FEVER and ρ=0.886 on FEVEROUS; p<2×10⁻⁴).

• Not every technique harms both stages. Repetition has the second-highest evasion rate (17.3%) but reduces BM25 Recall@5 by only 0.06 and its Contriever impact nearly vanishes (0.02). Appeal to Values (7.9% evasion) and Slogan (12.3% evasion) harm classification and retrieval, with |ΔRecall@5| of 0.29 on BM25 for Appeal to Values.

• Validation shows the attacks preserve labels. In a study annotating 690 persuasive claims (30 per technique, stratified by dataset), 8 techniques with label preservation rates ≤80% were excluded. Among the 15 retained techniques, less than 1.5% of adversarial claims flipped the original label, and attacks add fewer than 1.5 tokens per claim on average.

Methodology in Plain English

The researchers start from a taxonomy of 23 persuasion techniques grouped into 6 high-level categories (Attack on Reputation, Justification, Distraction, Simplification, Call, and Manipulative Wording), drawn from computational social science work on propaganda. The abstract describes the final experimental set as 15 techniques grouped into 5 categories; the introduction states 15 of the 23 techniques were retained after validation.

For each claim, an LLM attacker (Qwen2.5-7B-Instruct) rewrites the claim using one persuasion technique at a time. The prompts include the technique definition, the original claim, formatting and label-preservation constraints, and two few-shot examples. This produces a set of variants per claim.

Because persuasion changes emphasis and framing rather than factual content, the authors do not require strict semantic equivalence. They note that automated similarity metrics like BERTScore are poor filters here (a Slogan variant scoring 0.45 BERTScore still kept its False label). Instead they validate label-invariance: the persuasive variant must keep the same True/False label. Human annotation on the 690-claim sample determined which techniques to keep.

Two attacker models are tested. The Blind attacker applies a random technique (expected degradation); the Oracle attacker has query access to the classifier and picks the variant most likely to cause an error (worst case).

Classification is evaluated on RoBERTa-Base plus two decoder-only LLMs (Llama-3-8B-Instruct and Mistral-7B-Instruct), in a claim-only setting and a gold-evidence setting, using Accuracy, Macro F1, ROC AUC, and Attack Success Rate split into Evasion (False to True) and Sabotage (True to False). Six baselines are compared: Synonym Substitution, Word Swap, Character Perturbations, Back-Translation, Paraphrase, and Informal Style Transfer. Retrieval is measured with Recall@k on BM25 (sparse) and Contriever (dense), both indexed over Wikipedia snapshots. All attacks are applied only to the held-out test sets.

Why This Matters

Impact on research. The paper reframes robustness evaluation for fact-checking: instead of treating attacks as noise injection, it treats rhetoric itself as the attack surface. It also shows the failure is not confined to one model family or one retrieval paradigm, which weakens the assumption that evidence grounding alone solves adversarial vulnerability.

Real-world applications:

• Fact-checking organisations and platforms can use these findings to stress-test their verification pipelines before deployment.
• Social media platforms can treat vague, rhetorically loaded reformulations of false claims as a distinct moderation signal, not just a wording variant.
• Retrieval system developers can prioritise robustness to entity obfuscation and vagueness in query understanding.
• Policy and media-literacy work can point to concrete evidence that rhetoric alone can defeat automated verification.

Industry relevance. AFC systems are deployed at scale, and the paper shows that a 7-billion-parameter open-weight model is enough to mount the attack, with the same class of openly available models used by both defenders and attackers. The result that dense retrieval mitigates but does not defend against worst-case persuasion has direct implications for anyone choosing or procuring retrieval infrastructure.

Future Directions

• Building defences against persuasion attacks, with particular focus on Manipulative Wording techniques, which the authors name as the priority.
• Extending evaluation to cross-encoder and late-interaction retrievers such as ColBERTv2, since sensitivity profiles may differ from BM25 and Contriever.
• Testing whether the vulnerability holds in other languages and domains, since the study is restricted to English and Wikipedia-based benchmarks, and social media or news genres may behave differently.
• Investigating how attacker scale changes the results, since the adversarial variants came from models of around 7B parameters and more capable generators might make currently weak techniques stronger.

Target Audience

Researchers and practitioners working on automated fact-checking, adversarial robustness in NLP, disinformation detection, and information retrieval. The paper is also relevant to platform integrity teams and fact-checking organisations who need to understand how persuasive reformulations can defeat automated verification, and to anyone studying how persuasion and propaganda techniques operate in computational settings.

Authors’ abstract

Automated fact-checking (AFC) systems are susceptible to adversarial attacks, enabling false claims to evade detection. Existing adversarial frameworks typically rely on injecting noise or altering semantics, yet no existing framework exploits the adversarial potential of persuasion techniques against AFC systems, which are widely used in disinformation campaigns to manipulate audiences. In this paper, we introduce a novel class of persuasive adversarial attacks on AFCs by employing an LLM to rephrase claims using persuasion techniques. Considering $15$ techniques grouped into $5$ categories, we study the effects of persuasion on both claim verification and evidence retrieval using a decoupled evaluation strategy. Experiments on the FEVER and FEVEROUS benchmarks show that persuasion attacks can substantially degrade both verification performance and evidence retrieval. Our analysis identifies persuasion techniques as a potent class of adversarial attacks, highlighting the need for more robust AFC systems.