Building Resilient Information Ecosystems: Large LLM-Generated Dataset of Persuasion Attacks

AI summary

Overview

Research area: AI safety and ethics, specifically the use of large language models to generate persuasive (adversarial) content targeting government and research agency communications.

Technical level: Intermediate. The paper is readable without a background in adversarial NLP, but it assumes familiarity with concepts such as persuasion taxonomies and Moral Foundations Theory.

Scope: The paper introduces and releases a dataset of 134,136 LLM-generated persuasion attacks written against 972 press releases from ten agencies, and analyzes the moral foundations those attacks resonate with.

What This Paper Is About

Government and commercial agencies publish news to build public trust, but generative AI models can produce persuasive counter-messaging at speed and scale, leaving agencies reactive and largely unaware of how those attacks are constructed. The authors generate a large, labeled dataset of adversarial persuasion attacks on real agency press releases using three LLMs, then analyze the moral values those attacks appeal to. The goal is to give organizations a resource for anticipating attack vectors and building more resilient communications.

Key Contributions

1. A large LLM-generated persuasion attack dataset containing 134,136 attacks, generated by GPT-4, Gemma 2, and Llama 3.1 against 972 press releases from ten agencies, covering 23 persuasive techniques from SemEval 2023 Task 3.
2. A dual-medium design: every attack was crafted in two forms, press release statements and social media posts, to represent both long-form and short-form communication strategies.
3. A domain and topic labeling of the 972 source articles using GPT-4o Mini 2024-07-18, with per-agency breakdowns of the top domains and topics.
4. A moral resonance analysis of the generated attacks using the moral foundations dictionary under Moral Foundations Theory, reported per technique, per model, and comparatively across models.

Main Findings

• GPT-4's moral profile: Averaged across all 23 techniques, GPT-4 attacks resonate most with Care (0.32), followed by Authority (0.26) and Loyalty (0.26), while Fairness (0.12) and Purity (0.04) are weaker. Flag Waving shows the strongest Care resonance (0.38), Appeal to Authority the strongest Authority resonance (0.42), Appeal to Time the strongest Loyalty resonance (0.40), Causal Oversimplification a notable Fairness resonance (0.28), and Obfuscation the highest Purity resonance (0.09).

• Gemma 2's moral profile: Gemma 2 averages Care (0.42), Authority (0.27), Loyalty (0.20), Fairness (0.08), and Purity (0.03). Exaggeration shows the strongest Care resonance of any technique in the study (0.62); Repetition shows the strongest Authority resonance (0.51); Appeal to Time aligns with Loyalty (0.35); Straw Man shows the strongest Fairness resonance (0.14); Questioning the Reputation shows the highest Purity resonance (0.05). Several attacks align more with Authority than Care, including Straw Man, Conversation Killer, Loaded Language, Repetition, Guilt by Association, and Appeal to Hypocrisy.

• Llama 3.1's moral profile: Llama 3.1 is the only model whose average top foundation is Loyalty (0.32), followed closely by Care (0.31) and Authority (0.24), with Fairness (0.09) and Purity (0.03) trailing. Appeal to Time has the highest Loyalty resonance across the study (0.53); Flag Waving the strongest Care resonance (0.39); Causal Oversimplification the greatest Fairness resonance (0.18); and Slogans the highest Authority resonance (0.30). Seven techniques additionally carry strong Authority resonance: Slogans, Conversation Killer, Loaded Language, Repetition, Doubt, Guilt by Association, and Appeal to Hypocrisy.

• Cross-model comparison: All three models emphasize Care, but to different degrees, with Gemma 2 highest (0.42) versus GPT-4 (0.32) and Llama 3.1 (0.31). Llama 3.1 places the strongest emphasis on Loyalty through Appeal to Time (0.53), compared with GPT-4's Appeal to Time (0.40) and Gemma 2's (0.35). For Authority, Gemma 2's Repetition leads (0.51), followed by GPT-4's Appeal to Authority (0.42) and Llama 3.1's Slogans (0.30).

• Source article labeling: The 972 articles came from ten agencies, with 100 articles each except IARPA, which had 72 at the time of collection. Defense dominated the domain labels for most agencies, reaching 99% for DoD, 98% for DIU, 97% for DARPA and DTRA, 90% for AFRL, 88% for ARL, 86% for NRL, and 61% for NGA. IARPA was led by Technology (79%) and SNL by R&D (68%).

Methodology in Plain English

The authors gathered 972 publicly available press releases from ten agencies, then used GPT-4o Mini to tag each article with domains and topics. They built a layered prompt template to instruct an LLM to write a critique of each article using one specific persuasive technique. The template combined a description of the attack fallacy, a persona, a perspective guideline, an argument guideline, and a writing style guideline. The exact prompt text is deliberately withheld on ethical grounds; only the structural overview is given.

Each of the three models, GPT-4 0613, Gemma 2 9B Instruct, and Llama 3.1 8B Instruct, generated 23 attacks per article, one per technique, in both a press release medium and a social media medium. All attacks were framed from a harmful perspective aimed at diminishing the article's messaging. Generation ran both locally and through API calls.

To study what the attacks appeal to, the authors applied Moral Foundations Theory, which identifies five foundations: Care, Fairness, Loyalty, Authority, and Purity. Using the moral foundations dictionary, they counted virtue words tied to each foundation in each attack and computed the proportion of words belonging to each foundation, producing a moral resonance score per technique, per model, and overall.

Why This Matters

Impact on research. Existing work has examined LLM persuasion broadly, but the authors argue there is a gap in detailed, model-specific analysis of which persuasive tactics different LLMs deploy, especially in government communications. This dataset links generated attacks to a named persuasion taxonomy and to moral foundations, giving researchers a concrete benchmark for studying adversarial messaging.

Real-world applications.

• Communications teams at agencies can study which moral appeals (Care, Authority, Loyalty) attacks tend to use against their own press releases and pre-test counter-messaging accordingly.
• Analysts monitoring information operations can use the labeled attacks as reference material when identifying similar tactics in the wild.
• Model developers and safety researchers can compare how GPT-4, Gemma 2, and Llama 3.1 differ in persuasion style when prompted adversarially.
• Evaluation pipelines can use the two-medium design (press release versus social post) to test detection or mitigation tools across long-form and short-form content.

Industry relevance. The authors frame the dataset as a way to build "reputation armor" and to shift organizations from a reactive posture to proactive defense, anticipating attack vectors from observed moral resonance patterns. Distribution Statement A indicates the work is approved for public release with unlimited distribution.

Future Directions

• Extending the analysis of moral resonance into broader socio-emotional-cognitive signatures, such as subjectivity, toxicity, emotions, attitudes, and intent, as the authors suggest.
• Investigating how model-inherent biases shape the moral framing and technique selection of generated attacks, and how much these vary by topic or agency.
• Testing whether moral resonance scores correspond to actual persuasiveness with human audiences, since the paper notes that resonance may not always match real-world outcomes.
• Exploring how context variables, such as the tone of the targeted press release, influence the form the generated attacks take.

Target Audience

Government and defense communications and public affairs staff, information operations and influence-detection analysts, AI safety and alignment researchers studying persuasion and adversarial generation, and NLP researchers who work with persuasion taxonomies or Moral Foundations Theory. Readers looking for a ready-made dataset with per-technique and per-model breakdowns will get the most from it; readers seeking human-persuasion outcome data will not find it here, since the paper analyzes generated text and moral word ratios rather than measured audience effects.

Authors’ abstract

Organization's communication is essential for public trust, but the rise of generative AI models has introduced significant challenges by generating persuasive content that can form competing narratives with official messages from government and commercial organizations at speed and scale. This has left agencies in a reactive position, often unaware of how these models construct their persuasive strategies, making it more difficult to sustain communication effectiveness. In this paper, we introduce a large LLM-generated persuasion attack dataset, which includes 134,136 attacks generated by GPT-4, Gemma 2, and Llama 3.1 on agency news. These attacks span 23 persuasive techniques from SemEval 2023 Task 3, directed toward 972 press releases from ten agencies. The generated attacks come in two mediums, press release statements and social media posts, covering both long-form and short-form communication strategies. We analyzed the moral resonance of these persuasion attacks to understand their attack vectors. GPT-4's attacks mainly focus on Care, with Authority and Loyalty also playing a role. Gemma 2 emphasizes Care and Authority, while Llama 3.1 centers on Loyalty and Care. Analyzing LLM-generated persuasive attacks across models will enable proactive defense, allow to create the reputation armor for organizations, and propel the development of both effective and resilient communications in the information ecosystem.