From Insight to Exploit: Leveraging LLM Collaboration for Adaptive Adversarial Text Generation

AI summary

Overview

• Research area: Adversarial robustness of large language models, specifically zero-shot prompt-based text classifiers used for sensitive content moderation tasks.
• Technical level: Intermediate. Readers should be comfortable with LLM prompting, zero-shot classification, and basic adversarial-example concepts, but no specialized mathematics is required.
• Scope: The paper introduces and evaluates two automated, LLM-driven pipelines — Static Deceptor (StaDec) and Dynamic Deceptor (DyDec) — that generate natural-looking adversarial text to make target LLMs misclassify spam, hate speech, toxic comments, and fake news.

What This Paper Is About

Zero-shot LLMs are increasingly used as classifiers for sensitive tasks such as spam and hate speech detection, but their robustness against adversarial inputs is not well understood. Prior attacks either rely on prompt injection that is obvious to humans and detectable by LLMs, or on static, unguided perturbations (typos, random characters, synonyms) that modern LLMs resist. This paper asks whether an LLM's own reasoning and generation abilities can be used to craft subtle, adaptive, task-prompt-independent adversarial examples that reliably fool a target LLM.

Key Contributions

1. Two fully automated attack pipelines that use LLMs' inherent capabilities rather than external heuristics or manual tuning: Dynamic Deceptor (DyDec), which uses a Reasoning LLM, a Red LLM, an Attacking LLM, a Similarity Checker LLM, and the Target LLM, and Static Deceptor (StaDec), which folds the reasoning and red-team roles into the Attacking LLM itself.
2. Adversarial examples that deceive the target LLM while remaining semantically similar to the original input and appearing natural, without dependence on knowing the target task prompt.
3. Demonstrations of strong transferability: adversarial examples crafted against one model also fool models not involved in the attack design.
4. Evaluation of three existing defense families against the attacks, plus a discussion of potential mitigation via stronger safety alignment.

Main Findings

• PromptAttack is largely ineffective: The LLM-based PromptAttack of Xu et al. (2023) achieved attack success rates of 0.00%–1.15% across all five dataset configurations and both state-of-the-art target LLMs in Table 1.
• Prompt injection works unevenly: CombinedAttack (Liu et al. 2024) reached 100.00% on Hate Speech and LIAR with Llama-3-70B, but only 0.00%–5.81% on GPT-4o across most tasks; the paper attributes this to GPT-4o's instruction hierarchy training versus Llama-3-70B treating system and user instructions as a single stream.
• Both new attacks work across tasks and models: For Llama-3-70B, DyDec/StaDec reached 63.54%/73.96% (SMS Spam), 98.88%/89.66% (Hate Speech), 86.00%/60.00% (Toxic Comment), 100.00%/100.00% (LIAR), and 61.28%/75.42% (Spam Detection). For GPT-4o, they reached 51.51%/25.25%, 96.81%/92.00%, 75.00%/75.00%, 65.48%/63.10%, and 47.36%/44.06% respectively.
• Black-box attack remains effective: With GPT-4o as the attacking LLM and Llama-3-70B as an unknown target, DyDec achieved 53.13% (SMS Spam), 93.10% (Hate Speech), 53.00% (Toxic Comment), 60.87% (LIAR), and 47.81% (Spam Detection).
• Strong transferability: Successful adversarial examples transferred at 81.96% and 86.27% (SMS Spam), 100.00% and 100.00% (Hate Speech), 90.90% and 68.49% (Toxic Comment), 100.00% and 72.00% (LIAR), and 98.90% and 99.30% (Spam Detection) for Llama-3-70B and GPT-4o respectively, despite neither target being involved in the design process.
• Early iterations already succeed: Figure 5 shows a notable attack success rate in the early rounds, even before any feedback iterations, and Figure 6 shows substantial overlap in which samples different attacks successfully fool, suggesting some samples are inherently vulnerable.
• StaDec is cheaper but lower quality: DyDec cost $0.070 per input on GPT-4o and 0.117 GPU-hours per input on Llama-3-70B (3 × NVIDIA RTX A6000 GPUs); StaDec cost $0.015 per input on GPT-4o and 0.025 GPU-hours on Llama-3-70B. Manual inspection found StaDec's Llama-based examples were sometimes much shorter fragments that still received high similarity scores, prompting the authors to recommend GPT-4o as the Similarity Checker.
• Perplexity defenses fail: At a 1% false positive rate, PPL-based detection (Table 4) and windowed PPL detection (Table 5) showed high false negative rates against DyDec and StaDec in almost all cases, because the adversarial text is natural-sounding. Windowed PPL did partially catch CombinedAttack on Llama-3-70B for SMS Spam (37.83% FNR) and Spam Detection (7.00% FNR).
• LLM-based detection catches injections but not the new attacks: CombinedAttack produced 0.00% FNR and 0.00% FPR, while DyDec and StaDec showed high FNR and non-negligible FPR (for example, GPT-4o FPR of 26.02% and 31.58% on Toxic Comment).
• Paraphrasing mitigation is limited: Mitigation rates were generally low, though higher for CombinedAttack on Llama-3-70B Spam Detection (88.50%) and LIAR (55.07%). Paraphrasing was less effective for hateful and toxic content because LLMs tend to neutralize such intent during paraphrasing.
• Similarity thresholds: The similarity threshold was set to 6 for Llama-3-70B and 7 for GPT-4o based on score distributions; up to 8 feedback iterations were run per attack, after which an unsuccessful attack was marked as failed.

Methodology in Plain English

The attack starts by feeding an input the target LLM already classifies correctly and asking why it made that prediction. In DyDec, a Reasoning LLM explains the factors behind the prediction, a Red LLM turns that explanation into dynamic instructions for attacking those specific factors, and an Attacking LLM rewrites the text accordingly. A Similarity Checker LLM scores how close the rewrite is to the original; if the rewrite fails to flip the target's prediction or falls below the similarity threshold, feedback goes back to the Reasoning and Red LLMs and the loop repeats. StaDec removes the Reasoning and Red LLMs and gives the Attacking LLM static instructions plus the same feedback each round, making it lighter but less deliberate.

The authors tested on four sensitive tasks: SMS Spam, Hateful or Offensive Speech Detection, Toxic Comment Classification, and Fake News Detection (LIAR). They used GPT-4o and Llama-3-70B for both classification and attack, added a black-box setting (GPT-4o attacks, Llama-3-70B unknown target), and applied saved adversarial examples to other models to test transferability. They also compared against PromptAttack and CombinedAttack, and evaluated PPL detection, windowed PPL detection, an LLM-based detector, and paraphrasing defense.

Why This Matters

This work reframes adversarial text generation as a self-assessment tool: instead of hand-designed perturbation rules, the same class of model being evaluated is used to probe its own weaknesses. The paper argues the pipeline improves as LLMs improve, which is the opposite of static attacks that decay as models generalize better. It also shows that prompt-injection attacks are easy to detect, while fluent, meaning-preserving perturbation is not.

• Content moderation: Platform operators can use these pipelines to find spam, hate speech, or toxic comments that would slip past automated filters before adversaries do.
• Spam and phishing filtering: SMS and email filters that rely on zero-shot LLM classification can be stress-tested against messages rewritten to evade detection.
• Fake news and misinformation screening: The LIAR results (100.00% ASR for both attacks against Llama-3-70B) show how factual-claim classifiers can be manipulated while retaining apparent semantic similarity.
• Model deployment risk assessment: Since the attacks transfer to models not involved in the design process, vendors can use them to estimate risk for a target model they cannot inspect internally.

Industry relevance follows directly: the black-box setting matches how real attackers operate, the cost figures ($0.070 and $0.015 per input) make the attack economically realistic, and the failure of PPL and LLM-based detectors indicates that current commercial guardrail approaches are insufficient.

Future Directions

• Extending the attack to a broader range of LLM architectures and to multi-modal systems, as the authors propose.
• Developing robust defense mechanisms, since existing defenses perform poorly against these attacks.
• Exploring real-world applications where the pipeline dynamically analyzes and refines content to match specific criteria.
• Improving reliability of the automated pipeline itself, particularly reducing Similarity Checker LLM errors and addressing neutralizing effects such as omitted contact details in spam, less aggressive hate speech, and reliance on fictional scenarios in fake news.

Target Audience

Researchers and practitioners working on LLM security, adversarial robustness, and content moderation; machine learning engineers deploying zero-shot LLM classifiers in sensitive domains; and policy or trust-and-safety teams who need to understand the practical limits of current detection and defense mechanisms.

Authors’ abstract

LLMs can provide substantial zero-shot performance on diverse tasks using a simple task prompt, eliminating the need for training or fine-tuning. However, when applying these models to sensitive tasks, it is crucial to thoroughly assess their robustness against adversarial inputs. In this work, we introduce Static Deceptor (StaDec) and Dynamic Deceptor (DyDec), two innovative attack frameworks designed to systematically generate dynamic and adaptive adversarial examples by leveraging the understanding of the LLMs. We produce subtle and natural-looking adversarial inputs that preserve semantic similarity to the original text while effectively deceiving the target LLM. By utilizing an automated, LLM-driven pipeline, we eliminate the dependence on external heuristics. Our attacks evolve with the advancements in LLMs and demonstrate strong transferability across models unknown to the attacker. Overall, this work provides a systematic approach for the self-assessment of an LLM's robustness. We release our code and data at https://github.com/Shukti042/AdversarialExample.