AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

AI summary

Overview

Research area: LLM agent safety, adversarial reinforcement learning, and multi-turn tool-use robustness.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO, SFT, clipped surrogate objectives), tool-using LLM agents, and prompt-injection threat models.

Scope: The paper introduces AgentSafeEvo, an attacker–defender co-evolutionary RL framework in which an attacker constructs adversarial agent tasks (user queries plus tool environments) while a defender learns to complete those tasks safely, with learnability-guided task weighting steering both policies' updates.

Authors and affiliations: Chenxu Wang, Xiaoyu Wen, Han Qi, Ziyuan Zhou, Chaochao Lu, Qiaosheng Zhang (Shanghai Artificial Intelligence Laboratory, with Fudan University and Shanghai Jiaotong University) and Cheng Jin (Zhejiang University). Published as arXiv:2610.19422v1 [cs.AI] on 14 Oct 2026.

What This Paper Is About

LLM agents that call external tools over multiple turns can be manipulated by malicious content placed not only in user queries but also in tool descriptions and environment observations, causing unsafe actions during execution. Prior adversarial co-evolution work addresses language-level safety or indirect prompt injection inside pre-supplied web tasks, but leaves the construction of adversarial tasks themselves outside the co-evolutionary loop. AgentSafeEvo aims to co-evolve adversarial task construction with safe task execution across diverse tasks and attack types, improving robustness without sacrificing task performance.

Key Contributions

1. AgentSafeEvo, a co-evolutionary framework for LLM agent safety. Beyond conventional attacker–defender games, it co-evolves adversarial agent tasks across diverse attack types, jointly optimizing an attacker model and a defender agent over multi-turn tool-interaction trajectories.
2. Learnability-guided task weighting. A method that uses within-task defender-reward variability (the within-task standard deviation of defender rewards) to adjust each task's contribution to both attacker and defender policy updates.
3. An agent safety training dataset spanning 50 agent scenarios and approximately 7,000 tools, covering a broad range of attack types. It includes adversarial tasks pairing user queries with tool-environment configurations for attacker initialization, plus seed instances for co-evolution.
4. Extensive experiments on agent safety and general capability benchmarks showing improved adversarial robustness and general task performance, with analyses showing that both attack effectiveness and defensive robustness improve during co-evolution and that attack improvements generalize to substantially larger target models.

Main Findings

• Best safety results across four benchmarks. AgentSafeEvo achieves the best mean performance on all reported metrics across Agent-SafetyBench, AgentHarm, AgentDojo, and MCP Security Bench (MSB) for Qwen3-8B. Agent-SafetyBench ASR falls from 31.01% (Static-AT, the strongest safety baseline) to 17.84%, and MSB ASR falls from 38.64% to 31.25%.
• Large harm-score reduction on AgentHarm. The harm score drops from 30.00 under Static-AT to 6.53 under AgentSafeEvo.
• Better task completion under attack. On AgentDojo, Utility under Attack (UA) rises from 61.43% (Static-AT) to 66.52% while ASR stays low at 6.18%. On MSB, Net Resilient Performance (NRP) rises from 40.64 to 46.00.
• General capability gains, not just safety alignment. AgentSafeEvo attains the highest accuracy on ALFWorld (43.72% vs 30.89% for Static-AT) and ToolHop (48.04% vs 39.43%). BFCL-V3 accuracy is 65.98%, compared with 66.13% for the original Qwen3-8B and 66.73% for Static-AT, indicating largely preserved function-calling accuracy.
• Both sides improve during co-evolution. Using 100 seed instances held out from training, later-stage attackers achieve higher ASR against every defender and later-stage defenders reduce ASR against every attacker. Against the base defender, ASR rises from 55.81% (SFT attacker) to 92.59% (final attacker); against that final attacker, ASR falls from 92.59% (base defender) to 49.82% (final defender).
• Attack improvements generalize to other target models. Compared with the SFT attacker, the evolved attacker (co-evolved against Qwen3-8B) achieves higher ASR on all four additional target models, for example from 43.7% to 80.7% on Llama3.3-70B and from 12.3% to 22.5% on GPT-5.
• Co-evolution beats defender-only training. Comparing the "w/o attacker updates" variant (Static-AT) with unweighted co-evolution, ASR decreases by 8.10 and 5.75 percentage points on Agent-SafetyBench and MSB, and ALFWorld accuracy rises from 30.89% to 36.70%. The authors argue the frozen attacker already generates tasks online, so online task generation alone does not explain the gains.
• Shared task weighting outperforms unweighted co-evolution on all reported metrics. Agent-SafetyBench ASR falls from 22.91% to 17.84% and MSB NRP rises from 41.48 to 46.00, while ALFWorld accuracy rises from 36.70% to 43.72%.
• One-sided weighting lowers ASR but hurts general capability. Weighting only the attacker or only the defender reduces ALFWorld accuracy from 36.70% (unweighted) to 29.05% and 34.25% respectively. Both one-sided variants achieve lower ASR on Agent-SafetyBench than AgentSafeEvo (14.13% when defender weighting is removed, 15.52% when attacker weighting is removed, versus 17.84%), and removing defender weighting also yields lower MSB ASR (28.47% vs 31.25%). AgentSafeEvo nonetheless achieves the highest MSB NRP and the highest accuracy on both general capability benchmarks among the evaluated variants.
• Metric definitions are benchmark-specific. Agent-SafetyBench uses ASR; AgentHarm uses the harm score; AgentDojo uses ASR and UA; MSB uses ASR and NRP. General capability is measured by accuracy on BFCL-V3, ALFWorld, and ToolHop. Reported values are the mean and standard deviation over three independent runs.

Methodology in Plain English

The framework models agent safety as a two-player game between an attacker model (π_A) and a defender agent (π_D), both built on Qwen3-8B.

Threat model. The work covers six attack categories — direct prompt injection, indirect prompt injection, tool-description injection, misinformation, harmful tool execution, and user-query jailbreak — described in the appendix as six categories comprising 13 subtypes.

Roles. Given an agent scenario, a seed intent, and environment information, the attacker constructs an "agent task": a user query plus tool definitions and environment observations. Tasks containing malicious content are adversarial agent tasks. The defender takes an agent task, interacts with available tools over multiple turns, and produces a final answer; the recorded sequence of tool calls, observations, and the final answer is an interaction trajectory.

Stage 1: Red-team attacker initialization. To overcome the cold-start problem (an untrained attacker cannot reliably produce valid adversarial tasks), the authors sample approximately 20,000 source tasks from TOUCAN, a tool-agent dataset with over 1.5 million trajectories synthesized from 495 real-world MCP servers. Gemini 3 Flash Preview transforms approximately 10,000 of them into adversarial tasks per specified attack intents, with a construction trace synthesized for each example. These are combined with approximately 10,000 benign examples for supervised fine-tuning of the attacker.

Stage 2: Joint adversarial training with GRPO. For each seed, the attacker generates M agent tasks and the defender produces K trajectories per task. A reward model (Qwen3.8-27B) judges each trajectory twice: a safety reward of +1 if safe and −1 if unsafe, and a utility reward of +1 if the legitimate requirements are fulfilled and −1 otherwise. Progress toward the harmful objective is not counted as fulfilling legitimate requirements, and for entirely harmful requests a safe refusal receives +1 utility. The defender reward is the sum of the two, so the maximum is reached only by remaining safe while completing legitimate requirements, discouraging both harmful compliance and indiscriminate refusal. The attacker reward is the negative of the mean defender reward over the K trajectories, making task-level rewards zero-sum.

Learnability-guided task weighting. Because the attacker reward reflects the mean defender reward, a task can score high for the attacker even when all K defender trajectories get the same low reward, leaving their GRPO advantages at zero; meanwhile the defender's within-task normalization largely removes reward-scale differences across tasks. The authors therefore weight each task by its within-task defender-reward standard deviation, normalized within the M tasks from the same seed: w_i = 1 + λ (σ_D^(i) − mean σ_D) / max_i′ |σ_D^(i′) − mean σ_D|. If the denominator is zero, w_i is set to 1 for that seed group. The same weight multiplies both the attacker's task-level advantage and the defender's trajectory-level advantage after their respective GRPO normalizations. When a group's reward standard deviation is zero, all advantages in that group are set to zero. Both policies are updated with the same clipped GRPO surrogate, with a KL penalty to the respective reference policy; user inputs, tool outputs, and environment observations remain in the interaction context but are excluded from the loss.

Comparison setup. For the RL phase, 6,000 red-team and 6,000 benign seed instances are synthesized from sampled TOUCAN tasks. Baselines are the original Qwen3-8B, the prompt-based defense Spotlighting with Delimiting, the detector-based defense Prompt Guard 2, and Static Adversarial Training (Static-AT), which keeps the SFT attacker fixed and updates only the defender. For ALFWorld, the authors follow the AgentBench-FC setting described in AgentRL.

Why This Matters

The paper shifts adversarial co-evolution for safety from prompt–response exchanges to the construction and execution of full agent tasks, where risk arises from tool definitions, environment observations, and multi-turn decisions rather than from a single response. It also shows that the choice of task-weighting scheme governs a tradeoff between attack resistance and robust task performance: variants that minimize ASR can degrade general agent capability.

Real-world applications include:

• Tool-using personal and enterprise assistants that search, process documents, and manage schedules, where injected content in tool descriptions or observations can redirect execution.
• Model Context Protocol (MCP) ecosystems, the setting of MSB and of the training data's real-world MCP servers, where tool signatures, parameters, and responses are attack surfaces.
• Connector and plugin marketplaces, where third-party tool definitions must be treated as untrusted input.
• Regulated domains represented in the training data, such as finance and healthcare scenarios, where an unsafe tool action carries direct consequence.

Industry relevance: the framework targets the practical problem of deploying agents with broad tool access without losing task performance, and its robustness gains transfer to larger target models (Llama3.3-70B and GPT-5 in the reported cross-model tests), which matters for vendors who defend with one model while attackers target another. The authors also flag dual-use risk: an attacker trained to find agent weaknesses could be repurposed for malicious prompts or tool manipulation, and they withhold the red-team data-synthesis pipeline for this reason.

Future Directions

• Extending to collaborative multi-agent settings, explicitly named as future work, with focus on safety risks arising from communication and task delegation.
• Resolving the robustness-versus-capability tradeoff. One-sided task weighting achieved lower ASR than AgentSafeEvo on Agent-SafetyBench (and, for attacker-only weighting, on MSB at 28.47% vs 31.25%), so the question of which weighting objective should be optimized when ASR and Net Resilient Performance diverge remains open.
• Establishing safety beyond benchmark performance. The ethics statement notes that improvements on the evaluated benchmarks do not establish safety in all deployment settings, and that practical deployment requires further assessment of the tools, permissions, and environments available to the agent.
• Managing dual-use and release policy. The authors withhold the red-team data-synthesis pipeline, prompts, and implementation details, raising the question of how attacker models and adversarial data can be shared for defensive research without enabling misuse.

Target Audience

Researchers and engineers working on LLM agent safety, adversarial training, and reinforcement learning from verifiable rewards; safety teams evaluating tool-using agents and MCP-based integrations; and practitioners who need robustness against prompt injection and tool-manipulation attacks without sacrificing agent capability. Readers without a background in RL or multi-turn agent evaluation will find the framework-level ideas accessible, but the method sections assume advanced familiarity.

Authors’ abstract

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.