SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment

AI summary

Overview

Research area: Safety alignment for LLM-based tool-using agents, combining runtime harness engineering with reinforcement learning policy optimization.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning (SFT), reinforcement learning from verifier rewards, group-relative advantage estimation, KL-regularized policy objectives, and prompt-injection threat models.

Scope: SafeEvolve is a self-evolving framework that couples iterative updates to an agent's external execution harness (safety prompts and a hierarchical safety skill bank) with two-stage SFT-RL policy training, using completed on-policy trajectories as shared evidence for both.

What This Paper Is About

LLM agents built for multi-step tool use can fail safely in two distinct ways: they may comply with harmful user requests, or they may be hijacked by adversarial instructions hidden in webpages, files, or tool outputs. Existing defenses split into two camps — editing the external harness (instructions, skills, guardrails) while the model stays frozen, or updating policy parameters while the harness stays fixed — and neither alone reliably links runtime control to intrinsic model safety. SafeEvolve's goal is to make the harness and the policy evolve together in a continual loop, so that safety experience from actual rollouts is both externalized into auditable, versioned harness artifacts and internalized into the model's behavior.

Key Contributions

1. SafeEvolve framework. An experience-driven self-evolving framework for agent safety alignment that turns completed on-policy trajectories into safety capability through a continuous loop coupling harness refinement with policy optimization.

2. Two interleaved evolution processes. Harness refinement converts trajectory-level safety evidence into bounded, versioned, auditable and reversible artifact updates across a safety prompt and hierarchical skills; policy optimization uses a two-stage SFT-RL paradigm (harness-use SFT cold start followed by harness-augmented RL) to apply the evolved harness during multi-step execution.

3. Verifier-decomposed, task-typed rewards. Instead of a binary success label, SafeEvolve defines reward primitives from rule-based verifier feedback — a utility score, a safety score, and execution-invalidity penalties — and composes them differently for clean tasks, malicious-query tasks, and environment-injection tasks.

4. Empirical safety-utility tradeoff. Experiments across two backbones and three agentic safety benchmarks show SafeEvolve achieving a stronger safety-utility tradeoff than generic post-training baselines and agentic safety alignment baselines.

Main Findings

• Best safety-utility tradeoff on Qwen3.5-4B. SafeEvolve reduces AgentDojo ASR from 2.37% (base) to 0.79% while raising clean utility from 59.79% to 61.86%. On AgentHarm it cuts the harmful score from 56.45 to 12.27 and raises refusal from 28.98% to 83.83%. The paper describes this as a 3x ASR reduction on AgentDojo with slightly higher benign utility. Note this is the one place where the benign AgentHarm score falls: SafeEvolve records 71.31 versus the base 83.09.

• Second backbone, larger safety gains but lower benign AgentHarm score. On Qwen3-4B-Instruct-2507, SafeEvolve reaches AgentDojo utility 60.82, utility-under-attack 52.05, ASR 2.42; AgentDyn utility 25.00, utility-under-attack 15.14, ASR 4.87; AgentHarm harmful 15.47, refusal 71.93, benign 63.43. The base values were 44.33 / 35.91 / 13.38, 20.00 / 21.38 / 19.60, and 34.96 / 7.39 / 43.57 respectively.

• Generic post-training baselines are narrower. SFT and DPO reduce injection ASR but give limited defense against malicious queries. GRPO improves some safety metrics at heavy cost to tool-use utility — on Qwen3.5-4B its AgentDojo utility drops to 30.93 and utility-under-attack to 26.48, and its AgentHarm harmful score rises to 63.82.

• Safety alignment baselines expose their own tradeoff. AgentAlign attains very low AgentHarm harmful score on Qwen3-4B-Instruct-2507 (2.85) with refusal 87.50, but its AgentHarm benign score collapses to 15.84 and its AgentDojo utility to 18.56.

• Harness evolution helps even with a frozen policy. With the policy frozen on Qwen3.5-4B, evolved skills give AgentDojo utility 64.95 / utility-under-attack 60.72 / ASR 0.92, and AgentHarm harmful 16.80 / refusal 76.97 / benign 77.46. Evolved prompts give AgentDojo 60.82 / 58.17 / 1.27 and AgentHarm 43.49 / 48.85 / 83.88 versus the base 59.79 / 60.04 / 2.37 and 56.45 / 28.98 / 83.09. Evolved skills are described as the more reliable safety gain across backbones.

• Policy-only RL is unstable. On Qwen3.5-4B, model-only optimization with RL lowers AgentDojo utility under attack from 60.04 to 26.48 and raises the AgentHarm harmful score from 56.45 to 63.82. Co-evolution with evolved prompt lowers the harmful score to 33.88 but reduces AgentDojo utility under attack to 41.97. Co-evolution with evolved skills retains 56.77 utility under attack while reducing AgentDojo ASR from 2.37 to 0.79 and harmful score to 12.27 — better than harness-only evolution on both axes (harmful 16.80 to 12.27; ASR 0.92 to 0.79).

• Skill bank grows in a controlled way. The active skill bank grows from 26 to 47 entries, mainly through task-specific and common-mistake skills while general skills remain fixed. Accepted edits resolve ambiguous booking IDs and referenced deletion targets, add missing tool arguments, use exact targets from nested sources, or avoid fabricated update fields.

• Retrieval quality matters more than having a skill bank. Ablation on the default agent: default retrieval gives AgentDojo utility-under-attack 60.72 / ASR 0.92, AgentDyn 17.06 / 4.38, AgentHarm harmful 16.80 / refusal 76.97. Removing skill-bank retrieval reverts to the base numbers (60.04 / 2.37, 15.45 / 6.88, 56.45 / 28.98). General-skills-only lowers AgentDojo ASR to 0.74 but sacrifices AgentDojo utility (56.19) and is weaker on AgentHarm (harmful 30.59, refusal 65.71). Dropping dynamic-skill priority barely changes injection ASR (0.90) but raises harmful compliance to 32.03 and lowers refusal to 64.00.

• Evolution is not tied to one proposer model. Using GPT-5.5, DeepSeek-Chat, and GLM-5.1 as harness proposers, all proposer-evolved harnesses move away from the base point toward lower attack or harmful-compliance rates while preserving or improving the paired utility axis. AgentHarm movement is especially consistent; prompt-injection benchmarks show larger proposer-dependent variation.

• Cross-policy harness transfer is uneven. Harnesses evolved on Qwen3.5-4B transferred without further training are weakest on the 1.7B target, where they do not repair near-zero AgentDyn utility; strongest on the 4B target; and on the 8B target safety improves across benchmarks but attacked utility can decline.

• Not reported in the provided content. The exact values of the reward weights λ_U, λ_S, and λ_US are stated to be reported in Appendix A.5, which is not included in the supplied text. The case-study trajectory excerpt is also truncated mid-episode.

Methodology in Plain English

Setting. An agent is formalized as a policy (the language model) plus a harness: instructions, available tools, retrieved skills, permission and action policies, instruction-priority and information-flow constraints, and a rendering/execution controller. Tools and environment interfaces stay fixed; the evolvable safety-relevant components are mainly the safety prompt and the retrieved safety skills.

Where the experience comes from. Tasks are run in lightweight finite-state Python simulators generated with LLMs. Each task specifies a user request, an environment state, callable tools, and verifier metadata. The suite covers clean tasks, environment-injection tasks (benign goal, adversarial observations), and malicious-query tasks (the user request itself is harmful).

Harness evolution. The frozen policy rolls out under the current harness, producing trajectories annotated with success/failure buckets and metadata (domain, scenario, attack type, tool family, task type). A proposer picks exactly one harness component and generates a bounded mutation, changing only that component. The parent and candidate harnesses are then evaluated on the same task and environment set. A candidate is accepted only if its average task-typed return meets the parent's plus a margin δ and a gate function does not reject it for regressing safety, clean-task utility, or execution quality. Accepted edits are kept as versioned component changes with supporting evidence and rollback metadata.

Policy optimization, stage one. Harness-use SFT is a short cold start. Rollouts are collected under the evolved skill bank, only trajectories passing both safety and utility verifiers are kept, and training is applied only to assistant responses and tool calls, with system instructions, user messages, rendered skills, and tool observations serving as context. This teaches the policy when a retrieved skill is relevant, how to execute it, and how to ignore irrelevant skill context.

Policy optimization, stage two. Harness-augmented RL runs under the evolved harness. Because a rollout reveals whether the benign task was completed, whether the specified risk succeeded, and whether tool calls were valid, the reward is decomposed: utility score for clean tasks; safety score for malicious queries; and a weighted combination of utility, safety, and their product for injection tasks, since injection requires both ignoring the injected instruction and preserving the original goal. Trajectories are sampled in groups, group-relative advantages are computed, and the policy is updated with a clipped objective plus a KL penalty against a fixed reference policy.

The loop. Each rollout batch is generated under an explicit policy-harness pair, so behavioral changes can be attributed to policy updates, harness updates, or their interaction. The current batch updates the policy while recent rollout evidence (a sliding window of rounds) drives the next harness revision.

Experimental setup. Backbones are Qwen3.5-4B and Qwen3-4B-Instruct-2507. Each run trains for 200 rollout steps; each rollout batch samples 32 tasks with 8 rollouts per task, yielding 256 trajectories per update. Maximum prompt length is 4096 tokens and the learning rate is 1×10⁻⁶. The skill bank contains general skills, task-specific skills, and common mistakes. Baselines are SFT, DPO, GRPO (trained with the same environment reward), MetaSecAlign, and AgentAlign.

Why This Matters

Research impact. The paper reframes agent safety alignment as a co-evolution problem rather than a choice between external guardrails and parameter updates. It provides a concrete mechanism — bounded, single-component, gate-validated harness edits plus harness-conditioned RL — and shows empirically that the two paths are complementary rather than substitutable. The finding that co-evolution outperforms both harness-only and policy-only evolution is a directly testable claim for follow-up work, as is the observation that evolved harnesses transfer unevenly across policy scales.

Real-world applications.

• Enterprise assistants that read email, files, and web pages on a user's behalf, where injected instructions in retrieved content are a live risk.
• Coding and DevOps agents that execute tool calls against repositories and infrastructure, where unsafe or invalid actions are costly.
• Customer-facing booking, purchasing, and scheduling agents handling requests that may themselves be manipulative or policy-violating.
• Any deployed agent fleet that needs auditable, reversible safety configuration changes rather than opaque retraining cycles — the versioned, single-component, rollback-capable harness edits are designed for exactly this operational requirement.

Industry relevance. The framework targets on-policy deployment data that practitioners already generate, and produces harness artifacts with traceable evidence and rollback metadata, which fits change-management and compliance workflows. The cross-policy transfer results also matter for organizations running several model sizes: a single evolved harness can be reused, though the paper shows gains depend on the target policy and can trade off utility.

Future Directions

1. Closing the cross-policy compatibility gap. Transfer is weakest at the 1.7B target, where evolved guidance does not repair near-zero AgentDyn utility. Determining what policy capability is required to execute evolved harness guidance — and whether harness evolution can be conditioned on the target policy from the start — is an open question the paper explicitly raises.

2. Characterizing proposer dependence. Prompt-injection benchmarks show larger variation across GPT-5.5, DeepSeek-Chat, and GLM-5.1 proposers than AgentHarm does. The paper does not report a principled way to select or ensemble proposers, nor whether proposer quality bounds the achievable safety-utility frontier.

3. Sustaining benign performance alongside safety. The AgentHarm benign score falls on both backbones under SafeEvolve (to 71.31 on Qwen3.5-4B and 63.43 on Qwen3-4B-Instruct-2507). Whether this reflects necessary conservatism or an addressable limitation of the reward composition is not resolved.

4. Scaling and stress-testing the evolution loop. Runs use 200 rollout steps, 32 tasks per batch, 8 rollouts per task, and the skill bank grows to 47 entries. How the accept/reject gate behaves over much longer horizons, larger skill banks, and more adversarial environments — and when the gate should reject rather than accumulate — remains unexplored in the reported content.

Target Audience

Researchers and engineers working on LLM agent safety, agentic reinforcement learning, and tool-use evaluation will get the most from this paper, particularly those already familiar with GRPO-style policy optimization and prompt-injection threat models. It is also relevant to applied teams that need auditable, reversible safety configuration for deployed agents, and to benchmark designers interested in how safety and utility trade off under indirect prompt injection versus malicious user queries. Readers without a background in RL fine-tuning will find the method sections demanding, though the motivation and findings are legible at a higher level.

Authors’ abstract

The performance of LLM-based agents is jointly shaped by the base model and the harness used when interacting with the environment. This exposes them to safety risks in both harmful final responses and multi-step execution trajectories. Existing safety alignment mechanisms often rely on either external harness updates or policy optimization, yet applying either paradigm in isolation fails to bridge runtime control with intrinsic safety. We propose SafeEvolve, an experience-driven self-evolving framework for agent safety alignment. SafeEvolve leverages safety experience from completed on-policy trajectories to drive a continual loop of harness-policy co-evolution. On the harness side, SafeEvolve converts trajectory-level safety evidence into bounded, component-level updates across safety prompt and hierarchical skills, yielding auditable and reversible harness artifacts. On the policy side, SafeEvolve follows a two-stage SFT-RL paradigm, where harness-use SFT bootstraps the policy to actively leverage evolved harness artifacts, and harness-augmented RL further shapes autonomous safety behaviors during multi-step exploration via verifier-decomposed rewards. Through harness-policy co-evolution, SafeEvolve converts safety experience into an evolved runtime harness and improved policy behavior. Experiments on agentic safety benchmarks show that SafeEvolve achieves a stronger safety-utility tradeoff than existing baselines. For Qwen3.5-4B, SafeEvolve achieves a $3\times$ ASR reduction on AgentDojo while improving benign utility from 59.79% to 61.86%.