EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents

AI summary

Overview

Research area: Agentic AI security — system-level defenses (harnesses) for tool-using LLM agents, spanning indirect prompt injection and direct harmful requests.

Technical level: Advanced. The paper assumes familiarity with LLM agent loops (ReAct-style tool use), prompt injection threat models, and automated program/prompt search.

Scope: One-sentence scope: the paper proposes and evaluates a search framework that automatically synthesizes a distinct safety harness (a natural-language policy plus executable code logic) for each specific frozen model and target domain, rather than reusing one expert-designed defense everywhere.

What This Paper Is About

Existing agent safety harnesses — CaMeL, DRIFT, Progent, SafeHarness — are hand-built once by experts and then applied uniformly across very different models and application domains. The paper argues this is a mismatch: models differ in how much external enforcement they need before benign utility drops, and domains differ in which effects, state, and action sequences actually need governing (a filesystem policy says nothing about wash trading, and a transaction ledger says nothing about secret exfiltration). EvoSafeHarness addresses this by treating the choice of enforcement mechanisms and their thresholds as an optimization problem, searching a deployable harness around a frozen model in a target domain.

Key Contributions

1. A safety-specific harness optimization framework. EvoSafeHarness searches over a harness representation H = (P, C) — a natural-language policy P applied to the model's context and arbitrary executable code logic C valid under the application's adapter — for a frozen model M and domain D. It optimizes score = 100 · (U − ASR), which rewards security only when benign utility is preserved.

2. A four-component closed search loop. A Designer proposes and revises candidates using an explicit archive of prior source, scores, and per-task failure traces; a fresh-context Criticizer adversarially reviews each proposal; a staged Cascade Test Environment measures benign utility, direct attacks, and indirect attacks separately; and an Analyzer distills failed traces into experience. The archive, not hidden conversational memory, is the optimizer state.

3. Explicit defenses against three failure modes of security search — refuse-everything collapse (via separate benign evaluation), overfitting to benchmark artifacts such as filenames or recurring attack phrases (via a generalization mandate plus Criticizer review), and unsound synthesized enforcement code (via static and smoke gates before spending evaluation budget). Acceptance keys on attack invariants: provenance, scope, and intent mismatch.

4. A model × domain evaluation grid. Fifteen independently searched harnesses across five victim models and three DecodingTrust-Agent domains, plus results on Agent-SafetyBench, AgentDojo → AgentDyn (zero-shot transfer), and adaptive PAIR attacks via AgentCanary.

Main Findings

• DTAP grid, best score in 14 of 15 cells. Across the five-victim × three-domain grid, average ASR falls from 45.6% (no defense) to 10.0% at a 3.3-point utility cost (average utility 83.1% to 79.8%). Fixed baselines lag: CaMeL reaches 37.7% ASR at 69.7% utility, DRIFT 42.4% at 77.1%, and Progent nearly matches safety at 10.5% ASR but collapses utility to 56.4%.

• Per-channel breakdown on DTAP. Direct ASR falls from 50.9% to 12.6% and indirect ASR from 40.4% to 7.4%. The single cell where EvoSafeHarness does not win is Sonnet 4.6/finance, which is already at 2.9% ASR undefended. Each domain uses 30 benign, 35 direct, and 35 indirect tasks.

• The victim model is the largest single security variable. Undefended ASR ranges from 4.8% (Sonnet 4.6) to 71.0% (DeepSeek-V4-Flash), which the paper uses to argue that a defense fixed at design time cannot know which regime it will face.

• Statistical support. A paired McNemar test rejects no-change in 13 of 15 cells (the exceptions are two Sonnet cells already below 3%); pooled over the grid, 385 attacks are neutralised against 11 newly opened (p ≈ 2×10⁻⁷⁸); benign-utility intervals overlap in every cell.

• AgentDojo → AgentDyn. On AgentDojo, EvoSafeHarness reaches 82.8% mean utility at 0.0% ASR — described as twice the utility of CaMeL at the same zero-ASR operating point (CaMeL: 41.0% utility, 0.0% ASR). The same harness, with no re-search, transfers unchanged to unseen AgentDyn suites at 75.0% utility and 0.0% ASR, versus DRIFT at 38.9%/1.6% and Progent at 9.7%/2.9%.

• Agent-SafetyBench. Evaluated on a held-out split of 240 tasks, each run under six conditions (clean plus five attack channels). Mean over three victims: EvoSafeHarness posts the lowest unsafe-behaviour rate (12.3), the lowest attack success rate (7.4), and the highest utility under attack (63.4), compared with SafeHarness (18.9 / 10.5 / 44.8), Progent (20.6 / 14.1 / 49.0), DRIFT (24.5 / 16.4 / 61.6), CaMeL (23.3 / 15.9 / 61.1), and no defense (24.6 / 17.2 / 61.6). It attains the best score for every victim.

• Adaptive attacks. A frozen harness keeps mean ASR below 20% against three adaptive PAIR attackers with a refinement budget of 16 (reported in §6.3 of the paper; per-attacker numbers are not in the supplied content).

• Ablations. The security warm start, the fresh-context Criticizer, and the nested cascade each contribute measurably. Removing the Criticizer leaves the search score unchanged or higher while lowering held-out score by up to 31 points — direct evidence of benchmark-artifact overfitting.

• Why deployment-specific design helps (qualitative). Holding the domain fixed and varying the victim, Sonnet 4.6 reaches zero held-out ASR with a lightweight policy and two semantic checks, whereas GLM-5 benefits from deterministic gates, provenance state, and verdict caching. Holding GLM-5 fixed and varying the domain, filesystem safety depends on command effects, sensitive paths, and data flow, while finance must distinguish trades from money egress and retain transaction history because individually permissible actions can form a harmful sequence.

Methodology in Plain English

Each search starts from a domain specification that acts as the Designer's task contract: it pins down the frozen victim, backend, tools and judges; the direct and indirect threat semantics and which runtime context is trusted under each; the train-only evaluation cascade and score; the generalization mandate; and the adapter through which a defense may act. The specification states what must remain invariant and how candidates will be judged — not which defense architecture to build.

The Designer reads an archive of prior candidate source code, scores, Criticizer verdicts, and raw failure traces, then chooses a parent harness and applies a free-form edit: revise the policy, add trajectory state, add or remove a judge or gate, change control flow, or rewrite the code logic entirely. It may change one mechanism at a time so score changes stay interpretable. It never sees the held-out split. The archive is warm-started by distilling eight security design experiences from mature defenses (for example, that tool output must remain untrusted data, and that a tool call's effect must be checked against the trusted user request before execution), but these are guidance rather than a template — returned harnesses frequently contain mechanisms absent from the warm start.

Before scoring, a Criticizer in a clean, independent context is asked where the cheapest rephrase/rename/relocate evasion lies, whether any rule depends on a literal benchmark token or one domain's tool names, and whether the harness would protect an agent in an untuned domain. Flagged evasions are repaired before evaluation budget is spent.

Survivors enter a nested four-stage cascade that reuses a single ordered evidence stream: Stage 0 static import and structural check; Stage 1 a small smoke prefix that must complete without exceptions; Stage 2 a confidence gate that stops a candidate only when a stratified bootstrap's one-sided 95% upper bound places it more than δ score points below the no-defense reference; and Stage 3 extension to the full search sample, whose aggregate is the search score. Resource use is tracked at two levels — the outer optimizer (an agentic coding model that inspects the archive and edits code) and the inner evaluator (victim-model episodes) — because they use different models, tokenizers, and cache semantics.

For evaluation, the undefended agent is a bare loop, and every baseline (CaMeL, DRIFT, Progent, and SafeHarness on Agent-SafetyBench) is ported once and frozen across all victims and domains, while EvoSafeHarness is searched separately for each cell. Victims are: DTAP — Sonnet 4.6, GLM-5, Kimi-K2.5, Qwen3.7-plus, DeepSeek-V4-Flash; Agent-SafetyBench — DeepSeek-V3.2, Kimi-K2.6, GLM-5.2; AgentDojo/AgentDyn — MiMo-V2.5-Pro, DeepSeek-V4-Pro, GLM-5.2; AgentCanary — DeepSeek-V4-Flash driving the OpenClaw agent runtime. On AgentDojo and AgentDyn the attack family is important_instructions, a payload wrapped in authoritative-looking tags planted in a tool result.

Why This Matters

Impact on research. The paper reframes harness design from a one-shot architecture decision into a per-deployment search problem, and it demonstrates that adapting to a model and domain yields a strictly better safety–utility frontier than any fixed expert defense across four benchmark families. It also isolates the central hazard of automated defense search — rules that fire only because of a literal benchmark artifact — and shows the hazard is quantitatively large: removing the Criticizer raises the search score while dropping held-out score by up to 31 points.

Real-world applications:

• Filesystem and shell agents where a single mis-executed command can delete production data, establish persistence, or exfiltrate credentials — safety depends on command effects, sensitive paths, and downstream data flow.
• Financial agents where a sequence of individually permissible trades can collectively constitute wash trading, requiring destination constraints and a maintained transaction history rather than a per-action filter.
• Customer-service and telecom agents where harmful actions use ordinary in-scope tools, so tool-taxonomy-based defenses (CaMeL, DRIFT) provide no protection — the paper notes fixed defenses help most on os-filesystem, partially on finance, and not at all on telecom.
• Multi-tenant agent platforms that serve models of differing robustness behind a shared tool layer, where a harness strict enough for a weak model would over-block a strongly safety-trained one.

Industry relevance. The framework targets a practical gap: model-level safety training makes policy-compliant behavior more likely but does not provide a system-level enforcement boundary independent of model behavior. Because EvoSafeHarness is searched around a frozen model and ships as deployable code plus a policy, it fits teams that swap models or domains without retraining, and the zero-shot AgentDojo → AgentDyn transfer (75.0% utility, 0.0% ASR with no re-search) is evidence that a searched harness generalizes beyond the suites it was tuned on. Code and project pages are released at github.com/SaFo-Lab/EvoSafeHarness and andylinx.github.io/EvoSafeHarness.

Future Directions

• Beyond the three evaluated DTAP domains. DTAP's release contains fourteen domains, of which the cross-model grid evaluates three (os-filesystem, finance, telecom). Whether per-deployment search scales to the remaining domains, and to combinations of domains in a single agent, is open.

• Adaptive attackers with larger budgets and richer strategies. The paper reports mean ASR below 20% under three PAIR attackers with a refinement budget of 16. How the frontier shifts at higher budgets, or against attackers that target the searched harness's control flow rather than its prompts, is not established in the supplied content.

• Cost and search efficiency. The paper introduces two-level resource accounting (outer optimizer versus inner evaluator) and reports that the cascade yields measurable savings (§7.6), but the aggregate token, dollar, and wall-clock cost of searching fifteen harnesses is not reported in the available content — a practical prerequisite for adoption.

• Formal guarantees for synthesized enforcement code. The static and smoke gates catch import and runtime failures before budget is spent, but the authors themselves flag that an error skipping a check or failing open can create an untested path for unsafe tool execution. Verifying synthesized harness code, rather than stress-testing it, remains an open problem.

Target Audience

Researchers and practitioners working on agent security and safety, particularly those building or evaluating system-level defenses (harnesses, guardrails, capability policies, trajectory monitors) for tool-using LLM agents. It is also relevant to automated prompt/program optimization researchers (the DSPy, GEPA, Meta-Harness, NLAH line of work), since the paper shows what changes when the optimization objective becomes adversarial rather than task performance, and to ML platform and infrastructure engineers who must choose which models and tools to deploy together and need a defense that adapts per deployment. Readers unfamiliar with agent loops and prompt injection threat models will need background reading; the framing of the search procedure itself is accessible.

Authors’ abstract

Large Language Model (LLM) agents are turning language into real-world effects, making safety necessary against both indirect prompt injections and direct harmful requests. System-level safety harnesses add an enforcement layer beyond model-level defenses, but existing harnesses are usually designed once by experts and applied across heterogeneous models and domains. Effective protection is deployment-dependent: models differ in how much enforcement they need before utility declines, while domains differ in the effects, state, and action sequences that must be governed. A harness that is strict enough for one model may over-block another, and a policy that transfers across domains may miss application-specific safety relations. We present EvoSafeHarness, a safety-specific optimization framework that synthesizes a deployable harness for a frozen model in a target domain. It jointly searches a natural-language policy and executable code logic, guided by model behavior, domain specifications, and fresh-context adversarial review to reject benchmark-specific rules. Across four agent benchmark families, EvoSafeHarness achieves a stronger safety-utility frontier than fixed expert-designed defenses. On DecodingTrust-Agent, it reduces average attack success rate from 45.6% to 10.0% at a 3.3-point utility cost and achieves the best score in 14 of 15 cells. On AgentDojo, it reaches 82.8% utility at 0.0% ASR, twice CaMeL's utility at the same operating point, and transfers unchanged to unseen AgentDyn suites. It also achieves the best score on Agent-SafetyBench for every victim and keeps mean ASR below 20% under adaptive PAIR attacks with a refinement budget of 16. Analysis shows that domain semantics determine which safety relations and trajectory state are needed, while model and runtime behavior determine how and where those relations should be enforced.