SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agents

AI summary

Overview

Research area: AI agent security (cs.CR) — specifically the safety and adversarial robustness of language-model agents that execute tools against real environments.

Technical level: Advanced. The paper formalizes agentic safety as a partially observed control problem and evaluates learned attack and defense methods against four frontier target models.

Scope: The paper defines a shared "state-grounded execution" framework (SEAD) for attacks against tool-using agents, builds an attacker (DART) and a pre-execution defender (SAGE) from that framework, and evaluates them on an environment-verifiable benchmark built from MT-AgentRisk tasks.

What This Paper Is About

When an LLM agent uses tools, each executed action changes the environment — files, permissions, database records — so a later routine-looking action can become harmful because of state left behind by earlier ones. The defender sees only a partial view of that state before deciding whether a pending action should run, while the attacker supplies instructions but does not control which concrete actions the agent actually takes. The paper's goal is to derive attack and defense requirements from this single shared execution loop, rather than treating red-teaming and runtime guarding as separate problems, and then to build and measure methods that satisfy those requirements.

Key Contributions

1. A formal state-control view of agentic safety (SEAD). The paper models the attacker, target agent, and defender as roles inside one partially observed execution loop with a state-transition function, a tool-observation function, and a pre-execution gate that returns PASS or BLOCK. It defines the "harm-enabling" transition as the first action that moves the state from goal-unmet to goal-achieved, and defines state ambiguity as the case where two states consistent with the defender's evidence call for opposite decisions.

2. DART, a feedback-conditioned attack. DART (Decomposed Attacks with Runtime Feedback and Tree Search) performs tree search over executed target trajectories, decomposing harmful goals into locally plausible intermediate steps, proposing multiple alternative instructions from the same prefix, and adapting using observed tool execution. Each node's value combines an executable progress check, an introspection score, and an execution-feedback score (weights 0.20 / 0.30 / 0.50).

3. SAGE, a state-aware pre-execution defender. SAGE (State-Aware Gating through Exploration) couples the allow/block decision with optional private, read-only queries into the environment. It can investigate a pending action's likely effect before deciding, and it re-checks every replanned action after a block.

4. An environment-verifiable dataset and dual evaluation protocol. The paper builds 187 malicious tasks with controlled initial states, replayable tool environments, and task-specific executable checks, plus 95 benign trajectories derived from OpenAgentSafety, and evaluates both on recorded trajectories (offline) and in live defended interactions (online).

Main Findings

• DART outperforms all baselines under both scoring criteria. Across GPT-5.6 Luna, GPT-5.6 Terra, Gemini 3.8 Flash, and Claude Sonnet 5, DART achieves the highest attack success rate in every column of Table 2, exceeding the best competing baseline by 18.8–35.9 percentage points on semantic success and by 8.1–17.9 points on hard (executable-verifier) success. Its semantic ASR is 73.3 / 62.5 / 43.9 / 51.9 and its hard ASR is 54.7 / 51.1 / 33.2 / 38.7 for those four targets respectively.

• Multi-turn interaction alone does not make attacks stronger. The fixed-decomposition baseline MTA and the adaptive baselines STAC and Intent Hijacking do not consistently outperform a single-instruction Direct request. The paper attributes DART's advantage to conditioning on what the agent actually executed rather than only shaping dialogue.

• SAGE preserves benign behavior while intercepting harm offline. On recorded trajectories it passes all actions in 91 of 95 benign trajectories (B = 95.79%) and reaches H1 = 34.55% exact-closure interception and H2 = 58.79% timing-aware interception, giving the best harmonic scores among evaluated defenders: F1 = 50.78% and F2 = 72.86%. The abstract reports that SAGE preserves 95.79% of benign trajectories while intercepting 92.73% of harmful paths by the harm-enabling boundary.

• More aggressive blocking baselines trade utility for safety, or fail outright. Fine-tuned Binary keeps B = 100.00% but only H1 = 18.18% and misses the closure in 72.73% of harmful trajectories. Safiron scores B = 0.00%, blocking essentially all benign trajectories. TS-Guard reaches B = 71.58% and H2 = 38.14%, while blocking early in 78.18% of harmful cases versus 58.18% for SAGE.

• SAGE localizes the harmful transition more precisely. SAGE reaches the highest Exact rate among the listed methods (34.55%, versus 10.91% for untuned SAGE and 7.27% for TS-Guard), reduces Early blocks from 78.18% to 58.18% and Miss from 14.55% to 7.27%, with no Late blocks.

• Online, SAGE sharply reduces realized harm. Against DART on the 75-task subset, SAGE reduces semantic ASR from 65.3% without defense to 10.7%, and hard ASR from 48.0% to 4.0%. Fine-tuned Binary, despite its perfect benign all-pass rate offline, leaves hard ASR at 48.0%.

• SAGE transfers to environments it was never trained on. PostgreSQL and Web were deliberately excluded from defender fine-tuning. On the 30 PostgreSQL/Web tasks, SAGE cuts DART's semantic successes from 20 to 3 and hard successes from 9 to 1. No PostgreSQL attack achieves hard success across any of the four attack methods, and each method achieves at most two hard successes in Web.

• Residual risk concentrates in one attack family. Under SAGE, the largest remaining success rates come from Intent Hijacking at 13.3% semantic and 6.7% hard, compared with DART at 10.7% and 4.0%.

Methodology in Plain English

The authors start by writing down one loop that all three parties share. At each step, the attacker sends an instruction, the target model decides which tool call to make, and a defender either lets that call run or blocks it. If it runs, the environment changes and the agent sees some feedback; if blocked, nothing changes and the agent gets a refusal message. Because blocked calls cause the agent to try something else, the defender's decision changes how the whole interaction unfolds.

DART is built on the observation that the attacker cannot control the target's exact tool calls, so it should plan against what actually happened. It runs a tree search where each node is a real executed trajectory prefix, expands a node by proposing several candidate instructions aimed at the next intermediate goal, executes them, and scores the children. The score mixes an executable verifier check (weight 0.50), an LLM judgment of the observable execution feedback (0.30), and an LLM judgment of whether the proposal was sensible given the history (0.20). It then explores with UCT, backing up terminal rewards along the path.

SAGE is built on the observation that the visible history may be consistent with two different states that require opposite decisions. So before answering PASS or BLOCK, it may issue a bounded number of read-only queries — reading a script and its config, or checking a record — that reveal state without modifying the guarded task. It is fine-tuned from Qwen3-4B-Instruct-2507 on defender traces: actions before an annotated harmful closure are labeled PASS, the closure action itself is labeled BLOCK, and the queries are part of the supervised output. The same checkpoint is reused across domains and attack methods, with domain-appropriate read-only tools supplied at inference.

For measurement, the authors use two independent success criteria, because they can disagree: a "semantic" LLM-as-a-Judge protocol using an independent gemini-3-flash judge, and a "hard" criterion requiring the task-specific executable evaluator to pass on the resulting environment state. Defense is measured both offline on recorded trajectories — where ground-truth closure actions are known, so block timing can be scored — and online in fresh interactions where the attacker can react to refusals.

Why This Matters

Impact on research. The paper reframes agentic safety as state control rather than prompt filtering, and shows that the two standard success criteria (LLM judging and executable verification) can diverge substantially — DART's semantic ASR of 73.3% versus hard ASR of 54.7% on GPT-5.6 Luna is a concrete example. It also gives defense evaluations a paired safety–utility metric rather than a single blocking rate, and demonstrates that a defender trained on two domains can transfer to two unseen ones.

Real-world applications:

• Guardrails for coding and repository agents, where the harmful moment is a file write or a commit rather than a chat response.
• Database and operations copilots, where reading a record or a config is the difference between an authorized maintenance update and a destructive one.
• Web and browser agents navigating authenticated sessions, where uploading or disclosing the wrong file leaks credentials.
• Red-teaming pipelines that need to test whether an agent's safety holds when the agent can re-plan after being refused.

Industry relevance. Practitioners deploying tool-using agents need a gate that inspects a pending tool call before execution, and this work argues that such a gate needs its own read-only investigation ability, not just a policy prior. The finding that a perfect benign all-pass rate can coexist with 48.0% hard attack success under continued execution is a direct warning against evaluating runtime defenses on fixed trajectories alone.

Future Directions

1. Learning which queries are worth making. The paper frames the defender's investigation policy as part of the safety mechanism and explicitly raises learning which queries could change a decision, and when to stop investigating, as open problems.

2. Reusing evidence as state changes. The authors ask how evidence can be reused while tracking whether it remains valid as the environment evolves.

3. Extending the intervention interface. The limitations section notes that concurrent processes, compound actions, and assistant-output disclosure fall outside the current gate-and-read-only-query interface and require extensions.

4. Broader and more granular evaluation. The authors call for broader target-model coverage in defense evaluation (currently GPT-5.6 Luna, with PostgreSQL and Web held out from fine-tuning) and for controlled studies isolating individual search and investigation components.

Target Audience

This paper is most useful to agent-safety and security researchers, red-teamers who construct multi-step attacks against tool-using agents, and engineers building runtime guardrails or pre-execution gates for deployed agents. It also suits readers of prior work such as AgentHarm, AgentDojo, MT-AgentRisk, ToolSafe, StepGuard, and SafeMCP who want a unifying formal framing, and evaluation-focused researchers who care about the gap between LLM-judged and executable-verifier success. It assumes comfort with partially observed decision processes, tree search, and supervised fine-tuning; readers without that background will find the introduction and Section 3 accessible but the method sections dense.

Authors’ abstract

Language-model agents increasingly use tools to act on external systems. Earlier actions can alter files, permissions, database records, or other state, making a later routine-looking action harmful. Yet the visible interaction may not reveal the underlying state needed to assess that action. We formulate attack and defense as partially observed state control in SEAD, deriving their design requirements from this shared execution process. Because attackers supply instructions while the target chooses concrete actions, DART decomposes harmful goals into locally plausible steps and uses feedback from actual tool execution to guide trajectory search. The defender must decide before execution with incomplete state evidence. SAGE can therefore investigate relevant state through read-only queries before allowing or blocking each action, including those proposed after a block. We construct an environment-verifiable dataset integrating controlled initial states, replayable tool environments, and task-specific executable checks. Across four target models, DART improves semantic attack success by 18.8--35.9 percentage points over the competing baseline, with consistent gains under executable verification. On recorded trajectories, SAGE preserves 95.79% of benign trajectories while intercepting 92.73% of harmful paths by the harm-enabling boundary. In online attack-defense evaluation, it reduces DART's executable attack success from 48.0% to 4.0%. SAGE remains effective across four attack methods and generalizes to out-of-domain environments. Our code and data is available at https://github.com/EverywhereSafety/SEAD.