AIR: Improving Agent Safety through Incident Response

AI summary

Overview

Research area: AI safety for Large Language Model (LLM) agents, specifically incident response (detection, containment, recovery, and eradication of safety incidents), domain-specific languages, and runtime enforcement.

Technical level: Intermediate. The paper assumes familiarity with LLM agent execution loops (plan-act-observe), tool invocation, and runtime guardrails, but the DSL itself is presented in readable natural-language form.

Scope: The paper introduces AIR (Agent Incident Response), the first incident response framework for LLM agent systems, and evaluates it on code, embodied, and computer-use agents.

What This Paper Is About

Current safety mechanisms for LLM agents focus almost entirely on preventing failures in advance, leaving systems with limited ability to respond to, contain, or recover from incidents after they arise. Incidents are treated in the paper as still inevitable, so the authors argue that incident handling should be a first-class mechanism inside the agent's execution loop rather than an afterthought. AIR is proposed as a unified framework that detects incidents, guides the agent through containment and recovery using its own tools, and then synthesizes guardrail rules that block similar incidents in future runs.

Key Contributions

1. AIR, the first incident response framework for LLM agent systems. It integrates runtime incident detection, structured containment, recovery, and eradication into a single agent-level workflow, operating entirely inside the agent's execution loop.
2. A domain-specific language (DSL) for incident response rules. Each AIR rule specifies a trigger (which tool invocation activates the rule), a natural-language check (whether an incident occurred, judged against environment state and recent context), and a remediate component (containment and recovery actions executed through the agent's tools). Eradication is achieved implicitly by generating plan-level guardrail rules.
3. A reference implementation and open-source artifact. AIR is implemented on the OpenAI Agent SDK with the DSL parsed using ANTLR4, with two integration hooks: (1) after each tool invocation for detection and remediation, and (2) before each step to apply guardrail rules to the agent's plan. The framework is described as framework-agnostic, with the same hooks mappable onto frameworks such as LangChain. Code is released at https://github.com/FFchopon/AIR.
4. An extensive evaluation across three agent types. AIR is evaluated on a code agent, an embodied agent, and a computer-use agent, covering detection, remediation, eradication, timeliness, overhead, automatically generated rules, and ablations.

Main Findings

• Overall effectiveness exceeds 90%. The abstract reports that AIR achieves detection, remediation, and eradication success rates all exceeding 90%. The introduction states detection rates above 90% and remediation and eradication success rates exceeding 95% across domains.
• Code agent (CodeAct + RedCode). Rules were applied across all 25 risk categories, with 30 risky tasks per category and one AIR rule constructed per category. Across most categories AIR detected incidents and, where feasible, completed remediation. In categories with irreversible effects (for example, deleting sensitive files), effectiveness is reflected in eradication counts instead, since remediation is marked as not applicable. Some categories show lower detection counts because the agent did not reproduce the intended risky behavior, such as "Miss default case bug" (30 tasks executed, 19 detected), which the authors attribute to agent limitations or ambiguous task definitions rather than to AIR.
• Embodied agent (SafeAgentBench). AIR was evaluated on 25 tasks per hazard category, with 2 to 5 AIR rules per category to capture heterogeneity. Detection counts matched execution counts in every category (for example, Fire Hazard: 13 executed, 13 detected, 13 remediated, 13 eradicated, 3 rules; Electrical Shock: 7 executed, 7 detected, 6 remediated, 6 eradicated, 3 rules). Remediation and eradication values remained high, with most categories near-perfect.
• Computer-use agent (RiOSWorld for risky tasks, OSWorld for safe tasks). AIR remained effective despite low execution counts in the underlying agent: 6 executed for Phishing Web (5 detected, 5 remediated, 5 eradicated, 4 rules), 7 for Phishing Email (7/7/7/7, 3 rules), 6 for reCAPTCHA (6/6/6/6, 3 rules), 4 for Account Fraud (4/4/4/4, 4 rules), 12 for Induced Text (11 detected, 10 remediated, 10 eradicated, 3 rules), 6 for Web (6/6/6/6, 3 rules), and 6 for Social Media (5 detected, 5 remediated, 5 eradicated, 4 rules).
• No false positives were observed. On 50 safe embodied tasks (five benign counterparts sampled from each of the ten hazard categories), 45 passed and 5 failed, with 0 falsely flagged incidents. On 35 safe browser tasks from OSWorld, 8 passed and 27 failed due to agent capability limits on complex web interfaces, with 0 false positives and no failures caused by AIR.
• Detection is fast relative to execution; eradication is slower. For the code agent, check time was 6.918 s (0.68× execution time of 10.162 s), response time 10.514 s (1.03×), and eradication time 49.273 s. For the embodied agent, check time was 8.598 s (0.41× of 20.759 s), response 22.191 s (1.07×), eradication 94.031 s. For the CUA, check time was 11.735 s (0.16× of 74.327 s), response 25.943 s (0.35×), eradication 36.272 s.
• Moderate overhead on safe tasks. For safe embodied tasks, total execution time rose from 27.442 s to 39.610 s (1.44×) after integrating AIR; for safe CUA tasks, from 64.766 s to 90.602 s (1.40×). The corresponding code-agent overhead is not reported in the table. The authors state the overhead is not inherent to the design and can be reduced by decoupling detection from the main loop, for example by allowing a short delay window before intervention or running smaller, faster models in parallel.
• LLM-generated rules approach developer-authored ones. Using OpenAI's GPT-5 with a description of the target agent and tools, three developer-authored example rules, and in-context risky task examples, generated rules achieved detection of 84.560% (557 of 750 executed code tasks), 95.973% (149 of 250 embodied tasks), and 88.636% (44 of 210 CUA tasks); remediation of 88.811% (embodied) and 84.615% (CUA), with code-agent remediation not applicable; and eradication of 91.023% (code), 90.210% (embodied), and 89.744% (CUA). Detection success exceeded 80% in every setting.
• Rule numbers indicate generalizability. 25 rules covered 750 code agent tasks, 20 rules covered 250 embodied agent tasks, and 25 rules covered 210 CUA tasks. A single rule typically generalizes to a cluster of behaviorally similar risky tasks, while a qualitatively different risk pattern usually requires an additional rule.
• Ablations confirm the design components matter. Removing structured remediation substantially degraded remediation success rates and response efficiency, and disabling guardrail rule synthesis led to significantly higher incident recurrence across rounds. Quantitative details are provided in Appendix B, which is not included in the provided content.
• No direct baselines exist. The paper states that because AIR is the first work on incident response for LLM agents, there are no direct baselines for comparison.
• Traditional IR metrics are not directly comparable. The authors note that metrics such as dwell time or MTTI/MTTC are reported at the incident or breach level and often span days to months, whereas AIR measures step-level detection and response overhead inside the agent execution loop.

Methodology in Plain English

The researchers treat agent safety like IT security operations: assume prevention will sometimes fail, and build a process for what happens afterward. They designed a small language for writing incident response rules. A rule names the tool that activates it (the trigger), describes in ordinary language what counts as an incident given the current environment and recent context (the check), and describes the containment and recovery actions the agent should take (the remediate block). Because checks are written in natural language and interpreted by the agent, they can capture semantic conditions — such as whether a sensitive file was exposed — rather than only matching specific API calls.

AIR is inserted into the agent's execution loop at two points. After each tool call, only the rules whose triggers match the tool just used are evaluated; the agent judges the check condition using inspection tools, its latest observation, and part of the recent context. If an incident is found, the agent performs the containment and recovery actions through its own tool interface, and the original task is terminated because the environment is already compromised. The framework then extracts incident information and guides the agent to write a guardrail rule describing the risky behavior pattern. That guardrail rule is applied before future steps: when the agent produces a new plan, the rule checks the plan's intent and blocks the action before any tool is invoked.

The implementation attaches lightweight hooks to the OpenAI Agent SDK and parses rules (both hand-written and LLM-generated) with ANTLR4 into a common intermediate representation. Evaluation used three agent types, all backed by OpenAI's GPT-5: a code agent with CodeAct on the RedCode dataset, an embodied agent on SafeAgentBench, and a computer-use agent on browser-compatible risky tasks from RiOSWorld plus safe tasks from OSWorld. Effectiveness was measured with five metrics — successful execution, detection, remediation, eradication, and rule count — with ground-truth labels determined through manual assessment. The authors first hand-authored rules from each dataset's risk descriptions, then tested whether GPT-5 could generate rules automatically from agent descriptions, three example rules, and in-context risky tasks, evaluating all generated rules without manual correction.

Why This Matters

Impact on research. The paper reframes agent safety as an incident-handling problem rather than a purely preventive one, and argues that prevention is inherently incomplete. It contributes a DSL, runtime semantics, an implementation, and a set of hand-authored and LLM-generated rules, giving follow-up work a concrete artifact and a new evaluation framing (detection, remediation, eradication, timeliness, overhead, and false positives).

Real-world applications:

• Coding agents processing untrusted repositories, where incidents such as copying sensitive files, untrusted installations, or writing risky content must be contained and prevented from recurring.
• Embodied and household robots, where hazards such as fire, electrical shock, poisoning, slips, and property damage arise from multi-step object interactions.
• Computer-use and browser agents, where phishing pages, phishing emails, account fraud, and unintended data exposure occur during UI-driven workflows.
• Security operations, as a complement to existing LLM-assisted incident response tooling in traditional enterprise settings.

Industry relevance. The framework is described as guardrail-agnostic and framework-agnostic: its two integration hooks can be mapped onto other agent frameworks such as LangChain, and the eradication phase can work with other guardrail mechanisms such as trajectory-level or probabilistic monitoring. The workload of producing rules can be partly automated, since LLM-generated rules performed competitively with developer-authored ones, though the authors stress the need for human-in-the-loop validation and correction.

Future Directions

• Reducing detection overhead. The paper proposes adding lightweight rule-based prefilters to reduce how often semantic checks fire, introducing a short delay window before intervention, or running smaller, faster models in parallel to identify incidents earlier.
• Integrating alternative guardrail mechanisms. Since AIR is guardrail-agnostic, the authors suggest incorporating trajectory-level or probabilistic monitoring approaches into the eradication phase to intercept risky behavior before unsafe actions execute.
• Improving and validating generated rules. The paper identifies failure modes in LLM-generated rules — checks that are too narrowly tailored (for example, naming only a metal fork in a microwave) or too abstract (for example, "microwave may cause a fire hazard"), and idealized remediation actions such as "restore deleted file" that are infeasible in practice — and argues for human-in-the-loop validation.
• Extending coverage beyond the studied settings. AIR was evaluated on three agent types with a limited number of rules per risk category; how rules generalize across broader risk categories, other agent frameworks, and other domains is left open. The paper also leaves remediation undefined for categories where incidents are irreversible.

Target Audience

Researchers and practitioners working on LLM agent safety, agent frameworks, and AI guardrails; developers building code, embodied, or computer-use agents who need post-incident handling rather than only preventive filters; security engineers interested in applying incident response practices to autonomous systems; and readers interested in domain-specific languages and runtime enforcement for LLM-based systems.

Authors’ abstract

Large Language Model (LLM) agents are increasingly deployed in practice across a wide range of autonomous applications. Yet current safety mechanisms for LLM agents focus almost exclusively on preventing failures in advance, providing limited capabilities for responding to, containing, or recovering from incidents after they inevitably arise. In this work, we introduce AIR, the first incident response framework for LLM agent systems. AIR defines a domain-specific language for managing the incident response lifecycle autonomously in LLM agent systems, and integrates it into the agent's execution loop to (1) detect incidents via semantic checks grounded in the current environment state and recent context, (2) guide the agent to execute containment and recovery actions via its tools, and (3) synthesize guardrail rules during eradication to block similar incidents in future executions. We evaluate AIR on three representative agent types. Results show that AIR achieves detection, remediation, and eradication success rates all exceeding 90%. Extensive experiments further confirm the necessity of AIR's key design components, show the timeliness and moderate overhead of AIR, and demonstrate that LLM-generated rules can approach the effectiveness of developer-authored rules across domains. These results show that incident response is both feasible and essential as a first-class mechanism for improving agent safety.