OpenSec: Measuring Incident Response Agent Calibration Under Adversarial Evidence

AI summary

Overview

• Research area: AI agent evaluation and reinforcement learning environments for cybersecurity, specifically defensive incident response (IR) under adversarial prompt injection.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning, agent evaluation metrics, and security operations concepts, though the core argument is stated in accessible terms.
• Scope: The paper introduces a dual-control RL environment, OpenSec, that scores four frontier LLMs on whether they exercise restraint when deciding to contain threats, and reports that all of them over-trigger.

What This Paper Is About

Existing security benchmarks measure whether an AI agent can perform a task, not whether it knows when to perform it. The paper's core claim is that this conflates action execution with correct execution, hiding calibration failures: a model can correctly identify a real threat and simultaneously isolate legitimate systems alongside it. OpenSec is built to isolate that gap by scoring agents on executed containment actions against known ground truth while an attacker advances a kill chain and injects misleading evidence into the artifacts the agent must read.

Key Contributions

1. OpenSec environment: A dual-control RL environment with deterministic, execution-based scoring designed for training and evaluating incident response agents.
2. Taxonomy-stratified scenario design: Scenarios organized into three families with explicit sampling weights (direct_harm 0.50, data_exfil 0.30, adaptive 0.20), combined with trust tiers (untrusted, corroborated, verified) and prompt injection payloads across all scenarios to support curriculum learning.
3. A four-model baseline evaluation: Four frontier models evaluated on 40 standard-tier episodes each, revealing consistent over-triggering with 45–82.5% false positive rates and evidence-gated action rate (EGAR) below 55% across all models.
4. Evidence that calibration is not capability: Sonnet 4.5 shows partial calibration (62.5% containment, 45% FP, TTFC 10.6) where GPT-5.2 is uncalibrated (100% containment, TTFC 4.1), indicating calibration varies independently of raw capability.

Main Findings

• Every frontier model over-triggers. All four models execute containment in 62.5–100% of episodes with 45–82.5% false positive rates. EGAR ranges from 37.5% to 54.2%, meaning most containment actions are taken without first gathering trusted evidence about the target entity.
• GPT-5.2 is the only model classified as uncalibrated. It executes containment in 100% of episodes with an 82.5% false positive rate and a time-to-first-containment (TTFC) of 4.1 steps, acting after investigating only 27% of the episode.
• Sonnet 4.5 is the most restrained. It contains in 62.5% of episodes with a 45% false positive rate and a TTFC of 10.6 steps (70% of the episode). Gemini 3 (75% containment, 57.5% FP, TTFC 8.6) and DeepSeek 3.2 (92.5% containment, 65% FP, TTFC 9.0) fall between the extremes.
• The gap is restraint, not detection. All models correctly identify the ground-truth threat when they act. The failure is acting on the right target and wrong targets simultaneously.
• High aggregate rewards mask operational failure. Reward scores range from 2.37 to 3.45 despite the false positive rates, because the reward design penalizes missed containment not at all and caps false positive penalties by category. Models that exhaust the action space can score well while failing operationally.
• Blast radius is consistent across models. The ratio of false positive to correct containment actions ranges from 0.42 to 0.44, meaning that when false positives occur, their magnitude does not depend on how much the model investigated first.
• Injection vulnerability is tier-dependent. T2 (contextualized, domain-specific) payloads are the most effective attack surface across all models at 15–25% violation rates, while T1 (obvious override) payloads rarely succeed (0–7%). DeepSeek shows the highest T3 (complex multi-step or multilingual) vulnerability at 10%. Sonnet 4.5 has zero T1 and T3 vulnerability.
• Report submission lags containment. Time-to-report ranges from 12.1 (GPT-5.2) to 13.5 (Sonnet 4.5) steps against a default max_steps of 15.
• Preliminary RL training did not fix calibration. A Qwen3-4B-Instruct model trained with GDPO reached 75% containment with a 70% false positive rate and 47.5% correct containment — acting more often but less accurately than Sonnet 4.5 — while report submission dropped to 25%, suggesting reward shaping issues.

Methodology in Plain English

OpenSec is a simulated incident response environment with two actors sharing state. A defender agent reads evidence from SQLite logs, alerts, and emails using tools such as query_logs, fetch_email, fetch_alert, isolate_host, block_domain, reset_user, and submit_report. A separate attacker policy, itself an LLM constrained by a hard state machine, advances a fixed kill chain (phish_sent to creds_used to lateral_move to data_access to exfil_attempt) and can be replay-cached for reproducibility.

Each episode starts from a scenario seed containing a known ground truth and a timeline of artifacts. Seeds are drawn from 160 training seeds and 60 evaluation seeds (20 trivial, 20 easy, 20 standard); the baseline evaluation uses the 40 standard-tier seeds. Artifacts carry trust_tier and source fields, and every scenario contains injection payloads.

Scoring is based on executed actions rather than what the agent writes in its report. The reward sums four components: attribution (+1 per correctly identified field, −0.5 per unknown), containment (+1 per correct action, −0.5 per false positive capped by category), injection safety (−2 per violation after exposure to injected content), and efficiency (−0.1 per step). Injection violations are counted per-episode when a tool call's parameters contain a substring matching content that appeared only in injected text, such as a domain name or host identifier. The design deliberately penalizes incorrect containment more than inaction, and does not penalize missed containment at all, because isolating the wrong server takes production down.

Six metrics are reported per model: containment rate, false positive rate, evidence-gated action rate (EGAR), time-to-first-containment (TTFC), blast radius, and per-tier injection violation rate (T1, T2, T3). Time-to-report (TTR) is reported additionally in the operational analysis.

Why This Matters

The paper argues that offense scales faster than defense — frontier agents reportedly generate 40+ working exploits across 6 scenarios for roughly $30–50 in compute (Heelan, 2026), where the limiting factor is token throughput rather than expertise. Adversaries who understand that IR agents over-trigger can embed prompt injections in malicious artifacts specifically to induce false-positive containment.

Impact on research: The paper reframes evaluation from capability to calibration, arguing that aggregate reward scores are not a sufficient deployment metric and that benchmarks which conflate action execution with correct execution may simultaneously overestimate calibration and underestimate action willingness. It also contributes a standardized, execution-scored environment to a field where the Survey of Agentic AI and Cybersecurity (Lazer et al., 2026) identifies benchmark standardization as a key gap. The dual-control framing connects to τ²-Bench, where Barres et al. (2025) report a 28-point performance drop when shifting from reasoning-only to dual-control mode.

Real-world applications:

• SOC alert triage and containment: Deciding whether to isolate a host or block a domain when the evidence trail includes attacker-controlled content.
• Agentic SOC deployment decisions: Choosing between models for autonomous action authority, where the difference between a 45% and an 82.5% false positive rate is operationally significant.
• Prompt injection risk assessment: The per-tier results indicate that contextualized T2 payloads, not crude overrides, are the primary attack surface in security workflows.
• Curriculum-based agent training: The taxonomy-stratified seeds, trust tiers, and injection_type metadata provide infrastructure for staged training of restraint.

Industry relevance: The paper cites Omdia (2025) tracking over 50 agentic SOC startups, and notes LLMs achieving 94% precision on alert classification in controlled settings (Srinivas et al., 2025). It positions high false-positive rates as a core pain point that LLMs are meant to solve, and asks whether agents with action authority make it better or worse.

Future Directions

1. Trust-aware evaluation: The trust_profile field and EGAR metric provide infrastructure for measuring whether models appropriately weight evidence by provenance tier. EGAR currently uses trust tiers only for evidence gating; analyzing behavior as a function of evidence provenance quality remains future work.
2. Injection robustness training: The environment supports targeted injection curricula via injection_type metadata, offering a path toward robust behavior through adversarial exposure.
3. Calibration training: The preliminary GDPO experiment suggests calibration is trainable but insufficient when learned directly from a multi-component reward; the paper points toward a two-stage SFT + RL pipeline or a curriculum approach, with SFT warmup on successful trajectories, curriculum staging, and explicit verification gates as likely improvements.
4. Broader statistical grounding: The authors state that wider confidence requires additional seeds and replications beyond the 40 standard-tier seeds per model, and that the defensive capability thresholds are provisional, calibrated against observed frontier model behavior rather than human expert baselines.

Target Audience

Researchers and engineers working on autonomous security agents, agent evaluation methodology, and reinforcement learning environments; security operations leaders assessing whether to grant action authority to LLM-based tooling; and alignment researchers interested in calibration, restraint, and prompt injection robustness. Readers seeking benchmark results on offensive cyber capability would find the paper's scope narrower than expected — the environment is log-centric and does not execute real exploits or malware.

Authors’ abstract

As large language models (LLMs) improve, so do their offensive applications: frontier agents now generate working exploits for under $50 in compute (Heelan, 2026). Defensive incident response (IR) agents must keep pace, but existing benchmarks conflate action execution with correct execution, hiding calibration failures when agents process adversarial evidence. We introduce OpenSec, a dual-control reinforcement learning (RL) environment that evaluates IR agents under realistic prompt injection scenarios with execution-based scoring: time-to-first-containment (TTFC), evidence-gated action rate (EGAR), blast radius, and per-tier injection violation rates. Evaluating four frontier models on 40 standard-tier episodes each, we find consistent over-triggering: GPT-5.2 executes containment in 100% of episodes with 82.5% false positive rate, acting at step 4 before gathering sufficient evidence. Claude Sonnet 4.5 shows partial calibration (62.5% containment, 45% FP, TTFC of 10.6), suggesting calibration is not reliably present across frontier models. All models correctly identify the ground-truth threat when they act; the calibration gap is not in detection but in restraint. Code available at https://github.com/jbarnes850/opensec-env.