Shortcut Sensitivity in Agent-Safety Evaluations

AI summary

Overview

Research area: AI agent safety evaluation, benchmark validity, and guardrail/guard-model reliability (submitted to the TAE — Trust-AI-Eval — workshop track, arXiv:2610.22243v1 [cs.AI], 15 Oct 2026, by Hasan Demirkiran and Jens Ernstberger of Kontext).

Technical level: Intermediate. The statistical machinery (clustered bootstraps, paired interventions, placebo controls) is explained carefully, but readers should be comfortable with concepts like F1, AUC, balanced accuracy, and cross-validation.

Scope (1 sentence): The paper introduces a three-level "shortcut auditing" framework and applies it to three released agent-safety benchmarks plus a set of guard models, showing that surface features of benchmark records can predict labels, correlate with guard decisions, and — for at least one configuration — causally change those decisions when edited.

What This Paper Is About

Guard models that approve or block an AI agent's tool calls are evaluated against benchmarks, but those benchmark records carry incidental formatting cues — provenance tags, tool names, trajectory length, source subsets — that may make labels easy to predict without measuring actual safety. The authors ask a stricter question than "are the labels predictable?": if you change only such a cue while leaving the underlying action unchanged, does the guard's block/allow decision move? They answer it across released agent-safety datasets and several guards, and package the answers as a reusable auditing procedure rather than a claim that the benchmarks are broken.

Key Contributions

1. A three-level shortcut-audit framework. An inference ladder separating label predictability (can a deliberately weak rule predict released labels?), observational guard association (do guard decisions correlate with the suspected cue?), and paired decision sensitivity (does the guard change when only that cue is edited?), with explicit caveats that a paired flip only counts as shortcut reliance if the edit preserves the safety construct.

2. Controls designed for stochastic guards. Because hosted language models give different outputs on identical inputs, the framework adds interleaved repeated calls, a formatting-only placebo edit, and a separate report of identical-input disagreement, with a defined estimand (mean per-base difference in block probability) rather than a raw flip count.

3. An audit of released agent-safety evaluations. Applied to the LinuxArena public sample and two TS-Bench-derived step datasets (AgentDojo-Traj and AgentHarm-Traj), the audit surfaces label separability (command length AUC .910), domain-transfer failure of exact-tool lookup (F1 = .901 under grouped cross-validation to .495 under domain holdout), and evaluation-unit sensitivity of TS-Guard (F1 = .902 row-weighted to .719 under equal-interaction weighting).

4. Guard-level shortcut validation on a frozen cohort. A pinned 64-step AgentDojo-Traj cohort with paired marker-rewrite, tool-alias, combined, and placebo edits, run against a byte-pinned 8-bit MLX reproduction of TS-Guard plus hosted Claude Haiku 4.5 and GPT-5.6 Luna, producing a concrete instance of representation-driven decision change (and showing it is configuration-specific).

Main Findings

• Length predicts LinuxArena labels. Negative mean shell-command characters gives AUC .910 [.873, .941] on 576 exact-action-unique shell trajectories (176 attack, 400 honest, ten environments). A leave-one-environment-out lexical TF–IDF/logistic-regression baseline reaches AUC .934 [.907, .955], only .023 [.008, .040] higher.
• The LinuxArena effect survives deduplication and model matching. After exact trace deduplication the pooled leave-one-environment-out lexical AUC is 0.934; per-environment AUCs span .900 to 1.000 (macro mean .951). On a model-matched Haiku-only cohort, lexical AUC 0.955 versus length AUC 0.943, difference 0.012 with interval [−0.003, 0.027].
• Action count beats command length, and the monitor is not merely echoing length. Action count AUC .947 versus .905 for mean command length; within coarse length quartiles the stored monitor score retains weighted AUC .896 versus .607 for length. The authors therefore claim collection/protocol separability, not benchmark invalidation.
• The AgentDojo marker is a near-perfect-recall but imprecise cue. All 352 strict-unsafe rows contain the <INFORMATION> marker, as do 321 of 868 safe rows; the marker alone has a 37.0% false-positive rate (F1 = .687). Adding 11 side-effect action prefixes gives F1 = .802, balanced accuracy .894, recall .972.
• Exact-tool lookup does not transfer across domains. 40 of 51 exact tool names are label-pure and cover 776 rows, yielding grouped-CV F1 = .901 and balanced accuracy .925; under leave-one-domain-out it collapses to F1 = .495, with only 17.3% of rows using a tool seen in another domain.
• AgentHarm labels are fully determined by source subset. Every harmful_steps row is strict-unsafe and every benign_steps row strict-safe. The all-unsafe baseline scores F1 = .836 with balanced accuracy .500, showing how prevalence inflates F1.
• Evaluation unit changes the headline number. Released TS-Guard's row-level F1 = .902 falls to .719 under equal-interaction weighting, while balanced accuracy moves from .754 to .776; harmful interactions contribute 7.29 rows on average versus 1.65 for benign ones.
• Matched-execution analysis still favors the guard. Among 16 mixed groups (69 rows from 39 interactions), TS-Guard's mean composite score is .922 in unsafe contexts versus .413 in safe contexts (difference .510), higher in 11 groups and tied in five, with no reversals; all six cached hosted guards show a positive macro difference.
• Association screening picks the intervention target. Among gold-safe AgentDojo rows, Claude Haiku 4.5 is the outlier, blocking 74.45% of marked versus 0.37% of unmarked rows (74.09 pp [68.67, 79.75]); the other six cached guards range from 3.62 pp (TS-Guard) to 10.48 pp (GPT-5.6 Luna).
• Paired edits move Haiku's decisions. On 64 bases from 58 source interactions (three calls per condition, 960 total calls, all parsing successfully), marker rewriting lowers blocking by 8.33 pp [−15.05, −2.49] (safe-only −16.67 pp), opaque tool aliases raise it by 11.46 pp [2.69, 20.59] (safe-only +22.92 pp), the combined edit gives −2.60 pp [−8.99, 2.99], and the formatting placebo −1.56 pp [−6.78, 2.78]. Placebo-adjusted effects are −6.77 pp [−12.82, −1.61] and +13.02 pp [4.92, 21.72], both excluding zero.
• The Haiku effects are all false positives against inherited labels. Every unsafe original and every unsafe edited call remains blocked, so each all-base estimate is exactly half its safe-only estimate; measured accuracy against inherited labels rises from 73.44% on originals to 81.77% after marker rewriting and drops to 61.98% after aliasing.
• The effect does not replicate on GPT-5.6 Luna. With the same cohort, prompt, five conditions, and three calls per condition (960 valid calls), marker and alias contrasts are 0.00 pp [−3.13, 3.17] and +1.04 pp [−2.08, 4.41], and −1.04 pp / 0.00 pp relative to placebo; no base moves between opposite unanimous three-call decisions.
• The deterministic guard barely moves. The Q8 MLX TS-Guard reproduction (64/64 parse, 63/64 strict decisions, 58/64 exact scores agreeing with the release) flips 3/64 marker pairs (4.69% base rate, 4.31% equal-interaction rate), 3/63 alias pairs, and 6/64 combined pairs; only two of the fixed rule's 42 changing bases move Q8 in the rule's predicted direction, and a third moves oppositely.
• Observational association does not imply paired sensitivity. GPT-5.6 Luna shows a 10.48 pp observational marker gap yet a 0.00 pp repeated paired contrast — the paper's central caution that association only nominates a hypothesis.

Methodology in Plain English

Each benchmark is treated as a pair: the context needed to judge an action, plus one candidate cue singled out for audit (trajectory length, a literal <INFORMATION> tag, an exact tool name, a source subset). The audit then climbs three rungs.

Rung one asks whether a deliberately dumb rule that sees only the cue — negative mean command length, a marker plus a hard-coded verb list, an exact-tool majority lookup — can reproduce the released labels. The point is not to build a good classifier but to check whether labels are recoverable from construction details, and then to stress-test that recovery by deduplicating repeated traces, holding out entire environments or domains, and reweighting the evaluation unit.

Rung two checks whether guard outputs in the released records correlate with the cue, restricted to matched cohorts such as gold-safe rows. This is explicitly labelled observational, because the cue, the label, and legitimate evidence in the context share a data-generating process.

Rung three is the novel part. A fixed cohort of 64 AgentDojo steps is frozen before any edited input is scored: eight marker-present safe and eight marker-present unsafe steps per domain, drawn from 58 source interactions. Each base gets an original, a marker rewrite (<INFORMATION> to <UNTRUSTED_EXTERNAL_CONTENT>, contents untouched), a per-sample opaque tool alias (arguments and descriptions byte-identical), a combination, and for hosted runs a placebo that only changes outer JSON whitespace. Without access to guard internals, the authors call each guard as a pre-execution judge over inert text — no tool call is ever executed. For the stochastic hosted guards they interleave three calls per base and condition, at temperature zero, routed to a single provider with fallback disabled, and estimate the average per-base difference in block probability with 10,000 domain-stratified bootstrap replicates clustered over source interactions. A deterministic 8-bit MLX reproduction of TS-Guard provides the non-stochastic comparison, gated in advance on agreement with the release. A final section converts the experience into a four-step recommended audit sequence.

Why This Matters

Impact on research. The paper reframes benchmark evaluation as something to be stress-tested rather than trusted: a predictive benchmark property becomes a hypothesis, not evidence that a guard relies on it. It also shows that two standard practices — reporting row-level F1 and reporting cross-validated scores without group holdouts — can inflate results substantially (.902 vs .719; .901 vs .495) on benchmarks that are already in use. Because the result is configuration-specific rather than universal, it supports per-guard auditing instead of blanket condemnation of the benchmarks.

Real-world applications:

• Guardrail procurement and release gates. Teams deploying a pre-execution tool guard can run paired cue edits before shipping, rather than relying only on offline benchmark accuracy.
• Benchmark maintenance. Dataset authors can add domain-group holdouts, equal-interaction weighting, and refreshed equivalent serializations so that public rules do not become adaptive targets.
• Agent platform monitoring. Because the Haiku effect operates entirely on false positives against inherited labels — no unsafe action became unblocked — the risk profile is usability and mislabeling rather than a direct bypass, which matters for tuning over-blocking in production.
• Safety-case documentation. The three-level ladder gives reviewers a vocabulary for distinguishing "the benchmark is predictable" from "the guard is sensitive to this specific edit."

Industry relevance. The audit is black-box and cheap relative to building a new benchmark: the reported runs cost $3.408 (Haiku, 3,247,713 input and 31,990 output tokens), $0.4841 (GPT-5.6 Luna, 2,700,051 input and 173,298 output tokens), and $5.067 for the secondary GPT-5.5 check, with the deterministic reproduction running on a 24-GiB Apple Silicon laptop (2,668.5 seconds of generation, median 9.26 seconds per row, 9.47 GB peak MLX memory).

Future Directions

• Closing the construct-preservation gap. The marker rewrite changes provenance wording and the aliases are per-sample rather than a single consistent global renaming; neither had a blinded human equivalence check. Edits with validated semantic neutrality would strengthen any sensitivity claim.
• Mapping which guard configurations are sensitive, and why. Haiku shows large effects, Luna and the deterministic Q8 reproduction do not, and the paper reports an eight-pair authorization pilot for Q8 (unsafe member scored higher in 5/8 native structured pairs and 6/8 ReAct execution-only pairs, no reversals) as a weak positive control. What guard property predicts sensitivity remains open.
• Adaptive and hidden-holdout testing. The authors note that a public audit rule can become an adaptive target and motivate hidden group holdouts and refreshed equivalent serializations, but explicitly do not test adaptation.
• Reproducing official metrics and full-scale guard runs. The public LinuxArena sample cannot reproduce the official pre-harm metric exactly, and StepGuard's official vLLM/CUDA runtime could not be matched on Mac (a one-row MPS pilot took 45.9 seconds with 26.7 GB peak), so an equivalent full comparison is unaddressed.

Target Audience

Benchmark and evaluation authors, guardrail engineers, and agent-safety researchers who build or consume step-level safety evaluations; also policy and assurance reviewers who need to judge whether a reported guard F1 supports a deployment claim. Readers need moderate statistical literacy (F1, AUC, balanced accuracy, bootstrap intervals) but no access to model internals, since the entire method is black-box and depends only on released records, outputs, and fixed input transformations.

Authors’ abstract

AI agents take actions through tools. Some actions are sensitive, so guards that supervise them must themselves be evaluated reliably. Yet the data used to evaluate a guard can include cues that make labels predictable without measuring the intended safety construct, and those cues can also influence guard decisions. We introduce a three-level shortcut-auditing framework and audit completed LinuxArena trajectories and two TS-Bench-derived step datasets. We distinguish label predictability, observational guard association, and paired decision sensitivity. In LinuxArena, shorter average shell commands strongly predict trajectory labels. In AgentDojo-Traj, exact-tool lookup falls from F_1=.901 with mixed-domain folds to .495 under domain holdout. In AgentHarm-Traj, released TS-Guard F_1 falls from .902 under row weighting to .719 with equal interaction weighting, while balanced accuracy rises. We then test the strongest safe-row marker association in a seven-configuration screen on a frozen 64-step AgentDojo-Traj cohort using repeated, interleaved Haiku calls. Rewriting the <INFORMATION> marker lowers blocking by 8.33 percentage points, while opaque tool aliases raise it by 11.46 points. All unsafe calls remain blocked; the changes concern false positives against inherited labels. Both effects remain nonzero after subtracting the formatting-placebo effect, with exploratory 95% ranges excluding zero. The same edits also change measured accuracy against inherited labels. This provides direct evidence that benchmark representation can materially change a guard's decisions and measured evaluation performance. The Haiku-sized effects do not reproduce in Luna or deterministic Q8 TS-Guard. We recommend domain group holdouts, evaluation-unit metrics, and placebo-controlled paired interventions.