Exposure Is Not Commitment: Double-Stratified Evaluation of Latent Monitors for Tool-Using Agents

AI summary

Overview

Research area: Natural Language Processing — specifically LLM interpretability and agent security (indirect prompt injection, latent-state monitoring, evaluation methodology).

Technical level: Advanced. The paper assumes familiarity with linear probes, AUROC/AUPRC, cross-validation, bootstrap intervals, residual-stream activations, and benchmark design for tool-using agents.

Scope (one sentence): The paper proposes a "double-stratification" evaluation protocol that separates injection exposure from behavioral commitment — in both the evaluation population and the probe's training contrast — and applies it to latent monitors across three 7B model families on AgentDojo.

What This Paper Is About

Tool-using language agents can be hijacked by indirect prompt injection, where attacker-controlled content in an email, webpage, or tool result gets treated as an instruction and turns into an unsafe tool call. A growing line of work probes model internals to catch that unsafe action before it executes, but published results conflict sharply. The author argues much of that conflict is an artifact of how monitors are trained and evaluated: a monitor trained on a naive "unsafe versus everything else" contrast may simply learn to detect that untrusted content is present (exposure), not that the model will actually comply (commitment), and an unstratified AUROC therefore overstates what the monitor knows.

Key Contributions

1. The double-stratification protocol, which separates exposure (injected content in context) from commitment (the model's own subsequently generated unsafe tool call) across both the evaluation population and the training contrast, plus the central demonstration that on the primary Qwen workspace grid, naive mixed-contrast training produces exposure detectors that barely predict commitment.

2. Commitment-contrast retraining recovers pre-action prediction in the same representations — logistic probe AUROC 0.853 [0.799, 0.897] at n = 240, matched by a TF-IDF text baseline at 0.855 [0.803, 0.901], and absent from the model's own zero-shot verbalized forecast — offered as one candidate explanation for recent pre-action-probe negative results.

3. Replications on two further model families showing that what an unstratified contrast learns is population-dependent: Mistral-7B learns commitment with inverted exposure ranking, and OLMo-2-7B reads both axes, while the commitment contrast recovers signal in every case.

4. A baseline hierarchy of trained text baselines, a zero-shot self-judge, and NLA-text classifiers on identical folds, locating where representation matters (exposure, transfer, sanitized or unavailable context) and where it does not (in-distribution commitment), released as a thin provenance-tracked instrumentation layer with documented negative results.

Main Findings

• Naive contrast on Qwen learns exposure, not commitment. Under mixed unsafe-versus-rest training on the full workspace grid (141 unsafe, 99 attacked-but-safe, 40 clean), the activation probe reaches exposure AUROC 1.000 [1.000, 1.000] but only 0.672 [0.605, 0.743] on strictly pre-action attacked-only commitment; trained NLA text reaches 0.817 exposure and 0.556 [0.483, 0.629] pre-action, covering chance.

• The monitors fire on nearly everything exposed. The probe flags 141/141 unsafe actions before the action step (NLA text 136/141; raw NLA taxonomy only 49/141), but also fires early on 99/99 and 94/99 attacked-but-safe trajectories and on 18/40 and 1/40 clean ones — threshold noise around perfect exposure ranking.

• Commitment-contrast retraining recovers the signal. On exposed cases only (141 unsafe versus 99 attacked-but-safe, strictly pre-action), the centroid probe rises from 0.672 to 0.763 [0.691, 0.819], an added L2-regularized logistic probe reaches 0.853 [0.799, 0.897], and both hold under user-task-grouped folds (0.747 and 0.795). Per-step signal appears at t−2 (0.878 [0.833, 0.921]) and t−1 (0.871 [0.821, 0.917]).

• Trajectories cap the achievable lead time. Because the trajectories have three steps with the unsafe action at step three, two steps is the maximum observable lead — the paper reports it as the ceiling the derived metric admits, not a general limit.

• A competent text baseline matches the probes in-distribution. TF-IDF logistic over visible prompt text reaches 0.855 [0.803, 0.901] (0.780 grouped) versus the logistic probe's 0.853 (0.795 grouped); visible-text naive Bayes reaches 0.793 [0.737, 0.850] (0.673 grouped). Paired deltas put logistic minus TF-IDF at −0.003 [−0.054, 0.050], i.e. parity.

• Representations separate on exposure. The activation probe holds 1.000 while trained text reaches 0.908 [0.861, 0.950] (naive Bayes) and 0.790 (TF-IDF); this is where an auditor has no textual alternative.

• The model's own zero-shot forecast fails. Qwen judging its own next action detects exposure nearly perfectly (0.969–1.000) but predicts compliance at only 0.615 [0.563, 0.668] in-distribution, 0.442 [0.374, 0.516] on held-out templates, and 0.641 [0.509, 0.760] on the held-out suite.

• NLA text degrades under single-sample verbalization. Commitment AUROC drops to 0.570 [0.500, 0.644] (naive Bayes) and 0.518 [0.444, 0.594] (TF-IDF); a released three-sample rerun recovers part of it (0.704 and 0.683), still trailing raw activations and raw visible text. The verbalizations themselves are largely confabulated — all four exhibit excerpts were labeled BENIGN_TASK_EXECUTION, including both unsafe cases, and their specific referents do not appear in the task or the injected content.

• The naive contrast does different things on different models. Mistral-7B (480 attacked, 167 unsafe) shows pre-action commitment 0.835 [0.793, 0.874] with inverted exposure ranking 0.282 [0.223, 0.343]; commitment-contrast probes reach 0.896 [0.866, 0.923] (0.856 grouped). OLMo-2-7B (480 attacked, 201 unsafe) reads both axes under the naive contrast (exposure 0.957, pre-action 0.785) and reaches 0.912 [0.887, 0.935] (0.905 grouped) under the commitment contrast, with TF-IDF at 0.921.

• Commitment monitors transfer, weakly and unevenly. The centroid probe fitted on the workspace grid reaches 0.756 [0.651, 0.848] on 160 attacked cases under four held-out templates (33 positives) and 0.837 [0.745, 0.918] on the travel suite (15 positives); the logistic probe trades its in-distribution edge for brittleness (0.572, 0.684), and the visible-text naive Bayes collapses (0.599, 0.372).

• Ranking quality does not imply cheap gating. Out-of-fold FPR@90TPR on the primary grid: logistic 0.343, TF-IDF 0.394, naive Bayes 0.505, centroid 0.626, NLA text 0.838.

• Replay separability is strong but is a construction contrast. On 200 replay records (40 clean, 160 attacked), trained NLA text reaches 0.929 [0.872, 0.980] and the activation probe 0.999 [0.997, 1.000]; exact-prefix-only and normalized-replay-only ablations preserve this (0.962/1.000 and 0.968/0.999), as do suite and template transfer (0.927–1.000). The paper explicitly labels these results as construction-contrast evidence, matching the situation-not-action pattern reported by Fomin et al. (2026).

• Tool policy is unbeatable once the call is visible. The hybrid reaches 0.989 in the unstratified generated-action view precisely because the generated call is visible to it, which is why the author argues latent monitors must be judged on lead time rather than head-to-head detection.

Methodology in Plain English

The author treats latent-state auditing as post hoc instrumentation layered onto an existing benchmark rather than a new benchmark or a runtime defense. Each AgentDojo case is expanded into a trajectory of audited decision points; at each point the pipeline captures a residual-stream activation from the prompt prefix, runs an NLA verbalizer to turn that activation into text, attaches labels and provenance metadata, and then trains and scores detectors. Prompt fidelity is the key engineering requirement: every activation record carries a provenance tag — normalized_replay (prompt reconstructed from exported case fields), exact_prefix (the exact model prefix at the audited point), or agentdojo_model_generated_next_action (the model generated the labeled action) — so that an apparent latent signal cannot be confused with a reconstruction artifact. Manifest discipline caught early prompt leakage that inflated results, which was fixed by removing proposed tool calls and attack state from capture prompts.

Six monitor families are compared: transcript keyword/rule scoring, tool policy, raw NLA keyword taxonomy, trained NLA text, a raw activation probe, and a fixed-weight hybrid. The commitment experiments add an L2-regularized logistic probe, a visible-text naive Bayes, a TF-IDF logistic baseline (unigram plus bigram, top 5,000 features, identical folds), and a zero-shot LLM judge prompted over the identical visible context. Evaluations use 5-fold cross-validation with 1,000-resample bootstrap intervals, and a leakage-controlled variant with folds grouped by user task. Behavioral labels are mechanical: the first generated tool call naming a tool in the injection task's inferred unsafe-tool set, re-derived from released exports at 280/280 agreement.

Because the project mixes mock tests, replay, and model-generated trajectories, results are reported on an explicit evidence ladder so each claim is made only at the level its artifact supports. Replications use Mistral-7B-Instruct-v0.3 and OLMo-2-1124-7B-Instruct at layer 23 of 32, chosen by matched relative depth to Qwen's layer 20 of 28 with no layer search, activations only, since the NLA verbalizer is Qwen-specific.

Why This Matters

Impact on research. The paper offers a mechanism-level account of why pre-action probe results conflict across 2026 papers: a mixed attacked-versus-clean contrast can learn exposure, an inverted exposure signal, or both axes depending on the model and population, so a headline AUROC can be certified on a detector that carries little compliance signal. It also validates the corresponding negative result (Fomin et al., 2026) as a contrast artifact rather than an absence of signal, while showing that a competent text baseline can match the recovered probes — a methodological caution for any interpretability claim resting on a weak baseline. The distinction matters against exposure probes such as Dong et al. (2026), where detecting injection presence is the goal; the author argues auditing needs the finer distinction even though a runtime defense does not.

Real-world applications:

• Pre-execution gating for self-hosted agents that send email, delete files, or exfiltrate data, where an internal signal could add lead time the transcript does not provide.
• Auditing agent deployments where visible context has been sanitized or is unavailable, the setting where activations separate most clearly from trained text.
• Validating third-party guardrail claims — a judge-shaped monitor certified on attacked-versus-clean can be tested for whether it predicts compliance or only exposure.
• Defense-in-depth design alongside capability control and information-flow defenses (CaMeL, FIDES), for the residual probabilistic surface those by-design approaches leave open.

Industry relevance. Most frontier agents are API-only, so activation access is unavailable and the technique applies to open or self-hosted models — the paper says this explicitly. The released artifacts, run commands, and manifests make the protocol reusable as an evaluation layer over existing agent-security suites without changing their threat model, and the author positions it as relevant to tool-integration layers such as MCP, OWASP defense-in-depth guidance, and NIST's request for information on securing agent systems.

Future Directions

• Adaptive attacks as a further stratum. The paper notes a monitor-aware attacker could suppress the commitment direction or sit below thresholds, and that all reported AUROCs are non-adaptive upper bounds — adaptive populations are conjectured but unmeasured.
• Live rollouts rather than generated next-action exports. Environment transitions may change the unsafe-action distribution and timing, and the reported AUPRC and FPR figures are at benchmark prevalence with clean-traffic false-alarm costs of the commitment monitors unmeasured.
• Broader coverage and better verbalization. Unsafe generation concentrates in two suites and two high-yield templates, leaving transfer tests with only 33 and 15 positives; NLA results use one verbalizer at layer 20 with one sample per activation, leaving multi-sample and multi-layer verbalization open.
• Scoring argument-level hijacking and testing frontier judges. The protocol accommodates argument-level hijacking of a required tool but does not yet score it, and the zero-shot self-forecast gap covers only the 7B model judging itself.

Target Audience

Agent-security and LLM-interpretability researchers who build or evaluate pre-action monitors; benchmark designers who need to know why an attacked-versus-clean contrast can mislead; safety engineers deciding whether a latent probe is worth deploying alongside deterministic defenses on self-hosted models; and reviewers or practitioners auditing claims that a hidden-state detector "catches" prompt injection. Readers need comfort with probe training, AUROC/AUPRC, bootstrap intervals, and cross-validation to get the most from the tables, though the exposure-versus-commitment distinction is stated in plain terms and is useful on its own.

Authors’ abstract

Tool-using language agents are vulnerable to indirect prompt injection, and a growing literature probes model internals to catch unsafe actions before they execute, with sharply conflicting results. We argue much of the conflict is an evaluation artifact, and propose a double-stratification protocol: separate injection exposure from behavioral commitment, in both the evaluation and the training population. On the primary Qwen AgentDojo grid, monitors trained on the naive mixed unsafe-versus-rest contrast mostly learn exposure: they flag exposed trajectories whether or not the agent complies, and barely predict which cases turn unsafe. Retrained on exposed cases only, the same probe families predict the model's own unsafe tool call up to two steps early, the maximum the three-step trajectories admit (logistic AUROC 0.853, matched by a trained TF-IDF text baseline at 0.855). Mistral-7B reaches 0.896 and OLMo-2-7B 0.912; their naive contrasts learn different things: inverted exposure ranking on one, both axes on the other. Representations separate on exposure (probe 1.000 versus trained text 0.908) and where visible context is sanitized or unavailable. Training contrast alone reproduces the shape of recent pre-action-probe negative results, one candidate explanation for the conflict: the signal exists, but the model's own zero-shot forecast and NLA verbalizations do not reliably recover it. An unstratified AUROC overstates what a latent monitor knows, and what a naive contrast learns is population-dependent: evaluations should stratify both.