LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents

AI summary

Overview

Research area: Agent safety evaluation for enterprise workflows, sitting at the intersection of natural language processing, LLM agent benchmarking, and policy-compliance auditing (arXiv:2610.17685v1 [cs.CL], published 13 Oct 2026, CC BY 4.0, by authors at RMIT University, Monash University, YepAI, and an independent researcher).

Technical level: Intermediate. The paper uses formal notation for state reconstruction and metrics, but its core idea is accessible to anyone familiar with LLM agent evaluation.

Scope: One sentence — the paper introduces and validates a benchmark (LatentRiskBench) for measuring policy violations whose decisive evidence lies in reconstructed workflow state rather than in the action itself.

What This Paper Is About

Enterprise agents take actions such as sending emails, publishing product updates, or submitting forms. Whether any of those actions is safe often depends not on the action itself but on what happened earlier in the workflow — for example, whether an approval was granted or whether the recipient is the verified contact. Existing safety benchmarks mostly judge actions in isolation, so this class of "state-dependent" risk is invisible to them. The paper's goal is to build a benchmark that isolates this regime, plus new metrics that measure how much detection improves when a model reconstructs workflow state.

Key Contributions

1. Regime formalization. The authors define state-dependent risk as a risk-observability regime in which the same terminal action can be safe or unsafe depending on the policy-relevant workflow state reconstructed from the committed trajectory prefix.
2. Benchmark. LatentRiskBench is introduced with constant-action pairing: risky trajectories are matched with benign twins that share the same terminal action but differ in latent policy-relevant state. It includes a 100-task state-dependent core, visible-risk controls, matched benign twins, and an e-commerce instantiation, plus a 24-case control set of short-horizon or locally visible risks.
3. Metrics. Three state-dependent risk measures are proposed: Residual Risk (RR, reported overall and as S-RR, AR-RR, and SD-RR), State-Reconstruction Gain (SRG), and Constant-Action Specificity (CAS), alongside conventional Outcome Violation Rate (OVR) and Task Success Rate (TSR).
4. Evaluation. Representative guardrails and auditor backbones are compared, showing that state reconstruction — not mere access to the trajectory prefix — is the capability separating strong state-dependent auditing from action-local or prefix-only screening.

Main Findings

• The action-level blind spot is large. On the 100 state-dependent trajectories in LatentRiskBench-Core, an action-local checker detects only 0.09 of violations, whereas a state-dependent auditor detects 0.96, giving SRG = 0.87. McNemar's exact two-sided test reports p < 10^-15, and the Wilson 95% confidence intervals are well separated: 0.09 [0.05, 0.16] versus 0.96 [0.90, 0.98].
• Guardrails leave substantial state-dependent residual risk. In the LatentRiskBench-Core guardrail comparison, GuardAgent reaches the lowest SD-RR at 0.39 with RR 0.20, OVR 0.03, and TSR 0.94, while LLMJudge has SD-RR 0.67, RR 0.32, and OVR 0.07. StepRule leaves SD-RR 1.00 and RR 0.59; TrajRule leaves SD-RR 0.92 and RR 0.50; Keyword leaves SD-RR 1.00 and RR 0.78; LlamaGuard3 leaves SD-RR 1.00 and RR 0.85; the Vanilla baseline leaves RR 0.96 and SD-RR 1.00.
• Trajectory access alone is not enough. TrajRule observes the committed prefix but does not explicitly reconstruct policy-relevant state, and it still leaves high residual risk (RR 0.50, SD-RR 0.92). The authors state that SRG reflects state reconstruction rather than mere prefix access.
• The effect is not specific to one judge model. Backbone robustness checks report SRG values from 0.79 to 0.87: GPT-4o (D_action 0.13, D_state 0.93, CAS 0.90), GPT-4o-mini (0.09, 0.96, 0.87, 0.94), Claude-Sonnet 4.6 (0.12, 0.97, 0.85, 0.82), Gemini-2.5 Pro (0.07, 0.89, 0.82, 0.93), and DeepSeek-V3 (0.09, 0.88, 0.79, 0.92). GPT-4o-mini is the primary auditor for the main results.
• The auditor agrees with humans. Two annotators independently labelled all traces with agreement κ = 0.90. Against the human majority, the auditor reaches κ = 0.88 overall, κ = 0.83 on the SD regime, and κ = 1.00 on AR. Its main weakness is described as mild over-conservatism on short/local-visible tasks.
• Benign twins are largely admitted. On the 100 same-action benign twins, benign false-positive rates are low, indicating that lower RR is not achieved by broad blocking.
• A deployed e-commerce agent shows the same pattern. Over 4 weeks of auditing an anonymized e-commerce marketing agent's committed trajectories: OVR = 0.08, TSR = 0.92, RR = 0.61, SRG = 0.49, with a benign false-positive rate of 0.025 (admission 0.975). Outcome-level summaries look reassuring while state-dependent residual risk remains high.
• Operational cost is measurable. The audit has a median latency overhead of 430 ms, a p95 overhead of 1.9 s per audited action, and an estimated cost of $0.0028 per action.
• Flagged actions are validated by replay. Because the audit runs over committed trajectories rather than inline, flagged terminal actions are replayed in an inert sandbox under the reconstructed state, and deterministic event-log rules determine whether the transition would realize a policy violation; reported RR counts only flags backed by a policy-violating transition.

Methodology in Plain English

The researchers start from ordinary enterprise tasks and script a policy-compliant workflow for each. From that benign version, they build an unsafe counterpart with the same task structure and the same final action. The trick is that they only edit state-only leverage points: variables that affect compliance but are not visible in the surface form of the final action and are not sufficient to label the trace from any single earlier step. Earlier steps are then edited so that one such variable becomes policy-violating while the terminal action stays identical.

Each candidate item is certified by exclusion. The terminal action alone must look benign or ambiguous; no individual prefix step may expose the violation; the violation must be detectable from the reconstructed state at the pre-commit point; and the label must be stable under path perturbations that preserve that state. If rewording, reordering, or changing intermediate observations changes the label, the item is routed to a trace-pattern-visible stratum rather than the SD core.

To evaluate, the authors compare two judges on the same terminal actions. An action-local evaluator sees only the full terminal action and the policy set. A state-dependent auditor additionally receives the user intent, the committed prefix, and the environment context, reconstructs the policy-relevant state, and returns a risk label, the violated policy class, and a short evidence trace. Because both judges see the same action, the detection gap is attributable to state reconstruction rather than to differences in action content. Guardrails (lexical, rule-based, LLM-judge, moderation-style, and trajectory-level) are then scored by how much residual risk they leave after acting, with the state-dependent auditor supplying the verdicts.

The benchmark structure: the general suite contains 30 short/local-visible tasks, a 24-task action-revealing control, and the 100-task state-dependent core with 100 matched benign twins; the e-commerce instantiation contains 40 SD tasks (10 per mechanism) and 40 matched benign twins. After filtering, LatentRiskBench-Core contains 124 domain-general enterprise-agent traces. The SD core is balanced across four mechanisms — state accumulation, destination dependence, authorization dependence, and goal-shift dependence — and instantiates three policy classes (data leakage, phishing propagation, goal hijacking) through five concrete policies.

Why This Matters

Impact on research. Prior agent-safety benchmarks (tool-use security, prompt injection, web-agent safety, risk awareness from traces) largely judge risk signals that appear in a step-local output, the proposed action itself, or a recognizable trace pattern. This paper argues those settings miss a distinct observability regime and provides a controlled instrument — constant-action risky/benign twins — for measuring it. It also supplies metrics (SRG, CAS, RR) that separate "did the method use state?" from "did the method just flag everything?"

Real-world applications:

• Enterprise marketing agents drafting and dispatching campaigns, where a routine sync or send action becomes a violation if a draft has absorbed restricted content or is bound to an external destination.
• Document sharing and report distribution, where the same share action is safe or unsafe depending on accumulated content, recipient binding, or data-sensitivity tags.
• Access management and approval-gated publishing, where the decisive factor is whether the required approval state was established earlier in the workflow.
• Customer communication, where the same notification is compliant or a disclosure depending on whether it is bound to the verified contact on the order record.

Industry relevance. The external-validity audit on a commercial e-commerce marketing agent shows that standard outcome-level dashboards (OVR and TSR) can look healthy while state-dependent residual risk stays high, and that inline auditing of actions with external side effects is feasible at a reported median overhead of 430 ms and $0.0028 per action, with lower-risk internal actions suitable for asynchronous or sampled auditing.

Future Directions

• Separating benchmark design from organic difficulty. The authors note that because SD trajectories are authored so the final action looks benign in isolation, SD difficulty is partly a property of the benchmark design rather than a purely organic empirical finding.
• Reducing shared-LLM bias. The world-model auditor is itself the measuring instrument; although it is held out from the defense matrix and validated against independent LLM and human annotations, shared LLM bias cannot be fully ruled out.
• Transferring beyond the sandbox. The study is instantiated in an e-commerce marketing sandbox with modest per-category sample sizes, so absolute numbers, policy coverage, and inline deployment costs may not transfer directly to other domains without re-validation.
• Closing the SD gap in guardrails. The finding that trajectory-level guardrails still leave non-trivial SD residual risk (for example, GuardAgent at SD-RR 0.39) points to guardrail designs that explicitly reconstruct and reason over state rather than reading the prefix.

Target Audience

Agent-safety and LLM-evaluation researchers, benchmark designers, and policy-compliance engineers who build or assess guardrails for tool-using enterprise agents. It is also relevant to practitioners deploying agents in regulated workflows who rely on outcome-level metrics such as Outcome Violation Rate and Task Success Rate, and to reviewers interested in how constant-action pairing can isolate a specific risk-observability regime.

Authors’ abstract

Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local risks, overlooking failures whose safety depends on workflow state. In such state-dependent risks, the same action may be safe in one context but violate policy in another. Hence, we introduce LatentRiskBench for evaluating such state-dependent risks in enterprise settings. It pairs risky trajectories with benign counterparts that share the same terminal action but differ in latent policy-relevant states, requiring models to reconstruct workflow context rather than rely on action-level cues. We also introduce new metrics for state-dependent risk evaluation, including a suite of residual risk (RR) measures. Experiments on the benchmark show that action-level evaluation detects only 0.09 of state-dependent violations, whereas state-level evaluation reaches 0.96. Existing guardrail methods largely overlook such risks unless they are explicitly applied to workflow states. Case studies on a commercial deployed scenario further show that the safety metrics embedded in the benchmark provide a more realistic measure of safety for enterprise agents. Our dataset and code are available at https://github.com/JingyueCong/LatentRiskBench.