Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

AI summary

Overview

Research area: LLM safety and alignment, specifically detecting when a language model's response violates an organization's internal policy — approached through representation analysis and out-of-distribution (OOD) detection rather than content moderation or fine-tuning.

Technical level: Advanced. The method requires familiarity with Transformer hidden states, whitening transforms, covariance estimation, Mahalanobis distance, and standard OOD scoring baselines.

Scope: A training-free method that scores policy compliance by whitening in-policy LLM activations and thresholding the Euclidean norm of the resulting representation, evaluated across five open-weight models and two benchmarks.

What This Paper Is About

Existing guardrails and content-moderation tools are built for coarse safety categories and do not scale to the dozens of nuanced, context-dependent rules that organizations actually enforce. Flexible alternatives such as LLM-as-a-judge evaluators and fine-tuned compliance detectors are accurate but add latency and training cost. This paper reframes policy-violation detection as an out-of-distribution problem in the model's activation space: if policy-compliant behavior forms a consistent region of hidden-state space, violations should appear as deviations from it, and those deviations can be detected without any parameter updates.

Key Contributions

1. Activation-space policy compliance framing. The authors formulate policy-violation detection as an OOD problem over LLM hidden activations, moving the decision away from output-level judgments of generated text.
2. Whitening-based compliance scoring. They propose a training-free, calibration-based method that estimates a whitening transform from in-policy activations, standardizes them, and scores compliance by the Euclidean norm of the whitened vector.
3. Scalable, low-overhead detection. The approach scales to large policy sets, adds minimal inference overhead, and supports continuous policy updates in enterprise settings. It works both when the monitored generator's internals are accessible and when a surrogate model must be used for black-box/API deployments.
4. Comprehensive empirical evaluation. Comparisons against LLM-as-a-judge, fine-tuned detectors, linear probes, and competing OOD baselines across multiple LLMs, noisy conditions, long-context interactions, and adversarial prompt injection.

Main Findings

• Best result of 86.0% F1. Using Qwen2.5-7B-Instruct, the method reaches 86.0% F1 on the DynaBench test set, the paper's top score. The abstract reports gains of up to 9.1 points over fine-tuned baselines and up to 16 points over LLM-as-a-judge methods.
• State of the art on four of five backbones. Full backbone results: Mistral-7B-Instruct-v0.2 66.8%, Gemma-2-9B-it 75.2%, Llama-3.1-8b-instruct 75.6%, Qwen3-8B 78.4%, Qwen2.5-7B-Instruct 86.0%. Only the Mistral backbone was relatively weaker, though it still beat several baselines.
• Comparisons. Fine-tuned detectors scored DynaGuard-1.7B 65.2%, DynaGuard-4B 72.0%, DynaGuard-8B non-CoT 72.5%, DynaGuard-8B 73.1%, and LlamaGuard-3 20.9%. LLM-as-a-judge scored GPT-4o-mini 70.1% and Qwen3-8B 60.7%. Linear probes trained on Qwen3-8B representations scored 44.7% per-category and 63.8% joint.
• Representations beat generated tokens. Applying the whitening method to the hidden states of fine-tuned DynaGuard models improved F1 over their own generation-based classifiers: DynaGuard-1.7B 65.2% to 77.6%, DynaGuard-4B 72.0% to 78.5%, DynaGuard-8B 73.1% to 80.6%. The gap between DynaGuard-8B and the base Qwen3-8B was only 2.2% F1 in favor of the fine-tuned model, suggesting much policy knowledge is already in the base weights.
• Errors spread across categories. Most errors came from high-volume categories: user experience, content controls, regulations, and tool use. Rare categories were harder per-sample — transactions (F1 0.67, FPR 0.83), non-player characters (F1 0.36, FPR 0.75), age appropriate (F1 0.44, FPR 0.60) — but together represent less than 2% of the dataset.
• Generalization to tau-bench. On 1,000 synthetic airline-domain dialogues monitored with Qwen2.5-7B-Instruct as a surrogate, the method substantially outperformed both the fine-tuned detector and the LLM-as-a-judge baseline in F1. On 25 real tau-bench policy-violating trajectories paired with an equal number of compliant ones, the automatically calibrated threshold gave only moderate F1, but separation was strong (AUC 0.87); with an ideal threshold the method surpassed GPT-4o-mini.
• Much faster and cheaper. Average runtime per sample: GPT-4o-mini 1.47s, DynaGuard-8B 2.71s, surrogate-based whitening 0.98s (Llama-3.1-8B) and 0.92s (Qwen2.5-7B), same-representation whitening 0.05s and 0.03s. Preprocessing takes under 30 minutes and roughly $23 as a one-time cost, and inference is approximately 7x cheaper than API-based LLM-as-a-judge.
• Robust to prompt injection. Under the Combined Attack from Liu et al. (2024), the detector degraded by only 4.23% relative F1, versus 23.11% for GPT-4o-mini and 21.07% for DynaGuard-8B.
• Whitening beats other OOD scores. On Qwen2.5-7B and Llama-3.1-8B respectively: Mahalanobis 67.2 / 65.8, KNN 78.5 / 66.2, Energy Score 66.4 / 72.1, Whitening 82.2 / 74.3 — leading the best competing baseline by 3.7 and 2.2 points.
• Stable under parameter changes. With Llama 3.1-8b, 100 samples per category gave 75.6% F1 and 750 samples gave 79.1%; varying top-K from 10 to 50 produced only minor fluctuation (72.4%–76.7%).
• Layer choice is category-dependent. Different categories peak at different depths — some early (e.g., Information Leakage), others mid-to-late (e.g., Transactions) — motivating category-specific layer selection. Category-specific whitening consistently outperformed a single shared transform.
• Safety alignment correlates with separability. Models scoring better on SORRY-Bench (Xie et al., 2024) and HarmBench (Mazeika et al., 2024), such as Llama- and Qwen-based variants, showed stronger in-policy/out-of-policy separation; Mistral-7B-Instruct, weaker on those benchmarks, showed reduced separation.

Methodology in Plain English

The method treats policy compliance as a shape-matching problem in the model's internal state.

Offline step. For each policy category, the authors use GPT-4o to generate a contrastive dataset of paired "good" (in-policy) and "bad" (out-of-policy) responses for each rule — 100 examples per category, split 80/20. From the in-policy examples in the 80% subset, they extract the model's hidden activations at every layer and compute a mean and covariance. They then compute a whitening transform (via PCA-based whitening), which recenters and decorrelates the activations so every dimension is comparable and the in-policy activations roughly form a zero-mean, unit-covariance cloud. Using the held-out 20% mixed set, they pick the layer where in- and out-of-policy examples separate best, and set a threshold that maximizes Youden's statistic (J = TPR − FPR). The transform and threshold for that single layer are all that gets deployed.

Online step. At runtime, a response's activation at the chosen layer is centered, whitened, and reduced to the top K components (K = 15 by default). The compliance score is just the Euclidean norm — how far the point sits from the in-policy cloud. If the score exceeds the threshold, the response is flagged. Mathematically, the squared norm in whitened space equals the Mahalanobis distance in the original space; the low-dimensional version is the Mahalanobis distance computed in the top-K principal subspace.

Surrogate mode. Because benchmarks like DynaBench supply pre-generated dialogues rather than the models that produced them, the authors monitor interactions with an observable surrogate model and run the same detector on its activations. This is the black-box setting.

Validation. A separate LLM validator (GPT-5.1) checks agreement between benchmark labels and the generated contrastive labels. A manual audit of 100 generated calibration examples (50 DynaBench, 50 tau-bench) with blinded annotators showed 95% agreement with the validator. GPT-4o is used only offline; swapping it for open-weight Llama-3.1-8B for data generation reduced F1 by only 0.9 points.

Why This Matters

Impact on research. The paper provides evidence that decision-relevant policy information is already present in base-model activations, not something fine-tuning uniquely creates. It also shows that hiding states should be modeled as an in-distribution/OOD split rather than fit with supervised classifiers — linear probing on the same representations underperformed badly. This connects representation analysis with practical AI governance and suggests a cheaper alternative to the fine-tune-plus-judge paradigm.

Real-world applications:

• Enterprise policy enforcement: monitoring customer-facing or internal LLM agents against dozens of organizational rules simultaneously, with policy updates handled by refreshing statistics rather than retraining.
• Regulated industries: legal, financial, medical, and HIPAA-relevant settings where compliance rules change frequently and must be audited.
• Agentic pipelines: checking multi-turn tool-using agents, including noisy real executions with imperfect tool calls.
• Runtime monitoring for API-based systems: the surrogate-model mode allows policy monitoring even when the deployed generator's internals are inaccessible.

Industry relevance. The cost profile is the headline for deployment: sub-second latency, negligible runtime overhead beyond the 0.03–0.05s of reading internal representations, roughly $23 of one-time preprocessing, and about 7x cheaper inference than an API judge. The 4.23% degradation under prompt injection matters for security teams, since text-level evaluators degraded by 21–23%.

Future Directions

• Handling distribution shift in the decision threshold. On real tau-bench trajectories, score separation generalized well (AUC 0.87) but the automatically calibrated threshold did not, implying lightweight recalibration from a few representative samples is needed when the deployment distribution differs from calibration.
• Low-false-positive operating regimes. The authors used a balanced operating point; systematic evaluation of thresholds tuned for strict FPR requirements remains open.
• Proactive enforcement rather than detection. The method flags non-compliance but does not intervene in generation. Using activation-space signals to steer generation toward compliant regions is proposed as a next step.
• Extending to less separable models and broader evaluation. Performance depends on how clearly policy information is encoded; weaker-aligned models such as Mistral-7B-Instruct produced reduced separation, and the paper notes rare categories (under 2% of the dataset combined) would benefit from broader evaluation.

Target Audience

LLM safety and alignment researchers, particularly those working on representation engineering and guardrails; ML engineers and platform teams responsible for deploying policy-compliant LLM applications in regulated enterprises; and AI governance practitioners who need auditability and drift monitoring without the cost of fine-tuned detectors or LLM-as-a-judge pipelines. Readers should be comfortable with activation-space methods and OOD detection literature; the paper does not report total benchmark dataset sizes beyond the category ratios and per-category construction details, so those seeking exhaustive dataset statistics will need the DynaBench reference.

Authors’ abstract

As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we derive policy-violation scores directly from normalized representations of LLM hidden activations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks. It achieves up to 86.0% F1, outperforming both fine-tuned and LLM-as-a-judge baselines while requiring substantially less computation. Our code is publicly available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection