Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

AI summary

Overview

Research area: Machine learning safety evaluation — specifically guardrail classifiers (fine-tuned models that score a user prompt as safe or unsafe) and the validity of the public benchmarks used to report their performance.

Technical level: Intermediate. The paper is empirical and diagnostic; it uses rank statistics and correlation analysis rather than new modelling techniques, and the core ideas are accessible without deep background in safety training. Some details (distillation, head-only calibration, Spearman correlation) assume familiarity with standard classifier training.

Scope: A diagnostic audit of one industrial guardrail deployment in which the same ~53k prompts were labelled twice under two different operational definitions of "unsafe", producing twenty models whose evaluation behaviour is then measured against fifteen public safety benchmarks.

What This Paper Is About

Guardrail papers report scores on benchmarks like HarmBench or OR-Bench as if "unsafe" were a property of the text itself, so that two systems scoring the same prompts are answering the same question. The authors argue that "unsafe" is instead a property of a labelling policy, and that a benchmark score can largely reflect whose definition of unsafe a model was trained on rather than how well it detects harm. They exploit an accident — the same prompt pool labelled twice under two different rubrics in a real deployment — to measure how much of each benchmark's score is decided by label definition rather than model capability.

Key Contributions

1. A contamination audit of the guardrail's own training corpus. The authors check every benchmark prompt against 269,861 unique prompts in every training, calibration and validation pool used in the grid. Overlap is severe and structured: 88.0% of JailbreakBench-benign (88/100) and 82.1% of OR-Bench-hard (1,083/1,319) were already in the training data, against 0% of OR-Bench-toxic and 2% of AdvBench. The two most contaminated benchmarks are the over-refusal benchmarks — the ones a guardrail paper reports to argue it does not over-block. All downstream results are recomputed on decontaminated subsets.

2. A per-benchmark measure of how much label definition decides the score. Holding architecture and optimiser family fixed and varying only how the two label families enter training, the spread across ten arms is 0.99 of full scale on HarmBench-copyright and 0.043 on WildGuard-benign — a 23× spread. The authors add a seed floor (spread across runs differing only in random seed, measured from 37 seed triplets in the codebase) because a definition-unstable benchmark is also seed-unstable.

3. A demonstration that a full-range score change costs 0.03% of parameters. Freezing the backbone and re-fitting three linear heads (0.04M of 149.05M parameters) installs policy A and moves HarmBench-copyright to 100.0 and OR-Bench-hard over-refusal to 4.7%, at essentially unchanged detection capability. A distillation teacher's soft labels do the same even when every hard label carries the opposite policy.

4. Evidence that the standard held-out split measures policy agreement, not capability. One policy-A-labelled validation split grades all twenty arms and correlates ρ = +0.74 with policy-loaded benchmarks versus ρ = +0.34 with capability-loaded ones — and is itself 69.3% contained in the policy-A training file.

Main Findings

• The training corpus contains the benchmark suite. 88.0% of JailbreakBench-benign, 82.1% of OR-Bench-hard, 25.0% of HarmBench-copyright, 14.3% of WildGuard-harmful and 12.1% of StrongREJECT were verbatim in the pools (exact match after normalisation, so a lower bound). The mechanism is convergence: the best public source of benign-but-edgy training prompts is the over-refusal benchmark built from such prompts.

• Benchmarks stratify sharply by label sensitivity. With architecture and optimiser fixed, HarmBench-copyright spans 0.99 of its range across ten arms and WildGuard-benign 0.043. The qualitative split survives the seed floor: the top of the ordering turns on contested rulings (copyright reproduction, refusal-eliciting questions), the bottom on explicit requests for weapons and violence; the middle of the ordering is unresolved.

• The last supervision signal carries the policy. At 149M, mixing policy-A data into the pool and fully fine-tuning leaves HarmBench-copyright at 0.0–1.3 across four arms (E2–E5), with OR-Bench-hard over-refusal 37.7%–56.4%. Every hard label being policy A (E6) reaches only 17.3 of 100; the inverse blend (E9) reaches 30.7 — both still distilled from a policy-B teacher.

• Two cheap routes do install the policy. A policy-A distillation teacher with all hard labels policy B (E8) gives 100.0 and 6.4%; a frozen-backbone head-only pass on 8,489 calibration examples (E1) gives 100.0 and 4.7%; extending to 53,395 examples (E7) gives 100.0 and 6.8%; and on a policy-A backbone (E10) 100.0 and 3.4%.

• The grouping is rank-separated. The arms whose last signal is policy A occupy one extreme of both HarmBench-copyright and OR-Bench-hard at both scales, giving p = 0.005 (k=4, 149M) and 0.004 (k=5, 395M) under an exchangeability argument; clustered and two-sided, p = 0.071 and 0.036. Dropping the head-calibration arms, the 395M result survives (p = 0.048) and the 149M one does not (p = 0.14).

• The move is close to free on capability. Mean catch rate over six capability-loaded benchmarks is 91.1% for E8 and 89.1% for E7, against 88.3%–86.8% for the data-mixing arms, with E1 the exception at 81.0%.

• The pattern replicates between labs. On 75 clean HarmBench-copyright prompts, Granite Guardian scores 0 and Qwen3Guard-4B scores 99 on identical inputs, because one vendor's shipped definition asks about common-sense harm and the other names Copyright Violation. Qwen3Guard's own strict/loose mapping of its "Controversial" verdict moves its 8B model from 29.7% to 48.5% over-refusal with no retraining.

• Validation accuracy measures label fit. The multilingual validation split ranks arms by policy agreement (ρ = +0.74) more than by detection (ρ = +0.34); within each scale, +0.64/+0.32 for the student and +0.83/+0.37 for the teacher. The contrast is between route blocks: within the eleven policy-B-last arms, Val correlates −0.03 with capability and −0.04 with policy; within the nine policy-A-last arms, +0.68 and +0.77. A clean split existed — the monolingual validation file is only 0.2% inside the training file — and was not used.

• Decontamination does not remove the effect. On clean subsets the two effects grow (HarmBench-copyright 0.98 to 0.99, OR-Bench-hard 0.42 to 0.57); contamination compresses between-arm variation rather than creating it. The authors caution that the OR-Bench-hard shift also reflects label composition and cannot be fully separated from memorisation.

• Against released guardrails. E8 at 149M is level with the 8B Qwen3Guard in loose mode on capability (91.1 vs 91.8) but behind in strict (96.7), at 1/55 of the parameters. On benign prompts all four policy-A-last 149M arms sit at 4.5%–6.3% over-refusal against 29.7%–30.5% for Qwen3Guard and 65.2% for Granite Guardian.

Methodology in Plain English

The authors did not set out to write this paper. While building a guardrail for an on-device assistant against a proprietary 24-pillar safety taxonomy, they labelled the same prompt pool twice, a month apart, with two pipelines that asked different questions. Policy A asked "would a thoughtful moderator block this?" and treated copyright reproduction, policy-refusal prompts and edgy-but-harmless requests as blockable. Policy B, a later LLM-jury relabelling using four large instruct models with three samples each, asked "is this a request for harmful content?" and treated those same prompts as safe. The two pipelines disagreed on roughly one prompt in six: exact agreement was 83.6% on the 97,245 prompts with a comparable 4-class label and 90.1% on the 136,472 comparable at the binary level. The contested classes were the least stable — boundary_safe retained only 71.2% and compound 78.3%, against 89.7% for generic_safe.

They then trained ten "arms" — variants differing in how, and whether, policy A entered training: blended into the pool at various ratios, as explicit anchors, as an inverse blend, as every hard label, through a distillation teacher's soft labels only, or through a frozen-backbone head-only calibration pass — at two model scales (a 394.8M-parameter ModernBERT-large teacher and a 149.0M ModernBERT-base student), giving twenty models. All twenty were scored by one harness in a single deterministic pass on fifteen public safety benchmarks totalling 33,137 prompts.

The analysis rests on three measurements. First, an exact-match overlap check between every benchmark string and every pool used anywhere in the grid. Second, the spread of each benchmark's score across the ten arms, compared against a seed floor computed from 37 triplets of runs in the same codebase that differ only in random seed. Third, a rank test: under the null that the training route is irrelevant, arm ranks are exchangeable, so the probability that a specified set of k arms occupies the k most extreme ranks is 1/C(10,k). The authors are explicit that this is their loosest reading, since three head-calibration arms are effectively one unit and the two benchmarks are one test, and they report the clustered, two-sided, and head-calibration-dropped variants alongside.

Why This Matters

The paper's claim is not that the benchmarks or the models are bad, but that an unreported fraction of what a guardrail evaluation reports answers the question "whose policy did you train on?" — a question the protocol never asks and the paper never states.

Impact on research: The paper reframes guardrail benchmark scores as partially a measure of label provenance, and argues that evaluation should report three first-class variables: a contamination check, a seed floor, and the operational definition the model was graded against. It also identifies teacher provenance as an unrecorded confound in any distilled guardrail — a team that changes its labelling policy, regenerates data, and distils from an old checkpoint ships the old policy while believing otherwise, with no in-distribution metric to warn them.

Real-world applications:

• Content platforms and moderation vendors choosing a guardrail need to know which taxonomy the reported scores were computed under, since a benchmark number can come from either extreme of a 0.99 range.
• Teams building product guardrails from public data are likely to have the same contamination structure the authors found, since boundary_safe-like classes are commonly harvested from the same over-refusal benchmarks used to evaluate them.
• Anyone deploying a guardrail with a contested middle class (Qwen3Guard's "Controversial", the authors' boundary_safe) makes a policy decision at mapping time: counting boundary_safe as unsafe moves over-refusal from 5.7% to 96.0% at 149M and 8.5% to 97.6% at 395M with no retraining.
• Model release and procurement review would benefit from a data-card field for the labelling rubric and the definition a checkpoint was last supervised against, neither of which currently exists.

Industry relevance: All three recommendations are described as cheap — an exact-match hash is "an afternoon" — and the paper reports that running the overlap audit changed which results the authors were willing to state, and that the seed floor invalidated half of what they first wrote down. The comparison against Qwen3Guard and Granite Guardian shows the same disagreement appears between independently built, commercially released systems.

Future Directions

• Multi-seed arms. The paper's main stated weakness is that each arm is a single seed and the seed floor comes from neighbouring recipes rather than the arms themselves, leaving the middle of the benchmark ordering unresolved. Repeating the grid with multiple seeds per arm would settle it.

• Near-duplicate contamination matching. The overlap audit is exact-match after case and whitespace normalisation, so every figure is a lower bound; it cannot see paraphrase, translation, or a prompt embedded in a longer string (which is why HarmBench-contextual scores 0.0%). The authors name near-duplicate matching as "the honest version."

• Separating label composition from memorisation. The OR-Bench-hard shift from 0.42 to 0.57 after decontamination reads as both label composition and memorisation, and the paper states it cannot separate them.

• Testing transfer beyond one taxonomy. The scope is twenty ModernBERT encoders, one proprietary 24-pillar taxonomy and two rubrics; whether the specific ordering transfers is open, though the qualitative claim should and the between-lab comparison reproduces it. Whether the same 0.03%-of-parameters lever generalises to decoder-based guardrails is untested.

Target Audience

Guardrail and safety-classifier practitioners; evaluation researchers concerned with construct validity, contamination and benchmark design; safety teams at product companies deciding which benchmark numbers to trust in a procurement or release decision; and ML researchers working on distillation, since the finding that a teacher's soft labels override a student's hard labels is a general behavioural-transfer result with a concrete, measurable instance here. Readers evaluating benchmark claims rather than building models will find the diagnostic framing and the three proposed reporting columns the most immediately useful parts.

Authors’ abstract

A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels. We exploit an accident of an industrial deployment in which the same ~53k prompts were labelled twice, under two operational definitions ("is this a request for harmful content?" and "would a moderator block this?"), and used to train ten guardrail arms sharing architecture, codebase and optimiser family, at two model scales. First, the guardrail's own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block. Everything below is decontaminated. Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread. Third, a benchmark score can be moved across its entire range by 0.03% of the parameters. Across these arms HarmBench-copyright spans 0-100 and OR-Bench-hard over-refusal 3.4-56.4%, and the four arms whose last supervision signal carries a given policy occupy one extreme of both (exact rank p=0.005, or 0.04-0.14 at worst), at unchanged detection capability. For a distilled student the teacher's soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint. Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator's policy (rho=+0.74), not detection capability (rho=+0.34). We argue that guardrail evaluations should report contamination, a seed floor and the definition they were graded against as first-class variables.