Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

AI summary

Overview

• Research area: AI safety evaluation, with methods borrowed from psychometrics (the science of measurement). The paper audits a safety benchmark rather than proposing a new model.
• Technical level: Intermediate. The framing is accessible, but the analysis relies on item response theory (IRT), multidimensional IRT, and differential item functioning, which assume some familiarity with latent-variable modeling.
• Scope in one sentence: A construct-validity audit of HELM Safety's "harmful refusal" signal, focused on the HarmBench dataset within HELM Safety v1.17.0.

What This Paper Is About

AI safety leaderboards often compress many datasets, each targeting different harmful behaviors, into one overall score. The paper asks whether a single such score can legitimately be read as measuring one thing, specifically harmful refusal, which the authors define as a model's tendency to refuse dangerous or policy-violating prompts. Working within HELM Safety, the authors find that three of the four datasets that might target harmful refusal are saturated (nearly every model passes nearly every item), leaving only HarmBench to test, and then subject HarmBench to psychometric tests of whether one attribute stands behind its score.

Key Contributions

1. Applies a construct validity framework to an AI safety benchmark. Invoking Borsboom et al. (2004), the authors argue an attribute must exist before a test can measure it, and they audit whether HELM Safety's "harmful refusal" signal meets that bar.

2. Documents saturation in three of four candidate datasets. Across the HELM Safety pool, AnthropicRedTeam, SimpleSafetyTests, and the harmful subset of XSTest have mean pass rates between 0.92 and 0.94, leaving only HarmBench (pass rate 0.67) with enough variation to model at the item level.

3. Runs two item-level psychometric tests on HarmBench. A dimensionality analysis using multidimensional item response theory (MIRT), and a differential item functioning (DIF) analysis that conditions on estimated refusal ability before comparing developer groups.

4. Shows the composite collapses distinct behaviors. The authors argue HarmBench's score merges distinct harm behaviors into one number, and HELM Safety's top-line aggregate then merges HarmBench with other datasets again, so any score averaging over datasets and items can hide saturation and conflate behaviors.

Main Findings

• Three datasets are saturated. AnthropicRedTeam, SimpleSafetyTests, and the harmful XSTest subset have mean pass rates between 0.92 and 0.94, with almost every model passing almost every item, so only HarmBench supports item-level modeling in this pool.

• A one-factor account fails the model-light check. A Bernoulli-null eigenvalue check on the HarmBench item-correlation matrix shows that several later eigenvalues exceed the random-data cutoff, whereas under one factor plus noise only the leading eigenvalue should clear it. This is evidence against unidimensionality, not an exact factor count.

• One dimension is clearly too weak. The unidimensional 1D 2PL model reaches a held-out log-loss of 0.470 (SD 0.003). Every exploratory model from 2D through 10D improves sharply, with mean log-loss around 0.28.

• Interpretable multifactor models predict better with fewer quantities. The confirmatory 7D HarmBench-domain model has the best held-out log-loss at 0.255 (SD 0.006), and the confirmatory 3D response-process model is nearly identical at 0.258 (SD 0.006) while winning AIC and BIC. Against the strongest unidimensional specification, the 1D 3PL at 0.322 (SD 0.007), the 3D model reduces log-loss by 0.064 on average (SD 0.003; range 0.060–0.067), with all five splits favoring 3D.

• The predictive advantage is stable. Brier score falls from 0.096 (SD 0.002) to 0.078 (SD 0.002), a paired reduction of 0.0176 (SD 0.0005). The mean within-split SD of held-out log-loss is 0.00037 for the unidimensional 3PL and 0.00035 for confirmatory 3D, and every confirmatory 3D restart outperforms every unidimensional 3PL restart within each split.

• The named factors are related but not interchangeable. In the 3D model, standard and contextual factors correlate at 0.785, while copyright correlates with them at 0.451 and 0.603. In the 7D model, non-copyright harm domains are often correlated between about 0.65 and 0.85, while copyright ranges from 0.34 to 0.64 against the other domains.

• Copyright is not the whole story. Removing copyright items and comparing a unidimensional 2PL with a confirmatory 2D 2PL (standard versus contextual) still favors 2D in all five splits: 0.278 (SD 0.007) versus 0.293 (SD 0.009), a paired reduction averaging 0.0149 (SD 0.0043). Brier score also improved in every split, averaging 0.0851 versus 0.0891.

• Developer-linked DIF appears under the single score. Under the single score, OpenAI and Anthropic models matched on overall ability still differ on 13 items at the family-wise Mantel-Haenszel cutoff and 17 under the logistic sensitivity check.

• Scope-specific matching removes most flags. Under the 3D response-process model, OpenAI-versus-Anthropic flags collapse to 1 (Mantel-Haenszel) and 2 (logistic). In the 7D screens, most domains show no family-wise developer DIF; the notable residual is cybercrime/intrusion with 2 Mantel-Haenszel and 4 logistic flags, plus 1 logistic flag in misinformation/disinformation.

• The reduction has more than one explanation. The authors write that the drop in flags is consistent with aggregation effects but does not establish their cause, since scope-specific ability estimates use the same items being tested and can absorb domain-specific developer differences.

• Provenance comparison shows no family-wise flags. Closed or API-accessed versus open-weight-like models show no family-wise flags under the single score, the 3D scores, or the 7D scores, even though their mean scores differ widely at 0.776 against 0.507, because DIF compares groups only after ability matching.

• The conclusion is specific to one construct reading. Multidimensionality settles the validity question for the single-score interpretation in this pool, but the evidence rejects a single dimension rather than fixing an exact count.

Methodology in Plain English

The authors treat a benchmark score as a claim, not a fact. A score is only a tally of which prompts a model passed and which it failed; reading it as a measure of one attribute requires a warrant that a single attribute drives those pass/fail responses. They test that warrant two ways.

The internal test looks only at the response matrix, the table of which models passed which items. If one attribute drives everything, one latent axis should account for the pattern. The authors run a model-light eigenvalue check comparing the item-correlation matrix against random Bernoulli matrices matched to HarmBench's shape and pass rates, then fit exploratory MIRT models from one to ten dimensions with no preassigned item labels, and finally fit two confirmatory models with predefined structure: a three-factor model splitting standard, contextual, and copyright items, and a seven-factor model using HarmBench's own harm-domain labels.

The external test brings in information absent from the response matrix: which developer built each model. If harmful refusal alone drives responses, models matched on refusal ability should pass a given item at the same rate regardless of who built them. The authors use Mantel-Haenszel screening within three ability bands as the primary test, with a ridge-penalized logistic screen on continuous fitted ability as a sensitivity check, and calibrate flags against 5,000 simulated no-DIF matrices to get a family-wise cutoff. They restrict the developer screen to groups with at least eight models, leaving OpenAI at 21 models and Anthropic at 11.

Data come from HELM Safety v1.17.0, accessed 21 June 2026. Each item's score is a continuous value averaged from two LLM judges, gpt-4o-2024-05-13 and llama-3.1-405b-instruct-turbo, on a five-point rubric. The authors binarize strictly at 1.0, treating only a unanimous perfect score as a pass, which costs HarmBench 2 items of 400. Estimation uses variational inference in py-irt, with five repeated 80/20 response-level splits, twenty random initializations per model and split, and restart selection by training log-likelihood only. Held-out log-loss is the primary criterion.

Why This Matters

The paper argues that benchmark scores already shape pre-deployment evaluations, procurement guidelines, and model cards, yet those scores are rarely audited for what they measure. Its central claim is procedural: a number offered as a measure of one attribute should earn that reading before it is used to compare models.

Real-world applications the findings bear on:

• Procurement and vendor selection. If a composite safety score blends behaviors that do not move together, two models with the same total can differ sharply on any specific behavior, and a ranking built on the composite cannot support a component-specific claim.
• Model cards and deployment documentation. Reporting one number for a multi-behavior attribute may overstate what was actually demonstrated, whereas keeping behaviors separate preserves claims that the evidence does support.
• Pre-deployment evaluation design. The two tests need only item-level responses and a grouping like model family, which the authors describe as cheap and fast enough to run on any model release.
• Interpreting developer comparisons. A developer gap can come from the way a score lumps behaviors together rather than from one developer being safer overall, which matters when scores feed into comparative claims.

Industry relevance rests on the observation that safety scores are already decision inputs and that the audits proposed here are inexpensive relative to the cost of acting on a score that does not isolate what it claims to.

Future Directions

• Apply both tests beyond HarmBench. The authors note the evidence covers one benchmark at one release and that, while their tests run on any benchmark with item-level responses, they have not yet shown the same structure elsewhere.
• Replace the composite with separate behaviors. The paper's recommended reading of HarmBench for this pool is the narrow one that keeps behaviors apart, which points toward leaderboards that report attribute-level results rather than a single top-line number.
• Replicate against HarmBench's original scoring pipeline. The authors did not replicate the analysis using HarmBench's classifier-based attack-success-rate pipeline, either for the full dataset or a subset, so the item-level responses, dimensional structure, and DIF pattern could differ under that scoring.
• Address small, dependent model groups in DIF. With only 21 OpenAI and 11 Anthropic models, the authors flag the adequacy of group-specific item estimation, anchor selection, and uncertainty calibration as open issues, and note that within-developer clustering may inflate observed developer DIF.
• Test alternative scoring thresholds. The authors binarized a five-point rubric at a strict cutoff, which is conservative but discards gradation a different cutoff might surface.

Target Audience

This paper is most useful to AI evaluation researchers and benchmark developers who want to know whether their scores can support the inferences drawn from them, and to psychometricians interested in how measurement theory transfers to language model evaluation. It also speaks to policy analysts, procurement teams, and safety practitioners who consume leaderboard scores, since its practical recommendation is to demand attribute-level evidence before relying on a composite number. Readers need no psychometric background to follow the argument, though the model comparisons in the results section reward familiarity with IRT fit statistics.

Authors’ abstract

Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles. Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute. One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts. We examine whether it constitutes a single, measurable attribute in HELM Safety. Using a construct validity framework that stipulates that an attribute must exist before a test can measure it, we start with HELM Safety's four datasets that might plausibly target harmful refusal, but find that three are saturated. We subject the remaining dataset, HarmBench, to two psychometric tests to determine if a single attribute like harmful refusal could stand behind its score. First, multidimensional item response theory modeling strongly suggests that HarmBench does not measure a singular attribute. Second, a differential item functioning analysis finds items where models from different developers with the same refusal ability score differently. These flags largely disappear under scope-specific matching, a pattern consistent with aggregation effects but not sufficient to rule out domain-specific developer differences. Zooming out, HarmBench collapses distinct harm behaviors into one score, and the overall HELM safety aggregate further collapses HarmBench and scores from other datasets into a single top-line number. Any safety score that averages over datasets and items can hide saturation and conflate behaviors this way. We argue that a score should earn its single-attribute reading before models are compared with it.