Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming

AI summary

Overview

• Research area: AI red teaming, adversarial testing, and jailbreaking evaluation; evaluation methodology for generative AI systems.
• Technical level: Intermediate. The paper is argument- and framework-driven rather than a new-model paper, but it assumes familiarity with concepts such as attack success rates, estimands, statistical inference, and measurement theory.
• Scope: A position and methods paper arguing that attack success rate (ASR) comparisons in AI red teaming frequently do not support the comparative conclusions drawn from them, and providing a formal framework plus empirical case studies for when such comparisons are meaningful.

What This Paper Is About

AI red teaming increasingly reports attack success rates (the fraction of attacks judged successful in eliciting undesirable behavior) and uses them to claim that one system is more vulnerable than another, that a mitigation improved safety, or that one attack method is superior to another. This paper asks a simple but hard question: when can attack success rates be meaningfully compared? The authors argue that many such comparisons are apples-to-oranges comparisons or rest on low-validity measurements, using jailbreaking as the running example.

Key Contributions

1. A two-part sufficient condition for meaningful ASR comparison. The authors state that ASRs can be meaningfully compared only when there is (a) conceptual coherence — it is meaningful to compare the underlying population parameters (estimands) — and (b) measurement validity — the observed ASRs are valid measurements of those parameters.
2. A formal measurement-theory instantiation of jailbreaking. Drawing on social science measurement theory and inferential statistics, the paper recasts ASRs as measurements arising from systematization, operationalization, and execution, and defines a probabilistic threat model for jailbreaking, M = (s, D, C).
3. A conceptual coherence case study on aggregation. A three-part case study shows that differing aggregation schemes (for example, Top-1 versus one-shot) change the estimands being compared, making reported ASR gaps uninterpretable as evidence of method superiority.
4. An empirical and conceptual treatment of measurement validity. The authors document disconnects between the target prompt distribution and the prompts actually used, and between the "oracle" success criterion and the operational judge, including the bias introduced by judge error.

Main Findings

• Comparing ASRs is comparing estimands, not numbers. The authors distinguish descriptive claims (A_T = 0.65 > 0.4 = A_C as numbers) from inferential claims (generalizing to a population) and evaluative claims (for example, that a treatment or attack method is "superior"). The same structure applies to jailbreaking and to a hypothetical clinical trial where 3-year survival is 0.65 for treatment and 0.40 for control.
• GCG versus Generation Exploitation is an apples-to-oranges comparison. In Section 4.4 of Huang et al., the reported ASRs on Llama 2 13B-Chat are A_GE = 0.89 and A_GCG = 0.31. GCG is run for 500 steps and a single output is generated per prompt, producing a one-shot metric, while GE sweeps 49 configurations with 8 responses each (49 times 8 = 392 responses) and takes a Top-1 (of 392) metric.
• Repeated sampling under non-deterministic decoding can trivially inflate ASR. For prompts with a non-negligible one-shot success probability p_0 ≥ 0.01, the probability of at least one success in 392 attempts is 1 − (1 − p)^392 ≥ 1 − 0.99^392 = 0.98. Moving from one-shot to Top-1 aggregation is therefore enough to raise ASR substantially.
• Decoding configuration per se is not what drives the increase. Replicating Huang et al. for Llama 2 7B and 13B chat models on the same 100 base prompts from MaliciousInstruct, with 49 sampled responses per prompt from each of 49 configurations, one-shot ASR remains stable around 0.2 across configurations except at the highest temperatures, while Top-1 ASR increases with temperature. The authors note the original experiments considered temperature up to 1.0 and that they additionally tested t in {1.5, 2.0}.
• High entropy plus Top-1 aggregation explains the effect. As temperature increases, entropy of the per-prompt success probability distribution increases: many prompts' success probability decreases, but more prompts move away from effectively-zero success probability. The average one-shot success probability is mostly flat, yet Top-1 metrics under resampling grow rapidly because 1 − (1 − p_0)^K grows quickly for even moderate K.
• Sophisticated jailbreaks may not beat a simple resampling baseline. In the large-scale evaluation study of Chu et al., which compares Top-1 (of approximately 50) ASRs of 17 jailbreak methods across 8 models, the authors repeated the baseline experiment for Llama 2 7B Chat using the 160 publicly available base prompts, finding a Top-1 ASR over 50 repeated samples of baseline prompts at temperature 2.0 of 0.83. The best performing jailbreak identified in that study (LAA, reported ASR 0.88 with a plus-minus 0.04) cannot be directly compared because the judge systems differ, but the authors expect the judges to largely agree; ASRs for many other jailbreaks on Llama 2 were much lower, most below 0.6.
• Baseline ASRs are often implausibly high. Chu et al. report baseline ASRs ranging from 0.31 (Llama 2 7B Chat) to 0.52 (Vicuna 7B), with baseline ASRs as high as 0.8–0.94 for categories such as Adult Content and AI Usage Disclosure.
• Many "harmful" prompts are not clearly harmful. The 160 prompts in Chu et al. cover 16 violation categories (10 prompts each) drawn from usage policies of major LLM service providers (Google, OpenAI, Meta, Amazon, Microsoft). Inspecting the Adult Content and Political Activities prompts, the authors find that complying with many would not violate any reasonable interpretation of the actual usage policies. For the Political Activities category, of the 10 prompts just 1 is a clear violation of the OpenAI policy if complied with, 7 are borderline, and 2 are clearly non-violating — which the authors say makes the reported claim of ASR over 0.80 on GPT-3.5 and GPT-4 misleading, as a failure of face validity and content validity.
• Judge error introduces systematic bias. Because the operational judge J is a proxy for the oracle success criterion s, differential true positive rates or false positive rates across target systems can produce large ASR differences even when the estimands are identical. The paper analyzes how E(ASR) can deviate from α as TPR and FPR vary along an ROC curve with AUC 0.91, and under fixed judge accuracy P(J = s) of 0.7, 0.8, and 0.9.
• Success criteria are seldom systematically defined. The authors cite Feffer et al.'s notion of dissentive risks for contested outcomes such as stereotyping or hate speech, and Yu et al.'s observation that "a clear discussion of the principles determining whether a response is jailbroken remains absent." Without systematization, judge accuracy cannot be meaningfully interpreted.
• Reporting practices hinder reanalysis. ASR definitions are frequently ambiguous and computations are often omitted from accompanying code, and coherence or validity problems are often infeasible to correct after the fact unless experimental data is logged at a very granular level. The authors conducted their own experiments partly because reanalysis of logged data was insufficient.

Methodology in Plain English

The authors combine conceptual argument, formal framework construction, and original experiments.

First, they build an analogy between ASR comparisons and clinical treatment superiority studies, to show that both involve descriptive, inferential, and evaluative claims, and that the objects actually being compared are population parameters estimated by observed rates.

Second, they instantiate jailbreaking in a formal measurement-theory framework inspired by Adcock and Collier and by recent genAI evaluation work (Wallach et al.). In this framework, a jailbreaking protocol is the operationalized activity; ASRs are measurements; and the estimands are attack success probabilities defined by a probabilistic threat model. They define this threat model as M = (s, D, C), comprising an oracle success criterion s, a goal distribution D over base harmful prompts, and conditions C. The judge J:(R;P) maps to {0,1} is the operationalization of s, and the set of base prompts D is the operationalization of D. Measurement error is the gap between the ASR estimate and the target estimand; conceptual gaps arise from poor systematization.

Third, they walk through examples from well-cited jailbreaking studies to demonstrate conceptual incoherence, and run their own replication experiments. For the decoding-configuration study they use Llama 2 7B and 13B chat models with the same 100 MaliciousInstruct base prompts, collecting 49 sampled responses per prompt per configuration across 49 configurations, and comparing Top-1 ASR against one-shot ASR. For the baseline study they use the 160 publicly available base prompts from Chu et al.'s work with Llama 2 7B Chat, reporting a Top-1 ASR over 50 repeated samples at temperature 2.0.

Fourth, they analyze judge error analytically, showing how E(ASR) relates to α as TPR and FPR vary along an ROC curve with AUC 0.91, and under fixed judge accuracy of 0.7, 0.8, and 0.9. Related work is deferred to Appendix A, with additional measurement theory background in Appendix B, aggregation discussion in Appendix C, prompt analysis in Appendix D, and experiment details in Appendix E.

Why This Matters

• Impact on research: The paper challenges a widespread evidentiary practice in AI safety evaluation. If ASR comparisons are often conceptually incoherent or invalid, then a substantial body of published claims about relative system safety and attack method efficacy may not be supported by the reported evidence. It supplies a vocabulary (conceptual coherence, measurement validity, systematization, operationalization, execution) for diagnosing these problems.
• Real-world applications:
  • Evaluating whether a model provider's safety mitigation actually improved system safety, rather than merely changing aggregation or decoding settings.
  • Assessing whether a newly proposed jailbreak genuinely outperforms simple repeated resampling of the base prompt.
  • Auditing red-teaming prompt sets for face and content validity before using them to draw policy-violation conclusions.
  • Interpreting judge-based success adjudication when the judge's true positive and false positive rates may differ across systems being compared.
• Industry relevance: Red-teaming results feed into model release decisions, safety claims, and policy compliance narratives. The authors' finding that many prompts labeled "harmful" would not violate reasonable readings of provider usage policies, and that baseline ASRs can be as high as 0.8–0.94 in some categories, has direct implications for how providers and auditors present and consume safety evaluations.

Future Directions

• Documenting ASR computation. The authors recommend that studies state clearly how reported ASRs were computed, since conceptual incoherence is undetectable without this information; they note ASR definitions are frequently ambiguous and computations are often missing from code.
• Granular logging of experimental data. Because coherence and validity problems are often infeasible to correct post hoc, the paper calls for logging at a very granular level to enable reanalysis.
• Stronger baselines for new jailbreaks. Studies introducing new jailbreaks assessed with Top-1 (of K) metrics should demonstrate that they definitively outperform a repeated-sampling baseline; the authors also point to Best-of-N jailbreaking, which applies a randomly selected perturbation to the prompt each time, as a stronger baseline.
• Better systematization of contested concepts. The paper leaves open how to systematize success criteria for dissentive risks such as stereotyping or hate speech, and how to resolve conceptual debates about whether the refusal-as-safety paradigm is the right systematization — a paradigm the authors note is increasingly called into question.
• Congruent comparison designs. The authors suggest, as one example, computing Top-1 ASR over a subset of the 392 responses from the 500 τ's constructed across GCG's optimization steps, illustrating that more congruent comparisons are constructible even if not yet standard.

Target Audience

This paper is most useful to AI red teaming and safety evaluation researchers, benchmark and evaluation designers, and ML security researchers who report or consume attack success rates. It also serves model providers, auditors, and policy analysts who rely on jailbreaking results to make safety claims or compliance arguments, and methodologists interested in applying social science measurement theory and inferential statistics to machine learning evaluation.

Authors’ abstract

We argue that conclusions drawn about relative system safety or attack method efficacy via AI red teaming are often not supported by evidence provided by attack success rate (ASR) comparisons. We show, through conceptual, theoretical, and empirical contributions, that many conclusions are founded on apples-to-oranges comparisons or low-validity measurements. Our arguments are grounded in asking a simple question: When can attack success rates be meaningfully compared? To answer this question, we draw on ideas from social science measurement theory and inferential statistics, which, taken together, provide a conceptual grounding for understanding when numerical values obtained through the quantification of system attributes can be meaningfully compared. Through this lens, we articulate conditions under which ASRs can and cannot be meaningfully compared. Using jailbreaking as a running example, we provide examples and extensive discussion of apples-to-oranges ASR comparisons and measurement validity challenges.