Identically Trained Language Models Break on Different Jailbreaks

AI summary

Overview

Research area: AI safety and language model robustness — specifically jailbreak evaluation and the reproducibility of safety measurements in natural language processing.

Technical level: Intermediate. The paper is readable with a basic grasp of how language models are pretrained, safety-tuned, and evaluated, plus what a correlation coefficient means; no specialized mathematics is required.

Scope (one sentence): A controlled replication study asking whether two identically configured training runs of the same model produce the same jailbreak vulnerabilities, using 150 models spanning 25 data recipes, 3 pretraining seeds, and 2 post-training seeds.

What This Paper Is About

Jailbreak benchmarks typically evaluate a single trained checkpoint and treat the set of attacks that succeed against it as a stable property of the model's pretraining data and safety training. The authors ask whether that assumption holds: if you retrain the same model from scratch on the same data and give it the same safety post-training, will it fail on the same jailbreak prompts? The goal is to determine how much of a model's jailbreak profile is attributable to its data versus to the randomness of the particular training run.

Key Contributions

1. A replication-based framing of jailbreak evaluation. The paper reframes "which jailbreaks break this model" from a data property into a question about training-run variability, and designs an experiment that isolates pretraining run, data recipe, and post-training seed as separate factors.
2. A large controlled model suite. For each of 25 pretraining data recipes, the authors train 1B-parameter base models from 3 pretraining seeds, then apply identical safety post-training with 2 post-training seeds each — 150 models in total.
3. A per-prompt non-refusal correlation analysis. Every model is sampled 20 times per jailbreak prompt in a shared prompt set, and per-prompt non-refusal rates are correlated between every pair of models to quantify how similar their vulnerability profiles are.
4. Practical transfer and fingerprinting tests. The authors measure whether the best attack template found on one checkpoint remains best on another (over 91 templates), and whether a model's per-prompt refusal pattern can identify which pretraining run produced it (over 3,211 prompts).

Main Findings

• Post-training seeds barely matter. Two post-trained copies sharing the same pretraining run have per-prompt non-refusal rates that correlate at 0.984 on average, indicating near-identical jailbreak profiles.
• Pretraining seeds matter a great deal. Changing only the pretraining seed drops the correlation to 0.811 — a much larger change than changing the post-training seed.
• Data recipe matters only slightly more than seed. Switching to a different pretraining data recipe lowers the correlation only a little further, to 0.779, so the specific training run explains more variation than the training data does.
• The pattern is robust. The same ordering holds across all 25 data recipes and also appears at 150M and 300M parameter scales, not just at 1B.
• Attack rankings do not transfer across pretraining runs. The best of 91 attack templates found on one checkpoint remains the best after a new pretraining seed in only 27% of model pairs, compared with 67% when only the post-training seed changes.
• Each pretraining run leaves a detectable fingerprint. Using per-prompt non-refusal rates across 3,211 prompts, a model's pretraining run is identified among all 75 runs in every case tested; a random subset of just 50 prompts still succeeds 98% of the time.

Methodology in Plain English

The authors build a grid of models rather than a single model. They pick 25 different recipes for pretraining data, train three separate base models per recipe using different random seeds, and then give each base model the same safety post-training twice, again with different seeds. That yields 150 models whose differences can be attributed to a known factor.

Each model is then shown a shared set of jailbreak prompts, with 20 sampled responses per prompt, and the researchers record how often the model fails to refuse. That produces a per-prompt non-refusal profile for each model. Comparing two models means correlating these profiles: a high correlation means the two models are vulnerable to the same prompts, and a low correlation means they break in different places.

To connect this to practice, the authors test 91 attack templates against the models, rank them by effectiveness, and check whether the top-ranked template on one checkpoint is still top-ranked on another. Finally, they test whether the refusal profile itself is distinctive enough to serve as a signature, checking how often it identifies the correct pretraining run among all 75 runs, and how small a random subset of prompts still suffices.

Why This Matters

Impact on research. Jailbreak benchmark results are commonly reported for a single checkpoint and read as properties of a data recipe or a safety-training method. If a different pretraining seed with identical data produces meaningfully different vulnerabilities, then single-run benchmark numbers carry substantial hidden variance, and comparisons between methods may partly reflect seed luck. The work argues for multi-seed evaluation and for reporting run-to-run variability alongside benchmark scores.

Real-world applications:

• Safety evaluation and red-teaming. Red teams can expect that attacks tuned on one training run will frequently fail to transfer to a sibling model, so attack discovery must be repeated per deployed model rather than assumed portable.
• Model release gating and auditing. Organizations that approve releases based on jailbreak evaluations should account for the fact that a passing score on one run does not guarantee the same robustness for a retrained or re-run version.
• Provenance and attribution. The fingerprinting result suggests refusal behavior could help identify which training run — and potentially whose — a model came from, which is relevant to model supply chains and to detecting unauthorized retraining.
• Evaluation design for smaller budgets. Because a 50-prompt subset retains 98% identification accuracy in this setting, the fingerprinting approach may be cheap enough for routine use.

Industry relevance. Safety benchmarks are increasingly used as release criteria and in regulatory and procurement discussions. A finding that the dominant source of variation in jailbreak vulnerability is the pretraining run rather than the data recipe implies that evaluation claims made from one checkpoint are weaker than typically presented, and that reproducibility of safety properties deserves the same scrutiny as reproducibility of capability results.

Future Directions

• Explain the mechanism. The abstract establishes that the pretraining run matters most but does not say why; identifying which aspects of run-to-run nondeterminism drive divergent jailbreak profiles (data ordering, initialization, optimizer effects) is the obvious next question.
• Test at larger scale and with other alignment methods. The reported robustness checks cover 150M, 300M, and 1B parameters with one style of safety post-training; whether the same ordering holds for frontier-scale models and for other safety-tuning approaches is untested here.
• Turn fingerprinting into a practical tool. Open questions include how the refusal-profile signature survives further fine-tuning, quantization, or deliberate evasion, and how few prompts are needed in more realistic, less controlled settings.
• Rework evaluation protocol. A natural follow-up is to define how many pretraining seeds a credible jailbreak benchmark needs, and how results should be reported so that seed variance is visible rather than hidden inside a single number.

Target Audience

Safety and alignment researchers who run or consume jailbreak benchmarks; red-team practitioners who need to know whether discovered attacks will transfer; evaluation and benchmark designers concerned with reproducibility; ML training engineers who manage large multi-seed training campaigns; and policy or audit staff with enough ML background to interpret correlation-based robustness claims. Readers looking for mechanistic explanations of why runs diverge, or for results at frontier scale, will find this paper raises those questions rather than answering them.

Authors’ abstract

Jailbreak benchmarks test one checkpoint and treat which jailbreaks break it as a property of its pretraining data and safety training. We ask whether another pretraining run of the same data, given the same safety training, would break on the same jailbreaks. For each of 25 pretraining data recipes, we take 1B base models from 3 pretraining seeds and give each the same safety post-training with 2 post-training seeds, for 150 models in all. We then collect 20 responses from every model to a shared set of jailbreak prompts and correlate the per-prompt non-refusal rates between every pair of models. On average, two post-trained copies of the same pretraining run have per-prompt non-refusal rates that correlate at 0.984. A different pretraining seed lowers the correlation to 0.811, and a different data recipe lowers it only slightly further, to 0.779. These results indicate that which jailbreaks break a checkpoint depends much more on its pretraining run than on its data, a pattern that holds in all 25 data recipes and at 150M and 300M parameters. In practice, the best of 91 attack templates found on one checkpoint stays the best after a new pretraining seed in only 27% of model pairs, against 67% after a new post-training seed. Each pretraining run also leaves a fingerprint: across 3,211 prompts, a model's per-prompt non-refusal rates identify its pretraining run among all 75 runs every time, and a random subset of 50 prompts still does so 98% of the time.