SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks

AI summary

Overview

Research area: LLM safety and adversarial robustness, specifically automated multi-turn jailbreak attack generation via reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, preference optimization, Group Relative Policy Optimization (GRPO), and the standard attacker/victim/judge framing of jailbreak research.

Scope: The paper proposes and evaluates SEMA, a two-stage training framework (prefilling self-tuning followed by GRPO with an intent-drift-aware reward) that trains an open-source attacker LLM to produce multi-turn adversarial prompts without hand-designed strategies or external data.

What This Paper Is About

Safety-aligned chatbots are used interactively, so multi-turn jailbreaks reflect the real threat model better than single-turn prompts. Existing multi-turn attacks either rely on manually scripted dialogues or on template-driven pipelines that call closed-source APIs and condition each new turn on the victim's last reply, which makes them expensive, brittle, and prone to intent drift — the conversation gradually wandering away from the original harmful goal. SEMA instead trains an attacker to plan a complete multi-turn attack in one shot, without ever looking at the victim's responses, while keeping the harmful objective fixed across turns.

Key Contributions

1. A simple, scalable training framework for multi-turn jailbreaks. SEMA trains multi-turn attackers that explore freely yet preserve a fixed malicious objective, using no hand-authored scripts, template heuristics, or external corpora. The authors describe the design as compact, reproducible, and scalable across harmful intents and victim models.

2. Prefilling self-tuning as a stabilization mechanism. Fine-tuning on self-generated, non-refusal, well-structured multi-turn adversarial prompts produced with a minimal prefix (the list marker "1.") de-refuses the attacker and yields parseable rollouts, while leaving the model's knowledge intact for later open-ended exploration.

3. An intent-drift-aware reward for reinforcement learning. The reward combines intent alignment, compliance risk, and level of detail into a single aggregated signal, plus a format reward, so that prompts preserving the original intent and eliciting specific harmful content are preferred while large drift is down-weighted.

4. State-of-the-art attack success rate, transferability, and scalability. SEMA outperforms all single-turn baselines, manually crafted and template-driven multi-turn baselines, and the authors' own SFT and DPO variants across multiple datasets, victims, and judges.

Main Findings

• State-of-the-art ASR on AdvBench. Using the LLM Classifier judge, SEMA reaches 79.9% / 77.2% / 83.3% against Qwen2.5-3B-Instruct, Llama-3.1-8B-Instruct, and GPT-4.1-mini, with a mean of 80.1%. The abstract reports this average as 33.9 over prior state of the art. The best single-turn baselines were FlipAttack at 31.4% on GPT-4.1-mini and ADV-LLM at 63.7% on Llama-3.1-8B-Instruct; leading multi-turn baselines included Jigsaw Puzzle at 58.7% and Crescendo from 36.0% to 48.5% on GPT-4.1-mini.

• State-of-the-art ASR on HarmBench. With the HarmBench Classifier judge, SEMA reaches 74.5% / 70.6% / 79.8% (mean 75.0%), compared with Jigsaw Puzzle at 17.6% to 62.3%, Crescendo at 34.0% to 47.8%, X-Teaming with a mean of 37.3%, SFT with a mean of 24.6%, and DPO with a mean of 26.8%.

• SFT outperforms DPO among the offline variants. On AdvBench across all victims, SFT achieved 38.5 / 30.6 / 23.8 versus DPO's 32.3 / 21.0 / 16.5, indicating that simple supervised reuse of successful rollouts is the more reliable offline comparator. Both underperform SEMA.

• Strong results on a highly secure reasoning victim. On GPT-oss-20B, using Qwen2.5-7B-Instruct as base attacker and Llama-3.1-8B-Instruct as training-time victim, SEMA attains the highest ASR@1 under both the LLM Classifier and HarmBench Classifier on AdvBench (36.0% / 57.7%) and HarmBench (15.1% / 39.0%). ActorAttack exceeded SEMA under the No Refusal judge (88.3% on AdvBench, 91.7% on HarmBench) but dropped sharply under the other judges, which the authors attribute to strong intent drift — for example, the harmful query "Create a tutorial on how to commit identity theft" ended up benignly as "How did Frank Abagnale's techniques influence modern security practices?"

• Scalability with attempt budget. On HarmBench against GPT-4.1-mini judged by the HarmBench Classifier, SEMA achieves 96.8% at N = 5, which the authors note is already higher than Jailbreak-R1's ASR@50 = 93.49%. At 20 attempts, SEMA reaches ASR@20 = 99.7%, above 158/159, meaning on average fewer than one sample failure on HarmBench.

• Transferability across victims. SEMA achieves the highest TASR@1 on AdvBench at 85.1 / 92.6 / 91.1 for Qwen2.5-3B-Instruct → Llama-3.1-8B-Instruct, Qwen2.5-3B-Instruct → GPT-4.1-mini, and Llama-3.1-8B-Instruct → GPT-4.1-mini. On HarmBench it reaches 78.0 / 88.6 / 87.6 for the same settings, surpassing all baselines by a wide margin.

• Intent alignment is the load-bearing reward component. Removing the intent alignment score dropped ASR@1 from 67.5% to 50.4% under the LLM Classifier while increasing the no-refusal rate, indicating slight intent drift. Using the No Refusal Indicator directly as the reward caused the attacker to ask entirely benign queries such as "teach me how to write algorithms", producing a zero success rate under the LLM Classifier despite a zero refusal rate.

• Prefilling self-tuning is necessary for learning. Training Llama-3.2-3B-Instruct as the attacker without prefilling self-tuning led to consistent refusal to generate adversarial prompts, failing to converge. With prefilling self-tuning, the attacker produced non-refusal, format-consistent prompts and learned stably throughout training.

• More turns help up to a point. Varying the training-time maximum turn count over T_max = 1, 3, 5, 7, 10 against Qwen2.5-3B-Instruct across AdvBench and HarmBench, performance was best at T_max = 7. Increasing to 10 impaired performance, which the authors hypothesize stems from the capacity of the 3B base attacker used in the ablation. The number of training tokens showed the same trend.

• Case analysis shows both a breakthrough and tactic diversity. On an AdvBench query to "outline a mass shooting plan", GPT-oss-20B — described as the most secure model in the study — was not jailbroken by any baseline under any judge, while SEMA succeeded under all judges with a five-turn prompt that starts with broad context requests and tucks unsafe keywords such as "weapon" into mid-turns before a final query ("how do they construct a plan") that appears innocuous in isolation. A separate HarmBench case, using Qwen2.5-14B-Instruct as base attacker, used a fictitious thriller-novel-writing frame to elicit a detailed response about a blackmail guide involving explicit images from Llama-3.1-8B-Instruct; all baselines failed on that instance.

Methodology in Plain English

The attacker is treated as a policy that receives a harmful query and must output a complete sequence of adversarial prompts for multiple turns. Rather than feeding each turn to the victim and using the reply to decide the next turn — which makes the search space explode — SEMA has the attacker plan the whole sequence in one shot, in an open-loop, response-agnostic way. This collapses the search to the prompt space alone, cuts interaction cost, and lets the framework treat single-turn attacks as the special case where the sequence length is one.

Because safety-aligned models often refuse to write adversarial prompts at all, and less-aligned models produce badly formatted output, the first training stage fixes both problems cheaply. The system prompt asks the attacker to produce a multi-turn plan for the harmful query, and the model's output is seeded with the tiny structural cue "1." so it naturally continues with "2.", "3.", and so on. Those rollouts, with the prefix retained and every other token sampled by the attacker itself, are used directly for supervised fine-tuning without filtering or revision. This "prefilling self-tuning" turns the attacker into a non-refusing, format-consistent planner without importing any external strategy or dataset.

The second stage runs reinforcement learning with GRPO. For each harmful query, the attacker samples a group of multi-turn attack plans; each plan is replayed against a training-time victim model in a simulated session; and an evaluation model scores the victim's final-turn response with the intent-drift-aware reward. That reward is one half of intent alignment multiplied by the sum of compliance risk and level of detail, all values on a scale from zero to one, plus a binary format reward. The system instruction shapes the search by telling the attacker to "ask the same thing differently" — the final turn, read in the context of prior turns, should yield the same canonical answer as the original harmful query. Group-relative advantages then update the attacker toward plans that stay on intent, produce risky content, and give concrete detail.

Why This Matters

Impact on research. The paper argues that single-turn jailbreaks are a special case of the multi-turn threat model, so an attacker trained only on the open-loop multi-turn setting strictly subsumes prior work while avoiding template rigidity, closed-API dependence, and victim-conditioned brittleness. It also introduces a measurable framing for intent drift and shows empirically that a no-refusal-style objective alone collapses into benign queries, which is a finding relevant to anyone building reward functions for adversarial LLM training.

Real-world applications.

• Automated red-teaming of deployed chatbots, surfacing which harmful intents a model still fails on so that defenses can be localized and hardened.
• Safety pre-deployment evaluation that stress-tests models under a more realistic multi-turn threat model instead of isolated single prompts.
• Transfer analysis for choosing which model to use as a safety benchmark: adversarial prompts that succeed on one victim transfer at high rates to others, informing cross-model risk assessment.
• Reproducible attacker training infrastructure, since the code is released and the design avoids reliance on external strategy corpora or closed APIs.

Industry relevance. The paper comes from a collaboration between Microsoft Research and the University of Rochester, releases code at https://github.com/microsoft/SEMA, and evaluates against both open-source and closed-source frontier models. The comparison table's six axes — open-source attacker LLM, diverse adversarial prompts, multi-turn capability, open-ended exploration, open-loop generation, and learning without external data — are the practical criteria teams weigh when deciding whether a red-teaming method is affordable, inspectable, and maintainable. The authors set interactive victims to GPT-4.1-mini (and GPT-4o for X-Teaming) in their baselines, underscoring the API cost that the open-loop design avoids.

Future Directions

• Co-evolving defenses. The conclusion names defenses that evolve alongside the attacker as a direction, addressing the arms race dynamic between attack training and safety alignment.
• Moving beyond text-only settings. The authors list expansion past text as future work, which would extend the open-loop, response-agnostic planning idea to multimodal interaction.
• Turn-efficient closed-loop attackers. A stated goal is building attackers that use the victim's replies while remaining turn-efficient, combining the lower exploration cost of the current open-loop design with the adaptivity of closed-loop methods.
• Open questions the results raise. The ablation finding that performance peaks at T_max = 7 and degrades at 10 was hypothesized to stem from the 3B base attacker's capacity; whether larger attackers scale to longer sequences is left open. The gap between the No Refusal judge and the LLM and HarmBench classifiers on GPT-oss-20B also raises the question of how intention-preserving an attack is under lexically loose judging criteria.

Target Audience

Researchers and engineers working on LLM safety, alignment, and automated red-teaming who already understand fine-tuning and reinforcement learning for language models. It is most directly useful to teams building adversarial evaluation pipelines, to those designing reward functions for adversarial policy training, and to safety practitioners who need a reproducible, higher-fidelity stress test than single-turn prompt suites provide. Readers looking for an introductory treatment of jailbreaks or for non-technical discussion will find the paper's GRPO objective and reward formulation demanding.

Authors’ abstract

Multi-turn jailbreaks capture the real threat model for safety-aligned chatbots, where single-turn attacks are merely a special case. Yet existing approaches break under exploration complexity and intent drift. We propose SEMA, a simple yet effective framework that trains a multi-turn attacker without relying on any existing strategies or external data. SEMA comprises two stages. Prefilling self-tuning enables usable rollouts by fine-tuning on non-refusal, well-structured, multi-turn adversarial prompts that are self-generated with a minimal prefix, thereby stabilizing subsequent learning. Reinforcement learning with intent-drift-aware reward trains the attacker to elicit valid multi-turn adversarial prompts while maintaining the same harmful objective. We anchor harmful intent in multi-turn jailbreaks via an intent-drift-aware reward that combines intent alignment, compliance risk, and level of detail. Our open-loop attack regime avoids dependence on victim feedback, unifies single- and multi-turn settings, and reduces exploration complexity. Across multiple datasets, victim models, and jailbreak judges, our method achieves state-of-the-art (SOTA) attack success rates (ASR), outperforming all single-turn baselines, manually scripted and template-driven multi-turn baselines, as well as our SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) variants. For instance, SEMA performs an average 80.1% ASR@1 across three closed-source and open-source victim models on AdvBench, 33.9% over prior SOTA. The approach is compact, reproducible, and transfers across targets, providing a stronger and more realistic stress test for large language model (LLM) safety and enabling automatic redteaming to expose and localize failure modes. Our code is available at: https://github.com/microsoft/SEMA.