TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewards

AI summary

Overview

• Research area: AI safety and adversarial machine learning — specifically automated red-teaming of large language models through multi-turn jailbreak attacks.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning (policy gradients, advantage estimation, GRPO), preference optimization, and LLM safety evaluation.
• Scope: The paper introduces TROJail, a reinforcement learning framework that trains a small attacker LLM to conduct multi-turn jailbreaks against black-box victim models by optimizing the entire interaction trajectory rather than individual turns.

What This Paper Is About

Jailbreak attacks are attempts to bypass an LLM's safety guardrails and elicit harmful responses. Single prompts often fail, so attackers increasingly spread malicious intent across a multi-turn conversation — an approach that mirrors realistic user–model interaction. Existing trainable attackers, however, optimize each turn in isolation to maximize the harmfulness of that turn's immediate reply, which prevents them from discovering strategies that only pay off later in a conversation.

TROJail reformulates the task as a multi-turn reinforcement learning problem in which the attacker is rewarded for the harmfulness of the final response in the whole trajectory. Because that reward arrives only at the end, the authors add two intermediate "process rewards" that tell the attacker how useful each middle prompt is.

Key Contributions

1. A trajectory-level RL formulation of multi-turn jailbreaking. Rather than scoring each turn independently, TROJail maximally optimizes the harmfulness of the final-turn response, enabling the attacker to plan across the full conversation.
2. Two heuristic process rewards to combat sparse supervision. The first penalizes overly harmful intermediate prompts that trigger the victim's refusal mechanism; the second rewards a steady increase in the semantic relevance between intermediate responses and the original harmful request.
3. Integration into a multi-turn GRPO objective. The process rewards are combined with the outcome reward inside the advantage estimator, giving dense, per-turn credit assignment without requiring a separately trained reward model.
4. Broad empirical validation. TROJail improves attack success rates over single-turn, turn-level, and outcome-only RL baselines across four victim models and three benchmarks, while also demonstrating strong transferability to unseen victims.

Main Findings

• Trajectory-level optimization beats turn-level optimization by a wide margin. A naive GRPO baseline trained only on the final-response outcome reward reaches an average attack success rate (ASR) of 81.43, far above the best turn-level method (X-Teaming, 73.06) and single-turn methods (AutoDAN-Turbo, 60.80).
• Process rewards provide additional, complementary gains. Adding all three reward components (outcome, over-harm penalization, semantic relevance progression) raises the average ASR to 86.23, the best result across all benchmarks and victims.
• Explicit, task-informed process rewards outperform implicit ones. GRPO with an implicit process reward reaches 83.68, which is better than outcome-only training but still below TROJail's 86.23 — implicit rewards learned from sparse outcome signals fail to capture the specific patterns that make multi-turn attacks work.
• Intermediate responses are a poor reward signal on their own. The harmfulness reward spikes only at the final turn of successful attacks, whereas semantic relevance to the original prompt rises gradually and consistently across turns. This makes semantic drift a usable, dense training signal.
• Overly aggressive prompts are counterproductive. Controlled intervention experiments show that outcome reward rises when moderately harmful prompts are inserted into a trajectory, then collapses once prompt harmfulness crosses a threshold and triggers the victim's refusal behavior.
• Ablation confirms both process rewards matter. On Gemma-2-9B-IT, outcome reward alone yields 65.39 average ASR; adding only over-harm penalization gives 75.50; adding only semantic progression gives 70.43; combining both gives 77.75.
• The learned strategy transfers to unseen victims. Attackers trained against one victim still jailbreak other models. Training against more robust victims (Llama-3.1, Gemma-2) produces the most transferable attackers (82.22 and 84.55 average out-of-domain ASR), while training against easily jailbroken victims (Mistral, 55.86) generalizes worse.
• Performance scales with the turn budget. ASR increases monotonically with the maximum allowed turns, though it saturates around four turns for Mistral and Qwen2.5 and more gradually for Llama-3.1. TROJail was trained with a five-turn limit but keeps improving beyond it, and plateaus around 7–8 turns — earlier than training-free baselines.
• TROJail degrades gracefully on harder prompts. As prompts are grouped by how many baseline methods fail on them, all methods lose accuracy, but TROJail's decline is much shallower. It also allocates more turns on average to harder prompts, showing adaptive attack planning.

Methodology in Plain English

The authors treat the attacker LLM as a reinforcement learning agent. At each turn, the attacker sees the original harmful request plus the conversation so far and writes the next prompt; the victim model replies; the conversation continues up to a fixed turn limit.

The training loop works like this:

1. For each harmful request, sample several complete attack conversations (trajectories) from the current attacker policy.
2. Score each trajectory by how harmful the victim's final response is — this is the outcome reward.
3. Normalize these scores across the sampled trajectories to get an advantage for each trajectory.
4. Because that advantage is identical for every turn and therefore uninformative about which middle prompts mattered, compute two extra per-turn scores. The first is zero if the victim refused and otherwise equals the harmfulness of the victim's reply. The second is the cosine similarity between the sentence embedding of the victim's reply and the original harmful request, scaled by the turn index so later turns are weighted more.
5. Convert these per-turn scores into a cumulative process advantage by summing them from the current turn to the end of the trajectory, subtracting the mean and dividing by the standard deviation.
6. Add the process advantage to the outcome advantage, weighted by a coefficient, and update the attacker with a multi-turn version of GRPO using a clipped surrogate objective plus KL regularization against a reference policy.

The attacker is Qwen2.5-3B-Instruct, trained on 520 harmful prompts from AdvBench and evaluated against Qwen2.5-7B, Llama-3.1-8B, Gemma-2-9B, and Mistral-7B on HarmBench, StrongREJECT, and JailbreakBench. Success is judged by the HarmBench classifier with a 0.9 threshold.

Why This Matters

The paper's argument is that understanding adaptive, multi-turn adversarial behavior is a prerequisite for defending against it. Current safety alignment is mostly evaluated against single-prompt attacks, but real misuse unfolds over a conversation with a model that is actively adapting to the victim's responses. By showing that trajectory-level reinforcement learning produces markedly stronger attackers than turn-level training, the paper raises the bar for what a model's safety must withstand.

Real-world applications:

• Safety evaluation and red-teaming. Automated attackers like TROJail can be used by labs to stress-test models before deployment at a scale that manual red-teaming cannot match.
• Alignment training data generation. The attack trajectories expose specific failure modes — refusal-threshold sensitivity and semantic drift — that can be turned into training or preference data for safety fine-tuning.
• Serving-layer guardrail development. Understanding how attackers pace harm across turns informs the design of input and output filters that track conversation state rather than judging prompts in isolation.
• Agent and API security. Multi-turn models that browse, use tools, or execute code are exposed to longer interaction horizons where the vulnerabilities studied here become directly exploitable.

Industry relevance: Any organization deploying a chat model behind an API needs a quantified picture of its multi-turn vulnerability. TROJail provides both a measurable metric (ASR under an adaptive trajectory-optimizing attacker) and a method for producing it, which is relevant to model providers, safety teams, and regulators evaluating deployment risk.

Future Directions

• Explicit diversity optimization. The authors note that TROJail relies on entropy regularization rather than directly optimizing for diverse attack strategies. Multi-objective reinforcement learning that jointly optimizes effectiveness and diversity is a natural extension.
• Integration with defensive mechanisms. The current framework trains against a static victim. Training against adversarially hardened or continually updated safety policies would test whether the learned strategies remain effective against adaptive defenses.
• Better credit assignment across turns. The process rewards are heuristics derived from controlled experiments. Learning more principled per-turn value estimates — or validating the two chosen signals against other candidate feedback signals — remains open.
• Understanding when and why transfer works. The finding that attackers trained on more robust victims transfer better is suggestive but not explained. Identifying what a generalizable attack policy actually learns would inform both attack and defense research.

Target Audience

This paper is most useful to AI safety and alignment researchers, particularly those working on red-teaming, adversarial robustness, and reinforcement learning for language models. It is also relevant to model developers responsible for pre-deployment safety testing, to researchers studying multi-turn dialogue and agentic systems, and to policy or governance analysts who need concrete evidence about how LLM safety degrades over extended interactions. Readers without a reinforcement learning background will need to consult the GRPO and advantage estimation literature to follow the method section in full.

Authors’ abstract

Large language models have seen widespread adoption, yet they remain vulnerable to multi-turn jailbreak attacks, threatening their safe deployment. This has led to the task of training automated multi-turn attackers to probe model safety vulnerabilities. However, existing approaches typically rely on turn-level optimization, which is insufficient for learning long-term attack strategies. To bridge this gap, we formulate this task as a multi-turn reinforcement learning problem, directly optimizing the harmfulness of the final-turn response as the outcome reward. To address the sparse supervision of the outcome reward, we introduce TROJail, which employs two process rewards to evaluate the utility of intermediate prompts and integrate them into advantage estimation. These rewards (1) penalize overly harmful prompts that trigger the model's refusal mechanism, and (2) encourage steering the semantic relevance of responses toward the targeted harmful content. Experimental results show improved attack success rates across multiple models and benchmarks, highlighting the effectiveness of our approach. The code is available at https://github.com/xxiqiao/TROJail. Warning: This paper contains examples of harmful content.