Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment

AI summary

Overview

Research area: Safety alignment for Multi-modal Large Language Models (MLLMs), focused on multi-turn, multi-image conversations rather than single-turn visual question answering (VQA). The work sits at the intersection of vision-language modeling, reinforcement learning from human feedback, and AI safety.

Technical level: Advanced. The paper assumes familiarity with RLHF, DPO, PPO, GRPO, KL-regularized policy optimization, and multi-modal alignment pipelines.

Scope (one sentence): The paper introduces MINT-Safe, an open-source multi-modal multi-turn safety dataset, and TAD-Align, a two-stage alignment framework that uses a turn-aware dual-objective reward function to improve conversational safety in MLLMs without large-scale manual preference annotation.

What This Paper Is About

Existing MLLM safety alignment methods are built mainly for single-turn VQA, so they do not capture risks that accumulate across a dialogue, where an adversary can gradually reconstruct harmful intent turn by turn and bypass safety constraints that look adequate at any individual turn. The authors build a multi-turn, multi-image safety training dataset (MINT-Safe) using a multi-agent attack simulation pipeline with text-to-image (T2I) tool calls, then train a two-stage alignment framework (TAD-Align) that first teaches refusal behavior and then optimizes helpfulness and safety jointly at the turn level. The goal is to reduce harmful outputs in open-ended multi-modal conversation while avoiding the helpfulness loss and the prohibitive manual annotation cost typical of prior RLHF-based safety methods.

Key Contributions

1. MINT-Safe dataset. An open-source visual multi-turn training dataset comprising 11,270 multi-image dialogues and 500 refusal VQA pairs, built through multi-agent interaction with T2I tool-call augmentation. Dataset statistics report an average of 4 turns and 1.53 images per dialogue, and 20 dialogue categories.

2. TAD-Align framework. A dialogue safety alignment framework centered on a turn-aware dual-objective reward function. It uses rollout-based safety score variance to dynamically identify turns where the model behaves inconsistently on safety, and adaptively up-weights those turns during optimization.

3. Two-stage training recipe. A refusal template learning phase (supervised fine-tuning on roughly 500 instances, about 60% refusal scenarios and 40% professional task examples) followed by GRPO-based reinforcement learning that jointly optimizes helpfulness and safety over full conversation histories.

4. Empirical evaluation. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B across multi-modal multi-turn safety benchmarks and general capability benchmarks, reporting reductions of over 10% in Attack Success Rate (ASR), improvements of at least 8% in harmlessness and 13% in helpfulness on multi-modal multi-turn safety benchmarks, while preserving general model capabilities.

Main Findings

• Large ASR reductions on SafeMT for Qwen2.5-VL-7B. ASR on SafeMT drops from 48.92% for the base model to 28.06% with TAD-Align, while helpfulness rises from 0.5 to 0.6319 and harmlessness from 0.5 to 0.5819 (win rates, per Table 2).

• Strong gains on SafeMT for LLaVA-NeXT-7B. TAD-Align reaches 82.10% helpfulness and 69.18% harmlessness on SafeMT, described as a 32% enhancement in utility and 19% in safety over the base model, with ASR of 38.44% versus 48.95% for the base model.

• Cross-benchmark improvement beyond SafeMT. For Qwen2.5-VL-7B with TAD-Align, JailbreakV ASR falls to 11.87% (from 44.29% at base), MM-SafetyBench ASR falls to 17.26% (from 48.63%), and MMSafe-PO ASR falls to 12.00% (from 14.55%). For LLaVA-NeXT-7B, JailbreakV ASR is 35.16% (from 74.89%), MM-SafetyBench 43.33% (from 61.90%), and MMSafe-PO 10.55% (from 17.82%).

• Baselines underperform on multi-turn dialogue. Baseline methods (RLHF-V, Safe RLHF-V, MM-DPO, SPA-VL) show varying gains on JailbreakV and MM-SafetyBench but consistently fall short on SafeMT, where harmful intent is diluted across multiple turns. SPA-VL achieves low ASR in places but its helpfulness and harmlessness scores often fall below the base model, which the authors attribute to over-refusal.

• Refusal quality matters, not just refusal frequency. On OR-Bench (LLaVA-NeXT-7B), the score is 0.0750 for TAD-Align versus 0.0850 for the vanilla model and 0.1750 for SPA-VL. On PHTest-harmless, TAD-Align scores 12.76% versus 17.00% vanilla and 27.01% SPA-VL.

• Professional-domain behavior. In Table 3, TAD-Align lowers the reported domain scores relative to base models for Legal Opinion (LLaVA-NEXT: 0.3000 to 0.2077; Qwen2.5-VL-7B: 0.3000 to 0.1308), Health Consultation (0.1560 to 0.0826; 0.1927 to 0.0917), and Financial Advice (0.4012 to 0.3473; 0.4192 to 0.2036). The authors state the goal is not to give definitive professional decisions but to refuse harmful or misleading instructions and direct users to domain experts.

• General capabilities are largely preserved or improved. On Qwen2.5-VL-7B, TAD-Align gives MMDU 4.92 (base 4.85), ConvBench 59.97% (52.36%), ScienceQA 81.11% (81.80%), MathVista 52.56% (50.11%), Geometry3k 31.11% (27.79%), and MathVerse 30.71% (29.47%). On LLaVA-NeXT-7B, it gives MMDU 4.64 (base 4.15), ConvBench 22.24% (19.41%), ScienceQA 62.27% (61.33%), MathVista 29.00% (28.36%), Geometry3k 4.66% (5.49%), and MathVerse 14.87% (14.01%).

• Refusal template learning is necessary before RL. The ablation shows that turn-aware dual-objective RL alone produces only modest gains; adding refusal template learning first improves performance across benchmarks and dimensions. The one exception noted is Qwen2.5-VL-7B-Instruct, which shows slightly less safe performance on JailbreakV after RL than after refusal-template learning alone.

• Model-specific differences. LLaVA-NeXT significantly outperformed Qwen2.5-VL-7B-Instruct during training. The authors attribute this to LLaVA's earlier release and inherently more vulnerable safety mechanisms, plus weaker instruction-following that instruction tuning can substantially improve.

• Dataset quality control result. A random 10% of the retained dataset was manually annotated and reached 86% agreement with human annotators, which the authors state passes the standard.

Methodology in Plain English

The researchers first built a training dataset. They took harmful scenarios and goals from three public multi-modal safety datasets — the training subsets of JailbreakV-28k, BeaverTails-V, and SPA-VL — spanning 14 safety-relevant categories. Each scenario was then run through a three-role multi-agent simulation: an Attack Planner that designs the attack strategy and turn-level guidance, an Attacker that writes adversarial prompts turn by turn without overt harmful vocabulary, and a Responder that plays the role of a deployed MLLM under attack. When the Attacker decides extra visual context would help the attack, it calls an image generation tool, and the generated image is injected into the dialogue as a multi-modal payload. To clean the result, multiple LLM-based judges independently score each dialogue for safety and conversational consistency; only dialogues with unanimous approval are kept, and duplicate images produced by attack-driven generation are removed.

They also built 500 refusal VQA pairs. Of these, 300 are general harmful inquiries drawn from SPA-VL by category-stratified sampling across 14 safety categories, with refusal responses prefixed by "I'm sorry" generated by GPT-4o. The remaining 200 cover professional boundaries: 100 health consultation questions from JailbreakV and 100 financial questions from MME-Finance, with manually standardized refusals such as "I don't have the necessary certifications; please consult with a professional expert."

Training proceeds in two stages. First, refusal template learning: supervised fine-tuning on about 500 instances (roughly 60% refusal scenarios and 40% professional task examples) so the model learns to refuse with reasons before any RL begins. Second, GRPO-based reinforcement learning, which does not need a separate critic model because it estimates a baseline from group scores across multiple sampled responses.

The distinctive part is the reward function. For each dialogue turn, an evaluator (InternVL3-78B, run at temperature=0) assigns a helpfulness score and a safety score conditioned on the full dialogue. Because GRPO already generates 8 rollouts per instruction, the authors reuse that diversity: they compute the variance of safety scores across rollouts at each turn. High variance means the model sometimes refuses and sometimes answers, signaling ambiguous safety boundaries. They combine variance with mean safety as U_t = Var_t^safe · exp(−mean safety), so turns that are both uncertain and unsafe get more attention, while consistently safe turns are down-weighted. These uncertainty values are softmax-normalized into turn weights α_t, and the total reward is the weighted sum over turns of (β · helpfulness + safety), with β set to 0.1.

Experiments used 8 H800 GPUs with 8 rollouts per instruction, a global batch size of 128, and a PPO mini-batch size of 8. Only 7,000 dialogues were used for the turn-aware dual-objective training stage, for fewer than 15 epochs. Evaluation used win rate (GPT-4o pairwise comparison, adapted from Safe RLHF-V) as the primary metric and ASR as a complementary absolute metric on adversarial queries. SafeMT was evaluated with 8-turn dialogues.

Why This Matters

Impact on research. The paper shifts safety alignment evaluation from single-turn VQA toward realistic multi-turn, multi-image conversation, and provides an open dataset and framework for a setting where prior methods were shown to be fragile. It also offers an alternative to large-scale manual preference annotation by deriving reward signals from rollout variance.

Real-world applications:

• Conversational assistants that handle images, where users may probe a model's boundaries across many turns rather than in one query.
• Health and financial advisory settings, where the paper's design explicitly aims to make models refuse unsafe or misleading instructions and redirect users to certified professionals.
• Content moderation and red-teaming of deployed multi-modal chat systems, using the attack simulation pipeline and category taxonomy as an evaluation lens.
• Safety auditing of open-weight MLLMs being fine-tuned or deployed in consumer-facing products.

Industry relevance. Because the method reduces reliance on costly human preference annotation, it targets a practical bottleneck for teams aligning multi-modal models at dialogue scale. The reported preservation of general capabilities (MMDU, ConvBench, ScienceQA, MathVista, Geometry3k, MathVerse) matters for products where safety tuning often degrades task performance, and the release of a training dataset and a public model checkpoint lowers the barrier to reproducing multi-turn safety alignment.

Future Directions

• Robustness testing is missing. The limitations section states that the authors did not test on dedicated adversarial jailbreak benchmarks, so the model's actual robustness against intentional exploits remains uncertain and empirically unverified.

• Extension to reasoning models. The investigation is limited to general-purpose multi-modal models; how the training paradigm affects dedicated reasoning models, where safety constraints interact with complex logical deduction, has not been assessed.

• Reducing dependence on external judges. The current approach relies on a stronger external model for LLM-as-a-judge supervision, which introduces dependence on that teacher model's quality and biases. The authors suggest exploring self-correction, where a model scores its own generations, to reduce reliance on external judges and improve alignment autonomy.

• Scaling the training signal. Only 7,000 dialogues were used for the turn-aware dual-objective stage due to compute and time constraints, leaving open whether larger-scale training with the full 11,270-dialogue pool changes the results.

Target Audience

Researchers and engineers working on MLLM safety alignment, multi-turn dialogue safety, and RLHF-style optimization; teams building or deploying multi-modal conversational agents who need to handle adversarial multi-turn interaction; and dataset builders or red-teaming practitioners interested in automated, multi-agent construction of safety training data with T2I augmentation. Readers without a background in reinforcement learning or vision-language models will find the reward-function details and training setup difficult to follow.

Authors’ abstract

Despite remarkable capability in multi-modal understanding, deploying Multi-modal Large Language Models (MLLMs) in open-ended conversational scenarios introduces safety risks that remain poorly addressed by existing alignment methods. Unlike simple malicious visual question and answer (VQA) pairs , multi-turn interactions enable adversaries to incrementally reconstruct harmful intent across dialogues, progressively bypassing safety constraints in ways that are difficult to detect at any individual turn. Meanwhile, conventional reinforcement learning from human feedback (RLHF) approaches are unsuitable for this situation: designed primarily for VQA tasks, they neither capture cross-turn risk dynamics nor scale efficiently without costly manual preference annotation. To close this gap, we introduce \textbf{MINT-Safe}, an open-source visual multi-turn training dataset comprising 11,270 multi-image dialogues and 500 refusal VQA pairs, constructed via multi-agent interaction with text-to-image (T2I) tool-call augmentation. Building on MINT-Safe, we propose \textbf{TAD-Align}, a dialogue safety alignment framework centered on a turn-aware dual-objective reward function. Rather than treating all dialogue turns uniformly, TAD-Align leverages rollout-based safety score variance to dynamically identify turns where the model exhibits inconsistent safety behavior, and adaptively up-weights these turns during optimization. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B demonstrate reductions of over 10\% in Attack Success Rate (ASR), alongside improvements of at least 8\% in harmlessness and 13\% in helpfulness on multi-modal multi-turn safety benchmarks, while preserving general model capabilities.