When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models

AI summary

Overview

Research area: Security and robustness of generative models — specifically backdoor attacks on multimodal (image plus text) conditional diffusion models for instruction-guided image editing.

Technical level: Advanced. The paper builds on diffusion model fine-tuning, backdoor poisoning, cooperative game theory (Shapley values and Shapley interactions), and CLIP-based embedding evaluation.

Scope: This paper defines, measures, and empirically confirms "backdoor modality collapse," the finding that multimodal diffusion backdoors degenerate into reliance on one dominant modality (text) rather than benefiting from combined image and text triggers.

What This Paper Is About

Multimodal diffusion models can be conditioned on more than one input type at once, such as an image plus a text instruction. The intuitive assumption is that poisoning several modalities simultaneously — for example, stamping an image trigger and appending a text trigger — should produce a stronger backdoor than poisoning any single modality, because the triggers would reinforce each other. This paper tests that assumption directly and finds it false: the backdoor collapses onto whichever modality dominates (here, text), leaving the other modality's trigger largely irrelevant.

Key Contributions

1. Identifies a new phenomenon. The paper defines "backdoor modality collapse" (Definition 4.1): after backdoor training, activation is governed by triggers from a strict subset of modalities while triggers in the remaining modalities have negligible effect. The authors state this phenomenon was previously unexplored in the literature.

2. Introduces two diagnostic metrics. Trigger Modality Attribution (TMA) quantifies each modality's share of the backdoor effect using Shapley values, while Cross-Trigger Interaction (CTI) measures whether joint triggering is super-additive (synergy) or sub-additive (redundancy/interference).

3. Provides an exact bimodal formulation. For the two-modality case (M = 2), the Shapley attributions reduce to closed-form expressions requiring only four value-function evaluations, so no Monte-Carlo approximation is needed. The four evaluations correspond to clean, image-only poisoning, text-only poisoning, and joint poisoning.

4. Validates the phenomenon across many settings. Experiments span three trigger pairs, two poisoning protocols (OR and AND), and three poisoning ratios (1%, 5%, 10%), all on InstructPix2Pix fine-tuned with LoRA, and include a sanity check ruling out the explanation that the image trigger is simply non-functional.

Main Findings

• Text modality dominates almost completely. Across every trigger pair, protocol, and poisoning ratio, the text attribution is near 1 while the image attribution is near 0. For the "white-box + mignneko" pair at 5% OR poisoning, TMA for text is 0.9743 and for image is 0.0060. The lowest image attribution observed is 0.0040 (white-box + mignneko, 10% AND), and the highest is 0.1404 (eyeglasses + anonymous, 10% OR).

• Cross-modal interaction is never positive. Every CTI value in Table 1 is negative. Examples: −0.0089 for "white-box + mignneko" at 5% OR; −0.2174 for "eyeglasses + anonymous" at 5% OR; −0.0099 for "stop-sign + latte coffee" at 10% AND. The authors interpret this as redundancy or interference rather than synergy.

• The collapse pattern is consistent, not incidental. The same qualitative result appears across three distinct trigger pairs (white-box + mignneko, eyeglasses + anonymous, stop-sign + latte coffee), both OR and AND poisoning, and all three poisoning rates.

• Individual unimodal triggers are functionally valid. The sanity-check experiment (Table 2) shows both unimodal triggers work when used alone: text-only poisoning yields ASR of 0.996 on triggered samples for all three trigger pairs with clean ASR of 0.000; image-only poisoning yields triggered-sample ASR of 0.776 (WB + MN), 0.609 (EG + AN), and 0.698 (SS + LC), with clean ASR of 0.018, 0.036, and 0.001 respectively. The image trigger is weaker but functional, so the collapse is not explained by a broken trigger.

• Qualitative visualizations match the metrics. Using 5 fixed input images with an identical prompt across four poisoning scenarios, image-only poisoning usually fails to activate the backdoor, while the text trigger almost always succeeds regardless of whether the image is poisoned. Joint poisoning produces outputs nearly indistinguishable from text-only poisoning.

• The weaker modality contributes no unique successes. In the "eyeglasses + anonymous" case, every sample successfully attacked by the image trigger is already covered by the text trigger, which the authors offer as the mechanism behind the negative CTI values.

• Two proposed explanations. The authors hypothesize (i) an optimization imbalance in which the text modality produces stronger and more consistent gradients, letting training "short-circuit" onto the text shortcut, and (ii) feature-space misalignment in which the much higher dimensionality of the image modality creates a bottleneck, so fine-grained image features such as subtle trigger patterns are compressed or discarded as redundant noise.

Methodology in Plain English

The researchers treat the two input modalities — image (I) and text (T) — as players in a cooperative game. For any subset of modalities, they activate the triggers only in that subset, generate an output, and score it by comparing its CLIP embedding to the attacker's target image versus a clean reference image. The difference between those two similarity scores is the "payoff" of that subset.

Shapley values then fairly split the payoff of the full coalition among the modalities by averaging each modality's marginal contribution across all orderings in which it could be added. Because there are only two modalities, this can be computed exactly from four payoff numbers rather than approximated. The averaged Shapley value across the validation set is the TMA metric; the gap between the joint payoff and the sum of the individual payoffs (adjusted for the empty coalition) is the CTI metric.

To test this, the authors fine-tune InstructPix2Pix — an instruction-following image editing model built on Stable Diffusion — using LoRA. They poison training data either under OR poisoning (separate equal-sized subsets for text-only, image-only, and dual-trigger poisoning) or AND poisoning (a single subset with both triggers applied together), at ratios of 1%, 5%, and 10%. Training uses the CelebA dataset, split 90% training and 10% validation with evenly spaced indices, with identity reconstruction as the clean supervision. All runs use batch size 16 and learning rate 5×10⁻⁴, for 4,000 iterations except OR poisoning at 1%, which runs for 8,000 iterations.

Why This Matters

Impact on research. Current evaluations of multimodal backdoors typically report a single overall attack success rate under joint triggers, which can hide the fact that one modality is doing all the work. This paper argues that such numbers mask a fundamental reliance on a subset of modalities, and it supplies two metrics that decompose the effect. This reframes how backdoor strength in multimodal systems should be measured and gives defenses a clearer target: the dominant modality, not all triggers.

Real-world applications:

• Image editing services that accept an image plus a text instruction, where a text-only trigger such as an appended rare token — or even an imperceptible pattern like an extra whitespace, as the paper's example describes — could force an attacker-chosen output while the image-side trigger becomes irrelevant.
• Text-to-image generation pipelines, where the same collapse logic would make a single prompt-side trigger sufficient to hijack generation.
• Model supply chains and third-party fine-tunes: the paper's results are produced with LoRA fine-tuning, a widely used low-cost adaptation method, which means a compromised adapter can carry the backdoor.
• Security auditing and red-teaming of released diffusion models, where TMA and CTI could be adopted as standard diagnostic checks rather than relying on aggregate attack success rates alone.

Industry relevance. The findings imply that a defense designed to detect or neutralize image-space triggers may provide little protection if the deployed backdoor has collapsed onto text. For providers hosting image editing APIs or distributing fine-tuned diffusion weights, the practical lesson is that auditing only one modality's trigger channel is insufficient.

Future Directions

• Extend the analysis beyond image and text to additional modalities such as audio, as the authors explicitly propose.
• Apply the TMA and CTI framework to other diffusion models and to tasks beyond instruction-guided image editing.
• Develop defense strategies informed by the collapse finding — for example, defenses that specifically target the dominant modality or that exploit the negative cross-trigger interaction.
• Unravel the underlying optimization dynamics of multimodal backdoor training, to determine whether the proposed optimization-imbalance and latent-space-misalignment explanations are the actual mechanisms.

Target Audience

This paper is most useful to security and privacy researchers studying backdoors in generative models, machine learning researchers working on multimodal learning and modality imbalance, and practitioners who deploy or fine-tune diffusion models for image editing and text-to-image generation. It is also relevant to defense researchers and auditors who need principled metrics for evaluating whether a reported attack success rate reflects genuine cross-modal coordination or a single dominant modality. Readers need familiarity with diffusion models, backdoor poisoning, and basic cooperative game theory concepts to follow the formal sections.

Authors’ abstract

While diffusion models have revolutionized visual content generation, their rapid adoption has underscored the critical need to investigate vulnerabilities, e.g., to backdoor attacks. In multimodal diffusion models, it is natural to expect that attacking multiple modalities simultaneously (e.g., text and image) would yield complementary effects and strengthen the overall backdoor. In this paper, we challenge this assumption by investigating the phenomenon of Backdoor Modality Collapse, a scenario where the backdoor mechanism degenerates to rely predominantly on a subset of modalities, rendering others redundant. To rigorously quantify this behavior, we introduce two novel metrics: Trigger Modality Attribution (TMA) and Cross-Trigger Interaction (CTI). Through extensive experiments across diverse training configurations in multimodal conditional diffusion, we consistently observe a ``winner-takes-all'' dynamic in backdoor behavior. Our results reveal that (1) attacks often collapse into subset-modality dominance, and (2) cross-modal interaction is negligible or even negative, contradicting the intuition of synergistic vulnerability. These findings highlight a critical blind spot in current assessments, suggesting that high attack success rates often mask a fundamental reliance on a subset of modalities. This establishes a principled foundation for mechanistic analysis and future defense development.