Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance

AI summary

Overview

Research area: Generative computer vision, specifically guided sampling for diffusion models (text-to-image generation, controllable generation).

Technical level: Advanced. The paper combines diffusion sampling theory, self-attention mechanics, and optimal transport / Sinkhorn-Knopp theory, and includes formal proofs.

Scope (one sentence): The paper proposes Adversarial Sinkhorn Attention Guidance (ASAG), a training-free guidance method that deliberately degrades attention alignments inside a diffusion model by reversing the optimal-transport objective of Sinkhorn attention, and it evaluates this approach on unconditional and conditional image generation with SDXL and SD3 and on the downstream frameworks ControlNet and IP-Adapter.

What This Paper Is About

Guidance methods such as classifier-free guidance (CFG) improve diffusion samples by contrasting a desired output against a deliberately "weakened" or undesirable one, but existing ways of producing that weak signal (dropping the condition, injecting identity masks, applying Gaussian blur) are heuristic and lack a principled explanation for why they work. This paper asks what an optimal perturbation should look like, answers it using optimal transport theory, and shows that intentionally disrupting the transport cost inside self-attention layers yields an entropy-maximizing attention map that serves as a well-grounded undesirable path.

Key Contributions

1. ASAG, a new guidance method: A theoretically grounded diffusion guidance technique that adversarially disrupts attention by intentionally minimizing interaction between image embeddings, rather than heuristically corrupting attention maps.

2. A rigorous optimal transport analysis: The paper reinterprets attention scores through optimal transport theory and proves (Theorem 1, Lemma 1) that the adversarial Sinkhorn plan converges to the maximum-entropy uniform coupling as the regularization parameter goes to zero. The authors state this is the first work to reinterpret optimal transport from an adversarial perspective for perturbing attention scores in diffusion models.

3. Consistent empirical gains: ASAG improves unconditional and conditional generation across the reported metrics on MS-COCO, DrawBench, and HPD.

4. Plug-and-play compatibility with external frameworks: ASAG improves results when combined with ControlNet (Canny, depth, pose conditions) and IP-Adapter, with no additional training or fine-tuning required.

Main Findings

• Unconditional generation improves: On MS-COCO, ASAG reaches FID 92.01, KID 0.059, and Inception Score 10.54, versus Vanilla (122.07 / 0.086 / 7.052), PAG (108.63 / 0.067 / 10.46), and SEG (95.43 / 0.062 / 10.35).

• Conditional generation with SDXL improves: ASAG reports FID 23.30, CLIPScore 25.85, and ImageReward 0.459, compared with CFG (28.15 / 25.21 / 0.415), PAG (24.32 / 25.41 / 0.448), and SEG (26.80 / 25.39 / 0.431).

• Results transfer to SD3: ASAG reports FID 22.87, CLIPScore 26.33, and ImageReward 0.978, against CFG (24.19 / 26.03 / 0.931) and PAG (23.31 / 26.14 / 0.956). SEG is omitted here because it is not officially supported on SD3.

• Human-preference benchmarks favor ASAG: On DrawBench, ASAG reports CLIPScore 26.62, PickScore 21.99, ImageReward 0.316, and HPSv2 27.08. On HPD, ASAG reports 28.58 / 22.21 / 0.673 / 28.84. The paper describes ASAG with CFG as achieving state-of-the-art performance across all metrics on the human preference dataset.

• Structure is preserved where other methods diverge: Qualitatively, ASAG is reported to improve visual quality while retaining the structural intent of the original CFG or vanilla output, whereas other guidance methods often alter generation semantics.

• The adversarial cost direction matters: The ablation shows that maximizing similarity (cost M = 1 − QK^T) gives FID 111.53, KID 0.078, Inception Score 9.085 with roughly 10 Sinkhorn iterations, while ASAG's similarity-minimizing cost (M = QK^T) gives 92.01 / 0.059 / 10.54 with roughly 2 iterations. The uniform-plan extreme (P* = 1/n · 11^T) gives 92.11 / 0.058 / 9.710 but the paper reports a reduction in sample diversity.

• Overhead is small: Inference time is 1.551 seconds for ASAG versus 1.198 for CFG, 1.280 for PAG, and 1.513 for SEG, a reported increase of +0.35 seconds. Memory is 16.61 GB for ASAG versus 16.41 for CFG, 16.46 for PAG, and 16.61 for SEG, a reported increase of +0.20 GB.

• Guidance scale sweet spot: Performance improves as the guidance scale s increases up to a point and slightly degrades beyond it; the peak is at s = 1.5, which is the default used throughout.

Methodology in Plain English

Diffusion models work by repeatedly denoising an image, and at each step the model uses self-attention to decide which parts of the image should be similar to which other parts. The paper starts from an existing observation: softmax attention is equivalent to the first iteration of the Sinkhorn algorithm, a standard tool from optimal transport that produces a "doubly stochastic" attention map. The usual optimal transport reading of attention treats the attention score as a cost and minimizes it, which maximizes similarity between queries and keys and gives cleaner alignment.

ASAG flips this. Instead of minimizing the cost to maximize similarity, it defines the cost as the raw query-key product Q_t K_t^T and minimizes that, which drives similarity down. The resulting attention map collapses toward a maximum-entropy uniform plan, meaning the model's semantic preferences inside those attention layers are deliberately washed out. This degraded prediction becomes the "undesirable" branch ε̃_θ(x_t, c) in the standard guidance formula, and it is subtracted from the normal prediction with a guidance strength s.

The authors prove that as the Sinkhorn regularization parameter λ goes to zero, the adversarial transport plan converges to the uniform coupling 1/n² · 11^T, which uniquely maximizes Shannon entropy over the transport polytope. In practice they do not use that theoretical extreme, because it hurts diversity and stability; they use a small but finite λ (set to 1/√d) so the plan stays doubly stochastic and the disruption remains controlled. Only a few Sinkhorn iterations are needed because degradation does not require precise convergence, keeping overhead low. The method is applied to a subset of attention layers, and no model retraining or fine-tuning is involved.

Experiments use SDXL and SD3, with PAG and SEG at guidance scale 3.0 and ASAG at s = 1.5 with 25 sampling steps, all run on a single NVIDIA H100 GPU. Evaluation uses 30K samples from the MS-COCO validation set, 200 DrawBench prompts and 400 HPD prompts with 5 images per prompt, and 5,000 MS-COCO validation images for the guidance-scale ablation. Metrics include FID, KID, Inception Score (Diversity), CLIPScore, ImageReward, PickScore, and HPS v2.1.

Why This Matters

Impact on research. The paper supplies a theoretical explanation for a class of methods (perturbed attention guidance) that had accumulated empirical success without a principled justification. It reframes the question from "what perturbation works?" to "what is an optimal perturbation?", and it connects diffusion guidance to optimal transport in an adversarial rather than alignment-promoting direction. This gives follow-up work a formal target to build on.

Real-world applications:

• Text-to-image systems that need higher fidelity and better prompt adherence without retraining their base model.
• Controllable generation pipelines using ControlNet, where Canny, depth, and pose conditions need to be preserved while image quality improves.
• Multimodal conditioning with IP-Adapter, where the paper reports clearer structures and better fine-grained detail.
• Deployment of improvements to already-trained diffusion models, since ASAG is described as lightweight, plug-and-play, and training-free.

Industry relevance. The author is affiliated with Samsung Research, and the practical profile is notable for production settings: no retraining, a modest measured cost of +0.35 seconds and +0.20 GB over a CFG baseline, and compatibility with existing backbones (SDXL, SD3) and existing conditioning frameworks (ControlNet, IP-Adapter). That combination lowers the barrier to adopting the method inside an existing generation stack.

Future Directions

• Extension beyond image generation. The paper notes the growth of image and video generation and states that ASAG can generalize to a wide range of generative tasks, but the reported experiments cover images only, so video and other modalities remain untested here.

• Choosing the transport regime. The paper shows the finite-λ Sinkhorn plan beats both the similarity-maximizing variant and the uniform-plan extreme on diversity and stability, but how best to set λ and the iteration count across models and layers is left open.

• Layer and timestep selection. ASAG applies adversarial Sinkhorn attention to a specific subset of layers, following PAG and SEG. Which layers and which timesteps are optimal is not resolved by the reported ablation.

• Broader downstream integration. Results are shown for ControlNet and IP-Adapter only. Whether the same gains hold for other adapters, framings, or conditioning mechanisms is an open question, and SEG had to be omitted from SD3 and IP-Adapter comparisons, leaving that part of the comparison incomplete.

Target Audience

Researchers and engineers working on diffusion model sampling, guidance methods, and controllable image generation, along with anyone interested in optimal transport applications in deep learning. It is most useful to readers who already understand diffusion denoising, self-attention, and the basics of classifier-free guidance; the theoretical sections assume comfort with optimal transport notation, entropy regularization, and the Sinkhorn-Knopp algorithm.

Authors’ abstract

Diffusion models have demonstrated strong generative performance when using guidance methods such as classifier-free guidance (CFG), which enhance output quality by modifying the sampling trajectory. These methods typically improve a target output by intentionally degrading another, often the unconditional output, using heuristic perturbation functions such as identity mixing or blurred conditions. However, these approaches lack a principled foundation and rely on manually designed distortions. In this work, we propose Adversarial Sinkhorn Attention Guidance (ASAG), a novel method that reinterprets attention scores in diffusion models through the lens of optimal transport and intentionally disrupt the transport cost via Sinkhorn algorithm. Instead of naively corrupting the attention mechanism, ASAG injects an adversarial cost within self-attention layers to reduce pixel-wise similarity between queries and keys. This deliberate degradation weakens misleading attention alignments and leads to improved conditional and unconditional sample quality. ASAG shows consistent improvements in text-to-image diffusion, and enhances controllability and fidelity in downstream applications such as IP-Adapter and ControlNet. The method is lightweight, plug-and-play, and improves reliability without requiring any model retraining.