MINDiff: Mask-Integrated Negative Attention for Controlling Overfitting in Text-to-Image Personalization

AI summary

Authors: Seulgi Jeong, Jaeil Kim (corresponding author), Kyungpook National University, Daegu, South Korea arXiv: 2511.17888v1 [cs.CV], 22 November 2025 | License: CC BY 4.0 | Code: https://github.com/seuleepy/MINDiff

Overview

Research area: Computer vision and generative AI, specifically personalization of large-scale text-to-image diffusion models.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, cross-attention, and the DreamBooth personalization paradigm, though the core idea is explained through a conceptual diagram and a single modified attention equation.

Scope: The paper proposes an inference-time method — Mask-Integrated Negative Attention Diffusion (MINDiff) — that reduces subject overfitting in personalized text-to-image models by subtracting a masked "negative attention" term from the standard cross-attention output, without retraining or modifying model weights.

What This Paper Is About

When a text-to-image model such as DreamBooth is fine-tuned on a handful of photos of a specific subject, it tends to overfit: the subject's identifier token (for example, "sks") dominates the generated image, weakening the effect of the rest of the text prompt. The standard fix, a class-specific prior-preservation loss (PPL), requires extra training data and compute, and gives the user no way to steer the trade-off between subject fidelity and text alignment after training. MINDiff's goal is to achieve the same overfitting mitigation at inference time, while giving the user a tunable knob — the scale parameter λ — to balance identity preservation against prompt adherence.

Key Contributions

1. A new concept: negative attention. The authors introduce an auxiliary cross-attention branch whose output is subtracted from the main attention term, suppressing the subject's influence specifically in masked irrelevant regions. They state this is the first method to incorporate negatively weighted attention in a spatially controlled manner.

2. Overfitting mitigation without prior-preservation loss. MINDiff reduces overfitting in DreamBooth models without requiring the class-specific prior-preservation loss, and therefore without the additional training cost that PPL entails.

3. Inference-time controllability. A user-adjustable scale parameter λ lets users tune the trade-off between subject fidelity and text alignment at generation time, rather than fixing it during training.

4. No architecture or weight changes. Because the method operates entirely at inference time, it can be applied directly to models already fine-tuned with DreamBooth without modifying or re-training their weights.

Main Findings

• MINDiff versus PPL on SD 1.4: On the Stable Diffusion 1.4 backbone, at specific PPL weights (0.1 and 0.5), DreamBooth with MINDiff achieves higher scores in both text alignment and subject fidelity for certain scale ranges. The authors describe this as demonstrating MINDiff is more effective than PPL at mitigating overfitting.

• MINDiff versus NeTI: Compared to NeTI, a controllable baseline, DreamBooth with MINDiff consistently outperformed across all evaluated metrics.

• MINDiff on SD 2.1: The method offers comparable control to its SD 1.4 behavior. The improvement over prior-preservation loss is not as clear as on SD 1.4, but the authors note the inference-time controllability advantage remains valuable in practical applications.

• MINDiff on SDXL: The method also works effectively with a LoRA-based DreamBooth on SDXL, and the authors report it operates consistently across SD 1.4, SD 2.1, and SDXL.

• Prompt dilution, not inherent diversity loss: The paper reports that DreamBooth's reported degradation in output diversity is not inherent. Instead, strong influence from the learned identifier token weakens the effect of the text prompt, producing what the authors call prompt dilution.

• Background masking is essential (ablation): Without the spatial mask, negative attention removes the subject entirely — for example, "a sks dog" disappears and only the remainder of the prompt ("wearing sunglasses lying on a beach") is reflected. This shows negative attention effectively controls text influence, but that its influence must be properly constrained or subject loss occurs.

• λ behavior: As λ increases, subject influence is suppressed and generated images align more closely with the given text conditions. The authors found λ = 0.6 often provides a good balance across diverse prompts and subjects.

Methodology in Plain English

MINDiff starts from a model that has already been personalized with DreamBooth, and leaves those weights untouched. At inference, it changes how the cross-attention layers behave. Two text prompts are used: the main prompt y that describes the desired scene, and a subject prompt s containing the identifier token (e.g., "sks"). The image features become the Query; the main prompt produces the standard Key and Value matrices, and the subject prompt produces a second, auxiliary set of Key and Value matrices. The auxiliary branch is computed, scaled by λ, and subtracted from the normal attention output — the "negative attention" term. The full modified equation is:

Z = Softmax(QKᵀ/√d_k)V − λ · Mask ∘ Softmax(QK_sᵀ/√d_k)V_s

To decide where this suppression should apply, the method builds a spatial mask at each denoising step from the identifier token's attention map. Following MasaCtrl, attention maps are averaged across all heads and layers at the same timestep. MINDiff uses a 16×16 resolution for Stable Diffusion 1.4 (matching MasaCtrl's setting for SD 1.4), 24×24 for Stable Diffusion 2.1, and 32×32 for Stable Diffusion XL. The averaged map is binarized by assigning 1 to elements above the mean, which yields a subject mask; this mask is then inverted to isolate the background. That inverted background mask is what the negative attention multiplies, so subject suppression only happens in non-subject regions.

Evaluation used the DreamBooth Dataset, with each subject represented as [V]. On SD 1.4 the comparison set was Textual Inversion, DreamBooth, Custom Diffusion, and NeTI; on SD 2.1 it was AttnDreamBooth, DisenBooth, and Cones2; on SDXL, a LoRA-based DreamBooth was compared with and without MINDiff. Metrics were CLIP-T (cosine similarity between CLIP embeddings of the text prompt and the generated image) for text alignment, plus CLIP-I (similarity between generated and reference images) and the DINO metric from DreamBooth, which averages pairwise cosine similarity between DINO ViT-S/16 embeddings. The authors swept PPL weights of 0.1, 0.5, 0.75, and 1.0, and MINDiff scales from 0.1 to 1.0. Qualitative comparisons in Figures 3 and 4 use λ = 0.6, and NeTI results use a truncation value of 64 from the range (0, 128]. All images in each comparison were generated with the same random seed.

Why This Matters

Research impact. The paper reframes the DreamBooth overfitting problem: rather than accepting that fine-tuning destroys diversity and compensating with extra class data and training, it identifies prompt dilution from an overly dominant identifier token and corrects it at the attention level. Introducing negatively weighted, spatially masked attention as a controllable mechanism is a structurally distinct alternative to existing mask-based editing methods, which typically use masks to blend reconstructed and edited content. It also positions inference-time control as a first-class property for personalization, alongside training-time regularization.

Real-world applications (as grounded in the paper's framing and method):

• Personalized photo generation for people with limited mobility or accessibility needs — the introduction gives the example of generating an elderly individual with mobility impairments in wedding or travel scenarios while preserving their distinctive features and appropriately modifying the background.
• Custom avatars and character imagery — generating a consistent subject across many contexts and prompts without retraining per scene.
• Creative and design workflows — producing diverse scenes around a fixed subject or product identity while keeping prompt adherence controllable per image.
• Drop-in upgrades to existing fine-tuned models — any pipeline already holding a DreamBooth checkpoint can apply MINDiff without re-training or architectural change.

Industry relevance. Personalization fine-tuning and prior-preservation loss incur real training cost and require curated class data, which is a practical bottleneck for commercial generative tools. A method that adds controllability and reduces overfitting at inference time — with a single tunable parameter — is attractive for deployed image-generation products, and it extends across SD 1.4, SD 2.1, and SDXL, so it is not tied to a single model generation.

Future Directions

• Automating λ. The scale parameter is currently hand-tuned; the authors note that inappropriate values can cause generation failures such as missing the subject or failing to follow the prompt, and explicitly leave automatic tuning as future work.
• Resolving language drift. Strong entanglement between the subject and its class token persists: a model trained on "a photo of a sks dog" often generates the learned subject even when only "dog" is given at inference, meaning the two components are not fully disentangled.
• Choosing and improving the mask source. The mask depends on attention maps extracted from the identifier token, with resolution chosen per backbone (16×16, 24×24, 32×32); more robust or automatic region discovery is an open question.
• Broadening the evaluation. All reported personalization experiments use the DreamBooth Dataset; extending the comparison to other datasets, backbones, and multi-concept scenarios would test generality.

Target Audience

This paper is most useful to researchers and engineers working on diffusion-model personalization, controllable image generation, and attention manipulation — particularly those already using DreamBooth, Custom Diffusion, or LoRA-based fine-tuning who want to reduce overfitting without extra training. It is also relevant to practitioners building personalized image-generation products, since the method requires no retraining and adds only a single inference-time parameter. Readers without background in diffusion models and cross-attention will need to consult the cited prior work first, as the method is described at an intermediate level.

Authors’ abstract

In the personalization process of large-scale text-to-image models, overfitting often occurs when learning specific subject from a limited number of images. Existing methods, such as DreamBooth, mitigate this issue through a class-specific prior-preservation loss, which requires increased computational cost during training and limits user control during inference time. To address these limitations, we propose Mask-Integrated Negative Attention Diffusion (MINDiff). MINDiff introduces a novel concept, negative attention, which suppresses the subject's influence in masked irrelevant regions. We achieve this by modifying the cross-attention mechanism during inference. This enables semantic control and improves text alignment by reducing subject dominance in irrelevant regions. Additionally, during the inference time, users can adjust a scale parameter lambda to balance subject fidelity and text alignment. Our qualitative and quantitative experiments on DreamBooth models demonstrate that MINDiff mitigates overfitting more effectively than class-specific prior-preservation loss. As our method operates entirely at inference time and does not alter the model architecture, it can be directly applied to existing DreamBooth models without re-training. Our code is available at https://github.com/seuleepy/MINDiff.