StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance

AI summary

Overview

Research area: Computer vision — text-to-image diffusion models, visual style prompting, and training-free image style transfer.

Technical level: Advanced. The paper operates at the level of diffusion model internals (self-attention query/key/value manipulation, classifier-free guidance, denoising trajectories), so familiarity with diffusion sampling and attention mechanisms is assumed.

Scope: A training-free method for generating images that adopt the style of a reference image while avoiding the transfer of unwanted content from that reference.

Venue/status: arXiv:2510.06827v1 [cs.CV], published 08 Oct 2025, licensed CC BY-NC-ND 4.0. Work by Jaeseok Jeong and Youngjung Uh (Yonsei University) with Junho Kim, Gayoung Lee, and Yunjey Choi (NAVER AI Lab).

What This Paper Is About

Text-to-image diffusion models can be steered with both a text prompt and a reference image used as a "visual style prompt," but reference images carry both style and content. Existing methods struggle with content leakage: the pose, layout, shapes, or entire objects of the reference bleed into the output, reducing diversity and breaking alignment with the text prompt. StyleKeeper aims to separate style from content so that the reference supplies style while the text prompt fully controls content.

Key Contributions

1. CFG with swapping self-attention for T2I. The authors extend classifier-free guidance so that key and value features of self-attention layers come from a reference denoising process while the query comes from the original process, producing higher image quality and better text alignment than naive feature swapping.

2. Negative visual query guidance (NVQG). A new guidance term that deliberately simulates a content-leakage scenario — swapping queries instead of keys and values — and negates it, reducing the transfer of unwanted reference content.

3. Layer analysis for swapping self-attention. The paper identifies where in the network to swap features, finding that swapping should be restricted to the upblocks (starting at the 24th layer in SDXL) rather than all blocks, the bottleneck, or the downblocks.

4. Practical handling of real reference images. Stochastic encoding replaces DDIM inversion to map real images to latents, and color calibration matches the output's color statistics to the reference. The method also generalizes to image-to-image style transfer via ControlNet.

Main Findings

• CFG combined with swapping self-attention matters. Without CFG or NVQG, generated images suffer severe artifacts and misalignment with the prompt. Adding CFG with swapping self-attention raises image quality and makes text-specified content (e.g., "cat") clearer, but content leakage persists.

• NVQG removes content leakage. The paper reports that NVQG produces the best results, closely matching the text prompt while reflecting the style reference, and that CFG and NVQG together yield high-quality, coherent outputs aligned to both text and visual style prompts.

• Swapping location is decisive. Swapping in the bottleneck and downblocks causes content leakage (e.g., cat-like images despite a "dog" prompt), while swapping in the bottleneck alone disrupts results with scattered objects. The authors swap only in upblocks. Late upblocks attend to style-corresponding regions with more freedom to reassemble small parts, whereas early upblocks attend over wider, stylistically mixed regions.

• Not all upblock positions are equivalent. The paper reports a single fixed layer where four metrics abruptly change, and this point remains consistent regardless of the reference image; all results use this one fixed optimal layer.

• Stochastic encoding beats DDIM inversion. The reported latents from stochastic encoding lie closer to the standard Gaussian distribution (a P-value above 0.05 suggests the data likely follows it), whereas DDIM inversion accumulates numerical error and does not produce standard Gaussian noise. Color calibration further improves style similarity and text alignment.

• Quantitative style similarity. Using a Gram matrix, the reported style similarity scores are: Ours 0.791, IP-Adapter 0.768, StyleAligned 0.759, DB-LoRA 0.759, and StyleDrop 0.659. The paper states StyleKeeper achieves the best style similarity without compromising text alignment and free from content leakage, while also noting IP-Adapter shows higher style similarity but neglects text prompts significantly.

• Fewer intrusive changes to the original denoising process. With identical initial noise, the paper reports other methods severely alter pose, shape, or layout due to content leakage, while StyleKeeper reflects the reference style with minimal content change.

• I2I style transfer comparison. Compared with CrossAttn and StyleID, the paper reports CrossAttn often misses detailed reference style (rough, blocky appearance) and StyleID shows poor style reflection and color deficiency, while StyleKeeper reflects details without noticeable transfer of color values from content images.

• Ablation confirms each component. Removing swapping self-attention, NVQG, or color calibration individually degrades results in both real-reference and generated-reference settings; stochastic encoding outperformed DDIM inversion in all aspects.

Methodology in Plain English

The method runs two denoising processes side by side: one for the text prompt ("the original process") and one for the style reference ("the reference process"). In the original process's self-attention layers, the query — which encodes what is being generated — is kept, while the keys and values — which encode how it looks — are borrowed from the reference process. This is "swapping self-attention," and the authors restrict it to the upblocks, the later part of the network where attention can reassemble texture and color with fine granularity.

Because borrowed keys and values unavoidably carry some of the reference's content, the authors add a corrective term. They intentionally build a mirror-image score in which the query comes from the reference instead of the keys and values, which simulates content leakage, and subtract it with a negative weight. This is negative visual query guidance, and it acts like "not this content" guidance within the sampling equation.

To use real photographs rather than generated references, the authors add random noise to the reference according to the diffusion forward process (stochastic encoding) instead of inverting the image with DDIM, and they rescale the statistics of the predicted clean image to match those of the reference (color calibration). They evaluate using SDXL as the base model, with validation on SD v1.5, classifier-free guidance of 7.0, and DDIM sampling with 50 timesteps.

Why This Matters

The paper targets a specific failure mode that limits how useful style references are in practice — the inability to keep a reference's content out of the output — and addresses it without any additional training, which makes it cheap to adopt on top of existing diffusion models.

Real-world applications (grounded in the paper's described capabilities):

• Style-consistent content creation, such as generating new subjects in the visual style of an existing artwork or brand asset.
• Image-to-image style transfer, where StyleKeeper is combined with ControlNet to restyle a supplied content image.
• Generating diverse variations from a single reference and text prompt, since content leakage is what otherwise collapses output diversity.
• Use in personalized or themed generation pipelines, where the paper notes compatibility with DB-LoRA.

Industry relevance: The method is training-free and computationally light relative to fine-tuning approaches such as DreamBooth, textual inversion, or adapter training, which the paper criticizes for high computational cost and an inherent style–content trade-off. Experiments were conducted on the NAVER Smart Machine Learning (NSML) platform, and the work was supported by an IITP grant funded by the Korea government (MSIT) (No. RS-2024-00439762, Developing Techniques for Analyzing and Assessing Vulnerabilities, and Tools for Confidentiality Evaluation in Generative AI Models). The authors state their code will be released for reproducibility.

Future Directions

• Extending beyond images. The authors explicitly suggest exploring other domains such as video to expand applicability and research potential.
• Escaping pretrained model limits. StyleKeeper cannot generate elements outside the pretrained model's scope (the paper cites "stone golem" in Figure A24a as an example), raising the question of how to handle out-of-distribution content.
• Resolving style conflicts. When visual and textual styles conflict, the visual style dominates (Figure A24b), leaving open how to arbitrate between the two modalities.
• Reducing reliance on manual layer search. Since the method depends on a fixed optimal layer and a pre-selected upblock range, generalizing that choice across architectures and models is an open practical question.

Target Audience

Researchers and practitioners working on diffusion-based image generation, particularly those focused on style transfer, visual prompting, and training-free control of generative models. It is also relevant to engineers building production image-generation pipelines who need style reference control without training custom models, and to readers interested in the internals of attention manipulation in diffusion models.

Authors’ abstract

In the domain of text-to-image generation, diffusion models have emerged as powerful tools. Recently, studies on visual prompting, where images are used as prompts, have enabled more precise control over style and content. However, existing methods often suffer from content leakage, where undesired elements of the visual style prompt are transferred along with the intended style. To address this issue, we 1) extend classifier-free guidance (CFG) to utilize swapping self-attention and propose 2) negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. NVQG employs negative score by intentionally simulating content leakage scenarios that swap queries instead of key and values of self-attention layers from visual style prompts. This simple yet effective method significantly reduces content leakage. Furthermore, we provide careful solutions for using a real image as visual style prompts. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, reflecting the style of the references, and ensuring that resulting images match the text prompts. Our code is available \href{https://github.com/naver-ai/StyleKeeper}{here}.