Text to Sketch Generation with Multi-Styles

AI summary

Overview

Research area: Computer vision — text-conditioned sketch generation and exemplar-based style transfer using diffusion models.

Technical level: Advanced. The paper assumes familiarity with diffusion models, self-attention key/value (K/V) injection, classifier-free guidance, and AdaIN feature-statistic modulation.

Scope: The paper introduces M3S, a training-free framework built on Stable Diffusion v1.5 and SDXL that synthesizes sketches from text prompts while matching one or more referenced sketch styles.

What This Paper Is About

Existing sketch-generation systems can produce sketches from text, but they offer little precise control over how a sketch looks — stroke thickness, sparsity, or artistic style. This paper asks how to make a pretrained text-to-image diffusion model draw in the style of one or more supplied example sketches, without training any new network. The authors' answer, M3S (Multi-Style Sketch Synthesis), injects features from the reference sketches as auxiliary information rather than overwriting the model's own features, and adds a knob for blending multiple styles.

Key Contributions

1. A training-free multi-style sketch synthesis framework. Key and value features from referenced style sketches are treated as auxiliary information (concatenated with the target's own K/V) plus linear smoothing and a style-content guidance mechanism, balancing style consistency against content fidelity.
2. A joint AdaIN modulation mechanism for style tendency control. By modulating the latent noise with statistics from multiple references, users can continuously control how much each style dominates the output, using a single parameter.
3. A style-content guidance scheme that splits classifier-free sampling into separate content and style directions, letting users trade off expression against structure.
4. Implementation and validation on two backbones (Stable Diffusion v1.5 and SDXL) with experiments across six sketch styles, reported as outperforming state-of-the-art baselines.

Main Findings

• Direct K/V substitution fails on sketches. The authors report that swapping reference K/V matrices into target generation (as in StyleAligned-style approaches) causes structural incoherence and artifactual patterns, especially when reference and target have low structural similarity. Rigid statistical constraints (AdaIN applied to Q and K) also harm sketch quality.
• Concatenation plus linear blending works better. Concatenating reference features with target features improves line quality; adding linear blending with hyperparameter λ further reduces content leakage.
• M3S (SDXL) achieves the best average CLIP-T of 0.3514 in Table 1, described as superior text alignment. On DINO and VGG style metrics, M3S slightly trails AttentionDistillation, which the authors attribute to a deliberate style-content trade-off.
• Style-oriented parameters close the style gap. With reduced content guidance ω1, increased style guidance ω2, and λ = 0.0 (denoted "Ours (SDXL ∗)"), style metrics become comparable to AttentionDistillation while retaining slight CLIP advantages.
• Human preference favors M3S. In a 1–8 ranking study, M3S (SDXL) scored 6.19 and M3S (SD v1.5) 5.44, versus StyleAligned 2.77, CSGO 3.83, RB-Modulation 4.20, StyleStudio 4.22, AttentionDistillation 4.28, and InstantStyle 5.08. Of 58 submissions, 14 invalid responses were excluded, leaving 44 validated questionnaires (average completion 4m16s).
• Statistical significance. M3S (SD v1.5) significantly outperformed all baselines except InstantStyle (p = 0.26); with the backbone aligned to SDXL, M3S (SDXL) outperformed InstantStyle (p = 1.06 × 10⁻⁵).
• Ablation confirms the λ trade-off. At λ = 0, CLIP-T improves by 9.26% over basic K/V swap, but aesthetic score drops to 4.7952 — below the reference style average of 5.0549. Increasing λ raises aesthetic score and text alignment while eroding style fidelity; λ = 0.1 was selected as the default compromise.
• Multi-style blending behaves predictably. In Table 3, as η sweeps from 0 to 1, DINO similarity to the first reference rises and to the second falls, for both S5-S5 and QD-S5 reference pairings. M3S (SDXL) struggled to use QuickDraw's abstract features (lower DINO-ref1 than DINO-ref2 in QD-S5 pairs), a limitation the authors attribute to SDXL's high-fidelity optimization being incompatible with low-quality abstract datasets.
• Runtime. Approximately 40 seconds per sketch for M3S (SD v1.5) and 70 seconds for M3S (SDXL) on an A100 40GB GPU.

Methodology in Plain English

The pipeline starts by inverting each reference sketch into the diffusion model's latent space. During denoising of the target sketch, the model normally computes attention using only the target's own queries, keys, and values. M3S instead appends keys and values derived from the references to the target's keys and values, so the references act as supplementary context rather than replacements.

To stop the reference content from bleeding into the output, M3S blends each reference key/value linearly with the target's own key/value using a weight λ; higher λ means more of the target's own content is preserved. The authors call this linear smoothing.

For style blending, the latent noise image is modulated with AdaIN — a standard operation that shifts and scales features to match the reference's mean and variance. With two references, a parameter η linearly interpolates between the two AdaIN results: η = 1 favors the first reference's style, η = 0 the second, though the authors note that both styles still influence the result at the extremes because attention incorporates more than one stylistic feature.

A dual classifier-free guidance separates "content" and "style" directions with scales ω1 and ω2. ω2 ramps linearly from one third of its final value up to its final value over the denoising trajectory.

For sparse, abstract sketches, the authors add an edge-based regularizer: they estimate the denoised image at each step (Tweedie's formula), run Sobel operators to get horizontal and vertical gradients, penalize the negative sum of their magnitudes, and take a gradient step on the latent with a strength of γ = 60, clamping gradient values to [−0.001, 0.001]. Sampling uses DDIM with 100 steps.

Experiments cover six styles: four professional styles from the 4skst dataset (Styles 1–4), a web-collected set of 20 sketches (Style 5), and 50 abstract freehand sketches from Sketchy (Style 6). Fifty prompts were generated with DeepSeek using the template "A sketch of …", each paired with a randomly selected reference. Evaluation uses CLIP-T for text alignment, DINO feature similarity for style consistency, and VGG Gram matrix distance.

Why This Matters

Impact on research. The paper challenges the dominant paradigm in diffusion-based style transfer — replacing reference K/V matrices — and shows empirically that on sparse sketch domains this causes artifacts. It also shows that StyleAligned-style rigid statistical alignment degrades sketch quality, suggesting that style-transfer techniques tuned on natural images do not transfer cleanly to line art.

Real-world applications:

• Rapid concept sketching for industrial design and product prototyping, where a designer's house style or a client's reference sketch must be matched.
• Storyboarding and previsualization, where multiple artists' stroke styles need to be blended consistently across a sequence.
• Educational visualization, generating diagrams or illustrations in a consistent pedagogical drawing style.
• Creative ideation tools, where the interpolation between two reference styles (controlled by η) is meant to spark new directions.

Industry relevance. The framework is training-free, which matters for deployment: no per-style fine-tuning, no curated style datasets, just a few inference-time parameters. It runs on off-the-shelf Stable Diffusion v1.5 and SDXL weights and the implementation is released at https://github.com/CMACH508/M3S.

Future Directions

• Localized style control. The authors explicitly name this as future work: letting users assign specific styles to specific regions rather than relying on the model's automatic style assignment.
• Better handling of abstract reference styles. M3S (SDXL) failed to effectively use QuickDraw's abstract features, so improving compatibility with low-quality abstract sketch datasets is an open problem.
• Closing the style-fidelity gap. M3S trails AttentionDistillation on DINO and VGG metrics under default settings; whether that gap can be closed without sacrificing content fidelity remains open.
• Human evaluation at larger scale. The preference study retained 44 validated questionnaires; broader evaluation and the statistical comparison against InstantStyle (p = 0.26 for SD v1.5) leave room for more conclusive evidence.

Target Audience

Researchers and practitioners in generative computer vision and style transfer, particularly those working on diffusion-model attention manipulation, sketch synthesis, or personalized image generation. It will also interest graphics and design-tool engineers who want controllable, style-consistent sketch output without training custom models. Readers should already understand transformer attention and diffusion sampling; newcomers to those topics will find the methodology section dense.

Authors’ abstract

Recent advances in vision-language models have facilitated progress in sketch generation. However, existing specialized methods primarily focus on generic synthesis and lack mechanisms for precise control over sketch styles. In this work, we propose a training-free framework based on diffusion models that enables explicit style guidance via textual prompts and referenced style sketches. Unlike previous style transfer methods that overwrite key and value matrices in self-attention, we incorporate the reference features as auxiliary information with linear smoothing and leverage a style-content guidance mechanism. This design effectively reduces content leakage from reference sketches and enhances synthesis quality, especially in cases with low structural similarity between reference and target sketches. Furthermore, we extend our framework to support controllable multi-style generation by integrating features from multiple reference sketches, coordinated via a joint AdaIN module. Extensive experiments demonstrate that our approach achieves high-quality sketch generation with accurate style alignment and improved flexibility in style control. The official implementation of M3S is available at https://github.com/CMACH508/M3S.