A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space

AI summary

Overview

Research area: Generative computer vision, specifically style-controllable text-to-image diffusion, discrete representation learning, and autoregressive generative modeling.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), vector quantization and codebooks, contrastive learning, rectified flow matching, and autoregressive next-token prediction.

Scope: The paper defines and solves the "code-to-style" image generation task, in which a single numerical code specifies a visual style, using a discrete style codebook plus an autoregressive style generator named CoTyle.

Paper identity: arXiv:2511.10555v5 [cs.CV], dated 28 Nov 2025. Authors are affiliated with Beihang University, the Kolors Team at Kuaishou Technology, and South China Normal University. The work was conducted during the first author's internship at the Kolors Team, Kuaishou Technology.

What This Paper Is About

Existing style-aware image generators specify a style through long textual prompts, a reference image, or a fine-tuned LoRA, which makes styles hard to reproduce, hard to share, and unable to produce genuinely novel styles (they can only imitate images that already exist). The paper proposes a new task, code-to-style generation, where a single numerical code acts as a complete style controller. The goal is to build the first open-source system that turns such a code into a consistent, novel visual style, a capability previously seen only in industrial products such as Midjourney.

Key Contributions

1. Introduces the code-to-style image generation task, where diverse and consistent visual styles are conditioned solely on a numerical style code, and identifies that this task had only been explored industrially (for example, by Midjourney) with no open-source academic research.
2. Proposes CoTyle, the first open-source framework for this task, combining a discrete style codebook trained with contrastive loss, a T2I-DM conditioned on the codebook's embeddings, and an autoregressive style generator trained with next-token prediction over style indices.
3. Extends CoTyle beyond pure code conditioning, showing it also supports image-conditioned stylistic generation and smooth style interpolation by mixing subsets of style indices.
4. Provides an extensive evaluation using CSD for consistency and diversity, CLIP-T for text-image alignment, and QualityCLIP for aesthetics, comparing against Midjourney on the code-to-style task and against five image-conditioned methods.

Main Findings

• Code-to-style works: On 500 randomly sampled codes with 4 images each (2,000 images total), CoTyle achieved a Diversity score of 0.7764, Aesthetics of 0.7173, CLIP-T of 0.3119, and Consistency of 0.6007, while Midjourney (evaluated on 500 images manually collected from its website) scored 0.8088, 0.5948, 0.3090, and 0.4734 respectively. The paper reports that CoTyle achieves significantly superior style consistency but slightly inferior diversity.
• Better than image-conditioned baselines on consistency: CoTyle conditioned on reference images reached a Consistency of 0.5791, ahead of StyleStudio (0.4711), CSGO (0.5540), USO (0.4395), Flux-Kontext (0.4222), and InstantStyleXL (0.5753). Its Aesthetics (0.7178) and CLIP-T (0.3230) were competitive with those methods. Diversity is marked as not measurable or meaningful for image-conditioned methods, since it is constrained by the reference image.
• Textual branch injection beats visual branch injection: Injecting the style embedding into the DiT through the textual branch (Aesthetics 0.7178, CLIP-T 0.3230, Consistency 0.5791) outperformed visual-branch injection via OminiControl (0.7175, 0.3255, 0.5306). Qualitatively, visual-branch injection captured color cues such as red paper-cut tones but missed semantic style elements such as circular contours.
• Both loss terms in the codebook are necessary: Removing negative sample pairs dropped Consistency from 0.5791 to 0.4890, and removing the reconstruction loss dropped Aesthetics to 0.7001 and Consistency to 0.4102. The reconstruction loss was found essential to avoid codebook collapse.
• High-frequency suppression raises diversity: Without the suppression coefficient s(i), Diversity fell from 0.7764 to 0.7488 and Consistency from 0.6007 to 0.5301. Analysis of the codebook showed a long-tail distribution in which roughly 5–10% of indices are selected with significantly higher frequency than the rest; these act as placeholder tokens, and sampling only from them produces images nearly identical to a vanilla T2I-DM with no style.
• Style representation is order-invariant: During interpolation, whether tokens are combined by random sampling or by splitting evenly between two styles produced minimal difference in results, suggesting stylistic information is invariant to token order, unlike traditional image representations.
• Style interpolation is smooth: Blending style indices from two distinct styles at varying ratios yields gradual stylistic transitions, and this works with both predefined style codes and styles extracted from user-provided pixel images.

Methodology in Plain English

The system is built in three stages.

First, the researchers train a discrete style codebook, a large lookup table of style vectors, on pairs of images that share a style. Features are extracted with a vision transformer (ViT), and a contrastive loss pulls images of the same style into the same distribution while pushing different styles apart. A reconstruction loss keeps the codebook vectors aligned with the VLM image encoder's feature distribution, and standard vector-quantization commit and codebook losses are added, giving the overall objective L_style = L_contrast + alpha L_recon + beta L_vq.

Second, that frozen codebook is plugged into a text-to-image diffusion model. The style embedding is treated as if it were text: the team uses a vision language model (VLM) as the text encoder, replaces the original image features with the quantized style embedding, and feeds it into the Diffusion Transformer through the textual branch. Training uses image pairs where the style embedding from one image and the text prompt of the other are used to generate that second image, via rectified flow matching.

Third, because codebook embeddings only come from existing images, an autoregressive transformer is trained to invent new ones. Style indices extracted from images are treated as token sequences and learned with a next-token prediction objective. The style generator uses the Qwen2-0.5B architecture, trained from scratch.

At inference, the numerical style code seeds a random number generator. One initial token is sampled, the model autoregressively predicts the remaining N-1 tokens, the corresponding vectors are looked up in the codebook and decoded into a style embedding, and that embedding plus the user prompt drives rectified flow generation. A frequency-suppression coefficient down-weights the logits of over-used indices so the output does not collapse into a style-less, photorealistic default.

Key training settings: codebook vocabulary size 1,024, embedding dimension 64, 20,000 steps, batch size 128, learning rate 1e-5. DiT initialized from pretrained Qwen-Image, 60,000 steps, batch size 64, learning rate 4e-6. Style generator: 100,000 steps, batch size 64, learning rate 1e-5. Style references resized to 392×392 and encoded into 196 style tokens (N = 196).

Why This Matters

Impact on research: This is the first open-source framework for code-to-style generation, a task with no prior academic literature. It reframes style as a discrete, portable, composable latent quantity rather than a prompt, an image, or a set of LoRA weights, and it demonstrates that decoupling style specification from content specification is workable at scale. It also supplies a reproducible benchmark protocol (500 codes × 4 images for code-to-style; 500 prompt-reference pairs for image-conditioned evaluation) that later work can build on.

Real-world applications:

• Digital art, design, and media creation, where an artist can explore a vast space of novel styles by varying a single short code rather than writing long prompts or training LoRAs.
• Style sharing and reproduction: a numerical code is a compact, portable identifier, so a style can be distributed or reused without sharing pixel-level reference images or complex model weights.
• Consistent asset generation for creative pipelines, where the same code repeatedly yields images in the same style across different prompts and content.
• Style blending for brand or artistic control, using index-level interpolation to mix two styles at user-specified ratios.

Industry relevance: The paper positions Midjourney's style-code feature as the only prior art and places CoTyle as the open-source counterpart, with model weights and source code publicly released. That directly targets practitioners who want code-based style control without depending on a closed API.

Future Directions

• Broadening style diversity: The authors attribute CoTyle's slightly lower Diversity relative to Midjourney to the limited breadth of the curated style-paired training set, and suggest expanding the data with more abstract and diverse artistic sources, or adopting architectures such as mixture-of-experts models.
• Improving the codebook's capacity: Because styles must be quantized into a finite codebook, detailed stylistic subtleties and hybrid nuances may be attenuated or omitted. The appendix proposes working toward more balanced, semantically disentangled discrete representations.
• Better user navigation of the style space: Finding aesthetically or artistically engaging styles currently requires stochastic search across a vast code space; the authors call for more intuitive mechanisms to navigate or modify the latent style space.
• Cross-modal extension: The paper suggests the discrete feature extraction paradigm could be extended to other modalities, such as audio.

Target Audience

Researchers and graduate students working on diffusion models, style transfer, discrete representation learning, and autoregressive generation, who will care about both the new task definition and the codebook-plus-autoregressive design. Also valuable for applied machine learning engineers and product teams in generative media who need portable, reproducible style control without per-style fine-tuning, and for artists and designers who want a compact interface to a large space of explorable styles. Readers without background in vector quantization or diffusion transformers will find the method sections demanding.

Authors’ abstract

Innovative visual stylization is a cornerstone of artistic creation, yet generating novel and consistent visual styles remains a significant challenge. Existing generative approaches typically rely on lengthy textual prompts, reference images, or parameter-efficient fine-tuning to guide style-aware image generation, but often struggle with style consistency, limited creativity, and complex style representations. In this paper, we affirm that a style is worth one numerical code by introducing the novel task, code-to-style image generation, which produces images with novel, consistent visual styles conditioned solely on a numerical style code. To date, this field has only been primarily explored by the industry (e.g., Midjourney), with no open-source research from the academic community. To fill this gap, we propose CoTyle, the first open-source method for this task. Specifically, we first train a discrete style codebook from a collection of images to extract style embeddings. These embeddings serve as conditions for a text-to-image diffusion model (T2I-DM) to generate stylistic images. Subsequently, we train an autoregressive style generator on the discrete style embeddings to model their distribution, allowing the synthesis of novel style embeddings. During inference, a numerical style code is mapped to a unique style embedding by the style generator, and this embedding guides the T2I-DM to generate images in the corresponding style. Unlike existing methods, our method offers unparalleled simplicity and diversity, unlocking a vast space of reproducible styles from minimal input. Extensive experiments validate that CoTyle effectively turns a numerical code into a style controller, demonstrating a style is worth one code.