LLaDA-Image: Building Strong Image Generators with Fully Open Training Recipes

AI summary

Overview

Research area: Computer Vision — text-to-image generation, instruction-guided image editing, and multimodal understanding, with a focus on fully open training recipes.

Technical level: Advanced. The paper is a systems-level training report covering model architecture, data pipelines, optimization schedules, and distillation, rather than an introduction to image generation.

One-sentence scope: The paper documents how a 6B Diffusion Transformer, paired with a frozen diffusion-language-model understanding backbone, was trained from scratch through image-only pre-training, mid-training, text alignment, joint generation–editing, and few-step distillation, and releases the weights, code, and recipes.

What This Paper Is About

Most strong image generators are either proprietary, with undisclosed data and training recipes, or open models whose recipes are only partially described. The authors set out to answer a practical systems question: what does it take to train a strong image generator from scratch using a fully open, reproducible recipe, under a moderate data budget dominated by real image-only data rather than paired captions?

The goal is a single unified checkpoint that handles text-to-image synthesis, multilingual text rendering, and reference-preserving instruction editing, plus a distilled variant that runs in very few sampling steps.

Key Contributions

1. A data-efficient, image-only-first training pipeline. Visual-prior learning is decoupled from language alignment. A frozen dLLM-based vision-language model derives the conditioning signal from the same image region the DiT is asked to generate, so no external caption is needed early on, and high-resolution crops can be resized only mildly for 256² training. Image-only training accounts for more than 90% of the pipeline.

2. A unified understanding–generation–editing architecture. A dLLM-based vision-language model (LLaDA2.0-Mini backbone with a SigLIP-VQ vision encoder) interprets text, visual context, and reasoning traces. A Residual Query Adapter plus a lightweight Transformer connector feed generation-relevant hidden states into a pure single-stream DiT. Text-to-image prompts and editing instructions share that semantic pathway, while SigLIP-VQ features and a clean VAE latent supply reference signals for editing.

3. A stable, real-data-dominant progressive training schedule. Training moves from 256² image-only pre-training to aspect-ratio-bucketed 512² mid-training, then to paired language alignment at 512² and 1024², targeted refinement on text-rich and portrait data, and finally joint generation–editing training. Parameter-free RMSNorm throughout the DiT and the Muon optimizer stabilize the pipeline. The real-image share stays above 70% throughout supervised fine-tuning.

4. Efficient few-step inference and full release. TwinFlow distills the multi-step model into LLaDA-Image Turbo, which needs only 2–4 sampling steps. The authors release four checkpoints in total (Base and Turbo, plus an FP8 variant of each), along with training and inference code and detailed recipes.

Main Findings

• State-of-the-art open-source results on Qwen-Image-Bench: LLaDA-Image achieves overall scores of 53.53 on the English track and 53.38 on the Chinese track, which the authors describe as a new state of the art among open-source models on both tracks. Models in the comparison figure are ordered by their two-track average.

• Images alone are sufficient for pre-training: The authors state that large-scale image–text pairs are not required to pre-train LLaDA-Image, because a frozen VLM can extract high-level semantics from an image and use them as a self-conditioning signal.

• Real image-only data improves final quality: The paper reports that real image-only data can improve final generation quality and support visual-prior learning at scale without paired captions.

• A trade-off between early convergence and final realism: Compared with training predominantly on synthetic data, using a large proportion of real images leads to slower early convergence on standard benchmarks but substantially improves the realism of the model's final generated images after sufficient training.

• Parameter-free RMSNorm for stability: Replacing every normalization layer in the DiT with parameter-free RMSNorm is reported to substantially improve optimization stability over long training horizons.

• Data pipeline scale and composition: The generation pipeline processes approximately 220M samples, of which 98% are real images; image-only samples make up more than 90% of the total; pre-training and mid-training use only real images; and the real-image share of paired image–text SFT data stays above 70%. Table 2 notes that the 220M samples are counted cumulatively across the image-generation pipeline and refer exclusively to generation training.

• Filtering thresholds. Metadata filtering keeps only valid images with a total pixel count greater than 1024² and a file-size-to-pixel ratio of at least 0.15 bytes per pixel. Aesthetics filtering removes images with ArtiMuse scores below 60, and quality filtering removes images with DeQA-Score below 4.0.

• Captioning and validation. Captions are produced with Qwen3.6-35B-A3B and Qwen3-VL-235B-A22B-Instruct from image content alone; Qwen3.6-35B-A3B then compares each caption against its source image and removes captions that hallucinate objects, attributes, relations, or text, or that mis-transcribe visible text.

• Qualitative demonstration. A "Real or Generated?" reader challenge presents generated images; the reveal states that none of the candidates were real and that every image was produced by LLaDA-Image. The authors explicitly disclose that this is a qualitative demonstration, not a controlled perceptual study.

• Benchmarks used for evaluation. The paper states that evaluations on Qwen-Image-Bench, LongText-Bench, CVTG-2K, and GEdit-Bench demonstrate visual quality and prompt alignment, balanced Chinese–English text rendering, and competitive instruction-guided editing. The provided content does not report numeric results for LongText-Bench, CVTG-2K, or GEdit-Bench.

• Understanding-training configuration (Table 1). CoT supervised fine-tuning uses approximately 2.6M packed sequences of 16,384 tokens, 4 training epochs, 512² resolution, a global batch size of 512, masked answer tokens under block diffusion with block size b = 32, and a Gen:Und:Text data composition of 9:9:2. Optimization uses AdamW with β₁ = 0.9 and β₂ = 0.95, a learning rate of 1×10⁻⁵ decaying to 1×10⁻⁶ by cosine schedule, linear warmup at 1%, weight decay 0.1, and gradient norm clipping at 1.0.

• Generation-training configuration (Table 2). The pipeline spans pre-training, mid-training, SFT (512² alignment, 512²→1024² transition, refinement, editing), and distillation. Reported global batch sizes are 24,576 (PT), 6,400 (MT), 4,608 (512² alignment), 2,048 (512²→1024²), 2,880 (refinement), 2,688 (editing), and 256 (distillation). Sampling steps are 50 for mid-training and 2–4 for distillation. The optimizer is Muon throughout; weight decay is 0.0 and gradient norm clipping is 1.0. Reported learning rates span 4×10⁻⁴, 2×10⁻⁴, 5×10⁻⁵, 3×10⁻⁵, and 5×10⁻⁶, with EMA ratios of 0.9995 and 0.995 in the later stages.

• Timestep sampling. Supervised fine-tuning uses logit-normal timestep sampling with P_mean = 0.8 and P_std = 0.8, shifting probability mass toward higher-noise states where predictions are described as more sensitive and less stable.

• Reference conditioning for editing. The reference image deliberately bypasses the VLM. Instead, SigLIP-VQ features pass through a DiT-specific branch consisting of a dedicated feature embedder followed by two Transformer layers, and the clean reference image is also encoded by the FLUX.2 VAE and concatenated with the noised target latent.

Methodology in Plain English

The authors break a conventionally monolithic training job into ordered stages, each changing only one thing at a time.

1. Prepare a frozen understanding model. A vision-language model built on a diffusion language model (LLaDA2.0-Mini, with a SigLIP-VQ vision encoder) is first tuned with chain-of-thought supervised fine-tuning over packed 16,384-token sequences, mixing generation, understanding, and text-only data at a 9:9:2 ratio. This backbone is then frozen.

2. Learn a visual prior from images alone. At 256², random crops are taken from high-resolution images at or moderately above the native 256² size and resized with only mild downsampling. The frozen VLM reads the crop and produces a condition; the DiT learns to reconstruct the same crop's latent under a flow-matching objective. Because showing all image tokens would make the task a trivial copy, a random subset of image tokens is masked out, turning the task into predicting missing content from visible patches.

3. Move to 512² under the same self-conditioning. Mid-training raises the pixel budget while keeping the image-derived condition and the masking scheme, so the model adapts to higher resolution without simultaneously changing how it is supervised. Aspect-ratio bucketing is introduced here: images are assigned to a resolution bucket matching their aspect ratio, downsampled, and minimally cropped, keeping per-image pixel budgets and memory use roughly constant across data-parallel ranks.

4. Align text with images. SFT introduces paired captions in two progressive steps: first roughly 512² pixel buckets, then roughly 1024² buckets. Because resolution was already adapted during mid-training, this stage mainly teaches the model to associate textual concepts and relations with visual content.

5. Refine specific weaknesses. A refinement stage increases the proportion of text-intensive and portrait-centric examples to strengthen text rendering and human generation, while keeping synthetic data limited.

6. Add editing with a parallel reference pathway. For editing, the instruction text flows through the same VLM-and-connector path, but the reference image goes directly into the DiT through two complementary channels: SigLIP-VQ semantic features and the clean FLUX.2 VAE latent. Training then proceeds jointly on text-to-image and image-to-image at a 1:1 ratio.

7. Compress inference. TwinFlow distillation, built on distribution-matching distillation and self-adversarial flow training, turns the multi-step model into LLaDA-Image Turbo, needing only 2–4 sampling steps.

Throughout, all DiT normalization layers are replaced with parameter-free RMSNorm, and the Muon optimizer is used in every generation stage.

Why This Matters

Impact on research. The paper argues that the barriers to strong open image generators are not only modeling choices but data budget, training stability, and deployment cost. By releasing weights, code, and recipes — and by showing that more than 90% of generation training can be image-only with 98% real images — it makes those trade-offs inspectable and the recipe reusable rather than hidden behind a leaderboard number. It also demonstrates that multimodal understanding, generation, and editing can live in a single checkpoint without separate task-specific backbones.

Real-world applications:

• Design and content creation tools that combine text-to-image synthesis with instruction-guided editing, where the reference image must be preserved outside the edited region.
• Bilingual and multilingual content production, given the paper's reported balance between Chinese and English text rendering and its bilingual text-rendering examples.
• Low-latency deployment, where the 2–4-step Turbo variant and the FP8 checkpoints reduce inference cost relative to a 50-step sampler.
• Reproducible research baselines, where teams can retrain or modify the pipeline from a documented recipe instead of fine-tuning an opaque checkpoint.

Industry relevance. The paper's framing is explicitly practical: it targets a "moderate data budget," stable large-scale training, and efficient deployment, and it ships four checkpoints (Base, Turbo, and an FP8 variant of each) plus training and inference code through GitHub and Hugging Face.

Future Directions

• Controlled perceptual evaluation. The "Real or Generated?" challenge is disclosed as an informal qualitative demonstration, not a controlled study. A rigorous human perceptual study, ideally pre-registered, would substantiate the photorealism claims.
• Reconciling early convergence with final realism. The paper reports that real-data-dominant training converges more slowly on early benchmarks than synthetic-heavy alternatives while yielding stronger realism at convergence. Whether a hybrid schedule can capture both — faster early progress and the higher long-horizon ceiling — is left open.
• Scaling the recipe. The released model is a 6B DiT. It is not reported whether the same image-only-first pipeline and RMSNorm-plus-Muon stabilization hold at larger parameter counts, larger data budgets, or longer training horizons.
• Broadening evaluation coverage. The provided content reports Qwen-Image-Bench scores but no numeric results for LongText-Bench, CVTG-2K, or GEdit-Bench, leaving the precise extent of text-rendering and editing performance, and how it compares to closed systems on those axes, unresolved.

Target Audience

This paper is most useful to generative-model researchers and ML engineers who train or fine-tune image generation systems at scale: people who need to know how data composition, resolution schedules, normalization choices, optimizer selection, and distillation interact. It also serves teams evaluating open image generators for deployment, and researchers studying unified multimodal architectures that couple an understanding backbone to a diffusion generator. Readers without a background in diffusion models, flow matching, or large-scale distributed training will find the training-recipe sections dense.

Authors’ abstract

We introduce LLaDA-Image, a unified framework that pairs a 6B Diffusion Transformer (DiT) trained from scratch with a frozen vision-language understanding module built on the LLaDA2.0-Mini diffusion language model backbone. Instead of relying heavily on paired image-text data from the beginning, we first build a strong visual generative prior through image-only pre-training and mid-training. The generation pipeline comprises 220M samples, 98 of which are real images. For efficient and scalable optimization, we use parameter-free RMSNorm throughout the DiT together with the Muon optimizer. The resulting unified model produces highly photorealistic images while accurately following fine-grained editing instructions. We further distill LLaDA-Image into LLaDA-Image-Turbo, enabling fast inference in 2-4 sampling steps. On Qwen-Image-Bench, LLaDA-Image achieves overall scores of 53.53 and 53.38 on the English and Chinese tracks, respectively, setting a new state-of-the-art among open-source models on both tracks. To support further research on capable and efficient generative models, we release our model weights, training code, and detailed recipes.