Text-to-Image Generative Modeling Without Language Pretraining

AI summary

Overview

Research area: Multimodal generative modeling — specifically text-to-image (T2I) synthesis with autoregressive transformers and discrete tokenization.

Technical level: Intermediate. The paper assumes familiarity with tokenizers, autoregressive next-token prediction, and classifier-free guidance, but its central question is stated plainly enough for a general machine-learning reader.

Scope: The paper asks whether internet-scale language pretraining is actually necessary for strong text-to-image generation, and answers it by training a family of early-fusion autoregressive models (Halcyon, 0.1B–4B parameters) entirely from scratch on 100M image–text pairs.

What This Paper Is About

Nearly every modern text-to-image system depends on a language model that was pretrained on a massive text-only corpus — either as a separate text encoder (late fusion) or as part of a jointly trained multimodal model that also sees text-only data (early fusion). This language pretraining dominates the compute budget, accounting for over 80% of total pretraining FLOPs in representative T2I systems, yet its actual contribution to image generation has never been systematically quantified. The authors build a T2I system whose only language supervision is caption text paired with images, and run controlled experiments to measure what language pretraining actually buys.

Key Contributions

1. Halcyon, a from-scratch early-fusion T2I family. A decoder-only Transformer with a joint text–image vocabulary of 80,640 tokens (16,384 text, 64,000 image, 256 special), where the text tokenizer, image tokenizer, and generative model are all trained from scratch on GPIC (100M image–text pairs) — no pretrained language model at any stage.
2. A compute-normalized comparison against late-fusion baselines. The authors retrain LlamaGen (autoregressive, FLAN-T5 encoders) and JiT-T2I (diffusion, Qwen3-1.7B) on GPIC and compare generation quality (gFD) against total pretraining FLOPs, including the language model, image tokenizer, and image generator.
3. Controlled ablations isolating the value of language pretraining. Under a fixed pretraining budget, they vary the schedule and mixture of text-only, T2I, and image-to-text (I2T) sequences, plus a continual-pretraining experiment initialized from Qwen3-0.6B-Base (pretrained on 36T text tokens).
4. A free captioning capability. They show that replacing caption-dropped T2I examples with I2T sequences yields image captioning at no cost to T2I quality and no extra pretraining budget, because both sequence types share the same image prefix.

Main Findings

• From-scratch T2I beats language-pretrained baselines at far lower compute. Halcyon-0.52B reaches 48.0 gFD versus LlamaGen-XL/8 at 63.3 (FLAN-T5-XL 1.22B + 0.90B image generator), using 18.2× less total pretraining compute. Halcyon-1.6B reaches 35.3 gFD versus JiT-T2I-XXL/8++ at 41.0 (Qwen3-1.7B + 1.10B image generator), using 275× less compute.
• Quality scales cleanly with model size. gFD falls from 144.16 (0.1B) to 72.55 (0.25B), 48.02 (0.52B), 35.28 (1.6B), and 28.43 (4B). GenEval rises from 27.31 to 56.39 and DPG-Bench from 66.36 to 77.90 over the same range.
• Text-only pretraining does not help. Allocating α = 0.1, 0.2, 0.5 of a fixed budget to text-only pretraining degrades gFD from 48.0 to 51.56, 52.76, and 65.13 respectively. Giving the model a full extra T2I epoch after language pretraining (1.5× total compute) also fails to help (49.22 vs. 48.02).
• Continual pretraining from a large language model does not help either. Qwen3-0.6B-CPT, initialized from Qwen3-0.6B-Base (36T text tokens), scores 47.81 gFD versus 46.61 for the same architecture trained from random initialization. Lowering the peak learning rate from 7.5×10⁻⁴ to 1×10⁻⁴ makes it substantially worse (63.62).
• A much larger language encoder buys little. Swapping FLAN-T5-XL (1.22B) for FLAN-T5-base (0.11B) in LlamaGen changes gFD only from 63.3 to 66.7, despite a 13.5× reduction in language-model training compute (9.06×10²¹ vs. 6.69×10²⁰ FLOPs). On GPIC captions, FLAN-T5-XL has lower caption NLL than FLAN-T5-Base (2.76 vs. 3.24), so the encoder is genuinely better — it just does not transfer to generation quality.
• I2T as a separate task is not a free lunch. Allocating 10% of the budget to I2T leaves gFD roughly unchanged (48.35 staged, 48.18 mixed, vs. 48.02 baseline), while 50% hurts (54.22 staged, 75.97 mixed).
• Replacing caption dropout with I2T is a free lunch. Setting p_drop = 0 and η = 0.1 preserves the conditional and unconditional image-token counts, drops conditional text NLL by more than an order of magnitude (18.414 → 1.102 at 0.52B), and leaves T2I metrics comparable (e.g., 46.48 gFD, 44.60 GenEval, 76.14 DPG-Bench at 0.52B). Caption alignment rises with size from 0.090 to 0.559, against 0.891 for GPIC ground-truth captions.
• Instruction-tuned tokenizer quality matters more than language scale. The from-scratch image tokenizer beats Cosmos-DI-8x8 on GPIC Test-50K reconstruction (rFD 18.86 vs. 37.93, PSNR 25.61 vs. 25.38, SSIM 0.792 vs. 0.776, LPIPS 0.062 vs. 0.153), and switching LlamaGen to this tokenizer alone improves gFD from 147.0 to 63.3.
• Larger models need less guidance. The CFG scale minimizing gFD on GPIC Validation-50K drops from 7.5 at 0.1B to 4.5 at 4B, while validation gFD improves from 142.7 to 28.4.
• Longer captions help, up to a point. Medium and long captions consistently beat short ones across all sizes; medium and long are comparable from 0.52B onward (e.g., 4B: 27.91 medium, 27.77 long, 31.48 short).

Methodology in Plain English

The authors rebuild the entire T2I pipeline from raw data upward, deliberately excluding anything trained on web-scale text.

Image compression. A discrete image tokenizer is trained on GPIC images using a Cosmos-style encoder–decoder with finite scalar quantization, compressing each 256×256 image by 8× per spatial dimension into 1,024 tokens drawn from a 64,000-entry codebook. It is trained for one epoch with reconstruction and perceptual losses, then half an epoch more with an adversarial (PatchGAN) loss, with the decoder width increased from 128 to 192 relative to Cosmos-DI-8x8.

Text compression. A byte-level BPE tokenizer with a 16,384-entry vocabulary is trained from scratch on GPIC captions plus OpenWebText (OpenWebText is used only for the tokenizer, not for the generative models). Despite a vocabulary 9.3× smaller than Qwen3's, it compresses GPIC captions about as well: 4.774 bytes/token vs. 4.763 for Qwen3 and 4.152 for FLAN-T5.

One sequence space. Text tokens, image tokens, and special tokens share a single vocabulary and embedding table. Four sequence formats are used: text-only, image-only, image-then-caption (I2T), and caption-then-image (T2I). The model is a standard next-token predictor with a weighted cross-entropy that down-weights image tokens by λ_img = 0.5 so they do not dominate the loss.

Controlled budget comparisons. All variants train for exactly one epoch on GPIC with a global batch size of 1,024 and the same learning-rate schedule, so differences in data mixture or schedule do not come from differences in compute — only from how that compute is spent. Baselines (LlamaGen, JiT-T2I) are retrained in the authors' own setup using official codebases, and the authors deliberately modify them step by step (swap the image tokenizer, add LPIPS loss, shrink the patch size) until each baseline is at its strongest, so the final comparison is as fair as they can make it.

Total-cost accounting. Rather than comparing image generators alone, they sum FLOPs for the language model, image tokenizer, and image generator, which is what exposes the 18.2× and 275× gaps.

Evaluation. GPIC Test-50K is the primary benchmark (gFD, precision, recall, density, coverage, MMD); GenEval and DPG-Bench test compositional generation and prompt adherence. Guidance scales are swept per model on GPIC Validation-50K.

Why This Matters

Impact on research. The paper converts a widespread assumption into a measured claim. Prior work disagreed — Imagen reported gains from scaling pretrained text encoders, while Parti and Zhang et al. (2024) found little benefit — and modern systems drifted toward ever-larger pretrained language backbones anyway. By holding the data and compute budget fixed and re-running the comparison against compute-matched baselines, this work gives the field a clean empirical anchor: language pretraining is a cost, not a requirement, for T2I. It also reframes how T2I papers should report cost, since excluding language pretraining understates the true price of building a system by a large margin.

Real-world applications:

• Lowering the barrier to building T2I systems for groups without the resources to pretrain or license a large language model — regional labs, small studios, academic groups.
• Domain-specific image generators for medical imaging, satellite imagery, industrial inspection, or microscopy, where web-scale English text is a poor match for the vocabulary and the paired data is what actually carries the signal.
• Cheaper model iteration, since a from-scratch 4B model avoids carrying a separate pretrained text encoder through every retraining run.
• Multitask multimodal models on a single budget, using the caption-dropout-for-I2T swap to get captioning essentially for free alongside generation.

Industry relevance. The compute argument is directly commercial: 18.2× and 275× reductions in total pretraining FLOPs translate into proportionally lower training cost, and the finding that a 0.11B text encoder performs nearly as well as a 1.22B one (63.3 vs. 66.7 gFD) suggests that the standard late-fusion stack is substantially over-provisioned on the language side.

Future Directions

• Scaling beyond 4B parameters and beyond 256×256 resolution. All reported results are at 256×256 on a single dataset, GPIC; whether the no-language-pretraining recipe holds at higher resolutions and larger model sizes is not established by this paper.
• Reconciling the null result with cases where language pretraining might help. The paper's ablations cover its own architecture, GPIC, and one continual-pretraining setup from Qwen3-0.6B-Base. Whether language priors help for rare concepts, compositional reasoning, or text rendering in images — settings where paired data is sparse — remains open.
• Improving the from-scratch tokenizers. The image tokenizer still shows weak reconstruction of small text and fine glyphs, and caption alignment from the I2T variant (0.559 at 4B) remains well below ground-truth captions (0.891), leaving room to close both gaps.
• Rethinking what "language ability" means for generation. The paper argues the image generator must learn to map text representations to images from paired data regardless of how good the text encoder is; isolating exactly which linguistic capabilities, if any, do transfer from text-only pretraining is a natural follow-up.

Target Audience

Researchers and engineers working on diffusion or autoregressive image generation, multimodal pretraining, and tokenization; practitioners deciding whether to build on a pretrained language backbone or train from scratch; and compute-budget planners who need an honest accounting of where T2I training FLOPs actually go. Readers without a background in autoregressive token models will still follow the argument, but will get the most from the paper with some familiarity with classifier-free guidance and Fréchet-distance metrics.

Authors’ abstract

Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2$\times$ less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275$\times$ less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon