The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Models

AI summary

Overview

• Research area: Generative computer vision — specifically the utility of synthetic data produced by text-to-image (T2I) diffusion models, evaluated through the lens of prompt complexity.
• Technical level: Intermediate. The synthetic Gaussian experiments and theoretical derivations (§2, Appendix A.2) require familiarity with score-based diffusion and classifier-free guidance, but the large-scale empirical study is accessible to anyone who works with generative models and evaluation metrics.
• Scope: The paper introduces an evaluation framework that builds prompts of controlled complexity from existing image-caption datasets, then measures how prompt complexity affects the quality, diversity, and prompt consistency of images generated by six T2I models under several inference-time interventions.

What This Paper Is About

Prompt engineering is the primary way people interact with T2I models, yet no prior work systematically asks how the complexity of a prompt — how detailed or how specific it is — changes the usefulness of the images that come out. The authors address this by first showing, in a controlled toy setting, that generalizing to more general prompts is harder than generalizing to more specific ones, and then building a benchmark to test whether the same holds for real T2I models at scale. Their goal is to compare synthetic images against real images along the standard utility axes (quality, diversity, consistency) as prompt complexity varies.

Key Contributions

1. A synthetic motivating study with theory. Using a mixture of four Gaussians ("white dog," "white cat," "black dog," "black cat") and two conditional U-Net models trained from scratch on either fine-grained or general prompts, the authors show that generalization toward more general prompts is harder, and derive an explanation: general prompts act as an OR operator over fine-grained prompts, whose score function requires a conditional likelihood term that diffusion models do not learn (Equation 1). Fine-grained prompts act as an AND operator over general concepts, which can be approximated by general-prompt score functions alone (Equation 2).

2. A new evaluation framework for prompt complexity. The framework has five steps — captioning, pairing, alignment, sampling, generation — that transform a fixed image-caption dataset into K datasets with captions of increasing complexity, assemble real image sets semantically similar to each prompt, align the image modality across complexity levels, sample an equal number of prompts per level, and generate a matched number of synthetic images per prompt.

3. A large-scale empirical study across three datasets and six models. Datasets: CC12M (12 million image-caption pairs), ImageNet-1k (1,000 classes), and Densely Captioned Images (DCI, 7805 image-caption pairs). Models: LDMv1.5, LDM-XL, LDMv3.5M, LDMv3.5L, Flux-schnell, and Infinity. Interventions: CFG (vanilla guidance), CADS, interval guidance, APG, and prompt expansion.

4. A real-versus-synthetic utility comparison. Because the framework pairs each prompt with a set of semantically similar real images, the paper can compare the utility of synthetic and real data directly — something the authors state they are the first to do systematically for prompt complexity.

Main Findings

• Generalizing to general prompts is hard, both in theory and in the toy setting. With the model trained on fine-grained prompts and queried with the general prompt "cat," the authors report D_KL = 23.78, FD = 14.41, and VS_gen = 1.03 (against VS_ref = 1.82) when ω > 1; at ω = 1 this improves to D_KL = 1.20, FD = 2.48, and VS_gen = 1.43 (VS_ref = 1.82). The authors attribute this to diffusion models only learning the score function and not the likelihood weighting in Equation 1, so they effectively average the fine-grained scores.

• The reverse direction is easier. With the model trained on general prompts and queried with fine-grained prompts, at ω = 1 the results are D_KL = 1.51, FD = 6.64, VS_gen = 2.04 (VS_ref = 1.10), and at ω = 3 they reach D_KL = 0.93, FD = 1.32, VS_gen = 1.33 (VS_ref = 1.10). The authors note Equation 2 resembles CFG when ω ≈ M, and report no severe distributional shift or diversity reduction in this simple setting.

• Diversity decreases with prompt complexity, then plateaus. Vendi-score diversity falls as prompts get more complex for every model and intervention, but it does not collapse. On the long DCI prompts, diversity decreases and then plateaus, which is not observed in the shorter prompt ranges; for most LDM models the plateau begins around 30 words.

• Proper reference-free metrics require caution — diversity gains come with consistency costs. Both advanced guidance methods and prompt expansion produce higher diversity than vanilla guidance, and prompt expansion on short captions can exceed the diversity of real data — but at the cost of lower DSG prompt-consistency.

• Quality trends are non-linear. In CC12M and ImageNet-1k, aesthetics stay relatively stable across prompt complexity, especially for LDMv3.5L. Advanced guidance methods slightly lower aesthetics relative to vanilla guidance, while prompt expansion consistently raises it. On DCI, aesthetics first increase and then gradually decrease with prompt length — a sharp slope toward general prompts and a gradual decline toward fine-grained ones.

• Consistency falls monotonically with complexity. DSG scores decrease as prompt complexity increases in every case considered, indicating that T2I models struggle to honor the growing number of objects, attributes, and relations in longer prompts. Real data also shows decreasing consistency, which the authors attribute to the image-text pairing process.

• Reference-based metrics behave differently. As prompt complexity rises, precision, density, and coverage of synthetic data tend to improve in both CC12M and ImageNet-1k, suggesting detailed captions grounded in real images help generation stay within the reference dataset's support. Prompt expansion and advanced guidance both improve FDD over vanilla guidance, but the gains in coverage come at the expense of precision and density.

• The newer model does not win on every axis. LDMv1.5 has better overall reference-based performance (lower FDD) than LDMv3.5L across guidance methods, prompt expansion, and prompt lengths, even though LDMv3.5L is better on reference-free quality and consistency and only falls short in diversity. The authors read this as evidence that diversity is a key characteristic of real-world image distributions.

• Combining interventions gives the best trade-offs. Pairing prompt expansion with advanced guidance — particularly APG — further boosts diversity with comparable quality, consistency, and FDD. Interval guidance shows slightly lower aesthetics and considerably lower prompt consistency than CFG and APG for prompt lengths above 1.

Methodology in Plain English

The authors work in two stages.

Stage one — a controlled toy problem. They take a two-dimensional mixture of four Gaussians representing four categories and train two small conditional U-Net diffusion models from scratch (DDPM schedule, vocabulary included) on two different caption sets: fine-grained captions ("white dog," "black cat," etc.) and general ones ("dog," "cat," "white," "black"). They then swap the caption sets at inference to test generalization in both directions, sweeping the classifier-free guidance scale ω, and measure forward KL-divergence, Fréchet Distance, and Vendi score.

Stage two — a benchmark on real data. Because real datasets pair each image with one fixed caption, you cannot easily assemble real images matching an arbitrary prompt. The framework solves this: it rewrites each caption at K levels of increasing complexity (adding detail for CC12M, increasing class specificity for ImageNet-1k, using the naturally long captions of DCI); for every caption it retrieves semantically similar real images, keeping only sets with at least 20 images; it then removes images not shared across all complexity levels so the real data is comparable; it subsamples the same number of captions per complexity level; and it generates as many synthetic images per prompt as the smallest matched real image set. Prompts run up to length 50 for LDMv1.5 (bounded by its 77-token text encoder), and up to 100 for the LDMv3.5 models.

Evaluation uses reference-free metrics — aesthetic score for quality, Vendi score for diversity, DSG (Davidsonian Scene Graph) score for prompt consistency — and reference-based metrics — FDD computed in DINOv2 feature space, plus precision, density, and coverage. Human evaluations in Appendix E are reported to confirm the automatic metrics. Prompt expansion is implemented by using a large language model to expand each caption into N_gen different captions of at most thirty words.

Why This Matters

Prompt complexity turns out to be a meaningful, previously under-examined axis of control over synthetic data utility, and the paper shows the trade-offs are not intuitive: the images that look best by reference-free measures are not necessarily the images that best match the real data distribution.

Real-world applications:

• Synthetic training data for downstream models. The paper is motivated by using T2I output for downstream model training and model self-improvement; the finding that optimizing reference-free metrics hurts distributional fidelity argues for caution when curating synthetic training sets.
• Data augmentation for rare or imbalanced classes. The result that fine-grained, image-grounded captions improve precision, density, and coverage suggests how to write prompts when fidelity to a target distribution matters.
• Model and pipeline selection. The FDD-versus-aesthetic divergence between LDMv1.5 and LDMv3.5L gives practitioners a concrete reason to pick models based on the intended downstream use rather than on headline generation quality.
• Evaluation and benchmarking. The contributed framework, released with code at the linked repository, provides a way to audit T2I models along prompt complexity.

Industry relevance: Teams that generate image data at scale — for content creation, dataset construction, or model distillation — can use these results to decide when prompt expansion is worth the consistency penalty and when advanced guidance pushes samples outside the manifold of real data.

Future Directions

• Handling very general prompts better. The authors state explicitly that generating from very general prompts requires more investigation, and that the missing likelihood term in Equation 1 is the theoretical root of the difficulty.
• Closing the diversity gap without prompt expansion. The paper concludes that diversity is a key feature of real-world image distributions that current T2I models fail to capture unless prompt expansion is used at inference; how to achieve this intrinsically is open.
• Extending the framework to datasets with long captions. The authors list as a limitation that datasets with long captions need to be large and diverse enough to build paired image sets; because DCI has only 7805 image-caption pairs, they did not compare synthetic and real data utility on it.
• Understanding the reference-free versus reference-based mismatch. The paper documents that LDMv3.5L wins on reference-free quality and consistency while LDMv1.5 wins on FDD, but does not resolve what drives the discrepancy.

Target Audience

Researchers and practitioners working on text-to-image generation, synthetic data curation, and generative model evaluation. The paper is most directly useful to those building synthetic training sets or benchmarking T2I models, and to readers interested in the theory of why conditional score functions generalize asymmetrically across prompt specificity. Readers without a diffusion background can still follow the empirical sections (§3, §4) comfortably, while §2 and Appendix A.2 assume familiarity with score matching and classifier-free guidance.

Authors’ abstract

Text-to-image (T2I) models offer great potential for creating virtually limitless synthetic data, a valuable resource compared to fixed and finite real datasets. Previous works evaluate the utility of synthetic data from T2I models on three key desiderata: quality, diversity, and consistency. While prompt engineering is the primary means of interacting with T2I models, the systematic impact of prompt complexity on these critical utility axes remains underexplored. In this paper, we first conduct synthetic experiments to motivate the difficulty of generalization with regard to prompt complexity and explain the observed difficulty with theoretical derivations. Then, we introduce a new evaluation framework that can compare the utility of real data and synthetic data, and present a comprehensive analysis of how prompt complexity influences the utility of synthetic data generated by commonly used T2I models. We conduct our study across diverse datasets, including CC12M, ImageNet-1k, and DCI, and evaluate different inference-time intervention methods. Our synthetic experiments show that generalizing to more general conditions is harder than the other way round, since the former needs an estimated likelihood that is not learned by diffusion models. Our large-scale empirical experiments reveal that increasing prompt complexity results in lower conditional diversity and prompt consistency, while reducing the synthetic-to-real distribution shift, which aligns with the synthetic experiments. Moreover, current inference-time interventions can augment the diversity of the generations at the expense of moving outside the support of real data. Among those interventions, prompt expansion, by deliberately using a pre-trained language model as a likelihood estimator, consistently achieves the highest performance in both image diversity and aesthetics, even higher than that of real data.