T-DIG: Text-Conditioned Diversity for Image Generation

AI summary

Overview

Research area: Text-to-image (T2I) generation — specifically controllable diversity across a set of images returned for one prompt.

Technical level: Intermediate. The paper assumes familiarity with diffusion/flow-based image generators, LoRA adapters, preference/alignment scoring models (VQAScore, HPSv3, TPIPS), and reinforcement learning with GRPO.

Scope: The paper defines "text-conditioned diversity," where a user states in free-form text which aspect of an image set should vary, and introduces T-DIG, an iterative framework with LLM policies trained via GRPO to steer a frozen image generator or editor along that axis.

What This Paper Is About

Image platforms usually return several images per prompt so users can pick one that matches their intent, but T2I models are optimized for single-image quality, which shrinks the variation between those images. Diversity is also contextual: one user wants variety in background, another in artistic style, and a third later wants variety in clothing — so a fixed, one-size-fits-all notion of "diverse" does not serve them. The paper reframes diversity as a conditional generation problem: given a prompt plus a user-written "axis" of variation, produce K images that differ along that axis while still following the prompt.

Key Contributions

1. Task formulation: Diversity in T2I is posed as conditional generation, where the condition is free-form text written by the user — from broad axes like "artistic style" to prompt-specific requests that no fixed taxonomy covers.

2. Interactive framework (T-DIG): Users see default i.i.d. samples, receive suggested axes, write or pick an axis, set a set size (K ∈ {2, 4, 8, 16}) and a diversity strength, choose between generating a new set or editing the existing one, then iterate until satisfied.

3. Model-agnostic policy training: Two LLM policies (generation and editing) are trained with GRPO against a text-conditioned diversity reward plus calibrated alignment, prompt-length, and preference penalties. Because the generator stays frozen and need not be differentiable, T-DIG applies to open-weight, distilled, and API-only models.

4. A user-derived dataset: The authors describe releasing the first dataset for text-conditioned prompt diversity, with training and validation splits built from human-written axes (elicited at scale with a VLM) and a test set of 405 (prompt, axis) pairs collected from 58 participants using the tool, plus a human rating study against three baselines.

Main Findings

• Large on-axis diversity gains: Across four open-weight T2I models, T-DIG more than doubles diversity along the requested axis versus i.i.d. sampling on average (2.3×, up to 3.1×), while lowering base-prompt alignment by less than 2%.

• Axis-agnostic baselines fall short on one side: The most diverse baseline, CADS, reaches 1.8× at a 6% alignment drop; Scaling Group Inference preserves alignment best (under 1% drop) but gains only 1.3×, at up to 22× the sampling cost. PAG ranges from +0.03 to +0.09 TPIPS-Vendi over i.i.d., CADS from +0.08 to +0.17, training-free T-DIG (α = 0) from +0.08 to +0.14, and trained T-DIG (α = 1) from +0.14 to +0.26.

• Better quality–diversity trade-off: T-DIG loses at most 0.02 VQAScore per +0.1 TPIPS-Vendi gained, versus up to 0.08 for CADS and up to 0.41 for PAG. Its HPSv3 stays within −6% to +3% of i.i.d. on most models, whereas CADS loses 14–24%.

• Gain scales inversely with the model's own diversity: Qwen-Image and FLUX.2 [klein] are nearly deterministic under i.i.d. sampling (TPIPS-Vendi 0.12 and 0.14) and gain +0.26 and +0.25; SD3.5 Large and Medium start at 0.20 and 0.21 and gain +0.14 and +0.15. All four land at 0.35–0.40 at α = 1.

• Works without an axis too: On COCO-250 (250 prompts, no axis specified, "overall" axis), sweeping α from 0 to 1 raises DINO-Vendi by +0.21 to +0.39 (about 3× i.i.d. at α = 1) while losing only 0.01 to 0.02 VQAScore — versus CADS, which reaches +0.32 to +0.37 but drops VQAScore by 0.08 to 0.11.

• Policy size barely matters: 4B, 9B, and 35B-A3B Qwen3.5 policies reach nearly identical diversity; 9B matches or beats 4B on VQAScore by −0.002 to +0.010, and 35B-A3B adds no consistent gain. The untrained 4B policy emits malformed output on 1.7% of requests, but training removes these errors.

• One reward term is not enough: Ablating on FLUX.2 [klein], diversity-only reward pushes TPIPS-Vendi to 0.617 but collapses VQAScore to 0.228; adding alignment restores VQAScore to 0.857 but inflates prompts to 123 words; adding the length penalty cuts prompts to 34 words but leaves other reward hacks (HPSv3 falls to 11.38); the full reward holds TPIPS-Vendi at 0.454 with HPSv3 of 12.35.

• Humans prefer it: In pairwise ratings (roughly 2,400 ratings), raters found T-DIG (α = 0.5) sets more varied on 64–96% of screens in every comparison; T-DIG wins on "more varied and at least as well aligned" 57–83% versus i.i.d., 61–81% versus CADS, 43–76% versus Scaling Group Inference, and 48–58% versus training-free T-DIG. Mean diversity ratings are 7.4–8.3 out of 10 for T-DIG, against 2.4–3.9 for i.i.d.

• In the interactive study: Users rated T-DIG output diversity 3.95/5 and prompt alignment 4.09/5; 89% found requesting variations more useful than accepting i.i.d. outputs, and 96% enjoyed the tool.

• Cheap to serve: With vLLM on one H200, expanding a prompt into four takes a median 0.40 s (4B), 0.59 s (9B), and 0.51 s (35B-A3B). Against 5–11 s to render four images with SD3.5 and Qwen-Image, the 9B policy adds 6–12%; on the 4-step FLUX.2 [klein] (1.5 s), the 4B policy keeps overhead around 27%.

• Policies transfer to proprietary models: Applying policies trained on open-weight models to GPT Image 2 and Gemini 3.1 Flash Image on 50 test prompts, the Qwen-Image policy raises TPIPS-Vendi by +0.27 and +0.20, versus +0.19 and +0.15 for prompt expansion by GPT-5.5 and Gemini 3.8 Flash — roughly 40% more on GPT Image 2. VQAScore drops 0.02 and 0.05, and HPSv3 by 9% and 7%.

Methodology in Plain English

The pipeline wraps the image model rather than changing it.

Interaction loop. A user types a prompt and sees K independently seeded samples. A frozen suggestion VLM proposes axes of variation based on the prompt and those images. The user writes or picks an axis, sets K and a strength slider, and chooses whether to generate a fresh set or edit the current one. The new set can be accepted or fed back in for another round.

Two trained policies. The "Diversity Module" is a pair of LLM policies — one for generation, one for editing — each a rank-32 LoRA adapter on a frozen Qwen3.5 backbone. The strength slider scales the adapter weights by α_s ∈ [0, 1], so α = 0 is the untrained backbone. The policy outputs K axis-conditioned prompts (or K edit instructions), which a frozen generator or editor renders.

Data. Starting from the HPDv2 training split, exact duplicates and empty entries are removed, leaving 103,672 prompts; a GPT-5.6 Luna filter drops prompts with named real people, unsafe content, no depictable subject, or mostly style/quality descriptors, keeping only prompts with at least three meaningful axes (79.9% retained). Semantic clustering brings the pool to roughly 20,000 prompts. Axes are elicited from Gemini 3.8 Flash using human-written pilot responses as in-context examples (allowing up to four axes per prompt, with "overall" added for a random 10%). After held-out validation prompts are removed: 19,765 prompts and 76,089 (prompt, axis) pairs.

Reward design. Diversity along the axis is measured by embedding each image with TPIPS (a VLM trained on human judgments of similarity along a text-specified aspect) and computing a Vendi score over those embeddings — effectively the "number of distinct images" along that aspect — normalized to [0, 1]. Three penalties guard against cheating: an alignment hinge on VQAScore set to each prompt's own mean i.i.d. score; a length penalty capping extra words (w = 20) per expanded prompt; and a preference hinge on HPSv3, likewise calibrated per prompt, which catches tricks like writing the prompt as a caption inside the image. The editing policy adds a DreamSim drift penalty so instructions change the image along the axis rather than replacing it outright.

Optimization. GRPO with 8 completions per input, each a full set of K = 4 prompts, advantages computed over whole-set rewards. All K images in a set share one fixed initial latent, so reward differences come from the text rather than sampling noise. Training uses λ = 2, μ = 0.1, η = 0.05, learning rate 2×10⁻⁵, no KL term, 800 steps, 8 (prompt, axis) pairs per update, rendering and scoring 256 images per step.

Evaluation. Automatic metrics (TPIPS-Vendi, DINO-Vendi, DreamSim-Vendi, VQAScore, CLIPScore, ImageReward, HPSv3, LAION-Aesthetics V2) on the 405-pair interactive test set and on COCO-250, plus two-stage human evaluation: first collecting real requests, then pairwise ratings of T-DIG against i.i.d., CADS, and Scaling Group Inference.

Why This Matters

For research: The paper separates how much a set varies from along what it varies — a distinction most prior diversity work collapses into a single scalar like DINO or DreamSim distance. It also shows that a text-conditioned diversity metric (TPIPS) can be turned into a training reward, and that a frozen generator plus a small trained text policy is a viable alternative to sampler hacks, inference-time search, or fine-tuning the image model. It contributes human-grounded data (human-written axes, real user requests) that prior work approximated with templated prompts or object lists.

Real-world applications:

• Creative and design tools where a designer wants forty logo sketches that differ in typography style but agree on color palette.
• Marketing asset generation, where a campaign needs variants differing in setting, model, or framing while the product and copy stay fixed.
• Concept exploration and storyboarding, where the ask is deliberately open-ended — e.g., "crazy endings that would surprise me" — rather than an attribute list.
• Dataset and synthetic-data generation, where a practitioner needs controlled coverage along a named axis (pose, ethnicity, background) without hand-writing every variant.

Industry relevance: The method only trains a LoRA adapter on a language model, so it works on top of API-only models whose weights are unavailable. The transfer experiment on GPT Image 2 and Gemini 3.1 Flash Image, and the small latency overhead relative to rendering (6–12% for the 9B policy on standard samplers), speak directly to deployment in commercial T2I products that already return multi-image grids.

Future Directions

• Co-training the generator: The authors note T-DIG could not consistently produce variation in camera angles and framing, since prompt-level control alone cannot fix it, and suggest jointly tuning the generator with the policy as a next step.

• Perceptual calibration of the strength slider: About half the interactive-study participants found the slider poorly calibrated because it linearly scales the LoRA weights — low settings felt too subtle, high settings too sensitive. A perceptually calibrated control is left for future work.

• Avoiding saturation and reward hacking: All four T2I models converge to 0.35–0.40 TPIPS-Vendi at α = 1, and continuing training leads to hacks such as adding extra people or over-detailing prompts — leaving open how to push diversity further without gaming the metric.

• Reducing cost and extending transfer: Training a separate policy per proprietary model was judged too costly, so policies were transferred zero-shot; understanding what makes a policy transfer well (Qwen-Image's transferred best) and how much alignment can be preserved across model families remains open.

Target Audience

Researchers and practitioners working on controllable text-to-image generation, generative-model evaluation, and RL-based post-training of generators — especially those building multi-image products on top of frozen or API-only models. It is also useful for readers interested in conditional/perceptual similarity metrics and in how reward design (hinges, per-prompt calibration, anti-hacking penalties) behaves in set-level reinforcement learning. Readers without background in diffusion models or policy-gradient training will find the framework section accessible but the results tables dense.

Authors’ abstract

Text-to-image (T2I) platforms typically return several images per prompt so users can explore different visual interpretations and choose one that matches their intent. However, T2I models are usually optimized for single-image quality which comes at a cost of multi-image diversity, limiting this exploration. Diversity is also contextual: one user wants variations in background, while another wants them in artistic style, and the same user may later want variations in clothing. The generated images should differ along whatever axis the user cares about. In this paper, we introduce T-DIG, a text-conditioned diversity framework for image generation and editing. T-DIG is an iterative framework comprising multiple modules that lets users type prompts, inspect default generations, request variations using natural language, and continue refining outputs until they're satisfied. At its core, T-DIG uses an LLM policy that produces K axis-conditioned prompts or edit instructions for a frozen generator or editor. We train this policy with GRPO using a text-conditioned diversity reward and alignment penalties. Because the image generator remains frozen, T-DIG applies to open-weight, distilled, and proprietary models. We also release the first dataset for text-conditioned prompt diversity in image generation, with the training and validation splits built from human-written axes and a test set drawn from real user interactions with T-DIG. T-DIG far exceeds baselines in improving text-conditioned diversity, while preserving base-prompt alignment across a variety of metrics and human judgements. We show that user-directed diversity exploration can be added to a wide range of image generation models, open and closed, without the need to access their weights. Website: https://image-study.org/t-dig/