Evolve to Inspire: Novelty Search for Diverse Image Generation

AI summary

Overview

Research area: Computer Vision / generative AI — specifically evolutionary prompt optimization for text-to-image diffusion models, sitting at the intersection of quality-diversity algorithms, large language models, and text-to-image generation.

Technical level: Intermediate. The paper is readable without deep mathematical background, but familiarity with diffusion models, CLIP embeddings, novelty search, and prompt optimization baselines (APE, EvoPrompt, QDAIF) helps.

Scope: The paper introduces Wander, a novelty-search-based framework that evolves natural-language prompts with an LLM to produce a set of diverse images from a single starting prompt, and evaluates it against six prompt-optimization baselines.

What This Paper Is About

Text-to-image diffusion models such as Stable Diffusion, FLUX and GLIDE generate high-fidelity images, but they struggle to produce diverse images from one prompt — repeating the same prompt yields similar results, and manually rewriting prompts is unpredictable. The paper's goal is to generate a varied pool of images from a single simple prompt, prioritizing exploration and novelty rather than convergence on one "best" image. To do this, the authors evolve prompts with an LLM, quantify image novelty using CLIP embeddings, and steer the search with hand-written mutation strategies called "emitters."

Key Contributions

1. Wander, a novelty-search framework for image generation. The method operates directly on natural-language prompts, uses an LLM as the mutation and crossover engine, and quantifies novelty as the mean cosine distance between an image's CLIP embedding and its k-nearest neighbors in the pool.
2. Emitters as directed mutation strategies. The authors introduce ten human-designed mutation instructions (for example, "change the composition", "adjust the lighting") that guide evolution into distinct regions of prompt space, and show via ablation that using multiple emitters increases image diversity over a single fixed emitter or no emitter at all.
3. A demonstration that VLM-based quality-diversity fails for this task. Preliminary experiments applying QDAIF with a Vision-Language Model (GPT-4o-mini and Qwen-2.5-VL) to image rating found the feedback insufficiently nuanced or consistent, with similar images assigned to different MAP-Elites cells.
4. An empirical comparison against six baselines. Wander is benchmarked against APE, EvoPrompt-GA, EvoPrompt-DE, PhaseEvo, QDAIF and Lluminate on diversity, relevance, and token usage, including a comparison with the concurrent work Lluminate.

Main Findings

• Highest diversity among compared methods. Wander achieves a Vendi score of 3.60 ± 0.09 and an LPIPS score of 0.80 ± 0.01, higher than every baseline listed in Table 2. The next-best diversity comes from Lluminate (Vendi 3.29 ± 0.02, LPIPS 0.75 ± 0.01), followed by the Wander variants Wander-NE (Vendi 2.61 ± 0.10) and Wander-FE (Vendi 2.95 ± 0.25).
• Much lower token cost than the closest competitor. Wander uses 24,347 ± 649 tokens in Table 2, versus 175,902 ± 9,390 for Lluminate — the paper describes this as seven times fewer tokens (the body text reports an average of 24,363 ± 485 tokens for Wander).
• Baselines score lower on diversity. EvoPrompt-DE (1.42 ± 0.04), PhaseEvo (1.44 ± 0.05), APE (1.47 ± 0.03), EvoPrompt-GA (1.49 ± 0.02) and QDAIF (1.80 ± 0.02) all fall below both Wander variants on Vendi score.
• Slightly lower relevance than QDAIF. Wander's relevance score is 0.272 ± 0.003, compared with 0.297 for QDAIF and 0.292 ± 0.001 for EvoPrompt-DE. The authors argue from qualitative inspection that images remain aligned with the intended class, so the marginal drop does not compromise usability.
• Multiple emitters beat single or no emitters. In an ablation over 10 generations with n = 100 samples, using multiple emitters produced substantially higher diversity than a single fixed emitter per run or no emitters at all.
• Diversity grows over generations. UMAP visualizations over 30 generations, each containing 10 generated images, show increasing diversity, and similarity matrices and Vendi scores at generations 1, 15 and 30 show decreasing pairwise image similarity and rising Vendi score.
• More capable LLMs make better mutators. Using GPT-4o-mini (Vendi 4.2 ± 0.1, 61,402 ± 1,309 tokens), o4-mini (4.5 ± 0.1, 227,655 ± 7,114), GPT-4o (4.8 ± 0.1, 78,067 ± 2,031) and o3 (5.2 ± 0.1, 236,081 ± 8,300), the authors report that OpenAI's o3 achieves a 23% higher Vendi score than GPT-4o-mini, while reasoning models use roughly three times as many tokens as non-reasoning models.
• Occasional relevance drift. Roughly once per 5 runs, or 100 images, the novelty objective leads images to diverge from the initial prompt's core concept.

Methodology in Plain English

The framework starts with a fixed-size pool of prompt-image pairs, created by repeating the input prompt n times and generating an image for each. It then runs for a set number of generations, and each generation performs a fixed number of mutations in three steps.

First, emitter selection: the system picks a mutation strategy from a list of ten hand-written instructions (for example, "simplify and remove unnecessary information", "suggest a novel color scheme"). The paper describes a bandit-driven selection mechanism that continuously reassesses which emitters are working, and also compares it to random selection.

Second, prompt evolution: with a configurable probability (default 50%), the LLM either mutates a single prompt or performs crossover between two existing prompts, guided by the chosen emitter.

Third, pool update: an image is generated for each new prompt using FLUX-DEV, a CLIP embedding is computed, and a novelty score is calculated as the mean cosine distance between that image embedding and its k nearest neighbors in the pool. If a candidate is more novel than the least novel member of the pool, it replaces it. This keeps the pool at a fixed size and progressively pushes diversity upward.

The implementation uses GPT-4o-mini for prompt mutation and FLUX-DEV for image generation, with OpenAI's CLIP-ViT-B-32 for image and text embeddings. Evaluation uses the LPIPS perceptual distance, the Vendi score (the effective number of diverse samples, computed from eigenvalues of a normalized diversity matrix over pairwise cosine similarities), and a Relevance metric measuring the average cosine distance between text embeddings of the original and evolved prompts. In the main comparison each algorithm was run 10 times on each of 10 starting prompts, for 100 runs per method; the emitter ablation ran 10 times for each of 10 prompts; and the random-versus-bandit comparison ran for 30 generations.

Why This Matters

The work reframes prompt optimization for image generation away from "find the single best image" and toward "map out a spread of genuinely different images." That matters because existing prompt-optimization methods are built around a fitness objective — aesthetics or NLP task performance — and the paper shows directly that they produce narrower image pools than a novelty-driven approach, while a VLM-in-the-loop quality-diversity approach fails outright on this task.

Real-world applications suggested by the work:

• Ideation and creative exploration — generating many distinct visual directions from a single short brief for designers, illustrators and concept artists.
• Brainstorming aids for non-expert users — giving people who cannot write detailed prompts a way to see a broad range of interpretations of their idea.
• Data augmentation for computer vision — the paper names this as a potential downstream application of varied generated image pools.
• Red-teaming and safety research — the paper lists generating image model jailbreaks as another potential downstream use.

Industry relevance: Because Wander is black-box and model-agnostic, it can be layered on top of proprietary image APIs without access to model parameters, and unlike fine-tuning or prompt-expander training it does not need to be re-run when the underlying image model changes. Its token efficiency relative to Lluminate (roughly seven times fewer tokens) is directly relevant to the cost of deploying such a system at scale.

Future Directions

• Generating emitters automatically. The authors note emitters must be manually specified, which could bias or cap asymptotic diversity, and suggest an LLM could generate emitters instead.
• Extending to other modalities. The paper argues the approach generalizes to any domain where a meaningful distance metric can be defined on latent representations, such as text and audio; it demonstrates a text-based version used to inspire the paper's own title.
• Better steerability and drift control. The paper raises constraining the direction of exploration more strongly, and mitigating relevance drift through prompt tuning or an explicit relevance penalty during selection.
• Aesthetic and quality evaluation. The authors did not assess the aesthetic quality of generated images; they note Stable Diffusion 1.0 was prone to low-quality images during early experiments but that this was not observed with FLUX-DEV, and that aesthetic evaluation could give a more complete picture.

Target Audience

Researchers and practitioners in evolutionary computation, quality-diversity algorithms, and automatic prompt optimization will find the direct comparison against APE, EvoPrompt, PhaseEvo, QDAIF and Lluminate most useful. Generative AI engineers building creative tooling — particularly those wanting diversity from black-box image APIs without fine-tuning — are the practical audience. It is also relevant to HCI and creativity-support researchers interested in ideation tools, and to anyone studying the limits of vision-language models as automated evaluators, since the negative QDAIF/VLM result is reported in detail.

Authors’ abstract

Text-to-image diffusion models, while proficient at generating high-fidelity images, often suffer from limited output diversity, hindering their application in exploratory and ideation tasks. Existing prompt optimization techniques typically target aesthetic fitness or are ill-suited to the creative visual domain. To address this shortcoming, we introduce WANDER, a novelty search-based approach to generating diverse sets of images from a single input prompt. WANDER operates directly on natural language prompts, employing a Large Language Model (LLM) for semantic evolution of diverse sets of images, and using CLIP embeddings to quantify novelty. We additionally apply emitters to guide the search into distinct regions of the prompt space, and demonstrate that they boost the diversity of the generated images. Empirical evaluations using FLUX-DEV for generation and GPT-4o-mini for mutation demonstrate that WANDER significantly outperforms existing evolutionary prompt optimization baselines in diversity metrics. Ablation studies confirm the efficacy of emitters.