GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation

AI summary

Overview

Research area: Computer Vision / Text-to-Image Generation / Multi-Agent Systems / Automatic Prompt Optimization Technical level: Intermediate Scope: This paper introduces GenPilot, a plug-and-play, model-agnostic multi-agent framework that improves text-to-image generation by iteratively optimizing the input prompt at test time, without fine-tuning any underlying generative model.

What This Paper Is About

Text-to-image (T2I) models such as Stable Diffusion, FLUX, and DALL-E 3 often fail to faithfully render complex or lengthy prompts, producing images with semantic inconsistencies, missing objects, wrong counts, or broken spatial relationships. GenPilot solves this by treating prompt optimization as a search problem at inference time: a team of specialized AI agents analyzes errors in the generated image, generates candidate prompt edits, scores them, clusters them, and iteratively picks the best to feed back to the T2I model.

Key Contributions

1. GenPilot framework: A plug-and-play multi-agent system that performs test-time prompt optimization as a search over the text space, enabling interpretable, model-agnostic refinement of prompts without any model training.
2. Systematic error analysis and fine-grained verification: Combines prompt decomposition, VQA-based detection, caption-based comparison, and an integration step to localize and map semantic errors back to specific prompt segments.
3. Clustering-based adaptive exploration with memory: Uses K-Means with Bayesian posterior updates to identify high-potential prompt clusters, plus a memory module that stores past ratings and error summaries to guide later iterations.
4. Empirical validation and pattern library: Demonstrates consistent improvements across DPG-bench (up to 16.9%) and GenEval (up to 5.7%) on multiple T2I models, and releases 35 common error patterns with refinement strategies as a reusable resource.

Main Findings

• Consistent cross-model gains: Applying GenPilot improves the average score on every tested T2I model — DALL-E 3 (72.04 → 74.08), FLUX.1 schnell (68.16 → 73.32), SD v1.4 (53.16 → 62.12), SD v2.1 (57.24 → 61.72), SD 3 (58.81 → 62.89), and Sana-1.0 1.6B (73.98 → 75.38).
• Outperforms prior prompt methods: GenPilot beats Prompt Engineering (PE), naive test-time scaling (TTS), MagicPrompt, and BeautifulPrompt on average scores, while keeping comparable or better performance on subcategories like attribute binding and relations.
• Best on challenging prompts: On a curated 264-prompt DPG-bench subset (most below 0.81 baseline score), GenPilot delivers the largest gains — for example, +8.96 points on SD v1.4, showing it targets cases where models struggle most.
• Short prompts benefit too: On GenEval, GenPilot improves FLUX.1 schnell from 65.82% to 69.60% (vs. 66.59% for PE) and PixArt-α from 46.73% to 48.54%, with strong gains in position and color-attribute tasks.
• Memory and clustering matter: Ablations show removing either the memory module (66.05%) or clustering (66.27%) drops performance below the full system (69.60%), but both variants still beat the base model.
• Component flexibility: Swapping Qwen2.5-VL-72B for MiniCPM-V 2.0 or replacing the captioner with BLIP-2 still improves over baseline, though with smaller margins — confirming modularity.
• Error integration beats single-source analysis: GPT-4o ratings show integrated error analysis scores 4.62/5, versus 3.78 for VQA-only and 3.95 for caption-only, validating the combined strategy.
• Latency trade-off: Optimization adds inference-time overhead (e.g., ~69.6s of optimization time in a 3-iteration setup), mitigated with parallelization and early stopping.

Methodology in Plain English

GenPilot works in two stages:

Stage 1 — Error Analysis. The original prompt is broken into "meta-sentences" covering objects, relationships, and background. Two parallel checks run: a Visual Question Answering (VQA) branch asks yes/no questions about object existence, attributes, counts, and spatial relations, while a captioning branch generates a description of the image and compares it to the prompt. An integration agent merges inconsistencies into a single error list, and a mapping agent traces each error back to the specific prompt segment responsible for it.

Stage 2 — Test-Time Prompt Optimization. A refinement agent produces many candidate edits for the problematic sentences. Each candidate is merged into the full prompt, used to generate an image, and then scored by a multimodal LLM rater (checking attribute binding, relationships, position, background, style, and atmosphere). Candidates are grouped with K-Means, and a Bayesian update computes which cluster is most likely to contain good prompts; the winning cluster provides the next round's base. A memory module records all scores and error summaries, feeding them back so later rounds avoid past mistakes. The loop runs until convergence or a maximum iteration limit.

Why This Matters

This work reframes test-time scaling for image generation — most prior work scales noise, samples, or denoising steps — by scaling the text input itself. This makes optimization interpretable (you can read the modified prompt), training-free (works on any T2I model), and reusable across architectures.

Real-world applications:

• Creative and advertising content generation: Marketers can get precise, on-brief imagery from lengthy brand specifications without retraining models.
• Design and prototyping tools: Designers using prompts with complex spatial layouts or counting constraints get more reliable drafts.
• Accessible content creation: Users without prompt-engineering skills can rely on the agents to fix underspecified prompts automatically.
• Data annotation and synthetic dataset generation: Higher prompt-image fidelity improves the quality of synthetic training data for downstream vision models.

Industry relevance: Since GenPilot is model-agnostic and requires no fine-tuning, T2I service providers (Midjourney, Adobe Firefly, Stability, etc.) could bolt it on as a backend layer to boost user satisfaction without retraining expensive models. The 35 released error-pattern/refinement pairs also provide practical engineering knowledge for prompt controllability.

Future Directions

• Reducing inference latency: Optimization overhead is significant; future work could distill the multi-agent pipeline into a smaller, faster model or develop better early-stopping criteria.
• Reducing MLLM dependence: Since quality depends on the underlying multimodal LLM, robustness with weaker or open-source evaluators (e.g., MiniCPM-V) needs further improvement.
• Broader modality and task coverage: Extending test-time prompt optimization to video generation, 3D content, or editing tasks that also suffer from prompt-image misalignment.
• Learning the search policy: Instead of K-Means + Bayesian updates, reinforcement learning or learned proposal models could make exploration sharper and more sample-efficient.

Target Audience

This paper is most valuable for researchers and engineers working on text-to-image generation, prompt engineering, and multi-agent LLM systems. Practitioners building production image-generation pipelines who need better prompt-image fidelity without retraining models will find the framework directly applicable, while students entering the T2I field will benefit from the clear decomposition of error types and refinement patterns released in the appendix.

Authors’ abstract

Text-to-image synthesis has made remarkable progress, yet accurately interpreting complex and lengthy prompts remains challenging, often resulting in semantic inconsistencies and missing details. Existing solutions, such as fine-tuning, are model-specific and require training, while prior automatic prompt optimization (APO) approaches typically lack systematic error analysis and refinement strategies, resulting in limited reliability and effectiveness. Meanwhile, test-time scaling methods operate on fixed prompts and on noise or sample numbers, limiting their interpretability and adaptability. To solve these, we introduce a flexible and efficient test-time prompt optimization strategy that operates directly on the input text. We propose a plug-and-play multi-agent system called GenPilot, integrating error analysis, clustering-based adaptive exploration, fine-grained verification, and a memory module for iterative optimization. Our approach is model-agnostic, interpretable, and well-suited for handling long and complex prompts. Simultaneously, we summarize the common patterns of errors and the refinement strategy, offering more experience and encouraging further exploration. Experiments on DPG-bench and Geneval with improvements of up to 16.9% and 5.7% demonstrate the strong capability of our methods in enhancing the text and image consistency and structural coherence of generated images, revealing the effectiveness of our test-time prompt optimization strategy. The code is available at https://github.com/27yw/GenPilot.