Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

AI summary

Overview

Research area: Automatic prompt optimization for Multimodal Large Language Models (MLLMs), spanning computer vision, video understanding, and molecular property prediction.

Technical level: Advanced. The paper formalizes a new optimization problem, introduces Bayesian bandit machinery, and reports results across ten benchmarks and multiple backbone models.

Scope: The paper defines "multimodal prompt optimization" — jointly searching over textual and non-textual prompt components — and proposes MPO (Multimodal Prompt Optimizer) to solve it.

What This Paper Is About

Existing automatic prompt optimization methods only search over text, even though the models they are applied to (MLLMs) can accept images, videos, and molecules. This leaves much of the model's expressive capacity unused, because some information — such as the distinguishing features of a bird species — is conveyed far more directly by an image than by a long and potentially ambiguous description. The paper defines the new problem of optimizing a prompt as a pair of a textual and a non-textual component, and proposes a framework that refines both together.

Key Contributions

1. A new problem formulation. The paper defines a multimodal prompt as a pair $(t, m) \in \mathcal{T} \times \mathcal{M}$ — a textual prompt and a non-textual prompt — and defines multimodal prompt optimization as finding the pair that maximizes task performance, rather than optimizing over text alone.

2. Alignment-preserving exploration. MPO generates a single unified feedback signal from a failure set that encodes cross-modal weaknesses in textual form, then uses it both to revise the textual prompt and to produce a modality-specific condition that drives a generator (text-to-image, text-to-video, text-to-molecule) to create a matching non-textual prompt. Three operators — generation (create from scratch), edit (refine an existing prompt), and mix (blend multiple prompts) — diversify the search.

3. Prior-inherited Bayesian UCB selection. The paper measures a strong positive correlation between parent and child prompt performance (Pearson's $r = 0.88$) and uses the parent's posterior mean as a prior for the child's Beta distribution, warm-starting evaluation instead of treating each candidate as an independent arm.

4. A theoretical guarantee. Proposition 3.1 states that when the parent prior is more informative than uniform, the best-arm identification cost of Bayesian UCB is nonincreasing.

Main Findings

• MPO leads across all ten datasets. Averaged over all benchmarks, MPO reaches 65.1 versus 60.0 for the strongest text-only optimizer, ProTeGi, and 59.1 for SEE. Manual prompting trails further: Human averaged 44.1, Chain-of-Thought 40.8, and Few-Shot variants 47.2 (1-shot), 49.5 (3-shot), and 49.3 (5-shot).

• Gains hold per-modality. On PlantVillage, MPO scores 76.4 versus SEE's 69.0 and PE2's 67.9. On CUB, 78.6 versus 71.6 for both SEE and PE2. On VANE-Bench video VQA, 71.2 versus 62.4 for 1-shot. On the molecular Absorption F1, 64.5 versus SEE's 60.0; on BBBP accuracy, 75.3 versus ProTeGi's 72.1; on CYP inhibition accuracy, 64.3 versus SEE's 61.4.

• Both modalities contribute, and they reinforce each other. With human-written text and no MPO image, PlantVillage is 42.2 and CUB is 47.9. With MPO text only, 55.6 and 64.2. Using MPO text with an MPO image only raises these to 50.4 and 58.2. The full multimodal prompt reaches 76.4 and 78.6, showing the components are not merely additive.

• All three exploration operators help individually and together. On PlantVillage subtasks, SEE averages 69.0; generation alone reaches 73.3, edit 72.5, mix 74.8, and the full MPO 76.4.

• Prior inheritance saves evaluation budget. MPO matches the Uniform strategy's performance using only 30% of the evaluation budget (a 70% reduction), and matches standard UCB and the prior-free variant with 52% and 42% less budget respectively.

• Cross-modal alignment tracks performance. Using the DSG score, MPO achieves both the highest alignment score and the largest gains over the Human baseline, followed by the Sequential variant and Random Image Prompt, with In-Distribution and OOD Image Query variants lagging significantly behind.

• MPO avoids the text-only ceiling. On CUB, ProTeGi plateaus after the third iteration with only a 1.1-point additional gain, while MPO continues improving for an additional 6.4-point gain beyond the third iteration.

• Hidden states shift under multimodal prompts. PCA visualization shows text-only methods (including MPO's own text component) clustering in a similar region, while MPO's full multimodal prompt shifts hidden states into a distinct region.

• Generalization across backbones. With Qwen2.5-VL (72B) as base model MPO reaches 80.4; with Gemma3 (12B), 73.1; with InternVL-3.5 (14B), 73.2; with GPT-4.1 nano, 65.9. Varying the optimizer, MPO ranges from 69.1 (Qwen2.5-VL 7B) to 78.0 (GPT-4o). Varying the image generator, MPO ranges from 71.8 (SANA1.5 1.6B) to 76.6 (GPT-Image-Medium), all above SEE's text-only 69.0.

• Prior strength has an optimum. Small $S$ under-utilizes the parent prior; excessively large $S$ causes over-reliance, preventing adaptation to the child's actual performance. Performance peaks at an intermediate value (the paper uses $S=10$).

Methodology in Plain English

MPO runs an iterative beam search over prompts. It starts with a beam size of $b=3$ and runs $T=13$ iterations, keeping the explored-prompt count equal to baselines for fairness, with an evaluation budget of 100 per candidate.

At each round, the system looks at a failure set — queries where the model's answer differed from the correct answer — and asks an MLLM to produce one combined piece of feedback that covers weaknesses across both modalities. That single signal is used twice: to rewrite the text prompt, and to write a "condition" describing how the image, video, or molecule prompt should change. A modality-specific generator (GPT-Image for images, Wan2.1 1.3B for videos, GPT-4o mini for molecules) turns that condition into the new non-textual prompt, so the two parts stay coherent rather than being optimized separately.

From the top $b$ prompts, $b^2$ children are generated by splitting the work evenly across three operators. Generation creates a new non-textual prompt from scratch; edit refines a prior one; mix combines $K$ parents. In the first iteration, only generation runs, since no non-textual prompts exist yet.

Selection is where the Bayesian component matters. Each candidate's expected score is modeled as a Beta distribution. A newly created child inherits its prior from its parent's posterior mean, scaled by $S$ pseudo-observations. Each round, the candidate with the highest UCB score is evaluated on a small data batch and its posterior updated. When the budget runs out, the candidate with the highest expected score becomes a parent for the next iteration. Answer generation uses Qwen2.5-VL (7B) for image and video tasks and Qwen3 (8B) for molecular tasks, with GPT-4o mini as the prompt optimizer. All results are averaged over three independent runs.

Why This Matters

This work reframes prompt optimization as a search over multimodal inputs rather than text strings, which matters because the models being prompted are themselves multimodal. It provides an evaluation suite that crosses image classification, VQA, video action recognition, and molecular property prediction, and it supplies a budget-efficiency argument (via the Bayesian UCB analysis) for why inherited priors should be used when the search space is large and good candidates are sparse. Code is released at https://github.com/Dozi01/MPO.

Real-world applications implied by the evaluated benchmarks:

• Agriculture: identifying diseased leaves (PlantVillage) where visual symptoms carry the diagnostic signal.
• Medical imaging: radiology VQA (SLAKE), where text descriptions of findings are lossy compared to the image itself.
• Autonomous driving: driver action recognition (Drive&Act) and driving-scene VQA (DrivingVQA), where prompts must describe dynamic visual context.
• Drug discovery: absorption, blood-brain barrier penetration, and CYP inhibition prediction (TDC), where molecular structure prompts encode chemistry that text struggles to express.

Industry relevance: the generalizability results matter commercially — MPO still beats text-only optimization when paired with a lightweight 1.6B image generator (SANA1.5), and it scales upward with larger base models. That means the approach is not locked to a single expensive vendor stack.

Future Directions

• Extending beyond the three modalities tested. The paper covers images, videos, and molecules; audio and other modalities are mentioned as MLLM capabilities but not evaluated.
• Reducing dependence on external generators. MPO relies on separate text-to-image, text-to-video, and text-to-molecule modules; the results show sensitivity to generator quality (71.8 with SANA1.5 versus 76.6 with GPT-Image-Medium), leaving open how to close that gap.
• Understanding the hidden-state shift. The paper shows multimodal prompts move model representations into a distinct region but does not fully characterize what reasoning pathways this enables.
• Tuning the prior mechanism further. Since performance peaks at an intermediate prior strength and the optimum evidently depends on budget, a principled way to set $S$ automatically is not established.

Target Audience

Researchers and practitioners in prompt engineering, multimodal learning, and LLM/MLLM systems who want to move beyond text-only prompt optimization. It is also relevant to applied teams in domains where the essential signal is inherently visual or structural — medical imaging, remote sensing, autonomous driving, and cheminformatics — and to anyone interested in Bayesian bandit methods for efficient search over expensive-to-evaluate candidates. Readers should be comfortable with optimization formalism and bandit terminology; the intuition is accessible, but the methodology sections assume familiarity with beam search and Bayesian selection.

Authors’ abstract

Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.