A Retrospect to Multi-prompt Learning across Vision and Language

AI summary

Overview

Research area: Computer vision / vision-language pretraining, specifically prompt-based adaptation of Vision-Language Models (VLMs) such as CLIP.

Technical level: Intermediate to Advanced. The paper combines empirical embedding-geometry analysis, formal propositions, and an energy-based learning method, so familiarity with CLIP, prompt tuning, and contrastive learning is helpful.

Scope: The paper revisits multi-prompt learning for vision-language models, explains theoretically and empirically why multiple learnable prompts outperform a single prompt, and proposes an Energy-based Multi-prompt Learning (EMPL) method that balances in-domain accuracy with open-vocabulary generalization.

What This Paper Is About

Existing prompt-learning research for VLMs (e.g., CoOp, CoCoOp) has focused almost entirely on a single learnable prompt template, while multi-prompt variants remain under-explored despite being widely viewed as promising. The authors investigate why multiple prompts help, framing the question through the "constant modality gap" phenomenon and the risk that more prompts lead to overfitting on training classes. Their goal is a principled account of multi-prompt learning, plus a new algorithm that keeps in-domain performance high without sacrificing recognition of unseen classes.

Key Contributions

1. Extending the modality-gap analysis to learnable prompts. The authors extend the constant modality-gap phenomenon to learnable prompts and show empirically that adding prompts tends to close the modality gap, so the previously reported failure of over-extended prompt scales more likely stems from an overfitted model than from an inability to bridge the cross-modal disparity.
2. Identifying the cross-modal non-identifiability issue. They formalize two propositions showing that a single-prompt learner obeying a constant modality gap cannot distinguish images (or image groups) whose concepts are mutually exclusive, which explains why multi-prompt learning improves vision-language transfer.
3. Proposing Energy-based Multi-prompt Learning (EMPL). EMPL defines an energy-based prompt distribution over an image variable and a prompt variable, generating multiple prompt embeddings by drawing instances with a Stochastic Gradient Langevin Dynamics (SGLD) sampler; the objective combines a generic prompt-learning goal with an EBM uncertainty-modeling term balanced by a hyper-parameter lambda.
4. Comprehensive evaluation. Experiments cover three tasks and fifteen datasets, comparing CLIP, CoOp, CoCoOp, ProDA, and PLOT, and combining EMPL with CoOp and ProDA.

Main Findings

• Constant modality gaps persist under prompt learning. Measured on MSCOCO with the CLIP → CoOp → ProDA → ProDA(x2) → ProDA(x4) progression, both individual modality gap (IMG) and class modality gap (CMG) were approximated by constant vectors with trivial variances in magnitude and direction.
• More prompts shrink the gap. CoOp trained from CLIP further minimized the gap magnitude but underperformed ProDA with prompt augmentation; adding prompts notably closed the modality gaps. The paper therefore attributes the failure of overextended prompt scales to overfitting rather than to an inability to bridge the modality disparity.
• Single prompts are provably non-identifiable. Proposition 1 (individual-level) states that a single-prompt model satisfying the constant individual-level modality gap cannot distinguish two images with mutually exclusive concepts; Proposition 2 (population-level) extends this to groups. Both are stated informally in the main text, with formal statements and proofs in Appendix A.
• EMPL improves the base-to-new trade-off. CoOp with EMPL gained +7.71 on new classes (63.22 → 70.93) and reached a Harmonic mean of 76.38 (+4.73), while base accuracy changed only from 82.66 to 82.73 (+0.07). ProDA with EMPL reached a new-class accuracy of 73.27 (+0.98) and a Harmonic mean of 77.39 (+0.74), the highest trade-off among the reported methods. For reference, CLIP scored 69.34 / 74.22 / 71.69 (Base / New / H), CoCoOp 80.47 / 71.69 / 75.83, ProDA 81.56 / 72.29 / 76.65, and PLOT* 75.90 / 67.6 / 71.8.
• Per-dataset ablations are mixed but mostly positive. EMPL benefited 7 of 10 datasets for CoOp and 6 of 10 for ProDA in base-class performance, with negative effects on the minority. The authors link this to the conservative tendency of the open-vocabulary meta-learning objective.
• Cross-domain robustness improves in most settings. EMPL improved cross-domain accuracy in seven of eight situations. CoOp with EMPL moved ImageNet-Sketch from 47.99 to 48.64, ImageNet-A from 49.71 to 51.27, and ImageNet-R from 75.21 to 76.01, while dropping on the source ImageNet from 71.51 to 70.89. ProDA with EMPL moved ImageNet-A from 51.62 to 52.35 and ImageNet-R from 75.67 to 76.84, but dropped on ImageNet (71.41 → 71.17) and ImageNet-V2 (65.14 → 64.79). CLIP scored 66.73 / 60.83 / 46.15 / 47.77 / 73.96 and CoCoOp 71.02 / 64.07 / 48.75 / 50.63 / 76.18 on ImageNet / V2 / Sketch / A / R.
• Cross-dataset transfer averages rise. CoOp with EMPL raised the eleven-dataset average from 63.88 to 66.49, and ProDA with EMPL raised it from 65.89 to 66.81, versus CoCoOp at 65.74. The largest single jump was FGVCAircraft for CoOp (18.47 → 23.16).
• Theoretical property of the objective. Proposition 3 states that the EMPL objective encourages the marginal image-prompt distribution from training data and the distribution marginalized over unseen classes to be negatively correlated, which the authors interpret as an exploitation-exploration balance.

Methodology in Plain English

The authors start by looking at how CLIP places image embeddings and text embeddings in the same space. Prior work found that the offset between an image and its matching caption behaves like a near-constant vector — a "modality gap." The authors measure this gap for prompt-learning models (CoOp and ProDA) at increasing numbers of prompts, and confirm the offset stays roughly constant while its size shrinks as prompts are added.

Using that constant-gap assumption, they argue that a model relying on one prompt template effectively forces all its image-text pairs to share the same offset, which makes it impossible to separate images whose labels are mutually exclusive. They state this as two propositions.

They then ask how to get the benefit of many prompts without overfitting. Their answer is to treat prompts as samples from a probability distribution defined by an energy function that takes both an image and a prompt as input. Prompts with low energy correspond to well-matched image-prompt pairs. Sampling is done with Stochastic Gradient Langevin Dynamics, alternating updates to the image variable in feature space and the prompt variable in embedding space, adding Gaussian noise at each step. This avoids storing a large fixed collection of context vectors and adds few parameters beyond the base contexts.

Training uses a meta-learning objective over tasks, each with K′ observed classes and the remaining classes of the open vocabulary treated as unseen. The first objective term is standard prompt-learning log-likelihood over observed classes; the second term pushes down the energy of image-prompt pairs drawn from the current energy-based distribution. A hyper-parameter lambda balances the two. The resulting sampler both produces prompts for classification and supplies training instances for the energy term.

EMPL is orthogonal to existing prompt-tuning methods, so the authors implemented it on top of CoOp and ProDA, replacing their prompt generation with the SGLD-based sampler while keeping their original objectives as the first term. They note that applying it to CoCoOp and PLOT was prohibitively memory-intensive with the open-source versions.

Why This Matters

Impact on research. The paper supplies a geometric and theoretical account of why multi-prompt learning helps, rather than leaving it as an empirical curiosity. It also reframes the overfitting concern around prompt count, suggesting the real problem is model overfitting rather than an inherent limit on bridging modality disparity, and it introduces energy-based modeling and an SGLD sampler into vision-language prompt tuning.

Real-world applications (grounded in the benchmark domains and downstream tasks mentioned in the paper):

• Low-resource visual recognition where labeled data is scarce, since prompt learning adapts frozen VLMs with limited resources.
• Fine-grained recognition in specialized domains, reflected in the benchmark use of OxfordPets, StanfordCars, Flowers102, Food101, and FGVCAircraft.
• Satellite and remote-sensing image classification, corresponding to the EuroSAT benchmark.
• Open-vocabulary object detection, semantic segmentation, and scene graph generation, which the paper cites as existing prompt-learning applications that multi-prompt methods could upgrade.

Industry relevance. EMPL adds few parameters beyond the base context vectors and keeps the VLM backbone frozen, which suits deployments where retraining large vision-language models is impractical. Its domain-shift robustness results (ImageNet-Sketch, ImageNet-A, ImageNet-R) matter for products that must handle unusual or adversarial inputs.

Future Directions

• Extending EMPL to CoCoOp and PLOT, which the authors could not do because of the heavy memory consumption of those open-source implementations.
• Better understanding the negative effects EMPL produced on a minority of datasets, which the authors ascribe to the conservative tendency of open-vocabulary meta-learning toward observed classes.
• Further theoretical characterization of multi-prompt generalization beyond the negative-correlation property in Proposition 3, and of the conditions under which prompt scaling stops helping.
• Tuning and understanding the balance hyper-parameter lambda, which governs the trade-off between in-domain image recognition and open-vocabulary concept exploration.

Target Audience

Researchers and graduate students working on vision-language pretraining, prompt tuning, parameter-efficient adaptation, or open-vocabulary recognition. It is also relevant to practitioners who need to adapt CLIP-style models to new classification tasks with limited data and want a method that preserves zero-shot capability. Readers without a background in CLIP, contrastive learning, or energy-based models will need to consult the background section (Section 3) and the cited works first.

Note: The provided paper content is truncated. The cross-dataset transfer results discussion begins but is cut off mid-word, so the authors' analysis of Table 3 is not fully reported here; only the table values are available.

Authors’ abstract

The vision community is undergoing the unprecedented progress with the emergence of Vision-Language Pretraining Models (VLMs). Prompt learning plays as the holy grail of accessing VLMs since it enables their fast adaptation to downstream tasks with limited resources. Whereas existing researches milling around single-prompt paradigms, rarely investigate the technical potential behind their multi-prompt learning counterparts. This paper aims to provide a principled retrospect for vision-language multi-prompt learning. We extend the recent constant modality gap phenomenon to learnable prompts and then, justify the superiority of vision-language transfer with multi-prompt augmentation, empirically and theoretically. In terms of this observation, we propose an Energy-based Multi-prompt Learning (EMPL) to generate multiple prompt embeddings by drawing instances from an energy-based distribution, which is implicitly defined by VLMs. So our EMPL is not only parameter-efficient but also rigorously lead to the balance between in-domain and out-of-domain open-vocabulary generalization. Comprehensive experiments have been conducted to justify our claims and the excellence of EMPL.