ReBaPL: Repulsive Bayesian Prompt Learning

AI summary

Overview

Research area: Machine learning — Bayesian inference and parameter-efficient adaptation of large vision-language models (vision-language prompt learning).

Technical level: Advanced. The paper builds on stochastic gradient Hamiltonian Monte Carlo, Langevin dynamics, cyclical learning-rate schedules, and probability metrics (Maximum Mean Discrepancy, Wasserstein distance / optimal transport).

Scope: The paper proposes ReBaPL, a modular Bayesian extension that can be layered on top of any maximum-likelihood-based prompt learning method, and evaluates it on 11 classification benchmarks plus cross-dataset and domain-generalization protocols.

What This Paper Is About

Standard prompt learning methods (such as CoOp and CoCoOp) optimize a small set of continuous prompt vectors by maximum likelihood estimation, which makes them prone to overfitting on the few training samples available and weak at generalizing to out-of-distribution data and unseen classes. The authors argue that the space of good prompts is not a single point but a complex, multimodal landscape, and that finding only one solution wastes generalization potential. ReBaPL therefore reframes prompt learning as Bayesian inference and tries to sample many diverse, high-probability prompts from that posterior rather than committing to a single point estimate.

Key Contributions

1. Repulsive cyclical SGHMC (rcSGHMC): A new sampling algorithm that combines stochastic gradient Hamiltonian Monte Carlo with a cosine step-size schedule that alternates exploit and explore phases, plus an added repulsion term between prompt samples drawn from different cycles.
2. Representation-based repulsion: Instead of measuring distance between raw prompt parameters in weight space, the method defines the repulsion potential using probability metrics — Maximum Mean Discrepancy and Wasserstein distance — between the distributions of representations that different prompts induce, capturing functional rather than numerical similarity.
3. A plug-and-play Bayesian wrapper: Unlike prior Bayesian prompt learning methods, ReBaPL is described as a modular Bayesian extension that can be applied on top of any existing maximum-likelihood prompt learning method; the authors demonstrate it on top of MaPLe and MMRL.
4. Empirical validation across three protocols: Base-to-novel generalization on 11 benchmarks, cross-dataset transfer, and domain generalization, all under a 16-shot setting.

Main Findings

• Base-to-novel gains over MaPLe: MaPLe + ReBaPL improves the 11-dataset average from 82.03% to 83.28% on base classes, from 75.03% to 76.08% on novel classes, and from 78.37% to 79.52% on the harmonic mean, i.e. deltas of +1.25, +1.05, and +1.15. The paper reports that MaPLe is still slightly better than ReBaPL on some datasets (OxfordPets, StanfordCars, FGVCAircraft), but that the harmonic mean consistently improves.
• Gains over MMRL concentrate on novel classes: MMRL + ReBaPL moves the average from 85.54% to 85.74% (base), 76.52% to 77.44% (novel), and 80.59% to 81.38% (harmonic mean), with deltas of +0.20, +0.92, and +0.79.
• Large EuroSAT improvement: On EuroSAT, MMRL + ReBaPL improves novel accuracy from 77.20% to 83.63% (+6.43) and harmonic mean from 85.50% to 89.71% (+4.21). MaPLe + ReBaPL improves EuroSAT novel from 77.03% to 79.20% (+2.17) and harmonic mean from 84.05% to 86.51% (+2.46).
• Some regressions are reported: Not every cell improves. Examples include MMRL + ReBaPL on Caltech101 novel (-0.33) and harmonic mean (-0.10), on FGVCAircraft base (-0.57), and MaPLe + ReBaPL on OxfordPets novel (-0.36); the paper states that on average both base and novel improve.
• Best cross-dataset average: MMRL + ReBaPL achieves the highest average accuracy across the 10 target datasets at 67.62% (MMRL* at 66.87%, a delta of +0.75). MaPLe + ReBaPL reaches 66.77% versus MaPLe* at 65.63% (+1.14). The largest single transfer gain is on EuroSAT as a target: 50.90% for MaPLe + ReBaPL versus 45.70% (+5.20), and 52.97% for MMRL + ReBaPL versus 49.83% (+3.14).
• Domain generalization improves without hurting the source domain: MaPLe + ReBaPL raises ImageNet from 67.96% to 68.66% and improves ImageNetV2 (61.57 to 62.30), ImageNet-S (47.70 to 48.50), ImageNet-A (48.80 to 49.73), and ImageNet-R (75.33 to 75.40). MMRL + ReBaPL raises ImageNet from 70.13% to 71.00% and improves ImageNetV2 (62.20 to 62.50), ImageNet-S (47.80 to 48.40), ImageNet-A (48.90 to 49.63), and ImageNet-R (75.03 to 75.63). The paper notes MaPLe + ReBaPL gives the best ImageNet-A target result and MMRL + ReBaPL the highest source ImageNet accuracy.
• Ablation: The paper states it compares ReBaPL without repulsion (setting the force to zero) against repulsion based on Wasserstein distance and MMD, in order to isolate the benefit of repulsion. The numeric outcome of this ablation is not present in the available content.
• Choice of metric is method-dependent: The authors report that they used MMD for MaPLe + ReBaPL and Wasserstein distance for MMRL + ReBaPL, as these respectively performed best.
• Computational cost: MMD scales as O(n²) and Wasserstein as O(n³) in the number of samples, but because these are computed at the mini-batch level (e.g. 32 samples), the authors state the overhead is negligible. Inference over the C times K sampled models is described as embarrassingly parallelizable and adding only small overhead.

Methodology in Plain English

The researchers start from an existing prompt learning model (MaPLe or MMRL) that already learns visual and textual prompt tokens inside CLIP's two encoders. Rather than keeping a single set of learned prompts, they treat the prompts as random variables and try to draw samples from the posterior distribution over prompts — the set of prompts that are plausible given the data.

To do this they run a sampler (SGHMC) that behaves like a ball rolling over a landscape of model quality: it adds momentum and calibrated noise so that instead of settling in one spot it keeps moving through high-quality regions. They add two twists. First, the step size follows a cosine schedule and each cycle is split into an exploration phase (where noise is injected and the sampler wanders to new regions) and a sampling phase (where noise is switched off and the sampler refines what it found), with an importance indicator function deciding which phase applies at each iteration. Second, they add a repulsive force between the prompts sampled in the current cycle and those from the previous cycle, so that new samples are pushed away from old ones instead of collapsing onto the same solution.

The clever part is how "different" is measured. Instead of comparing raw prompt vectors, they compare the distributions of image representations that different prompts produce, using Maximum Mean Discrepancy or Wasserstein distance. Prompts that lead the model to see images in similar ways are pushed apart, which encourages functionally diverse solutions. At prediction time, the final answer is an ensemble over all C cycles times K samples, using uniform weights (each weight being 1/(C·K)) for simplicity.

Why This Matters

Impact on research: The paper argues that characterizing the whole posterior over prompts is more useful than chasing one regularized optimum, and that doing so with a repulsion mechanism based on representations rather than parameters sidesteps the known difficulty of defining distances in weight space, which is confounded by permutation invariance and data scarcity. Because ReBaPL is presented as a plug-and-play Bayesian layer over any maximum-likelihood prompt method, it offers a reusable recipe rather than a competing architecture.

Potential real-world applications (as implications of the method, not tested in this paper):

• Few-shot image classification in settings where only a handful of labels per category are available, which is the paper's own 16-shot regime.
• Deployment under domain shift, such as images captured by different cameras, sensors, or in different environments than the training data.
• Scientific or medical image triage, where classes are rare and data-scarce — the paper motivates Bayesian learning with reference to data-scarce applications.
• Adapting general-purpose vision-language models to specialized catalogs (products, aircraft, satellite imagery) without retraining the base model.

Industry relevance: Prompt learning is attractive commercially because it adapts a large pretrained model by learning a tiny number of parameters instead of fine-tuning the whole network. A method that improves robustness under distribution shift while keeping the base model frozen, and that can be bolted onto an existing prompt-learning pipeline, is directly relevant to teams deploying vision-language models in production where labeled data is limited and inputs drift over time.

Future Directions

• Choice of repulsion metric: The authors used MMD for one base method and Wasserstein for the other, selected empirically. Understanding when each metric is preferable, and whether other probability metrics would help, remains open.
• Learned ensemble weights: The paper uses uniform weights over the C·K sampled models and states this is done "for simplicity," implying that importance weighting of individual samples is an unexplored direction.
• Ablation completeness and broader baselines: The provided content cuts off in the middle of the ablation study, and the paper explicitly notes that VaMP could not be included because its code is not publicly available — a fair comparison with that method is therefore still outstanding.
• Extending beyond the tested base methods: ReBaPL was evaluated only on top of the multi-modal methods MaPLe and MMRL. Whether it delivers similar gains on unimodal prompt learning methods, or on foundation models outside vision-language classification, is not reported.

Target Audience

Researchers and advanced practitioners working on parameter-efficient adaptation of large foundation models, Bayesian deep learning, or sampling-based inference; engineers building few-shot image classification or domain-robust vision systems who already use prompt learning methods such as CoOp, MaPLe, or MMRL and want a drop-in robustness improvement; and readers interested in MCMC methods (SGHMC, Langevin dynamics, cyclical schedules) applied to representation learning.

Authors’ abstract

Prompt learning has emerged as an effective technique for fine-tuning large-scale foundation models for downstream tasks. However, conventional prompt learning methods are prone to overfitting and can struggle with out-of-distribution generalization. To address these limitations, Bayesian prompt learning has been proposed, which frames prompt optimization as a Bayesian inference problem to enhance robustness. This paper introduces Repulsive Bayesian Prompt Learning (ReBaPL), a novel method for Bayesian prompt learning, designed to efficiently explore the complex and often multimodal posterior landscape of prompts. Our method integrates a cyclical step-size schedule with a stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm, enabling alternating phases of exploration to discover new modes, and exploitation to refine existing modes. Furthermore, we introduce a repulsive force derived from a potential function over probability metrics (including Maximum Mean Discrepancy and Wasserstein distance) computed on the distributions of representations produced by different prompts. This representation-space repulsion diversifies exploration and prevents premature collapse to a single mode. Our approach allows for a more comprehensive characterization of the prompt posterior distribution, leading to improved generalization. In contrast to prior Bayesian prompt learning methods, our method provides a modular plug-and-play Bayesian extension of any existing prompt learning method based on maximum likelihood estimation. We demonstrate the efficacy of ReBaPL on several benchmark datasets, showing superior performance over state-of-the-art prompt learning methods.