BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically test-time adaptation and debiasing of vision-language models (VLMs) such as CLIP.

Technical level: Intermediate. The paper builds on prior test-time prompt tuning work (TPT, SEraser) and assumes familiarity with contrastive vision-language embeddings, Grad-CAM, KL divergence, and worst-group accuracy, though its core idea is describable in plain terms.

Scope: The paper proposes BiPrompt, a bilateral (visual plus textual) test-time prompt-optimization framework that suppresses spurious feature reliance in VLMs without retraining or domain supervision, and evaluates it on real-world out-of-distribution and simulated spurious-correlation benchmarks.

What This Paper Is About

VLMs like CLIP generalize well zero-shot but latch onto spurious shortcuts, such as background context, instead of causal object features, so they fail unpredictably under distribution shift. Existing test-time debiasing methods attack only one modality at a time (mostly visual, via random erasure) and leave the static text prompts, with their built-in class-name priors, untouched. BiPrompt addresses both sides at once by pairing attention-guided, structured visual erasure with a learnable normalization of text embeddings during test-time adaptation.

Key Contributions

1. Balanced Prompt Normalization: A learnable re-centering mechanism that interpolates each class text embedding toward the global semantic centroid (the mean of all class embeddings), using a learnable gating parameter α to encourage isotropic, less linguistically biased text representations.

2. Structured Spurious-Region Erasure: A replacement for random masking that uses Grad-CAM attention maps to split an image into foreground (causal) and background (spurious) views, then enforces prediction consistency between foreground and full image while promoting orthogonality between background and full image.

3. A bilateral test-time objective: A combined loss (cross-entropy plus the structured erasure term plus an entropy regularizer) that implicitly drives the conditional mutual information between spurious features and predictions toward zero, updated with only a few lightweight parameters (the gating parameter and normalization weights).

4. Broad empirical validation: Evaluations across three real-world OOD datasets, three simulated spurious-bias datasets, and two additional VLM architectures, reporting gains in both average and worst-group accuracy.

Main Findings

• Real-world OOD (Table 1): BiPrompt reaches an average Top-1 accuracy of 42.4% across Tiny-ImageNet, CUB-200, and ImageNet-A, a reported ▲16.6 over the Vanilla CLIP baseline (25.8) and 1.9% above the strongest SEraser variant, Blocks (40.5). Per-dataset results are Tiny-ImageNet 44.1 (vs Vanilla 23.2), CUB-200 31.0 (vs 12.1), and ImageNet-A 52.2 (vs 42.1).

• Gains over the best SEraser variant: BiPrompt improves by 2.1% on CUB-200 and 2.5% on ImageNet-A over the next-best SEraser variant.

• α-CLIP caveat: α-CLIP reaches the highest average in Table 1 at 57.3, driven by 76.0 on Tiny-ImageNet; the paper attributes this to its checkpoint having been retrained on ImageNet.

• Simulated spurious bias (Table 2): BiPrompt achieves the highest average and worst-group accuracy across Waterbirds, CamelDeer, and SpiderCrab, with a three-dataset average of 91.3 AVG (↑19.0 over Vanilla's 72.3) and 85.0 W.G. (↑35.5 over Vanilla's 49.5).

• Worst-group accuracy is where the method stands out: Waterbirds W.G. 66.6 versus Vanilla 40.0 (reported as ↑26.5) and SEraser 65.3; CamelDeer W.G. 92.8 versus Vanilla 66.4 (↑26.4); SpiderCrab W.G. 95.4 versus Vanilla 42.0 (↑53.4).

• Synthetic benchmarks favor BiPrompt strongly: SpiderCrab shows the largest average gain in Table 2, 97.4 AVG versus Vanilla 66.0 (↑31.4).

• Weaker baselines on these benchmarks: TPT scores 66.9 AVG on Waterbirds and 34.4 W.G., and 77.7 AVG / 55.3 W.G. on CamelDeer, illustrating the paper's argument that high-confidence spurious cues break entropy-based filtering.

• Architecture generality (Table 3): On Waterbirds with CLIP ViT-L-14, BiPrompt gets 88.4 AVG (▲4.7 over Vanilla 83.7) and 60.1 W.G. (▲27.2 over 32.9). On BLIP-2, BiPrompt gets 56.3 AVG (table reports ▲-1.4 against Vanilla 57.7) and 35.5 W.G. (▲7.3 against 28.2). Across the two models the average change is 72.4 AVG (▲1.7) and 47.8 W.G. (▲17.3).

• Framing of the mechanism: The paper states that the two modules jointly minimize the conditional mutual information between spurious cues and predictions, approximating I(z_s; y | z_c) ≈ 0, steering the model toward causal, domain-invariant reasoning without retraining or domain supervision.

Methodology in Plain English

The method keeps a pretrained VLM frozen and adapts only a small set of parameters at test time, per input image.

On the visual side, instead of erasing random image regions (the SEraser approach), BiPrompt computes a Grad-CAM attention map for the input and uses it as a soft mask. Multiplying the image by the mask gives a foreground view; multiplying by one-minus-the-mask gives a background view. The optimization then pushes the prediction on the foreground view to match the prediction on the full image, while pushing the background view's prediction to be orthogonal (via cosine similarity) to the full image's prediction. The intuition is to keep the causal content and actively decouple the spurious content.

On the textual side, standard class-name embeddings are anisotropic: dominant or frequent classes occupy favored directions in the embedding space. BiPrompt re-centers each class embedding by interpolating it with the global centroid of all class embeddings, using a learnable gate α, producing a more isotropic text space.

The full test-time loss is the standard cross-entropy term plus the structured erasure term plus the entropy regularizer carried over from SEraser, combined with weighting coefficients λ1 and λ2. Only the gating parameter and normalization weights are updated, with one or a few gradient steps per test sample. Inference then uses the similarity between the visual feature and the normalized text embeddings. The paper does not report the specific numeric values of λ1, λ2, β, the learning rate, dataset sizes, or the exact number of update steps.

Why This Matters

Impact on research. The paper argues that single-modality debiasing yields only partial robustness, and that visual and textual bias must be corrected together. It extends the test-time adaptation line of work (TPT, SEraser) by replacing random erasure with attention-guided structured erasure and by introducing textual isotropy as a debiasing lever, positioning its contribution as a lightweight path toward causally grounded foundation models. The emphasis on worst-group accuracy rather than only average accuracy connects it to the group-robustness literature.

Real-world applications include:

• Medical imaging: the paper explicitly motivates the problem with "hospital logos" being spuriously associated with particular pathologies, where worst-group reliability matters clinically.
• Autonomous navigation and robotics: scene understanding where a water background should not force a "boat" prediction for a land object.
• Wildlife and biodiversity monitoring: the paper's SpiderCrab and CamelDeer examples illustrate camera-trap or field imagery where habitat background correlates with, but does not determine, the species.
• Content moderation and media analysis: large-scale image tagging pipelines where background context could systematically mislabel certain scene-class combinations.

Industry relevance. BiPrompt requires no retraining, no architectural change, and no domain labels, and updates only a few parameters per test sample, which the paper describes as efficient and memory-friendly. That makes it attractive for deployed vision-language systems that must adapt to shifting distributions on the fly, and its demonstrated gains across CLIP ViT-B/32, CLIP ViT-L-14, and BLIP-2 suggest it is not tied to a single backbone.

Future Directions

• Reducing dependence on external components: the structured erasure relies on Grad-CAM, and the simulated-bias experiments use SAM to isolate and erase background regions. Making the framework fully self-contained, or robust to poor attention maps, is a natural next step.
• Extending beyond two modalities and classification: the current formulation targets image-text classification under background-object correlations; generalizing to tasks such as detection, segmentation, or captioning is unexplored.
• Consistency of the BLIP-2 result: the BLIP-2 average of 56.3 is reported as ▲-1.4 relative to its 57.7 Vanilla baseline, so whether the method reliably improves average accuracy on non-CLIP architectures remains an open question.
• Understanding the text-side mechanism: the paper motivates isotropy of text embeddings qualitatively; how the learned α behaves across datasets with different numbers of classes, and whether it transfers across domains, is not reported.

Target Audience

Researchers and practitioners working on test-time adaptation, distribution shift, spurious-correlation robustness, and trustworthy multimodal foundation models. It is most useful to readers already familiar with prompt tuning and contrastive vision-language models who want a lightweight, retraining-free debiasing method, and to applied engineers evaluating whether test-time prompt adaptation can improve worst-group reliability in deployed vision-language systems.

Authors’ abstract

Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address a single modality either visual or textual leading to partial robustness and unstable adaptation under distribution shifts. We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation. On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce orthogonal prediction consistency between causal and spurious regions. On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space. Together, these modules jointly minimize conditional mutual information between spurious cues and predictions, steering the model toward causal, domain invariant reasoning without retraining or domain supervision. Extensive evaluations on real-world and synthetic bias benchmarks demonstrate consistent improvements in both average and worst-group accuracies over prior test-time debiasing methods, establishing a lightweight yet effective path toward trustworthy and causally grounded vision-language adaptation.