SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias

AI summary

Overview

• Research area: Robustness of large vision-language models (CLIP-style models) for zero-shot image classification, specifically the problem of multimodal spurious bias.
• Technical level: Intermediate. The paper combines a probabilistic theoretical analysis of contrastive pretraining with a simple, training-free inference procedure built on standard CLIP prompt templates and cosine similarity.
• Scope: The paper defines multimodal spurious bias, proves that it biases zero-shot predictions toward the wrong class, and proposes a training-free prompt-selection method (SAGE) that improves worst-group robustness on four benchmark datasets and five backbone models.

What This Paper Is About

CLIP-style models often learn spurious cross-modal associations during pretraining, such as pairing "landbird" with "land background," and then misclassify images when that association no longer holds at test time (for example, a waterbird photographed on land). The paper's goal is to mitigate this multimodal spurious bias in the zero-shot setting, without fine-tuning, labeled downstream data, external large language models, or prior knowledge of which attributes are spurious. SAGE addresses this by scoring a fixed pool of 80 prompt templates per image and using the templates that best separate the classes.

Key Contributions

1. A formal definition and theoretical analysis of multimodal spurious bias. The paper introduces a latent textual spurious feature and defines multimodal spurious bias as the condition where the model's alignment behaves as if this spurious feature were always present (Definition 1). Theorem 1 then shows that when a spurious bias links the image, the wrong class text, and the spurious feature, the model's prediction is biased toward the incorrect class label.
2. The SAGE method. A training-free framework that computes a separation score for each candidate prompt template on each test image, defined as the difference between the highest and lowest class cosine similarities, and then performs zero-shot inference with the top-ranked template(s).
3. Extensive empirical validation. Experiments on four real-world benchmarks (Waterbirds, CelebA, PACS, VLCS) and five backbone models (CLIP-RN-50, CLIP-ViT-B/32, CLIP-ViT-L/14, ALIGN, AltCLIP), compared against zero-shot CLIP (ZS), ROBOSHOT, and TIE*.
4. Ablation and correlation analyses. A Pearson correlation analysis linking separation scores to worst-group accuracy, a comparison against random selection and full prompt ensembling, and a study of how the number of selected prompts affects performance.

Main Findings

• Theory: spurious bias flips predictions. Theorem 1 proves that when the text representation for the incorrect class shares a multimodal spurious bias with the image representation and the spurious feature, the ratio of predictive probabilities favors the wrong class (greater than 1), so the model is biased toward predicting the incorrect label. When a prompt induces this bias, the predictive probabilities for the two classes become arbitrarily close (low class separation).
• Separation score is the proposed proxy. Prompts less affected by spurious bias produce a large margin between the correct-class and incorrect-class probabilities, which motivates maximizing class separation as a practical, bias-agnostic selection criterion.
• Averaged results over five backbones (Table 1). SAGE achieved the best average worst-group accuracy (WGA) and harmonic mean (HM) on all four datasets:
  • Waterbirds: AVG 88.9, WGA 44.9, HM 59.7 (ZS: 84.1 / 36.7 / 51.1; ROBOSHOT: 71.4 / 41.5 / 52.5; TIE*: 84.3 / 38.4 / 52.8).
  • CelebA: AVG 83.4, WGA 80.6, HM 82.0 (ZS: 81.1 / 75.3 / 78.1; ROBOSHOT: 84.4 / 79.5 / 81.9; TIE*: 76.7 / 69.8 / 73.1). ROBOSHOT had the highest average accuracy here (84.4 versus SAGE's 83.4).
  • PACS: AVG 96.6, WGA 81.9, HM 88.7 (ZS: 96.2 / 75.5 / 84.6; ROBOSHOT: 96.1 / 78.0 / 86.1; TIE*: 95.7 / 77.2 / 85.5).
  • VLCS: AVG 75.8, WGA 33.8, HM 46.7 (ZS: 76.1 / 23.0 / 35.3; ROBOSHOT: 76.3 / 30.1 / 43.2; TIE*: 80.5 / 31.7 / 45.5). TIE* had the highest average accuracy on VLCS.
• Baselines require extra information. ROBOSHOT relies on prompting a large language model to specify spurious attributes per task, and TIE* directly uses spurious attributes to obtain pseudo spurious labels. SAGE requires neither.
• Separation score correlates positively with worst-group accuracy. On CelebA, across all five backbones, the Pearson correlation coefficient between a template's separation score and its resulting WGA was consistently positive (Figure 3); the magnitude of the coefficient is not reported in the text.
• SAGE beats random selection and full ensembling (Table 2). On Waterbirds, averaged over backbones: SAGE 88.9 / 44.9 / 59.7 (AVG / WGA / HM) versus Ensemble (K=80) 89.7 / 36.2 / 51.6 and Random 87.8 / 40.1 / 55.1. On CelebA: SAGE 83.4 / 80.6 / 82.0 versus Ensemble 80.2 / 73.2 / 76.5 and Random 79.8 / 74.9 / 77.3.
• One prompt is usually enough. The default is K=1 (single highest-scoring template). Sweeping K over 1, 5, 20, 40, 80 showed top-1 was best on CelebA (especially for CLIP-RN-50 and ALIGN) and on Waterbirds for CLIP-ViT-B/32, CLIP-ViT-L/14, ALIGN, and AltCLIP; CLIP-RN-50 performed best on Waterbirds at K=5.
• Design details. The method uses a fixed set of 80 prompt templates containing a "[CLASS]" placeholder, and zero-shot inference averages similarity scores over the selected templates.
• Compute. All experiments were conducted on NVIDIA Quadro RTX 8000 GPUs (48GB).

Methodology in Plain English

The approach starts from the observation that CLIP aligns images and text, so a prompt template effectively sets how each class is described to the model. Some templates accidentally drag in the spurious feature (for example, backgrounds), and some do not.

SAGE works as follows. First, it takes a pool of 80 generic prompt templates (things like "a photo of a [CLASS]" or "a blurry photo of the [CLASS]") and fills each one with every class name, producing text embeddings for each template-class pair. Second, for a given test image, it computes the cosine similarity between the image embedding and every class embedding under every template. For each template it calculates a separation score: the highest class similarity minus the lowest class similarity. Third, it ranks templates by this score and keeps the top K (K=1 by default). Finally, it performs zero-shot classification using those templates, averaging similarity scores across them when K is greater than 1.

The intuition is that a template which makes one class clearly stand out relative to the others is capturing the core, class-defining semantics rather than a background or other incidental feature. Everything is decided per image at inference time, so no training, labels, fine-tuning, or external model is needed. The authors also prove a theoretical result linking spurious bias to predictions being pulled toward the wrong class, and use that to justify separation as the right quantity to maximize.

Why This Matters

Impact on research. The paper targets an under-explored setting: debiasing in zero-shot multimodal classification, where no retraining data is available. It offers a probabilistic definition of multimodal spurious bias plus a theorem connecting that bias to prediction errors, and demonstrates that selecting prompts by class separation is a simple alternative to fine-tuning, LLM-based attribute discovery, or explicit spurious-attribute labels. It also introduces the harmonic mean (HM) of average accuracy and worst-group accuracy as a primary metric, penalizing models that look good on average but fail on the worst group.

Real-world applications (derived from the method's properties):

• Deployment of off-the-shelf vision-language models where retraining is impractical, since SAGE requires no model updates and no external knowledge.
• Domain-shift scenarios such as images from a different camera, location, or season than the pretraining distribution, where spurious background-label correlations break down.
• Fine-grained classification tasks with subtle attribute differences and imbalanced groups, the setting the CelebA analysis focuses on.
• Content moderation, medical imaging triage, or autonomous perception pipelines where worst-group failures are costlier than average-case errors and where labeled group annotations are typically unavailable.

Industry relevance. Because SAGE adds only a prompt-scoring and selection step at inference and works with any zero-shot vision-language model, it fits as a drop-in wrapper around existing CLIP-style deployments. The gain in worst-group robustness at little or no cost in average accuracy is directly relevant to teams that cannot fine-tune large models or cannot annotate spurious attributes. The code is released at https://github.com/wenqian-ye/spurious˙vlm.

Future Directions

• Explain the selection behavior. The paper notes observed trends in which templates are most frequently selected but leaves systematic analysis of those trends open.
• Adaptive K selection. The best number of prompts varied by dataset and backbone (top-1 generally best, K=5 best for CLIP-RN-50 on Waterbirds), suggesting a per-image or per-model rule for choosing K.
• Combine with other debiasing families. Since SAGE operates purely on prompts, combining it with fine-tuning-based, prompt-tuning-based, or latent-projection-based debiasing methods is a natural extension.
• Extend beyond the evaluated settings. The method is stated to generalize to other sets of prompt templates and model families, but the evaluation covers four benchmarks and five backbones; broader validation, including on other modalities and tasks beyond classification, remains open.

Target Audience

Researchers and practitioners working on trustworthy machine learning, vision-language models, and zero-shot or out-of-distribution robustness. It is also useful for engineers deploying CLIP-style models who need a lightweight, training-free way to improve worst-group performance, and for readers interested in how theoretical analyses of spurious correlations translate into practical inference-time algorithms. A basic familiarity with CLIP-style contrastive embeddings and cosine similarity is helpful; the theory section assumes comfort with probability notation.

Authors’ abstract

Large vision-language models, such as CLIP, have shown strong zero-shot classification performance by aligning images and text in a shared embedding space. However, CLIP models often develop multimodal spurious biases, which is the undesirable tendency to rely on spurious features. For example, CLIP may infer object types in images based on frequently co-occurring backgrounds rather than the object's core features. This bias significantly impairs the robustness of pre-trained CLIP models on out-of-distribution data, where such cross-modal associations no longer hold. Existing methods for mitigating multimodal spurious bias typically require fine-tuning on downstream data or prior knowledge of the bias, which undermines the out-of-the-box usability of CLIP. In this paper, we first theoretically analyze the impact of multimodal spurious bias in zero-shot classification. Based on this insight, we propose Spuriousness-Aware Guided Exploration (SAGE), a simple and effective method that mitigates spurious bias through guided prompt selection. SAGE requires no training, fine-tuning, or external annotations. It explores a space of prompt templates and selects the prompts that induce the largest semantic separation between classes, thereby improving worst-group robustness. Extensive experiments on four real-world benchmark datasets and five popular backbone models demonstrate that SAGE consistently improves zero-shot performance and generalization, outperforming previous zero-shot approaches without any external knowledge or model updates.