Caption-Driven Explainability: Probing CNNs for Bias via CLIP

AI summary

Overview

Research area: Explainable artificial intelligence (XAI) for computer vision, combining multimodal (vision-language) models with bias detection in convolutional neural networks (CNNs).

Technical level: Advanced. The paper assumes familiarity with CNNs, activation maps, residual networks, saliency-map XAI methods, contrastive language-image pre-training (CLIP), and embedding similarity.

One-sentence scope: The paper introduces a caption-based XAI method that performs "network surgery" to graft a standalone CNN into CLIP, so that text captions can reveal which high-level concept (such as color versus shape) dominates the standalone model's predictions.

What This Paper Is About

Standard XAI for vision relies on saliency maps, which simply highlight which pixels excite a model the most. That signal can be misleading when spurious features and genuine salient features occupy overlapping pixel regions, and a saliency map cannot by itself state with certainty whether a model is biased. This paper's goal is to identify the dominant concept driving a model's decision — for example, whether a digit classifier is really looking at shape or is secretly relying on color — and to use that insight to detect a covariate shift before the model is deployed.

Key Contributions

1. A caption-based XAI method. The paper proposes integrating a standalone model to be explained into CLIP via a novel network surgery approach, producing a caption-based XAI model that identifies the dominant concept contributing most to the standalone model's prediction.

2. The network surgery procedure. Activation maps from the standalone model's convolutional layers are selected, rescaled, and swapped into the CLIP image encoder, so CLIP can explain what the standalone image encoder focuses on while preserving as much of CLIP's learned concept space as possible.

3. Activation Matching. Because there are far more activation maps in the standalone model (22,720) than in the CLIP image encoder (3,840), the paper introduces a selection process that standardizes activation maps and scores their pairwise similarity via correlation, forming a 22,720 × 3,840 activation matching score matrix whose largest entries determine which maps are swapped.

4. A worked bias-detection and de-biasing demonstration. The method is demonstrated on a deliberately biased handwritten-digit task (MNIST digits five and eight), identifying color as the dominant concept, and it confirms the removal of that bias after de-biasing the data and retraining.

Main Findings

• Color is the dominant concept in the biased standalone model. Aggregating the caption-based model's correct/incorrect shape and color classifications by concept reveals color as the dominant concept for the standalone model — a property that is not apparent from the model's learning curves (Fig. 1).

• Shape becomes the dominant concept after de-biasing. When the dataset is de-biased using a color-to-grayscale pre-processor and a new standalone model is trained, the caption-based XAI model identifies shape as the dominant concept, confirming the color bias was removed.

• Only a fraction of the standalone model's activation maps can be transferred. Because only the last convolutional layers of stages 2, 3, 4, and 5 of the CLIP image encoder are available for swapping (four of 51 convolutional layers), the procedure incorporates 3840/22720 = 16.9% of all activation maps from the standalone model. The paper states that in an ideal network surgery procedure the probability for the concept color should be 100% and 0% for shape, so the concept probabilities must instead be compared relative to one another.

• A residual artifact appears with the grayscale model. The caption-based XAI model can still predict a red or green digit even for the grayscale-trained model, because grayscale images are still three-channel images with identical values on all channels, and because CLIP itself is not perfect. Consequently the "correct color" and "incorrect color" counts aggregate into "any color," which in an ideal world should equal zero.

• Claimed advantage over saliency maps. The paper states the most promising result is the superiority of the caption-based XAI method over saliency maps in situations where spurious and salient features are present in overlapping pixel spaces.

• Not reported: The paper does not report classification accuracy numbers, cosine similarity magnitudes, the number of test images, or training hyperparameters.

Methodology in Plain English

The researchers start from a "standalone" model they want to interrogate: a ResNet-50 pre-trained on ImageNet and fine-tuned for a binary MNIST task distinguishing handwritten fives from eights. That model has 49 convolutional layers, one fully connected layer, and two pooling layers.

The data used to develop this model is deliberately biased: every five is colored red and every eight is colored green. In the real-world setting the color assignments are random, which creates the covariate shift the paper wants to expose.

To explain the model, the authors borrow CLIP, which contains both a text encoder (a masked self-attention transformer) and an image encoder (OpenAI's modified, pre-trained ResNet-52, with 51 convolutional layers, one fully connected layer, and two pooling layers). CLIP's image encoder is altered through "network surgery": activation maps from the standalone model are substituted into selected layers of the CLIP image encoder. To preserve CLIP's concept space, the first stage is left untouched and only the last convolutional layer of each of stages 2 through 5 is eligible for swapping.

Because there are many more candidate activation maps in the standalone model than slots in CLIP, the authors first run the training images through both models and record every activation map. They compute each map's mean and standard deviation, standardize the maps, resize the smaller map to match the larger one with bilinear interpolation, and then compute correlation coefficients between every standalone map and every CLIP map. These coefficients populate a 22,720 × 3,840 score matrix; the largest entries indicate the best matches.

The chosen standalone maps are then rescaled back into the CLIP encoder's scale and bilinearly upscaled to the original size of the CLIP activation map they replace, so they integrate cleanly between neighboring layers.

Finally, four captions (describing shape and color variants) are used at inference. The change in cosine similarity between image and text embeddings before and after the swap is measured, so any initial CLIP bias is excluded and only the surgical influence remains. Aggregating the correct and incorrect shape and color classifications by concept reveals which concept dominates.

Why This Matters

Impact on research: The paper reframes XAI from a pixel-highlighting exercise into a concept-level diagnostic. It argues that a deeper understanding of dominant concepts in CNNs is fundamental to improving robustness, and that this method should be a prerequisite before deploying machine vision CNN models rather than merely a debugging tool.

Real-world applications (as framed by the paper):

• Medical imaging. The paper specifically highlights high-stakes environments such as medicine, where a patient could suffer from incorrect predictions, and notes the method could confirm or debunk a doctor's preconceived notions about what a model is learning.
• Models vulnerable to covariate shift. Any deployment where the data seen in the lab does not perfectly reflect real-world data — the paper's central risk scenario.
• Diagnosing spurious-feature reliance. Cases where spurious and salient features overlap in pixel space, which saliency maps handle poorly.
• Pre-deployment auditing of classifiers. Using the dominant-concept readout as a gate before releasing a model.

Industry relevance: The method builds on widely used, off-the-shelf components (ResNet-style CNNs and CLIP), makes its code publicly available at https://github.com/patch0816/caption-driven-xai, and targets the practical engineering question of whether a model will keep working once it leaves the lab.

Future Directions

• Improve the coverage of network surgery. Only 3840/22720 = 16.9% of the standalone model's activation maps can currently be transferred, and the paper acknowledges the ideal is for the color concept probability to reach 100%. Reducing this gap is an obvious next step.
• Address the residual color prediction in grayscale models. The caption-based XAI model can still output a color for grayscale inputs because three-channel grayscale images preserve identical channel values and CLIP is imperfect; the paper leaves this artifact unresolved.
• Extend beyond the two-digit binary demo. The demonstration is limited to MNIST digits five and eight with a red/green color manipulation; whether the method scales to broader, multi-class, or non-synthetic tasks is untested here.
• Validate the medical use case. The paper speculates that the method could confirm or debunk doctors' preconceived notions about model behavior, but no clinical evaluation is reported.

Target Audience

Researchers and practitioners in explainable AI, computer vision, and trustworthy machine learning, particularly those working on model bias detection, multimodal vision-language models, and robustness under covariate shift. It is also relevant to ML engineers responsible for auditing classifiers before deployment, and to readers interested in methods that go beyond saliency maps to concept-level explanations. The level of detail — activation map statistics, correlation scoring, layer selection — makes it best suited to readers already comfortable with CNN internals and CLIP-style contrastive models.

Authors’ abstract

Robustness has become one of the most critical problems in machine learning (ML). The science of interpreting ML models to understand their behavior and improve their robustness is referred to as explainable artificial intelligence (XAI). One of the state-of-the-art XAI methods for computer vision problems is to generate saliency maps. A saliency map highlights the pixel space of an image that excites the ML model the most. However, this property could be misleading if spurious and salient features are present in overlapping pixel spaces. In this paper, we propose a caption-based XAI method, which integrates a standalone model to be explained into the contrastive language-image pre-training (CLIP) model using a novel network surgery approach. The resulting caption-based XAI model identifies the dominant concept that contributes the most to the models prediction. This explanation minimizes the risk of the standalone model falling for a covariate shift and contributes significantly towards developing robust ML models. Our code is available at https://github.com/patch0816/caption-driven-xai