Data Descriptions from Large Language Models with Influence Estimation

AI summary

Overview

• Research area: Explainable AI and data-centric interpretability — specifically, using large language models (LLMs) to generate textual descriptions of image classes, selecting the most useful descriptions via influence estimation, and evaluating them through a new cross-modal transfer benchmark.
• Technical level: Intermediate. The paper assumes familiarity with CLIP-style vision-language embeddings, influence functions (TracIn), zero-shot classification, and prompting, but its pipeline is conceptually straightforward.
• Scope: A single paper (arXiv:2511.07897v1 [cs.AI], published 11 Nov 2025) by Chaeri Kim, Jaeyeon Bae, and Taehwan Kim of UNIST, proposing "proponent texts" selected by Influence scores For Texts (IFT) and a benchmark task called cross-modal transfer classification, validated on nine image classification datasets.

What This Paper Is About

Most explainable AI work asks how a model makes a prediction; this paper instead asks how the data — specifically image classes — can be explained in human-readable language. The authors use LLMs to generate textual descriptions of each image class (grounded with Wikipedia URLs), then introduce a scoring metric called Influence scores For Texts (IFT) to identify which of those descriptions actually help a model learn. They then propose and evaluate a new benchmark, cross-modal transfer classification, in which a model is first trained on images and then further trained on the selected text descriptions.

Key Contributions

1. A pipeline for generating class descriptions with external knowledge. A two-stage prompting process queries GPT-3.5 first for the appearance components of a class, then asks it to summarize each component using a corresponding Wikipedia URL, producing one-line noun-based descriptions that are more class-specific and less hallucinated than unconstrained generation.
2. Influence scores For Texts (IFT). IFT is defined as the sum of an influence score (computed with TracIn over image training samples) and a CLIP score (cosine similarity between CLIP image and text embeddings). Descriptions with the highest IFT are named "proponent texts" — ten per class are selected — and are used as weighted supervision.
3. A new benchmark task: cross-modal transfer classification. Based on cross-modal transferability, the model is trained first on CLIP image embeddings, then retrained on weighted proponent text embeddings in the same shared embedding space, with only the classifier updated while CLIP encoders stay frozen.
4. Extensive evaluation. Comparisons against CLIP zero-shot, Menon and Vondrick, LaBo, CuPL, and VDT-Adapter across nine datasets in both zero-shot and cross-modal transfer settings, plus ablation studies on IFT, Wikipedia grounding, a second CLIP backbone, and a GPT-4o qualitative evaluation of Helpful, Informative, and Relevant descriptions.

Main Findings

• Zero-shot superiority on most datasets. In Table 1, the authors' descriptions beat the baselines on seven of nine datasets — for example Miniimagenet 85.320% (vs. VDT-Adapter 85.199%), 102flowers 69.109% (vs. VDT-Adapter 68.742%), DTD 48.989% (vs. VDT-Adapter 48.457%), CIFAR-100 64.570%, EuroSat 39.148%, Food101 83.452%, and CIFAR-10 89.470%. The two exceptions are CUB 200 2011 (53.227%, below CuPL's 53.349%) and OxfordPets (88.679%, below CuPL's 88.814%).
• Largest gains in cross-modal transfer classification. In Table 2, the authors' method is the best on all nine datasets when text is added to image training — for example OxfordPets 93.396% (vs. 91.664% with images only), EuroSat 96.037% (vs. 94.296%), 102flowers 97.948% (vs. 96.459%), CUB 200 2011 75.130% (vs. 71.332%), and DTD 74.393% (vs. 72.074%).
• IFT outperforms its individual components. Table 3 shows that selecting proponent texts with the full IFT is better than using the influence score alone or the CLIP score alone on every dataset (e.g., EuroSat 96.037% for IFT vs. 94.222% for influence score and 94.407% for CLIP score; CUB 200 2011 74.525% vs. 72.834% and 72.696%). The paper presents lower-case examples of CLIP-only selections being error messages or irrelevant details (wheel material, bus decoration) and influence-score-only selections including interior bus structure.
• Wikipedia grounding helps. Table 4 shows providing Wikipedia URLs improves cross-modal transfer results on all nine datasets over the "No Wiki" setting (e.g., CUB 200 2011 74.525% vs. 72.092%; EuroSat 96.037% vs. 94.333%), and the qualitative examples show more class-specific descriptions and fewer hallucinations (for example, a long beak described correctly only with Wikipedia).
• Robustness across CLIP backbones. Table 7, using laion/CLIP-ViT-L-14-laion2B-s32B-b82K, reports the authors' method as best on all eight datasets listed there (CUB 200 2011 66.396%, OxfordPets 92.562%, CIFAR10 93.819%, CIFAR100 75.730%, EuroSAT 43.740%, Miniimagenet 86.435%, 102flowers 72.161%, DTD 59.989%). Food101 is not reported in that table.
• Better embedding alignment. t-SNE visualizations (Figure 3) of 102flowers, CUB 200 2011, and DTD classes show the authors' text embeddings aligning more closely with same-class image embeddings than baseline descriptions.
• GPT-4o evaluation favors the method. Across 100 randomly selected classes and 300 evaluation instances, the authors' descriptions have the highest Top-1 Rating for Helpful (31.25%), Informative (33.55%), and Relevant (28.29%, tied with VDT-Adapter), and the lowest (best) average rank for Helpful (1.6), Informative (1.57), and Relevant (1.69).
• Low training cost. Training with images only across all datasets takes about 2 hours on an NVIDIA 3090 GPU, while cross-modal transfer training with proponent texts takes less than 30 minutes.
• Note on reporting. For a few datasets the ablation table's IFT row differs slightly from the main cross-modal transfer table (for example 102flowers 97.192% in Table 3 vs. 97.948% in Table 2, and CIFAR-100 78.250% vs. 78.650%); the paper does not explain the difference.

Methodology in Plain English

1. Generate descriptions. For each dataset, the raw class name is inserted into a fixed question asking GPT-3.5 for the appearance components of the class. For datasets with a superclass and subcategories (such as CUB 200 2011, where "bird" is the superclass), the components of the superclass are requested instead. Each component is then summarized in one line of nouns using a corresponding Wikipedia URL, with an instruction to fall back on the model's own knowledge if the URL lacks relevant information.
2. Score images. A pre-trained ResNet34 is trained with SGD (mini-batch size 64, learning rate 0.1 divided by 10 every 30 epochs, 200 epochs total), with checkpoints saved every 10 epochs. TracIn is used over these checkpoints to compute influence scores showing which training images help predict each validation image.
3. Score texts. For each training image, the CLIP score — cosine similarity between CLIP image embeddings and CLIP text embeddings of a class description — is computed and added to the influence score. Averaging this sum over training and validation images gives the IFT for each description. The ten highest-IFT descriptions per class become the proponent texts.
4. Train cross-modally. A linear classifier is first trained on CLIP image embeddings (SGD, mini-batch size 64, learning rate 0.1, 30 epochs, CosineAnnealingLR with 200 maximum iterations) using ViT/32 CLIP as the encoder. It is then retrained on proponent text embeddings, with each class's loss weighted by its normalized IFT, while the CLIP encoders remain frozen. Because images and texts live in the same embedding space, no architectural change is needed.
5. Evaluate. The authors compare against CLIP zero-shot and four LLM-based description baselines (Menon and Vondrick, LaBo, CuPL, VDT-Adapter) in zero-shot classification and in cross-modal transfer, run ablations on the scoring metric and Wikipedia grounding, and have GPT-4o rate 100 randomly selected classes on helpfulness, informativeness, and relevance using Top-1 Rating and Ranking Average metrics.

Why This Matters

• Research impact. The paper shifts explainability from "why did the model predict this" toward "how can the data itself be described," and offers IFT as a reusable, cheap selection criterion over LLM-generated text. It also introduces cross-modal transfer classification as a concrete benchmark for testing whether language descriptions are actually informative.
• Real-world applications.
  • Building lightweight, low-cost classifiers in domains where labeled images are scarce but textual domain knowledge exists, since only a linear layer is trained and text training takes under 30 minutes.
  • Generating human-readable class documentation for datasets, aiding annotation guidelines and dataset auditing.
  • Improving zero-shot image tagging or retrieval systems that rely on prompt engineering, by replacing hand-crafted prompts with influence-selected descriptions.
  • Producing per-class textual explanations that indicate which visual factors drive a black-box model's decisions.
• Industry relevance. The pipeline uses off-the-shelf GPT-3.5 and frozen CLIP encoders and runs on a single NVIDIA 3090 GPU, making it practical for teams that want to squeeze extra accuracy out of existing vision models without retraining backbones or collecting more images. The GPT-4o-based evaluation protocol also offers a template for automated quality assessment of generated text.

Future Directions

• Prompt and LLM sensitivity. The extracted descriptions vary with the pre-defined prompts and the LLM used, which affects performance; the authors plan to experiment with various prompts and large language models.
• Scaling influence computation. Computational load grows with dataset size because of influence score calculation; the authors plan to explore more efficient ways to compute influence scores at scale.
• Bias in generated descriptions. Because the method relies on GPT-3.5/4, it may inherit biases from those models' pretraining data; Wikipedia grounding improves factual consistency and mitigates hallucinations but may not resolve deeper representational or social biases.
• Extended evaluation and datasets. The paper provides dataset partitions only in Appendix E (not included in the provided content) and does not report Food101 in the second-CLIP-backbone experiment, leaving room for broader replication.

Target Audience

Researchers and practitioners in explainable AI, data-centric machine learning, and vision-language modeling who are interested in LLM-generated supervision and influence functions. It is also useful for applied engineers seeking cheap ways to improve image classifiers with text, and for readers evaluating how to use LLMs as automated judges of description quality. Some familiarity with CLIP embeddings and influence estimation is helpful but not required to follow the main argument.

Authors’ abstract

Deep learning models have been successful in many areas but understanding their behaviors still remains a black-box. Most prior explainable AI (XAI) approaches have focused on interpreting and explaining how models make predictions. In contrast, we would like to understand how data can be explained with deep learning model training and propose a novel approach to understand the data via one of the most common media - language - so that humans can easily understand. Our approach proposes a pipeline to generate textual descriptions that can explain the data with large language models by incorporating external knowledge bases. However, generated data descriptions may still include irrelevant information, so we introduce to exploit influence estimation to choose the most informative textual descriptions, along with the CLIP score. Furthermore, based on the phenomenon of cross-modal transferability, we propose a novel benchmark task named cross-modal transfer classification to examine the effectiveness of our textual descriptions. In the experiment of zero-shot setting, we show that our textual descriptions are more effective than other baseline descriptions, and furthermore, we successfully boost the performance of the model trained only on images across all nine image classification datasets. These results are further supported by evaluation using GPT-4o. Through our approach, we may gain insights into the inherent interpretability of the decision-making process of the model.