A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models

AI summary

Overview

• Research area: Test-time prompt tuning (TPT) and calibration of large vision-language models (VLMs), specifically CLIP-style models.
• Technical level: Intermediate. The paper assumes familiarity with contrastive vision-language pretraining, prompt tuning, expected calibration error (ECE), and cosine-similarity geometry on the unit hypersphere.
• Scope: The paper proposes A-TPT, a test-time prompt tuning regularizer that maximizes the minimum pairwise angular distance between normalized class-wise textual features, and evaluates it for calibration across fine-grained, distribution-shift, and medical datasets with CLIP ViT-B/16 and CLIP RN50 backbones.

What This Paper Is About

Test-time prompt tuning adapts VLMs to new tasks using only unlabeled test images, but it tends to produce overconfident, poorly calibrated predictions. Prior calibration-focused TPT methods either spread textual features away from their centroid (C-TPT's Average Textual Feature Dispersion) or push them toward orthogonality (O-TPT), and the authors argue neither guarantees uniform angular separation between class-wise features. A-TPT instead directly maximizes the minimum pairwise angular distance between normalized textual features on the unit hypersphere, aiming to lower calibration error while preserving accuracy.

Key Contributions

1. A numerical optimization method (A-TPT) for calibrating test-time prompt tuning of VLMs. The authors position it as resolving what they describe as suboptimal behavior in the leading calibration techniques for TPT, namely C-TPT and O-TPT.

2. Angular diversity as the objective, implemented by maximizing the minimum pairwise angular distance between normalized textual features. The paper claims this improves calibration in both regimes the authors analyze: when the number of classes exceeds the embedding dimension (N > |D|) and when it is smaller (N < |D|).

3. Extensive empirical validation across datasets and backbones, including medical datasets and medical baselines (FPT, PLIP with Prompt Smooth, MedCLIP with BAPLe), with the reported results showing A-TPT surpassing state-of-the-art calibration methods. The paper also states A-TPT achieves better calibration than the zero-shot CLIP model itself.

4. Supporting analyses consisting of theoretical aspects (gradient derivation, computational complexity), t-SNE visualizations of textual features, reliability diagrams, and a study of 80 hard prompt styles relating angular distance to ECE.

Main Findings

• Angular distance correlates with calibration quality within the same accuracy group. In a study of 80 hard prompt styles (following C-TPT's setup on Caltech101 with CLIP RN50), well-calibrated prompts showed higher angular distance: "a [class]" gave Acc 83.2, ECE 5.66, AD 0.643; "a toy [class]" Acc 82.8, ECE 6.65, AD 0.600; "this is a photo of [class]" Acc 82.4, ECE 6.15, AD 0.622. Poorly calibrated prompts showed lower angular distance: "there are [class] objects" Acc 81.1, ECE 9.25, AD 0.543 and "the nearest shape in this image is [class]" Acc 80.1, ECE 10.84, AD 0.461.

• Applying A-TPT to tuned prompts increased both angular distance and accuracy while cutting ECE. On the same Caltech101/RN50 setup: "a toy [class]" with TPT gave Acc 83.9, ECE 6.18, AD 0.6216; "this is a photo of [class]" with TPT + A-TPT gave Acc 86.0, ECE 3.74, AD 0.6333; "a [class]" with TPT + A-TPT gave Acc 86.6, ECE 2.23, AD 0.6486; "there are [class] objects" with TPT + A-TPT gave Acc 84.1, ECE 2.12, AD 0.6585.

• Group-wise analysis on CLIP ViT-B/16 shows gains concentrated in the harder N > |D| group. Baseline ECE was 3.36 (Group 1, N > |D|), 5.44 (Group 2, N < |D|), 4.40 overall. TPT raised ECE to 12.60 / 9.89 / 11.25. C-TPT gave 5.58 / 5.25 / 5.42. O-TPT gave 4.27 / 4.44 / 4.36. A-TPT gave 2.92 / 3.60 / 3.26. Corresponding accuracies for A-TPT were 58.23 / 64.30 / 61.27.

• Fine-grained classification results (Table 2). For CLIP ViT-B/16, average ECE across the eleven reported fine-grained datasets was 4.43 (Baseline), 11.60 (TPT), 5.13 (C-TPT), 4.23 (O-TPT), and 2.61 (A-TPT), with A-TPT average accuracy 64.92. For CLIP RN50, average ECE was 5.61 (Baseline), 11.70 (TPT), 6.19 (C-TPT), 5.45 (O-TPT), and 2.92 (A-TPT), with A-TPT average accuracy 57.53. The paper's prose states the improvement differently, describing the ViT-B/16 average as dropping to 2.92 and the RN50 average as 2.79; these do not match the values in Table 2.

• Natural distribution shift results (Table 3). Across ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-S with CLIP ViT-B/16, average ECE was 5.04 (Baseline), 12.00 (TPT), 5.82 (C-TPT), 4.88 (O-TPT), and 3.92 (A-TPT), with A-TPT average accuracy 58.06. With CLIP RN50, average ECE was 7.46 (Baseline), 16.70 (TPT), 12.10 (C-TPT), 9.69 (O-TPT), and 7.82 (A-TPT). The regularization weight λ was set to 10.0 for these experiments.

• A-TPT also beats the zero-shot baseline in calibration. The paper reports lower ECE than the zero-shot CLIP baseline on both backbones under distribution shift, which it describes as a feat unmatched by other approaches.

• Medical dataset generalization. On ISIC'18 (N = 7) with FPT (512-d): Basline FPT Acc 98.43 / ECE 0.2328; FPT + O-TPT Acc 98.25 / ECE 0.1381; FPT + A-TPT Acc 98.31 / ECE 0.0794. On KatherColon (N = 9) with PLIP and Prompt Smooth (768-d): PS Acc 76.6 / ECE 15.54; PS + O-TPT Acc 76.2 / ECE 12.73; PS + A-TPT Acc 76.4 / ECE 8.86. With MedCLIP and BAPLe (768-d): on Covid (N = 2), Acc 99.90 / ECE 3.21 baseline, 99.62 / 0.91 with O-TPT, 99.78 / 0.42 with A-TPT; on BA (N = 10), Acc 82.5 / ECE 15.64 baseline, 81.36 / 5.97 with O-TPT, 82.19 / 2.34 with A-TPT.

• Combination with CoOp and CoCoOp (Table 7). With CLIP ViT-B/16 and CoOp, average ECE across ten datasets was 5.25 (Baseline + CoOp), 18.36 (TPT + CoOp), 10.04 (TPT + CoOp + C-TPT), 7.91 (TPT + CoOp + O-TPT), and 3.63 (TPT + CoOp + A-TPT). With CoCoOp, average ECE was 4.20 (Baseline + CoCoOp), 4.35 (TPT + CoCoOp), 3.68 (TPT + CoCoOp + C-TPT), and 3.22 (TPT + CoCoOp + A-TPT).

• Reliability diagrams show A-TPT fixes both over- and under-confidence. C-TPT displayed under-confidence on Food101 and over-confidence on DTD and Flowers102; O-TPT partially mitigated these but noticeable gaps persisted, particularly on DTD; A-TPT produced the most reliable predictions in these plots.

• Gradient behavior differs qualitatively from O-TPT. O-TPT's orthogonality loss produces gradients that shrink toward zero as the pairwise angular distance θ approaches 0, whereas A-TPT's gradient norm is angle-independent and stays stable at small θ.

• Computational complexity is comparable to O-TPT. The paper states A-TPT has the same asymptotic complexity as O-TPT and negligible runtime/memory overhead over C-TPT.

• Cosine similarity consistency (Figure 4). O-TPT showed higher but fluctuating cosine similarities when N < |D| and grouped features closer when N > |D|; A-TPT produced more consistent values in both regimes.

Methodology in Plain English

Standard zero-shot CLIP classification compares an image feature against a text feature for each class using cosine similarity, converts the similarities to probabilities with a softmax at a fixed temperature τ of 0.01, and predicts the highest-probability class. Test-time prompt tuning replaces hand-crafted prompts with learnable vectors and optimizes them at inference on unlabeled test images, using an entropy-minimization loss (L_TPT) that pushes the model toward confident predictions.

The A-TPT idea is to add a second term that controls the geometry of the class-wise text features. The text feature matrix E (size N × |D|) is row-normalized to Ê, and pairwise cosine similarities are computed as Cos = Ê Êᵀ. Angular distances are θ = arccos(Cos). The angular diversity score is

AD = (1/N) Σᵢ min_{j, j≠i} θᵢⱼ

so what is maximized is the smallest angular gap each feature has to any other feature. The final objective is p* = argmin_p (L_TPT + λ · L_A-TPT), where L_A-TPT = −AD and λ controls the regularization strength.

The conceptual grounding comes from the Tammes problem (best-packing: arranging points on a sphere so the minimum pairwise distance is maximized) and from prior work showing that uniformity of features on the unit hypersphere preserves information. The angular loss form is inspired by ArcFace. Directly maximizing angular distance, rather than minimizing cosine similarity, is argued to avoid the vanishing-gradient regime when features are nearly aligned.

Why This Matters

Calibration is what makes a model's confidence score usable as a proxy for correctness. The paper argues that TPT's accuracy gains come with overconfidence, which undermines the reliability and trustworthiness of VLMs in settings where uncertainty estimates drive decisions. A method that lowers calibration error without sacrificing accuracy, and that also beats the zero-shot baseline, addresses a practical gap in deploying VLMs in safety-critical settings.

Real-world applications the paper names or implies:

• Medical diagnostics — explicitly cited, and evaluated on ISIC 2018, KatherColon, Covid, and BA datasets using medical baselines (FPT, PLIP with Prompt Smooth, MedCLIP with BAPLe).
• Autonomous driving — cited as a domain requiring reliable uncertainty estimates.
• Other safety-critical deployment of VLMs — the paper frames calibration as a question of VLM reliability, trustworthiness, and safety more broadly.
• Zero-shot and out-of-distribution deployment — where labeled validation data for post-hoc calibration is unavailable, which the paper notes makes temperature scaling, Platt scaling, and conformal prediction impractical.

Industry relevance: the method is designed for test-time adaptation where no labels exist, works on top of existing pipelines (TPT, CoOp, CoCoOp, and medical prompt-tuning baselines), and the paper reports no significant overhead beyond O-TPT in runtime or memory, which matters for inference-time deployment.

Future Directions

• Clarifying hyperparameter selection. The paper sets λ to 10.0 for the natural distribution shift experiments and refers to Appendix A.8/A.10 for implementation details, but the summary content does not report a systematic study of how λ should be chosen across domains; a principled selection rule is an open question.

• Establishing formal guarantees. The method is motivated by the Tammes problem, but the paper reports theoretical aspects limited to gradient analysis and complexity rather than bounds on the achievable minimum angular distance for arbitrary N and |D|.

• Broadening the backbone and modality range. Experiments cover CLIP ViT-B/16 (512-d) and CLIP RN50 (1024-d) plus medical encoders (512-d, 768-d); scaling behavior with much larger embedding dimensions and other VLM families is not reported here.

• Extending beyond classification. The evaluation is confined to image classification tasks and datasets; whether angular diversity helps calibration for detection, segmentation, or other open-ended VLM tasks is not addressed.

Target Audience

Researchers and graduate students working on vision-language model adaptation, test-time optimization, and uncertainty calibration; practitioners who deploy CLIP-style models in label-scarce or safety-critical settings such as medical imaging and autonomous systems; and anyone following the line of work on TPT calibration that includes C-TPT and O-TPT. Readers need working familiarity with cosine similarity, softmax temperature, and expected calibration error to get the most from the method section and the tables.

Authors’ abstract

Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code will be made publicly available.