SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuning

AI summary

Overview

Research area: Computer vision — test-time prompt tuning (TPT) of vision-language models (VLMs), specifically the calibration of uncertainty estimates.

Technical level: Intermediate. The paper combines a geometric argument with a first-order gradient analysis, but the motivation and practical takeaway are accessible without deep mathematical background.

Scope: This paper proposes Semantic Orthogonal Calibration (SoC), a Huber-based regularizer for test-time prompt tuning that relaxes the full orthogonality constraint used by prior work in order to preserve semantic proximity between related classes and improve calibration.

What This Paper Is About

Test-time prompt tuning adapts CLIP-style vision-language models to new tasks at inference time using only unlabeled test images, typically by minimizing prediction entropy. This makes predictions more accurate but also overconfident, which is dangerous in safety-critical settings. Prior work (O-TPT) tried to fix this by forcing the class text embeddings to be fully orthogonal to one another, but the authors show theoretically and empirically that this rigid constraint pushes semantically related classes (e.g., "annual crop land" and "permanent crop land") too far apart and inflates confidence. SoC replaces that constraint with a smoother one that caps the repulsion applied to highly similar class pairs.

Key Contributions

1. A Huber-based TPT regularizer (SoC). Instead of penalizing all pairwise cosine similarities quadratically, SoC applies a quadratic penalty only below a margin δ and a linear penalty above it. This caps the repulsion for highly similar class pairs, preserving semantic structure rather than over-separating related classes.

2. A theoretical confidence bound. The authors derive a lower bound on softmax confidence in terms of the worst-case pairwise class similarity μ (Proposition 1, the "cosine coherence"), linking prototype geometry directly to prediction uncertainty.

3. A first-order analysis of orthogonality versus smooth repulsion. Under a single gradient step of size η, full orthogonality reduces worst-case similarity to approximately (1 − 4η)μ, while the Huber regularizer reduces it to (1 − 2η)μ when μ ≤ δ, or to μ − 2ηδ when μ > δ. O-TPT's reduction is strictly sharper whenever μ > δ (Corollary 1), meaning it inflates the confidence floor more aggressively.

4. A comprehensive empirical validation. Experiments over 11 classification datasets plus four ImageNet distribution-shift variants, two ViT backbones, 18 CLIP prompt templates, CoOp-initialized prompts, and one- versus two-step optimization show SoC improving calibration while keeping competitive accuracy.

Main Findings

• Better calibration than the state of the art. On the ViT-L/14 backbone across 11 datasets, SoC reaches a mean ECE of 5.4, compared with 7.7 for O-TPT, 10.0 for C-TPT, 14.9 for TPT, and 5.1 for zero-shot CLIP. The mean ECE improvements range from 2.3 relative to O-TPT to 9.5 relative to the original TPT.

• Competitive, sometimes better, accuracy. SoC achieves the best mean accuracy in that benchmark at 72.3, versus 71.4 (O-TPT), 72.0 (TPT), 72.1 (C-TPT), and 71.1 (zero-shot).

• Best ECE on all but one dataset. Compared with O-TPT, SoC wins on 10 of the 11 datasets. The exception is ImageNet, where SoC's ECE is 7.2 versus O-TPT's 5.5 (a +1.7 regression). The largest gains are on EuroSAT, where accuracy rises from 53.6 to 58.3 (+4.7) and ECE drops from 17.7 to 3.2 (−14.5).

• Robustness to natural distribution shift. Across ImageNet-A, ImageNet-v2, ImageNet-R, and ImageNet-Sketch, SoC matches O-TPT on average accuracy (71.3 versus 71.3) while lowering average ECE from 9.9 to 8.4 (−1.5). This is a 5.8 improvement over TPT (14.2) and a 4.2 improvement over C-TPT (12.6).

• Confidence inflation is visibly reduced. Reliability diagrams on Flowers102, EuroSAT, and FGVC-Aircraft show O-TPT producing systematically overconfident curves, while SoC's curves are flatter and closer to the diagonal. The per-class-pair analysis on EuroSAT shows O-TPT is most overconfident precisely on class pairs with high zero-shot similarity — the case the authors predicted from theory.

• More stable under a second gradient step. When applying two gradient updates instead of the standard one, SoC's ECE degrades by nearly 23%, while O-TPT's calibration deteriorates by 39%, nearly twice as much.

• Calibration holds across backbones. With ViT-B/16, SoC reaches 64.6 accuracy and 4.3 ECE, versus O-TPT's 64.0 and 4.8. With ViT-L/14, SoC reaches 72.3 and 5.4 versus O-TPT's 71.4 and 7.7. SoC improves zero-shot accuracy by +0.7 (ViT-B/16) and +1.2 (ViT-L/14) while keeping ECE close to zero-shot levels (4.3 vs 4.2; 5.4 vs 5.1).

• Less sensitive to prompt choice. Across 18 CLIP text prompts on DTD and FGVC-Aircraft, SoC yields lower ECE than O-TPT for nearly all prompts, though both remain sensitive to prompt initialization.

• Works with supervised prompt initialization. With CoOp-initialized prompts, SoC still beats O-TPT in both accuracy and ECE at 2-shot and 4-shot: 75.2 accuracy / 6.3 ECE at 2-shot (versus 72.9 / 7.4), and 78.0 / 5.4 at 4-shot (versus 76.6 / 5.6). The gap narrows as accuracy increases.

• Strong selective classification. Under varying confidence thresholds, SoC consistently beats TPT, C-TPT, and O-TPT in selective accuracy, with gaps often between 5% and 10%, and matches the zero-shot CLIP baseline at all thresholds despite having been adapted on unlabeled test data.

Methodology in Plain English

The researchers start from CLIP, where an image and each class name are encoded into a shared, L2-normalized space, and classification is decided by the highest cosine similarity. Test-time prompt tuning learns the text prompt for a batch of augmented views of a test image by minimizing the entropy of the averaged prediction — a single gradient step with AdamW, a learning rate of 0.005, and a batch size of 64 (i.e., 64 augmentations), using ViT-L/14 by default and ViT-B/16 for the backbone study.

Their diagnosis of O-TPT is that its penalty ‖S − I_K‖² drives every pair of class prototypes toward orthogonality with a force proportional to their similarity. For classes that genuinely overlap in meaning, that is the wrong direction: it destroys real semantic structure and mechanically pushes the worst-case similarity μ down, which raises the softmax confidence floor.

Their fix is to swap the quadratic orthogonality penalty for a Huber penalty on pairwise cosine similarity with margin δ. Below δ, the penalty is quadratic (s²/2); above δ, it switches to linear (δ(s − δ/2)), so the gradient stops growing with similarity and is capped at δ. The regularizer is averaged over the K(K−1)/2 pairs in the lower triangle of the similarity matrix and added to the TPT entropy loss with weight λ.

To justify this, they prove a lower bound on the maximum softmax probability in terms of μ and the temperature parameter α, then analyze one gradient step to compare how μ evolves under full orthogonality versus the Huber penalty. They validate empirically against TPT, C-TPT, and O-TPT under identical settings, measuring accuracy and expected calibration error (ECE), plus reliability diagrams, selective classification curves, and prompt-sensitivity sweeps.

Why This Matters

Impact on research. The paper reframes orthogonality as something to be enforced selectively rather than absolutely. Its confidence-floor bound connects prototype geometry to calibration in a way that is testable, and its first-order analysis explains why more aggressive separation is not automatically better. It also argues that calibration-aware TPT can approach zero-shot calibration without giving up the ability to adapt to new domains — a tension the authors say earlier work did not resolve.

Real-world applications (drawn from the settings the paper motivates):

• Healthcare, where image-based decisions require trustworthy confidence scores before acting on a prediction.
• Autonomous vehicles, where the model must know when it is uncertain about an out-of-distribution scene.
• Video surveillance, where novel or ambiguous categories are common and overconfident errors are costly.
• Remote sensing and land-use mapping, since EuroSAT is the dataset where SoC's improvement is largest — a satellite image dataset where semantically overlapping land-cover classes (e.g., annual versus permanent crop land) are exactly the failure mode the method targets.

Industry relevance. Any deployment pipeline that adapts a vision-language model on unlabeled data at inference time inherits the overconfidence problem that entropy minimization creates. SoC is a drop-in change to the loss function, requires no labels, and follows the same training configuration as prior TPT methods, which lowers the barrier to adoption. The selective-classification results are directly relevant to abstention and human-in-the-loop workflows.

Future Directions

• Choosing δ and λ. The main text does not report how the margin δ or the weight λ are selected; the paper says all remaining settings follow prior work and refers to an appendix for further details. A principled selection procedure — or a way to learn δ — remains open.

• Beyond one or two gradient steps. The theory covers a single update, and the empirical study covers two. Whether the smoothing advantage compounds or saturates over longer adaptation horizons is not established here.

• Combining with other calibration strategies. The paper contrasts SoC with post-hoc methods such as temperature scaling and with unsupervised logit normalization (SaLS) but does not report experiments combining them.

• The ImageNet regression. SoC's ECE is worse than O-TPT's on ImageNet specifically (7.2 versus 5.5), and the paper does not offer an explanation, leaving a question about when smooth repulsion is not the right inductive bias.

Target Audience

Researchers and practitioners working on vision-language model adaptation, test-time optimization, and uncertainty quantification. It is most useful for readers already familiar with CLIP, prompt tuning, and calibration metrics such as ECE, and for engineers building safety-critical or abstention-based systems on top of VLMs. Readers looking only for accuracy improvements will find the gains modest; the value here is in calibration and reliability.

Authors’ abstract

With the increasing adoption of vision-language models (VLMs) in critical decision-making systems such as healthcare or autonomous driving, the calibration of their uncertainty estimates becomes paramount. Yet, this dimension has been largely underexplored in the VLM test-time prompt-tuning (TPT) literature, which has predominantly focused on improving their discriminative performance. Recent state-of-the-art advocates for enforcing full orthogonality over pairs of text prompt embeddings to enhance separability, and therefore calibration. Nevertheless, as we theoretically show in this work, the inherent gradients from fully orthogonal constraints will strongly push semantically related classes away, ultimately making the model overconfident. Based on our findings, we propose Semantic Orthogonal Calibration (SoC), a Huber-based regularizer that enforces smooth prototype separation while preserving semantic proximity, thereby improving calibration compared to prior orthogonality-based approaches. Across a comprehensive empirical validation, we demonstrate that SoC consistently improves calibration performance, while also maintaining competitive discriminative capabilities.