Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions

AI summary

Overview

Research area: Test-time adaptation (TTA) for pretrained vision-language models (VLMs), specifically CLIP, under common image corruptions. Technical level: Intermediate (the motivation and method are accessible, but Section 3 assumes familiarity with variance decompositions, LayerNorm, and gradient-based analysis). Scope: The paper identifies and names the "variance collapse" phenomenon in CLIP image embeddings under corruption, explains it theoretically, and proposes a lightweight TTA method, Mint, that maximizes pseudo-label-based inter-class variance during testing. arXiv:2510.22127v1 [cs.CV], 25 Oct 2025, by Wenxuan Bao, Ruxi Deng (equal contribution), and Jingrui He at the University of Illinois Urbana-Champaign.

What This Paper Is About

CLIP generalizes well zero-shot but loses accuracy when images are corrupted (noise, blur, weather, digital artifacts). The authors first ask what actually happens to CLIP's image embeddings under such corruption, and they find that both within-class and between-class embedding variances shrink as corruption severity rises, a pattern they call variance collapse. Their goal is to build a simple test-time adaptation method that reverses this collapse and improves accuracy, without source data, labels, or large test batches.

Key Contributions

1. They identify and document "variance collapse" in CLIP image embeddings: as corruption severity increases, GT-intra, GT-inter, and GT-total variances all consistently decrease across corruption types.
2. They provide a theoretical analysis attributing the collapse to the visual encoder encoding corruption-related signals (structured shift and noise components) into the embedding space, which dilutes class-discriminative features; they also prove that maximizing pseudo-label inter-class variance (PL-inter) via LayerNorm updates suppresses shift-related weights and increases class-relevant weights.
3. They propose Mint, a TTA method that maximizes PL-inter variance on the fly using a mean accumulator (cumulative class and global embedding averages) and a gradient accumulator (averaged update direction), plus a training-free text-embedding adjustment using the accumulated class means.
4. They demonstrate that Mint improves CLIP accuracy across CIFAR-10-C, CIFAR-100-C, and ImageNet-C with ViT-B/32, ViT-B/16, and ViT-L/14, including at batch size 1, while being substantially faster than most training-based baselines.

Main Findings

• Variance collapse is consistent and universal in their experiments: Using the corruption benchmark (15 corruption types, 5 severity levels), both GT-intra and GT-inter variances decrease monotonically as severity increases, meaning embeddings become more similar regardless of class membership.
• Inter-class variance tracks accuracy most closely: Computed over 76 corruption settings (15 corruption types × 5 severity levels plus clean) on CIFAR-100-C, GT-inter variance correlates with accuracy at 0.98, versus 0.86 for GT-intra and 0.94 for GT-total.
• Theory explains the mechanism: In a balanced binary classification model with latents decomposed into class-relevant, task-irrelevant, structured shift (s·δ), and unstructured noise (s·Rademacher) components, increasing severity s strictly decreases GT-inter variance and (when |δ|₂ ≥ sqrt(d_noise/d_irr)·|μ|₂) also decreases GT-intra variance. Gradient ascent on PL-inter provably increases class-relevant LayerNorm weights and suppresses shift-related weights.
• Main accuracy results: Mint achieves 71.0% average accuracy on CIFAR-10-C with ViT-B/32 (CLIP: 59.0%), 44.1% on CIFAR-100-C with ViT-B/16 (CLIP: 35.8%), and 47.0% on ImageNet-C with ViT-L/14 (CLIP: 39.6%). Against the strongest baselines in each setting, this corresponds to absolute gains of 3.9%, 2.2%, and 3.1%.
• Robustness to very small batches: With one shared hyperparameter set across batch sizes 1 to 200, Mint scores 70.5% at batch size 1 on CIFAR-10-C (ViT-B/32), 43.1% at batch size 1 on CIFAR-100-C (ViT-B/16), and 45.8% at batch size 1 on ImageNet-C (ViT-L/14), with peaks of 71.0%, 44.6%, and 47.1% at larger batch sizes.
• It mitigates variance collapse: On CIFAR-100-C, Mint increases PL-inter variance by design, and GT-inter variance and accuracy both rise after adaptation across the four representative corruptions examined.
• Efficiency: On CIFAR-100-C (10,000 images per corruption type), Mint takes 1m07s and reaches 44.1% accuracy, versus WATT-S at 50m20s and 41.9%, CLIPArTT at 7m40s and 40.7%, TPS at 9m58s and 38.6%, TPT at 23m21s and 36.0%, and plain CLIP at 21s and 35.8%. Mint is slower only than CLIP and the training-free, augmentation-free baselines (TDA at 33s, DMN-ZS at 30s).
• Both accumulators matter: Ablation on CIFAR-10-C shows the mean accumulator is essential for estimating PL-inter variance at extremely small batches (including batch size 1, where without it gradients cannot be computed), and the gradient accumulator reduces gradient noise across batches; the two together give the strongest performance.
• LayerNorm scope: Updating all LayerNorm layers in the visual encoder yields the best performance (details in Appendix C.5).

Methodology in Plain English

The authors first measure how far CLIP's image embeddings for corrupted images drift from each other, using ground-truth labels to split total variance into within-class (intra) and between-class (inter) parts. To explain the pattern, they build a simplified mathematical model in which each image has four latent ingredients: class-relevant features, task-irrelevant features, a corruption-induced shift that grows with severity, and random noise. Passing these through a LayerNorm-style transform (with the demeaning step dropped and bias ignored, i.e., RMSNorm-like) and normalizing to unit length, they derive how the two variances behave as severity grows, and then derive the gradient of the pseudo-label-based inter-class variance with respect to the LayerNorm weights.

The key practical idea is that maximizing inter-class variance is equivalent to maximizing total variance while minimizing intra-class variance, which pushes each embedding away from the global mean and toward its own class mean, with the gradient direction roughly given by (class mean − global mean). Since online adaptation sees only tiny batches, class means computed from a single batch are unreliable. Mint therefore keeps running averages of the global mean and each pseudo-class mean (the mean accumulator), substitutes these into the objective for the current batch, and averages the gradient across batches (the gradient accumulator) so it mimics a larger effective batch. Only LayerNorm weights are updated, one gradient step per batch with Adam (learning rate 0.007 for ViT-B models, 0.015 for ViT-L/14). After each batch, the encoder and optimizer state reset to their initial values, but the accumulators carry over. Separately, the accumulated class means are blended with the original CLIP text embeddings via a prior weight K_prior = 10,000, and predictions are made by image-text similarity using the refined text embeddings.

Why This Matters

The paper connects a measurable property of embeddings to accuracy in a simple, actionable way, and shows that a single-objective, single-step update can beat more elaborate TTA pipelines while running far faster. It also provides theory for why a pseudo-label-based objective can still help despite noisy labels.

Real-world applications:

• Autonomous driving and robotics, where cameras encounter rain, fog, snow, motion blur, and sensor noise at test time.
• Surveillance and remote sensing, where deployment conditions differ from training data and labels are unavailable.
• Mobile or edge vision systems, where memory constraints force batch size 1 and repeated iterative adaptation is impractical.
• Medical or industrial imaging pipelines, where source data may be proprietary or unavailable and models must adjust on the fly.

Industry relevance: Mint needs no source data, no labels, and no large batches, which fits the common deployment situation where a pretrained VLM is used as-is but the input distribution drifts. Its 1m07s testing time on 10,000 CIFAR-100-C images, versus minutes to tens of minutes for most baselines, makes online streaming adaptation practical, and its design (updating only LayerNorm weights) keeps the compute and memory footprint modest.

Future Directions

• Extending the theoretical analysis beyond the balanced binary classification setting used in Theorem 3.1 and Theorem 3.2 to general multi-class, imbalanced scenarios.
• Testing Mint on distribution shifts other than common corruptions, and on VLM architectures beyond the CLIP ViT-B/32, ViT-B/16, and ViT-L/14 combinations evaluated.
• Exploring whether objectives beyond LayerNorm-only updates (for example, combining Mint with prompt or text-encoder adaptation) yield further gains; the authors note prompt/embedding-based methods are the dominant prior line of work.
• Better understanding and tuning the prior strength K_prior and learning rate: the paper states it studies sensitivity to these two hyperparameters across three datasets, but the content is truncated before those results are reported, so the conclusions of that analysis are not available here.

Target Audience

Researchers and practitioners working on test-time adaptation, domain robustness, or vision-language models will get the most from this paper. It is also useful for engineers deploying CLIP-like models in streaming, low-batch, or source-data-free settings, and for readers interested in a case study of connecting an empirical embedding statistic to a provable adaptation objective.

Authors’ abstract

Pretrained vision-language models such as CLIP achieve strong zero-shot generalization but remain vulnerable to distribution shifts caused by input corruptions. In this work, we investigate how corruptions affect CLIP's image embeddings and uncover a consistent phenomenon we term as embedding variance collapse, where both intra-class and inter-class variances shrink as corruption severity increases. We find that this collapse is closely tied to performance degradation, with inter-class variance strongly correlated with classification accuracy. To explain this phenomenon, we analyze how corruptions alter the structure of the embedding space. Our theoretical results suggest that the visual encoder tends to encode corruption-related signals, which dilute class-discriminative features and compress the representation geometry. We further show that maximizing inter-class variance, even when estimated from pseudo-labels, can provably enhance embedding quality. Based on this insight, we propose Mint, a simple test-time adaptation method that maximizes pseudo-label-based inter-class variance on the fly using a mean accumulator and a gradient accumulator. Mint operates effectively with small batch sizes and consistently improves performance across multiple corruption benchmarks and CLIP architectures. Our code is available at https://github.com/baowenxuan/Mint .