Bridging CLIP with DINO: Cross-Modal Information Maximization for Online Test-Time Adaptation

AI summary

Overview

Research area: Computer vision — test-time adaptation (TTA) of vision-language models, specifically CLIP-based zero-shot classification under distribution shift and image corruption.

Technical level: Advanced. The paper assumes working familiarity with contrastive vision-language pretraining (CLIP), self-supervised ViT backbones (DINO/DINOv2), entropy minimization for TTA, and calibration metrics such as ECE.

Scope (one sentence): The paper diagnoses a failure mode it calls modality bias in CLIP-based test-time adaptation and proposes CDIM, a three-level correction (representation, optimization, prediction) that pairs CLIP with a frozen DINOv2 branch, and reports gains on ImageNet-C, CIFAR10-C and CIFAR100-C.

What This Paper Is About

When CLIP is deployed on corrupted images and adapted online with the usual entropy-minimization objective, its visual encoder tends to snap distorted features onto the wrong class-name anchor — and does so with high confidence. Because entropy minimization uses the model's own prediction as its learning target, these confident mistakes get reinforced rather than fixed, producing a confirmation-bias loop that eventually collapses predictions onto a small subset of classes.

The paper's goal is to break that loop by (1) adding a text-free self-supervised backbone (DINOv2) as a second opinion, (2) replacing per-sample entropy minimization with a gated information-maximization objective, and (3) applying a training-free logit correction that counters the residual class-frequency skew at prediction time.

Key Contributions

1. Diagnosis of "modality bias." The authors name and characterize the tendency of CLIP's visual encoder, under corruption, to project distorted features onto incorrect textual anchors with high confidence, and they link it to confirmation bias in CLIP-based TTA. Figure 1 shows CLIP+TENT (entropy minimization) concentrating its errors at near-perfect confidence (>0.95) on ImageNet-C at severity 5.

2. A cross-modal logit ensemble (representation level). CLIP text prototypes are projected into the frozen DINOv2 feature space through a batch-level cross-covariance matrix, producing a parameter-free, batch-local text-to-DINO prototype set. The projection is deliberately one-directional (text into DINO space, never DINO features toward text), so the auxiliary branch is not exposed to the same text-biased projection.

3. GT-IM: Gated Test-time Information Maximization (optimization level). An information-maximization objective that pairs a per-sample sharpening term with a batch-level diversity term, plus an adaptive, quantile-based confidence gate that drops the k = max(1, ⌊ρB⌋) least-confident samples from the sharpening gradient while still computing the marginal term over the full batch.

4. Online logit adjustment, OLA (prediction level). A parameter-free, gradient-free anti-frequency correction that subtracts log π̄_c^(t), where π̄ is an exponential moving average of the model's own predicted class frequencies, initialized uniformly.

5. Empirical validation across three TTA protocols (online, continual, mixed-domain) and across natural-shift benchmarks (IN-A, IN-V2, IN-R, IN-S), plus ablations, calibration analysis and backbone-scale studies. Code is released at https://github.com/B1ueC1oud/CDIM.

Main Findings

• Online TTA gains over the strongest baseline (SubTTA): +4.8% on ImageNet-C (36.9 vs 32.1 mean accuracy), +3.5% on CIFAR10-C (79.8 vs 76.3), +3.7% on CIFAR100-C (48.2 vs 44.5), all with CLIP ViT-B/16 at severity 5.

• CLIP+TENT is a weak reference point: entropy minimization reaches only 25.2% mean accuracy on ImageNet-C, and on some individual corruptions it degrades below zero-shot (e.g. 5.2% on Gaussian noise vs 11.1% zero-shot).

• Calibration improves monotonically with each component: ECE (10 equal-width bins) drops from 0.400 (CLIP+TENT) to 0.210 with the DINOv2 ensemble alone, to 0.101 with GT-IM alone, to 0.281 with OLA alone, and to 0.048 for full CDIM.

• Confidence of errors moves to a usable range: under CDIM, incorrect predictions on ImageNet-C shift from >0.95 confidence to roughly 0.1–0.4, so they no longer act as near-one-hot self-training targets.

• Ablation shows complementary components (ImageNet-C / CIFAR10-C / CIFAR100-C): baseline CLIP+TENT 25.2 / 61.9 / 37.0; +ensemble only 28.7 / 63.1 / 40.0; +GT-IM only 33.8 / 76.9 / 45.2; GT-IM+OLA 36.3 / 79.2 / 47.5; full CDIM 36.9 / 79.8 / 48.2. GT-IM gives the largest single jump, reported as +15.0% on CIFAR10-C.

• Continual TTA is stable despite not being the design target: CDIM reaches 80.4 / 46.3 / 34.6 on CIFAR10-C / CIFAR100-C / ImageNet-C, described as +13.0% on CIFAR10-C and +0.5% on both CIFAR100-C and ImageNet-C over the strongest baseline (EATA reaches 34.1 on ImageNet-C; Mint 67.4 on CIFAR10-C).

• Mixed-domain results: CDIM beats Mint by +5.3% on CIFAR10-C (ViT-B/32), +0.3% on CIFAR100-C (ViT-B/16), and +1.7% on ImageNet-C (ViT-L/14); the authors attribute the largest margin on the 10-class benchmark to the marginal-entropy term mattering more when K is small.

• Robust to the choice of auxiliary model: on CIFAR10-C, substituting DINOv1 (78.8), DINOv3 (79.7), MoCo v3 (79.1) or MAE (79.3) for DINOv2 changes accuracy by at most 1.0%.

• Scales with backbone size: with DINOv2-base, CIFAR10-C accuracy rises 10.6% from ViT-B/32 to ViT-L/14 (76.6 → 77.1 → 87.1 across backbones for DINOv2-small; 77.4 → 80.3 → 88.7 for DINOv2-large); increasing DINOv2 scale adds up to 1.6%.

• Generalizes beyond synthetic corruption: on ImageNet variants CDIM scores 55.5 (IN-A), 63.3 (IN-V2), 79.2 (IN-R), 50.1 (IN-S), improving over Mint by up to 1.7% and over BATCLIP by up to 10.2%.

• Works with a small encoder: with TinyCLIP ViT-61M/32 on CIFAR10-C, CDIM reaches 78.8 vs 72.8 for BATCLIP (+6.0%), 65.7 for TENT, and 63.9 for unadapted CLIP.

• Batch-size sensitivity: at batch sizes of 50–100, CIFAR10-C accuracy exceeds 81% (Table 7), though the main table uses B = 200 to follow the prior protocol.

• Diagnostic plots support the mechanism: image–text alignment ⟨f_img, t_true⟩ is higher throughout adaptation and recovers within a few batches under CDIM, whereas CLIP+TENT drifts downward; CLIP+TENT puts roughly 25% of its predictions on a single class (C3) on CIFAR10-C, while CDIM stays near-uniform.

• Cost: the method needs one additional frozen DINOv2 forward pass per batch (Appendix A.4).

Methodology in Plain English

The pipeline runs four steps on every incoming test batch.

Step 1 — Two opinions instead of one. Each image goes through CLIP and through a frozen DINOv2 ViT-B/14. To let DINOv2 classify the same class names CLIP knows, the authors compute a cross-covariance matrix between the batch's CLIP image features and DINOv2 image features, use it to rotate the CLIP text prototypes into DINOv2's feature space, and renormalize. This mapping is computed fresh for each batch from that batch alone — no learned translator, no source data. Logits from the CLIP branch (scaled by CLIP's pretrained logit scale) and the DINOv2 branch (fixed scale τ_D⁻¹ = 20) are averaged 50/50.

Step 2 — Drop the least trustworthy samples. Each sample gets an uncertainty score E_i = −log Σ_c exp(z_ic^ens). In every batch, the k = max(1, ⌊ρB⌋) samples with the largest E_i are excluded from the sharpening gradient (ρ = 0.05). Because the cutoff is a per-batch quantile rather than a fixed entropy threshold, a genuinely hard batch still contributes its most informative samples instead of being rejected wholesale.

Step 3 — One gradient step on a two-part objective. The loss is the entropy of the retained samples' predictions (sharpening, averaged over retained samples only) minus the entropy of the full-batch mean prediction (diversity, pushing the average prediction toward uniform over the K anchors). The two terms are weighted equally, so there is no balancing hyperparameter. Only the channel-wise affine parameters (γ, β) of CLIP's normalization layers in both the image and text encoders are updated; the rest of CLIP and all of DINOv2 stay frozen. The projected DINO prototypes are treated as constants during backprop.

Step 4 — Fix the leftover frequency skew. The model keeps an exponential moving average (α = 0.9, initialized to 1/K) of its own predicted class frequencies. Each logit is reduced by log of that running prior, an anti-frequency penalty that damps whatever class the model has been over-predicting. This has no parameters and no gradient cost.

The intuition tying it together: the frozen DINOv2 branch is independent of text supervision, so it is less vulnerable to text-driven misprojection, and its presence partially anchors the batch-mean prediction that the diversity term operates on. The gate keeps noisy gradients out of the normalization-layer update, and the logit adjustment cleans up whatever skew survives.

Why This Matters

For research: The paper reframes CLIP-based TTA failures not as a generic "entropy minimization is unstable" problem but as an identifiable interaction between language-supervised feature geometry and self-training. That framing suggests a general recipe — bring in a supervision-free second view, gate the gradient, correct the prior — that could transfer to other vision-language adapters. The calibration results (ECE 0.400 → 0.048) are also a concrete argument that adaptation accuracy and adaptation trustworthiness should be reported together.

Real-world applications:

• Autonomous driving and robotics, where sensor artifacts, rain, snow and glare shift the input distribution and no labels are available at deployment.
• Industrial visual inspection and anomaly detection, where lighting or camera drift degrades a frozen classifier and source data cannot be shared.
• Medical imaging deployed across sites, where scanner differences act as a corruption shift and privacy rules block retraining on source data.
• Edge and mobile deployment, since the method adapts only normalization-layer affine parameters and works with a small encoder like TinyCLIP ViT-61M/32.

Industry relevance: The method requires no source data, no target labels, and no second pass over a batch, which matches deployment constraints. The extra cost is one frozen DINOv2 forward pass per batch and a K-dimensional state vector, and it is hyperparameter-light (ρ = 0.05, α = 0.9, τ_D⁻¹ = 20 fixed across all three benchmarks). That combination — no source data plus a bounded, non-learned correction — is the kind of thing that can be shipped rather than merely benchmarked.

Future Directions

• Cost reduction and distillation. The gains come with an extra frozen backbone forward pass. Distilling the DINOv2 branch's contribution into CLIP, or sharing computation between the two ViTs, is a natural next step the paper does not attempt.
• Choosing π^T rather than assuming uniform. OLA presumes the target class marginal is close to uniform and the authors state it is a balanced-error rule, not a Bayes-optimal estimate. Estimating a non-uniform target prior from the stream, or detecting when the uniform assumption fails, is an open question the paper raises but leaves open.
• Going beyond normalization-layer affine tuning. The learnable surface is restricted to channel-wise γ, β following prior work. Whether the three-level correction permits adapting a larger parameter set without re-triggering confirmation bias is untested in the provided content.
• Stronger continual-protocol evaluation. CDIM is explicitly not designed as a continual TTA method yet performs well there (80.4 on CIFAR10-C vs 67.4 for Mint). Making the design continual-first — for example, a principled treatment of the EMA prior across long, drifting streams — is a clear extension.
• Scope note: the provided text is truncated during Section 4.2, so any author-stated limitations or conclusion section is not reported here and cannot be summarized.

Target Audience

Graduate students and research engineers working on test-time adaptation, distribution shift robustness, or vision-language model deployment. It will be most useful to readers already comfortable with CLIP's contrastive objective and with entropy-minimization TTA baselines (TENT, TPT, BATCLIP, SubTTA), since the contribution is defined largely in contrast to those methods. Practitioners building corruption-robust classifiers who can afford a second frozen backbone will find the three-level decomposition directly implementable; readers new to the area should start with the TENT and BATCLIP references before this paper.

Authors’ abstract

Vision-language models such as CLIP enable zero-shot classification from arbitrary class names, but under test-time corruptions, their visual encoder projects distorted features onto incorrect textual anchors with high confidence: a phenomenon we term modality bias. Standard entropy minimization reinforces these overconfident errors, which creates a confirmation-bias loop and collapses the predicted label distribution. We propose an online test-time adaptation (TTA) method that breaks this loop at three levels. At the representation level, a frozen self-supervised backbone (e.g., DINO v2) is combined with CLIP through a cross-covariance logit ensemble. CLIP text prototypes are projected into the feature space of this text-free backbone, so the self-supervised branch classifies native image features and is less affected by the text bias of CLIP. At the optimization level, a gated test-time information-maximization objective pairs per-sample sharpening with batch-level diversity, while an adaptive gate excludes unreliable samples from the gradient. At the prediction level, an online logit adjustment corrects the residual class-distribution skew. CDIM outperforms the compared CLIP-based online TTA baselines on ImageNet-C, CIFAR10-C, and CIFAR100-C, and remains competitive in continual and mixed-domain TTA. Extensive analyses and ablation studies verify the contribution of each of its components. Our code is available at https://github.com/B1ueC1oud/CDIM