Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP

AI summary

Overview

Research area: Adversarial robustness of vision-language models (VLMs), specifically test-time defense for CLIP under zero-shot settings, with a focus on frequency-domain analysis of adversarial examples.

Technical level: Advanced. The paper assumes familiarity with CLIP's contrastive image-text embedding, adversarial attacks (PGD, AutoAttack), frequency-domain signal processing (Discrete Fourier Transform, low-pass filtering), and test-time optimization.

Scope: The paper proposes Contrastive Spectral Rectification (CSR), a training-free test-time defense that detects adversarial inputs via spectral consistency and then rectifies them with a few steps of spectral-guided contrastive optimization, evaluated on 16 classification benchmarks plus segmentation, captioning, and VQA. (arXiv:2601.19210v2 [cs.CV], published 2026-01-27.)

What This Paper Is About

CLIP and similar vision-language models generalize well to new tasks without fine-tuning, but tiny, imperceptible pixel perturbations can destroy their predictions. Existing defenses either require expensive adversarial fine-tuning that degrades benign accuracy, or purify inputs at inference time but collapse under strong attacks, run slowly, or only work for one narrow task. The authors investigate why adversarial examples are brittle in the frequency domain and turn that insight into an efficient, general-purpose test-time defense.

Key Contributions

1. A spectral diagnosis of adversarial fragility in CLIP. The paper shows that adversarial examples lose feature consistency sharply when mid-to-high frequencies are progressively attenuated, while benign examples (including Gaussian-noised ones) stay stable, and traces this to CLIP's spectral bias and hypersensitivity to mid-to-high frequency bands.

2. The CSR defense method. A test-time defense combining an input-adaptive gating mechanism (adversarial detection via spectral consistency) with a spectral-guided contrastive objective that optimizes a small rectification perturbation, using the low-pass filtered feature as a positive anchor and the original feature as a negative anchor, plus a greedy selection rule over optimization steps.

3. State-of-the-art robustness across 16 classification benchmarks. CSR reports an average of 6.9% improvement against PGD and 18.1% against the stronger AutoAttack (APGD), with a 0.7% average drop in clean accuracy (Table 1), while remaining far cheaper at inference than the slowest baselines.

4. Demonstrated cross-task and cross-backbone applicability. Beyond classification, CSR is evaluated on semantic segmentation (VOC2010), image captioning, and VQA (MS-COCO with LLaVA), and on CLIP ViT-B/32, ViT-L/14, and ViT-L/14@336px in addition to the default ViT-B/16.

Main Findings

• Adversarial examples are spectrally fragile. Using 1,000 images sampled from ImageNet, cosine similarity between original and low-pass filtered embeddings stays high for clean and Gaussian-noised images (negligible divergence even at an l-infinity noise budget of 4/255), but drops abruptly for PGD and AutoAttack adversarial examples, even at 1/255.

• CLIP's gradients concentrate in mid-to-high frequencies. Spectral Gradient Magnitude heatmaps show large gradient response outside the low-frequency center, and band-restricted attacks induce much larger representational shift in mid-to-high bands than in low bands (for example the 0-30 band), especially below 4/255.

• Low-frequency attacks are inefficient. Matching the damage of a high-frequency attack at 2/255 requires a low-frequency budget around 16/255, producing visible wavy artifacts; the adversarial gradient shows consistent negative cosine similarity with the low-frequency constraint gradient.

• Strong robustness under 10-step PGD (1/255) on 16 datasets. CSR averages 58.1% robust accuracy and 62.9% clean accuracy, versus 1.0% robust for original CLIP and 24.5% for TTC, the closest baseline. CSR beats the strongest baseline by 6.9% with a 0.7% average clean drop. Per-dataset examples: ImageNet 62.5 clean / 58.9 robust, CIFAR10 87.2 / 75.0, OxfordPets 88.9 / 65.9, EuroSAT 42.9 / 40.4.

• Robustness holds under stronger attacks. Under 50-step PGD and AutoAttack at 4/255 (Table 2), CSR reaches 66.4% average accuracy under AutoAttack, an 18.1% gain over the best test-time baseline, while preserving clean accuracy. TTC, Anti-Adv, and HD fall to near-zero under AutoAttack, and R-TPT drops to 28.4% on the General group.

• Fine-tuning baselines trade away benign accuracy. TeCoA and FARE degrade clean performance by roughly 20% and 13% respectively, and CSR is reported to outperform FARE by about 14% under PGD in cross-modal attack settings (60.3 vs 46.2).

• CSR is fast. On a single RTX 4090, CSR averages 15.17 ms per image (4.16 ms clean, 26.18 ms adversarial), compared with 3.37 ms for CLIP, 176.33 ms for R-TPT, 189.34 ms for HD, 32.42 ms for Anti-Adv, 5.42 ms for TTE, and 35.19 ms for TTC. R-TPT is described as roughly 12x slower than CSR.

• Cross-backbone consistency. On CLIP-B/32, CSR achieves 50.9 robust vs 76.1 clean on General, 41.1 vs 71.4 on fine-grained, 30.2 vs 38.3 on Scene, and 29.7 vs 35.8 on Domain; on CLIP-L/14 the corresponding numbers are 77.5 vs 82.2, 73.6 vs 83.5, 45.8 vs 44.2, and 45.9 vs 45

Authors’ abstract

Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong APGD with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.