Diversifying Counterattacks: Orthogonal Exploration for Robust CLIP Inference

AI summary

Overview

Research area: Adversarial robustness of vision-language pre-training models (VLPs), specifically test-time defense for CLIP inference. The work sits at the intersection of multimodal representation learning and adversarial machine learning.

Technical level: Intermediate. The core insight is intuitive (explore more varied counter-perturbations instead of one narrow gradient path), but the method involves gradient normalization, orthogonal projection, momentum updates, and a cosine-similarity-based gating score, so some familiarity with gradient-based attack/defense machinery helps.

Scope in one sentence: The paper proposes Directional Orthogonal Counterattack (DOC), a parameter-free, label-free test-time defense that generates more diverse counter-perturbations to neutralize adversarial inputs fed to CLIP, and evaluates it on 16 datasets against PGD, CW, and AutoAttack.

What This Paper Is About

CLIP and similar vision-language models are easily fooled by small, human-imperceptible perturbations, which is a problem when they are used in security-sensitive settings. The prior state-of-the-art defense, Test-Time Counterattack (TTC), fixes the adversarial image as an anchor and uses PGD to push a modified copy away from that adversarial embedding, but the paper argues this search is too narrow: counterattacks optimize a different objective than the attacks they are trying to cancel, so PGD-based counterattacks overfit to limited adversarial patterns. DOC's goal is to broaden that search so the defense generalizes across many kinds of adversarial perturbations without training, labels, or prompt engineering.

Key Contributions

1. DOC (Directional Orthogonal Counterattack): a counterattack generation scheme that adds a randomized component orthogonal to the primary gradient direction at each optimization step, combined with a momentum-based update, widening the explored perturbation space and increasing counterattack diversity.
2. A directional sensitivity score: a measure built from the averaged cosine similarity between an input's embedding and the embeddings of its randomly perturbed versions, used to decide whether an input looks adversarial and to adaptively scale counterattack strength through a soft gating function rather than a hard threshold.
3. A broad empirical evaluation: 16 datasets spanning general object classification, fine-grained classification, scene recognition, and domain-specific tasks, showing improved adversarial robustness over test-time defenses while maintaining competitive clean accuracy.
4. Demonstrated plug-in compatibility: DOC is applied on top of adversarially fine-tuned CLIP models (TeCoA, PMG-AFT, FARE) and improves their robustness without additional training cost.

Main Findings

• Robustness gain under PGD-10 (ε_atk = 4/255): averaged over 16 datasets, DOC reaches 31.02 percent robust accuracy against 21.22 percent for TTC and 0.06 percent for the original CLIP, an average gain of 9.80 percent over TTC and an improvement of 30.96 over original CLIP.
• Clean accuracy is largely preserved: DOC averages 58.26 percent clean accuracy versus 55.63 percent for TTC and 61.51 percent for the unmodified CLIP (a change of −3.25 relative to CLIP), so it keeps higher clean accuracy than the previous best test-time defense.
• Gains hold under stronger attacks (CW, ε_atk = 4/255): DOC averages 28.18 percent robust accuracy versus 20.61 percent for TTC (gain of 7.58), with average clean accuracy of 58.34 percent.
• AutoAttack results: the paper reports an average improvement of 4.1 percent across the 16 datasets under AutoAttack at ε_atk = 4/255.
• Works as a plug-in on fine-tuned models: combined with TeCoA, PMG-AFT, and FARE, DOC brings a 4 percent to 5 percent improvement over those baselines; combined with FARE it yields an average robust accuracy increase of over 18 percent relative to the original CLIP (evaluated with PGD-10 at ε_atk = 1/255).
• Ablation (average over 16 datasets, ε_atk = 4/255, five random seeds): with neither component, clean/PGD/CW/AutoAttack are 55.66/21.43/20.70/21.97; DSS alone gives 58.23/23.37/22.27/22.66; orthogonal gradient augmentation alone gives 55.38/31.83/29.02/26.07; combining both gives 58.27/31.04/28.15/25.89. The orthogonal component drives most of the robustness gain, while the directional sensitivity score mainly protects clean accuracy.
• Counterattack steps matter up to a small number: increasing the number of counterattack steps N consistently improves robustness up to N = 3 and saturates around N = 3 or N = 4, while clean accuracy stays stable.
• Diversity is the mechanism claimed: t-SNE visualizations and mean cosine similarity (lower values indicating higher diversity) show DOC produces more diverse counterattacks than TTC, which the authors link to the robustness improvement.
• Setup details: counterattack budget ε_ca = 4/255, batch size 256, 4 steps, default step size α_ttc = 3/255, run on a single NVIDIA 4090 GPU; adversarial fine-tuning baselines were trained on Tiny ImageNet.

Methodology in Plain English

The defense runs entirely at inference time and never changes CLIP's weights.

Step 1 — Orthogonal gradient augmentation. At each iteration, the method computes the gradient of the embedding-distance objective with respect to the adversarial input and normalizes it into a unit direction g. It then takes a random Gaussian vector r, subtracts its component along g, and renormalizes, producing r^⊥ that is mathematically orthogonal to the main gradient (their inner product is zero). The update direction is

Authors’ abstract

Vision-language pre-training models (VLPs) demonstrate strong multimodal understanding and zero-shot generalization, yet remain vulnerable to adversarial examples, raising concerns about their reliability. Recent work, Test-Time Counterattack (TTC), improves robustness by generating perturbations that maximize the embedding deviation of adversarial inputs using PGD, pushing them away from their adversarial representations. However, due to the fundamental difference in optimization objectives between adversarial attacks and counterattacks, generating counterattacks solely based on gradients with respect to the adversarial input confines the search to a narrow space. As a result, the counterattacks could overfit limited adversarial patterns and lack the diversity to fully neutralize a broad range of perturbations. In this work, we argue that enhancing the diversity and coverage of counterattacks is crucial to improving adversarial robustness in test-time defense. Accordingly, we propose Directional Orthogonal Counterattack (DOC), which augments counterattack optimization by incorporating orthogonal gradient directions and momentum-based updates. This design expands the exploration of the counterattack space and increases the diversity of perturbations, which facilitates the discovery of more generalizable counterattacks and ultimately improves the ability to neutralize adversarial perturbations. Meanwhile, we present a directional sensitivity score based on averaged cosine similarity to boost DOC by improving example discrimination and adaptively modulating the counterattack strength. Extensive experiments on 16 datasets demonstrate that DOC improves adversarial robustness under various attacks while maintaining competitive clean accuracy. Code is available at https://github.com/bookman233/DOC.