TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models

AI summary

Overview

• Research area: Adversarial robustness and test-time defense for vision-language models (VLMs), specifically CLIP-based image classification.
• Technical level: Intermediate (readers should know CLIP-style contrastive image-text models, adversarial attacks such as PGD, and the general idea of test-time adaptation).
• Scope: The paper proposes Test-Time Padding (TTP), a detect-then-adapt inference framework that uses spatial padding to separate clean from adversarial inputs and to restore attention disrupted by adversarial perturbations, evaluated across eight fine-grained datasets and three CLIP backbones.

What This Paper Is About

CLIP and similar vision-language models are easy to fool with small, deliberate image perturbations, and existing defenses either require expensive adversarial retraining or apply the same adaptation to every input, which hurts clean accuracy. TTP attacks this problem at inference time: it checks whether an image's CLIP feature changes much when the image is padded, treats a large change as evidence of an adversarial example, and then runs a short padded adaptation only on those detected as adversarial. The goal is to gain adversarial robustness without retraining and without giving up clean accuracy.

Key Contributions

1. A padding-based adversarial detector. The authors show that spatial padding restores attention patterns disrupted by adversarial perturbations, and that the cosine-similarity shift between CLIP embeddings computed before and after padding separates clean from adversarial inputs. They use a single universal threshold (cosine similarity 0.8) that transfers across datasets and CLIP architectures.
2. Trainable test-time padding for detected adversarial inputs. Instead of adapting text prompts, TTP optimizes instance-specific image padding parameters with a single gradient step at inference by minimizing average entropy over the lowest-entropy augmented views.
3. A similarity-aware ensemble. Each selected augmented view is weighted by the difference between its similarity to the padded adversarial embedding and its similarity to the original adversarial embedding, producing a more reliable final prediction.
4. A two-stage detect-then-adapt framework. Clean samples bypass adaptation by default and can optionally be routed through existing test-time adaptation methods (demonstrated with TPT) for further clean-accuracy gains, making TTP plug-and-play and compatible with prior TTA work.

Main Findings

• Robustness improvement over state-of-the-art test-time defenses. On eight fine-grained datasets with CLIP-ViT-B/32 under 100-step PGD with ε = 4.0, TTP reaches an average adversarial accuracy of 39.7% versus 35.3% for R-TPT (a 4.4% average improvement), 35.0% for MTA, and 34.3% for plain Ensemble, while zero-shot CLIP scores 0.0%.
• Detection quality is the key differentiator. TTC, which also uses a detect-then-defend design, reaches only 6.8% average adversarial accuracy on ViT-B/32 under the same attack. The paper attributes this to its weak detection: TTC uses an L2-distance threshold of 0.2 and fluctuates across datasets and backbones, whereas TTP uses a unified cosine-similarity threshold of 0.8 and reports nearly 100% detection accuracy across architectures and datasets.
• Transfer across CLIP backbones. TTP also leads on ViT-B/16 (42.9% average adversarial accuracy versus 39.9% for R-TPT) and ViT-L/14 (51.6% versus 49.6% for R-TPT), with clean accuracy essentially matched to the baselines.
• Clean accuracy is preserved. Average clean accuracy across backbones is 62.4% for TTP versus 62.6% for zero-shot CLIP, 61.3% for TTC, and 62.2% for R-TPT; combined with TPT, TTP reaches the highest average clean accuracy at 63.2%.
• Robustness generalizes to other attacks. Under CW, DeepFool, and FGSM attacks on Flowers and DTD, TTP attains the highest adversarial accuracy in every setting (for example, 54.1% versus 51.8% for R-TPT on Flowers averaged over the three attacks, and 38.7% versus 34.2% on DTD).
• Padding pattern matters for detection. With threshold 0.8, detection accuracy averages 98.5% using 0-padding (all black) and 98.7% using 255-padding (all white), compared with 95.8% for random padding; the authors recommend 0 or 255 padding for the detection stage.
• Padding size trades off detection and robustness. Increasing padding size steadily lowers the cosine similarity for adversarial examples while barely affecting clean ones, but excessively large padding narrows the gap again. On DTD adversarial accuracy peaks at moderate padding sizes and declines for larger ones.
• Every component contributes. Ablation on ViT-B/32 and ViT-B/16 shows fixed padding alone already provides most of the gain (37.5% and 38.0%), entropy minimization adds more (39.0% and 40.8%), the similarity-aware ensemble adds on its own (38.3% and 39.5%), and all three together give the best results (39.7% and 42.9%). With no components, adversarial accuracy is 0.0%.

Methodology in Plain English

The pipeline runs in three stages at inference, with the CLIP image encoder kept frozen throughout.

1. Detect. Encode the test image with CLIP, then encode the same image after applying a fixed padding operation, and compute the cosine similarity between the two embeddings. Clean images barely change; adversarially perturbed images change a lot. If the similarity exceeds the threshold of 0.8, the image is declared clean and classified directly. Otherwise it is treated as adversarial.
2. Adapt. For adversarial inputs, generate N augmented views using stochastic transformations (random resize, crop, color jitter, and AugMix augmentation). Compute the entropy of each view's prediction and keep the top-K lowest-entropy views as the confident set. A lightweight trainable padding module is applied to those views, and its parameters are updated with a single step that minimizes the average prediction entropy. Padding parameters are randomly initialized in [0, 10] (pixel values in [0, 255]), the padding size is 32, the learning rate is 5, and the augmented batch size is 64.
3. Ensemble. Each selected view gets a weight based on how similar its padded embedding is to the padded adversarial embedding minus how similar it is to the unpadded adversarial embedding, passed through a softmax. The final prediction is the class with the highest weighted vote across those views.

Evaluation follows the R-TPT protocol: eight fine-grained datasets spanning general objects, animals, plants, vehicles, textures, satellite imagery, and video actions, three CLIP backbones, PGD with 100 iterations at ε = 4.0, plus CW, DeepFool, and FGSM for additional attack coverage. All methods use only the CLIP backbone and AugMix augmentation, without extra foundation models or external knowledge. Experiments run on NVIDIA RTX 3090 GPUs.

Why This Matters

Adversarial fragility in large pretrained VLMs is a practical problem because most users cannot retrain these models and rely on public checkpoints. TTP offers a defense that works purely in input pixel space, requires no model modification, no labeled adversarial data, and no knowledge of the model's internals, making it easy to bolt onto deployed systems.

• Safety-critical perception: content moderation, medical image triage, or autonomous systems that use CLIP-style encoders and could be manipulated by imperceptible perturbations.
• Robust retrieval and tagging: image search, dataset curation, and auto-labeling pipelines where adversarial or corrupted inputs would otherwise be silently misclassified.
• Open checkpoint deployments: any application that consumes a public CLIP checkpoint without fine-tuning and wants a low-cost robustness layer.
• Composable test-time pipelines: since TTP routes clean samples untouched, it can be layered on top of existing test-time adaptation methods like TPT to improve clean accuracy as well.

For the research community, the paper's main conceptual push is that accurate detection should come first, and that adaptation should be applied selectively rather than uniformly — a critique aimed directly at the uniform-adaptation design shared by TAPT and R-TPT and at the weaker detection mechanism of TTC.

Future Directions

• Extending beyond classification. All experiments are on classification with CLIP; whether the padding-induced similarity shift generalizes to detection, segmentation, captioning, or retrieval is not reported and remains open.
• Padding-size selection. The paper shows a trade-off between detection separability and structural distortion, with a peak at moderate sizes, but does not report an automatic rule for choosing the size per dataset or model.
• Detection threshold universality. A single cosine-similarity threshold of 0.8 is used throughout; testing on larger or non-ViT architectures, or under adaptive attacks that account for padding, is not reported.
• Stronger adaptation pairings. The authors note clean accuracy could improve further by combining TTP with stronger test-time adaptation methods than TPT, which they demonstrate only as an example.

Target Audience

Researchers and practitioners working on adversarial robustness, test-time adaptation, and vision-language models; engineers deploying CLIP-based systems who need a lightweight robustness layer without retraining; and graduate students who want a clear example of a detect-then-adapt design combined with entropy-based test-time optimization.

Authors’ abstract

Vision-Language Models (VLMs), such as CLIP, have achieved impressive zero-shot recognition performance but remain highly susceptible to adversarial perturbations, posing significant risks in safety-critical scenarios. Previous training-time defenses rely on adversarial fine-tuning, which requires labeled data and costly retraining, while existing test-time strategies fail to reliably distinguish between clean and adversarial inputs, thereby preventing both adversarial robustness and clean accuracy from reaching their optimum. To address these limitations, we propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference. TTP identifies adversarial inputs via the cosine similarity shift between CLIP feature embeddings computed before and after spatial padding, yielding a universal threshold for reliable detection across architectures and datasets. For detected adversarial cases, TTP employs trainable padding to restore disrupted attention patterns, coupled with a similarity-aware ensemble strategy for a more robust final prediction. For clean inputs, TTP leaves them unchanged by default or optionally integrates existing test-time adaptation techniques for further accuracy gains. Comprehensive experiments on diverse CLIP backbones and fine-grained benchmarks show that TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy. The code for this paper will be released soon.