C-LEAD: Contrastive Learning for Enhanced Adversarial Defense

AI summary

Overview

Research area: Adversarial machine learning and computer vision — specifically, using contrastive learning (a self-supervised representation-learning technique) to harden image classifiers against adversarial perturbations.

Technical level: Intermediate. The paper assumes familiarity with convolutional neural networks, adversarial attacks (FGSM, PGD, C&W), and contrastive loss functions, but the core idea is explained through figures and an algorithm listing rather than dense theory.

Scope: The paper introduces C-LEAD, a two-stage training framework that pre-trains a CNN backbone with adversarially perturbed positive/negative pairs under an InfoNCE-style contrastive loss, then fine-tunes a linear classifier on CIFAR-10 (Suklav Ghosh, Sonal Kumar, Arijit Sur, IIT Guwahati; arXiv:2510.27249v2).

What This Paper Is About

Deep neural networks can be fooled by adversarial perturbations — tiny, often imperceptible changes to an input image that cause a wrong prediction. The standard fix, adversarial training, improves robustness but typically costs clean accuracy and requires class labels. This paper asks whether the pair-sampling machinery of contrastive learning can be repurposed as a defense: instead of pulling two augmented views of an image together, it pulls an image together with its adversarially perturbed versions, teaching the backbone to produce similar features for clean and attacked inputs.

Key Contributions

1. A contrastive-learning framework for adversarial defense. The authors describe this as a previously unexplored use of contrastive learning, adapting both the positive/negative pair sampling strategy and the objective function (InfoNCE) for adversarial defense rather than for standard self-supervised representation learning.

2. An adversarially perturbed sampling strategy. Positive pairs are formed from multiple perturbed versions of the same image; negative pairs from perturbed versions of different images. Perturbations are generated with FGSM, PGD, and C&W.

3. A two-stage training pipeline (Algorithm 1): Adversarial Contrastive Training (ACT) over a set number of pre-training epochs, followed by Task-specific Fine-tuning (TF) where the backbone is frozen and only a newly added linear layer is trained for classification.

4. Reported robustness gains on CIFAR-10. The paper states improvements over the untrained baseline of 40% for FGSM resistance, 53% for PGD resistance, and 41% for C&W resistance. Code is released at https://github.com/suklav/C_Lead.

Main Findings

• Untrained backbones are highly vulnerable. ResNet18 reports 87.89% clean accuracy but only 18.38% / 13.60% under FGSM, 12.60% / 10.34% under PGD, and 9.80% / 7.40% under C&W before any training. ResNet34 (89.96% clean) and ResNet50 (93.38% clean) show comparably low pre-training attack accuracy.

• Contrastive adversarial training substantially improves attack accuracy. For ResNet50, accuracy after training reaches 55.28% (FGSM, ε=0.03) and 50.86% (FGSM, ε=0.08); 68.67% (PGD, ε=0.03) and 57.56% (PGD, ε=0.08); 59.85% (C&W, ε=0.03) and 52.20% (C&W, ε=0.08). ResNet34 reaches 53.23%/49.16% (FGSM), 61.01%/58.60% (PGD), and 58.66%/51.79% (C&W).

• Deeper models benefit more. ResNet34 and ResNet50 outperform ResNet18. The paper reports that the contrastive training benefit is less pronounced with ResNet18, giving an average improvement of only 10%–15%, which the authors attribute to its limited capacity and feature-extraction ability.

• Seen attacks improve more than unseen ones. PGD and C&W, which are used during training, show significant gains, while FGSM — used as an unseen attack — still poses a challenge for the trained model.

• The headline improvement figure. The paper reports an average improvement of 40% compared to the baseline model in Table 1, and separately states the 40% (FGSM), 53% (PGD), and 41% (C&W) figures in its contributions list.

• Comparison with other adversarial-training defenses (ε=8, Table 2). On ResNet50, C-LEAD reports 55.28 (FGSM), 68.67 (PGD), and 59.85 (CW). Reference points include AT (Madry et al., 2019) on ResNet18 at 60.9 (FGSM) and 66.3 (PGD); PGD-AT/LAS (Jia et al., 2022) on ResNet34 at 56.02 (PGD) and 53.91 (CW); TRADES (Zhang et al., 2020) on ResNet50 at 53.49 (FGSM) and 63.87 (PGD); LAS-AT (Jia et al., 2022) on ResNet18 at 61.09 (PGD) and 58.22 (CW); and ROCl (Zhu et al., 2022) on ResNet50 at 67.59 (FGSM) and 66.76 (PGD).

• Acknowledged limitations. The authors state their models may fall short in accuracy compared to state-of-the-art models that use preprocessing techniques, model modifications, or ensemble learning.

• Hardware and dataset. Experiments used the PyTorch framework on an NVIDIA DGX Station A100 GPU with 40G memory, on CIFAR-10 — 60,000 32×32 color images across ten classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck).

• Hyperparameters. A 2-layer MLP projection head with a ResNet18-based encoder produced 128-dimensional vectors; temperature coefficient 0.1; batch size 512; cosine learning rate scheduler; SGD optimizer with momentum 0.9 and learning rate 0.4. The pseudo-label creation task used a linear layer over a ResNet18 base encoder with the Adam optimizer, batch size 128, learning rate 0.0001.

Methodology in Plain English

The approach borrows the skeleton of contrastive learning and swaps in adversarial examples as the "views."

Stage 1 — Adversarial Contrastive Training (ACT). A mini-batch of images is drawn from the dataset. Each image is turned into several variants: augmented versions (random crops, horizontal flips) and adversarially perturbed versions produced by PGD and C&W. The contrastive loss (InfoNCE, with a temperature parameter τ controlling how smooth the probability distribution is) is then computed so that the network is rewarded for placing an image and its perturbed versions close together in feature space, and penalized for placing perturbed versions of different images close together. The final loss averages the contrastive terms computed between the original and the PGD-perturbed image and between the original and the C&W-perturbed image. Similarity between samples can be measured with cosine similarity or Euclidean distance.

Stage 2 — Task-specific Fine-tuning (TF). The trained encoder is extracted and its weights frozen. A single linear layer is added on top and trained for image classification on CIFAR-10. Because only the linear layer's parameters are updated, the robust feature representations learned in stage 1 are preserved; the paper describes this as a filter that prevents adversarial attacks from propagating into downstream tasks and as producing feature representations for perturbed images that resemble those of the original anchors.

Why This Matters

Impact on research. The paper reframes adversarial defense as a representation-learning problem rather than solely a min-max optimization problem. If an encoder learns to map clean and perturbed inputs to the same region of feature space, robustness becomes a property of the backbone and could in principle be transferred to multiple downstream tasks without re-running full adversarial training per task. The paper also argues that label-dependence is a limitation of classical adversarial training, and positions self-supervised contrastive pre-training as an alternative route.

Real-world applications (as framed or implied by the paper):

• Autonomous driving, which the paper names explicitly as a safety-paramount application where susceptibility to tiny perturbations is a major problem.
• Security-critical deployments generally, which the conclusion cites as environments where adversarially trained models could be used.
• Image classification pipelines that need to resist manipulation of their inputs before deployment in practical scenarios.
• Any vision system built on a reusable backbone, where a single adversarially robust encoder could serve multiple downstream tasks.

Industry relevance. The released code, the use of standard architectures (ResNet18/34/50), the CIFAR-10 benchmark, and the two-stage pre-train/fine-tune recipe make the method straightforward to slot into existing transfer-learning workflows. The paper also notes that deeper, higher-capacity backbones respond best — which matters for practitioners choosing model size when robustness is a requirement. However, the authors concede that their approach does not yet match state-of-the-art accuracy from preprocessing, model modification, or ensemble methods, which limits immediate deployment claims.

Future Directions

1. Use adversarial training as a preprocessing step rather than only as a fine-tuning recipe, as the conclusion proposes.

2. Explore image preprocessing techniques to close the accuracy gap with state-of-the-art defenses that use preprocessing or model modifications.

3. Apply label smoothing to further enhance model resilience, as suggested in the conclusion.

4. Investigate ensemble methods for creating robust models suitable for real-time applications.

Additional open questions implied by the results: the paper reports that the contrastive training benefit is weak on ResNet18 (only 10%–15% average improvement) and that FGSM — an unseen attack during training — remains difficult, so scaling the approach to smaller backbones and improving transfer to unseen attack types are both unresolved. The conclusion also notes that smaller batch sizes improved training outcomes, though the experimental settings report a batch size of 512.

Target Audience

Researchers and graduate students working on adversarial robustness, adversarial training, or self-supervised and contrastive representation learning. It is also useful for practitioners building security-sensitive vision systems who want a concrete, code-backed recipe for hardening a ResNet backbone, and for readers interested in how self-supervised objectives can be repurposed beyond standard pre-training. Readers without background in adversarial attacks or contrastive losses will need to consult the cited FGSM, PGD, C&W, and InfoNCE references first.

Authors’ abstract

Deep neural networks (DNNs) have achieved remarkable success in computer vision tasks such as image classification, segmentation, and object detection. However, they are vulnerable to adversarial attacks, which can cause incorrect predictions with small perturbations in input images. Addressing this issue is crucial for deploying robust deep-learning systems. This paper presents a novel approach that utilizes contrastive learning for adversarial defense, a previously unexplored area. Our method leverages the contrastive loss function to enhance the robustness of classification models by training them with both clean and adversarially perturbed images. By optimizing the model's parameters alongside the perturbations, our approach enables the network to learn robust representations that are less susceptible to adversarial attacks. Experimental results show significant improvements in the model's robustness against various types of adversarial perturbations. This suggests that contrastive loss helps extract more informative and resilient features, contributing to the field of adversarial robustness in deep learning. The code is publicly made available on GitHub in the following link: https://github.com/suklav/C_Lead .