Panda: Test-Time Adaptation with Negative Data Augmentation

AI summary

Overview

Research area: Test-time adaptation (TTA) for pretrained vision-language models (VLMs), specifically robustness to image corruptions.

Technical level: Intermediate. The paper assumes familiarity with CLIP-style contrastive image-text models, embedding spaces, entropy-based adaptation, and standard corruption benchmarks, though the core idea is describable without heavy math.

Scope: The paper proposes and evaluates Panda, a test-time adaptation method that uses negative data augmentation (patch shuffling across a batch) to remove corruption-related components from image embeddings, improving accuracy and reducing prediction bias across three corruption benchmarks and ten baseline methods.

What This Paper Is About

Pretrained VLMs such as CLIP classify images well zero-shot but degrade significantly when images are corrupted (noise, blur, weather, compression). Existing test-time adaptation methods rely on positive data augmentation (PDA), which creates many semantic-preserving views per test image. The authors argue PDA is both expensive (typically K = 63 augmentations per image) and ineffective, because it operates on corrupted inputs and therefore never removes the corruption itself, leaving a persistent prediction bias where corrupted images are disproportionately assigned to certain classes. Panda instead generates class-agnostic negative augmentations shared across a batch, then subtracts their average embedding from each test image embedding to suppress corruption-related signal.

Key Contributions

1. The authors identify two limitations of PDA-based TTA: high inference cost from generating many augmentations per image, and a failure to reduce prediction bias under corruption.
2. They propose Panda, a TTA method that uses negative data augmentation — dividing batch images into patches, shuffling them into a shared pool, and recomposing them — to produce embeddings that retain corruption characteristics but discard object semantics.
3. They introduce an offset operation that subtracts the averaged negative-augmentation embedding (scaled by a hyperparameter β) from each original image embedding, with a theoretical result (Theorem 4.1) showing that offsetting improves classification accuracy and is maximized when β = r.
4. They show empirically that Panda outperforms PDA-based TTA methods and can be plugged into a wide range of existing TTA frameworks (Tent, ETA, SAR, DeYO, TPT, DMN-ZS, Zero, TPS, BAT) to improve their robustness at low computational cost.

Main Findings

• Consistent gains across all baselines and datasets: On CIFAR-10-C with ViT-B/32, Panda raises Tent from 62.8 to 71.1 (+8.3), SAR from 63.3 to 70.7 (+7.4), ETA from 64.9 to 68.3 (+3.4), DeYO from 65.5 to 67.2 (+1.7), TPT from 62.2 to 63.5 (+1.3), DMN-ZS from 61.6 to 63.1 (+1.5), Zero from 63.2 to 64.9 (+1.7), TPS from 63.7 to 65.6 (+1.9), BAT from 65.7 to 68.5 (+2.8), and CLIP alone from 59.0 to 61.6 (+2.6). The reported average improvement across all baselines on CIFAR-10-C is +3.3%.

• Gains hold on harder benchmarks: On CIFAR-100-C (ViT-B/32) the average improvement is +2.2% with a maximum of +4.1% (DeYO, 38.0 to 42.1). On ImageNet-C (ViT-B/16) the average improvement is +2.0% with a maximum of +2.9% (Tent, 25.3 to 28.2).

• Panda beats PDA methods head-to-head: Under identical settings, Panda reaches 71.1 on CIFAR-10-C, 38.4 on CIFAR-100-C, and 28.2 on ImageNet-C, compared with TPT (62.2 / 32.1 / 25.2), Zero (63.2 / 29.5 / 24.6), and TPS (63.7 / 33.1 / 25.1).

• Panda's NDA design beats an existing NDA approach: Replacing DeYO's NDA-based sample selection and weighting with Panda's NDA generation and offset gives 68.3 / 43.3 / 29.4 on CIFAR-10-C / CIFAR-100-C / ImageNet-C, versus 65.5 / 38.0 / 29.0 for the select-and-weight strategy, and 59.0 / 31.8 / 24.5 for CLIP alone.

• Low computational overhead: On ViT-B/32 with CIFAR-10, runtime increases by less than 10% for every baseline: CLIP 17s to 18s (5.9%), Tent 25s to 27s (8.0%), ETA 21s to 23s (9.5%), SAR 31s to 34s (9.7%), DeYO 27s to 28s (3.7%), TPT 22min21s to 22min39s (1.3%), DMN-ZS 22s to 23s (4.5%), Zero 8min51s to 8min55s (0.8%), TPS 9min32s to 9min37s (0.9%), BAT 28s to 30s (7.2%).

• Prediction bias is reduced: Measured as the L1 distance between the ground-truth label distribution and the soft prediction distribution, Panda reduces distributional distance consistently across all 15 corruption types in CIFAR-10-C. Positive data augmentation reduces distance in only 4 corruption domains and increases it to varying degrees in the remaining 11.

• Bias accumulation during adaptation is curbed: On the gaussian noise corruption of CIFAR-10-C, with the 10,000 test samples split into 10 consecutive chunks of 1,000, Tent progressively accumulates prediction bias, improves only slightly at first, then degrades toward model collapse. Tent + Panda maintains lower prediction bias and substantially higher accuracy across the whole stream.

• Robust to hyperparameters: Panda remains stable across different M/B ratios (M is the number of negative augmentations per batch, B the batch size), across a wide range of the offset ratio β, and across varying batch sizes, where Tent + Panda keeps a consistently large gain over Tent.

• Both components matter: Ablations comparing full Panda against (1) disabling Panda, (2) shuffling patches within individual images rather than across the batch, and (3) removing averaging of negative augmentation features show that all ablated variants perform significantly worse across baselines.

Methodology in Plain English

Panda operates only on the forward pass of an existing TTA pipeline, so it can be dropped into methods like Tent without changing their training logic.

For each test batch of B images, the method cuts every image into non-overlapping patches of size Hp × Wp. With a default patch size of 32 × 32 on images resized to 224 × 224, each image yields a 7 × 7 grid of patches. All patches from the batch go into one shared pool, which is shuffled, and M new "negative" images are assembled from that pool. Each patch is used at most once within the batch, and each recomposed image keeps the original H × W size. Because object structure is scrambled while corruption texture survives at the patch level, these images preserve corruption-specific features and discard object-relevant signals.

The original batch and the M negative images are both passed through the image encoder. The M negative embeddings are averaged into a single vector, which acts as a corruption prototype. Each original image embedding is then debiased by subtracting β times that average, with β > 0 controlling the offset strength. The debiased embeddings replace the original embeddings in the logits computation (scaled by 100 before the text-embedding dot product) and in any downstream adaptation such as entropy minimization.

The key efficiency argument is that M is shared across the whole batch rather than generated per image. The paper typically sets M = B/10 (Algorithm 1 uses M = ⌈B/10⌉), so a small number of augmentations serves many samples, in contrast to PDA's K = 63 per image and K + 1 forward-pass cost.

Theorem 4.1 provides the theoretical backing in a 1-D binary classification setting. Modeling the feature as v = v_cls + v_corr with v_cls ~ N(0,1) and v_corr ~ N(0, s²), the unaugmented accuracy is 1/2 + (1/π)·arctan(1/s). After offsetting with a negative feature n that is uncorrelated with v_cls (ρ = 0) but correlated with v_corr (ρ = r > 0), accuracy becomes 1/2 + (1/π)·arctan(1/(s·√(1 − r² + (β − r)²))), which is maximized at β = r. In other words, offsetting can shrink the corruption component down to √(1 − r²) times its original magnitude.

Experiments use CIFAR-10-C, CIFAR-100-C, and ImageNet-C (15 corruption types each) at severity level 5, with ViT-B/32 for the CIFAR benchmarks and ViT-B/16 for ImageNet-C, a batch size of 100, images resized to 224 × 224, and the fixed prompt "a photo of a {class}". Hyperparameters were tuned on the original baselines alone and then held identical for the Panda-integrated versions.

Why This Matters

Research impact: The paper reframes test-time adaptation around removing corruption from the representation rather than averaging it away. Its diagnosis — that PDA preserves corruption and can amplify prediction bias, which is dangerous for entropy-based methods that treat soft predictions as pseudo-labels — is a concrete critique of a dominant TTA design pattern. The theoretical offsetting result gives a principled justification for feature subtraction as a debiasing primitive, and the batch-shared augmentation idea offers a template for TTA methods that are cheap enough to deploy at scale.

Real-world applications:

• Deploying CLIP-style classifiers on camera or sensor feeds where images degrade from noise, blur, fog, frost, or compression, without access to labeled target data.
• Autonomous driving and robotics perception under adverse weather and lighting, where corruption-induced class bias can systematically skew which objects are recognized.
• Medical imaging pipelines where scanning artifacts shift the input distribution away from the pretraining data.
• Content moderation, catalog tagging, or search ranking systems that run vision-language models on large volumes of user-uploaded images of unknown quality and need low-latency adaptation.

Industry relevance: The sub-10% runtime overhead is the central practical selling point. PDA-heavy methods such as TPT, Zero, and TPS take minutes per evaluation (22min21s, 8min51s, and 9min32s respectively in Table 3), whereas Panda adds fractions of a second to baselines that already run in seconds. Because Panda only modifies the forward pass, an engineering team can integrate it into an existing TTA stack rather than replacing it. The released code at https://github.com/ruxideng/Panda lowers the barrier to adoption.

Future Directions

• Determining how the patch size Hp × Wp and the M/B ratio should be selected automatically for new datasets, rather than fixed at 32 × 32 and B/10, since the paper only reports sensitivity analyses rather than a selection rule.
• Testing whether the offsetting idea transfers beyond CLIP-style VLMs to other pretrained backbones and beyond image corruption to natural distribution shifts such as domain or geographic shift.
• Investigating whether a learned or adaptive β, rather than a fixed offset ratio, could improve on the theoretical optimum β = r, which the paper cannot observe directly at test time.
• Clarifying the interaction between Panda's NDA and DeYO's NDA, given that Table 4 replaces DeYO's select-and-weight strategy with Panda's offset and reports different ImageNet-C numbers (29.4) than the DeYO + Panda combination in Table 1 (31.2).

Target Audience

Researchers and graduate students working on test-time adaptation, distribution shift robustness, and vision-language models; practitioners deploying CLIP-style models in production under latency constraints; and engineers looking for a low-overhead drop-in component to stabilize an existing TTA pipeline. Readers without a background in contrastive VLMs or entropy-based adaptation will need to consult the cited CLIP and Tent papers first.

Authors’ abstract

Pretrained VLMs exhibit strong zero-shot classification capabilities, but their predictions degrade significantly under common image corruptions. To improve robustness, many test-time adaptation (TTA) methods adopt positive data augmentation (PDA), which generates multiple views of each test sample to reduce prediction variance. However, these methods suffer from two key limitations. First, it introduces considerable computational overhead due to the large number of augmentations required per image. Second, it fails to mitigate prediction bias, where the model tends to predict certain classes disproportionately under corruption, as PDA operates on corrupted inputs and typically does not remove the corruption itself. To address these challenges, we propose Panda, a novel TTA method based on negative data augmentation (NDA). Unlike positive augmentations that preserve object semantics, Panda generates negative augmentations by disrupting semantic content. It divides images into patches and randomly assembles them from a shared patch pool. These negatively augmented images retain corruption-specific features while discarding object-relevant signals. We then subtract the mean feature of these negative samples from the original image feature, effectively suppressing corruption-related components while preserving class-relevant information. This mitigates prediction bias under distribution shifts. Panda allows augmentation to be shared across samples within a batch, resulting in minimal computational overhead. Panda can be seamlessly integrated into existing test-time adaptation frameworks and substantially improve their robustness. Our experiments indicate that Panda delivers superior performance compared to PDA methods, and a wide range of TTA methods exhibit significantly enhanced performance when integrated with Panda. Our code is available at https://github.com/ruxideng/Panda .