Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You

AI summary

Overview

Research area: Test-time adaptation (TTA) for computer vision — specifically gradient-free adaptation of frozen neural networks under distribution shift, plus a reliability certificate attached to that adaptation mechanism.

Technical level: Advanced. The method builds on Gaussian discriminant analysis, between-class/within-class scatter, Mahalanobis geometry, and closed-form affine transport of class statistics; the evaluation spans corruption benchmarks, architecture-dependent baseline behavior, and controlled throughput/memory measurements.

Scope: The paper introduces CASTER, an analytic feature-space adaptation method that requires no backward pass, no optimizer state, and no stored source feature bank, and analyzes an empirical "transportability certificate" that decides when that transport should be applied — and shows the certificate does not transfer to gradient-based adaptation such as Tent.

What This Paper Is About

Most test-time adaptation methods assume you can update model parameters at inference time, which is impossible on inference-only accelerators, frozen third-party models, or memory-constrained deployments; on the four LayerNorm backbones tested here, the standard BatchNorm-only Tent and EATA configurations have no eligible parameters at all and reduce exactly to the frozen source model. The authors ask whether adaptation can instead be done analytically in feature space, leaving the learned network completely frozen, by transporting stored source class statistics to each target batch with a single class-shared affine map. Because that transport can be catastrophic (on ImageNet-C, batches contain only 64 samples for 1000 classes), they also derive and stress-test a certificate that decides when the transport is geometrically supported, and they examine what that certificate can and cannot justify.

Key Contributions

1. CASTER, a gradient-free TTA mechanism. It stores per-class moments (mean, covariance, prior) in a discriminative subspace rather than a source feature bank, estimates a shrinkage-stabilized affine transform from target-batch moments, and transports the source class Gaussians before evaluating a Gaussian discriminant. On identical frozen features it outperforms k-NN in 27 of 28 backbone–dataset cells with a median 18× smaller retained state, and it remains applicable on architectures where the BatchNorm-only Tent/EATA configurations evaluated here are inert.

2. A scale-consistent residual-to-margin transportability certificate. The certificate divides the maximum inverse-transported centroid residual by the minimum source-class separation, both measured in the same Mahalanobis metric so the ratio is invariant to invertible linear rescalings of feature coordinates. The paper explicitly separates two claims often conflated: detecting degenerate transport (supported) and ranking adaptation benefit (not supported).

3. Evidence that the certificate is mechanism-specific. Applied to Tent on matched CIFAR-10-C cells, the certificate accepts only 4.3% of updates and preserves 0.6% of Tent's available gain; a more permissive source-statistic referee accepts 46.3% and still keeps only 32.3%. A structural explanation is given: a criterion computed from the pre-update frozen representation cannot in general certify a mechanism that changes that representation.

4. A fail-closed evaluation protocol. All reported aggregates are regenerated from raw logs by a verification script that aborts when a required artifact is missing or inconsistent; runs are admitted only if the checkpoint exists, clean-source accuracy passes a sanity threshold (CIFAR-10 ≥ 85%, CIFAR-100 ≥ 55%), and the source moments came from that exact checkpoint.

Main Findings

• Source-statistic transport beats k-NN on frozen features. Across four ImageNet-pretrained backbones (ViT-B/16, ResNet-50, DeiT-B, Swin-T) and seven datasets (Oxford-IIIT Pets, DTD, Flowers-102, Food-101, CIFAR-10, CIFAR-100, Tiny-ImageNet), CASTER wins 27 of 28 cells. The only exception is ViT-B/16 on CIFAR-10, where the two differ by less than 0.04 top-1 points. Largest margins are on Flowers-102, roughly 18–22 points on three of four backbones.

• Storage scales with classes, not source examples. A k-NN bank grows with the number of source examples and reaches 781 MiB on Tiny-ImageNet; CASTER's state is O(C·k₀²) and independent of the number of source examples. The advantage is not universal: on Flowers-102 only 1020 training images are stored by k-NN, while k₀ = C − 1 = 101 makes CASTER's covariance state comparatively large — the paper reports that inversion explicitly.

• Unconditional transport is unsafe in low-coverage regimes. On ImageNet-C (1000 classes, batch size 64), unconditional transport loses 21.2 top-1 points relative to the frozen head; Source-GDA and the non-transported Gaussian estimator lose 19.0 points and T3A loses 33.5. CASTER's certificate rejects every target batch, so the deployed policy returns the frozen-head prediction and incurs zero additional harm. SAR is the only evaluated adaptive method that improves over the frozen head in this experiment (+4.6 points).

• The certificate detects catastrophe but does not rank benefit. Across 307 admissible evaluation cells, every transport losing more than 10 top-1 points has certificate value above 3.9. Separation is not perfect: at least one non-harmful cell has a certificate as high as 8.9. Within the benign regime the raw score is positively correlated with transport gain, whereas a score meant to rank beneficial updates would need the opposite ordering — so on CIFAR-10-C, where catastrophic transport is absent, thresholding does not beat applying transport to every batch.

• The threshold is not delicate. Pooled over admissible cells, unconditional transport averages −3.35 points relative to the frozen head; gating at the shipped τ = 0.8 gives +1.69 points, and any τ ∈ [1.0, 3.9] gives between +1.85 and +1.92 points. A roughly fourfold threshold interval stays within 0.3 points of the best observed value. τ = 0.8 is kept as a conservative operating point.

• The gate's value is asymmetric. On ResNet-50 across nine datasets, gating costs little where transport helps (0.1 points on Flowers-102, 0.2 on Oxford-IIIT Pets, at most 2.2 on CIFAR-10-C) but recovers most or all of the loss where transport is destructive (+10.2 points versus always transporting on CIFAR-100-C, +21.2 on ImageNet-C). Pooled over the admissible ResNet-50 cells: always transporting 55.70 top-1, never transporting 59.05, certificate gating 60.05.

• Corruption benchmarks split by normalization family. Aggregated over 19 corruptions and 5 severities, on the BatchNorm backbones (ResNet-18/50) gradient methods are very strong: CIFAR-10-C BN gives Tent 85.7 ± 0.60, EATA 85.8 ± 0.60, SAR 85.7 ± 0.60, CASTER 85.7 ± 0.51 against a frozen head of 72.0 ± 0.55; CIFAR-100-C BN gives Tent/EATA/SAR 62.9 ± 0.58 and CASTER 63.4 ± 0.57 against a frozen head of 48.4 ± 1.07. On the LayerNorm backbones (ConvNeXt-T, ViT-B/16, DeiT-B, Swin-T) the standard Tent and EATA configurations reproduce the frozen head exactly, while CASTER reaches 87.4 ± 0.62 on CIFAR-10-C LN (frozen head 86.7 ± 0.82) and 68.1 ± 0.61 on CIFAR-100-C LN (frozen head 65.1 ± 0.61; SAR 66.1 ± 0.85). The authors do not claim a universal advantage over gradient-based TTA.

• The certificate does not transfer across mechanisms. On 57 matched CIFAR-10-C cells with ResNet-50 (19 corruptions × 3 severities), ungated Tent reaches 86.87 (+15.35) with 2.97% harm; with the certificate gate it falls to 71.61 (+0.09), accepting 4.3% of updates and keeping 0.6% of the available gain. The referee gate gives 76.47 (+4.95), accepting 46.3% and keeping 32.3%. EATA shows the same qualitative behavior as Tent, within 0.03 top-1 points. Cell-level certificate acceptance rate is anti-correlated with Tent's ungated gain (ρ = −0.86), whereas for CASTER's own transport the analogous relationship is positive (ρ = +0.72). By contrast, gated CASTER reaches 87.50 (+15.98) with 0.92% harm at 91.0% acceptance.

• The method is cheaper. On a controlled CIFAR-10-C fog/severity-3/ResNet-50 cell, CASTER runs at 1076 images/s versus 667 for Tent, 649 for EATA, and 386 for SAR — 1.6× faster than Tent/EATA and 2.8× faster than SAR. Its 1249 MiB peak is 22.7% of the 5510 MiB used by the gradient-based methods (4.4× less), and it retains 90% of the frozen model's throughput.

• A correction to an earlier protocol. An earlier analysis used fine-grained test streams that were inadvertently class ordered, leaving 97–100% of many batches dominated by one class; after shuffling to the intended i.i.d. stream those cells are benign and the certificate reflects the corrected protocol.

Methodology in Plain English

The authors leave the trained network untouched and do all adaptation outside it. First, from labeled source data, they find a compact set of directions that best separate the classes (from the generalized eigenvectors of between-class and within-class scatter), keeping at most k₀ = min(k_max, C − 1) directions, and for each class they store only a mean, a covariance, and a prior in that subspace. At test time, from each target batch they estimate a single affine map — essentially a whitening-and-recoloring plus a shift — that moves source batch statistics onto target batch statistics, with shrinkage (λ) to stabilize the covariance square roots and a fallback to a diagonal version when the batch is small relative to the subspace size (B < 2k). They then push each source class Gaussian through this map, add a small regularization (β) to the covariances, and classify with the resulting Gaussian discriminant. No gradients, no optimizer state, and no stored source features are needed.

Before trusting that transport, they run a check. They take the target samples the transported classifier is most confident about, map them back into source coordinates, and measure how far each pseudo-class centroid moves relative to the stored source mean. They divide the largest such residual by the smallest separation between any two source class means — using the same Mahalanobis metric in both numerator and denominator, which makes the ratio dimensionless and invariant to rescalings of the feature coordinates. If this ratio exceeds a threshold, or if there are not enough retained samples or classes, the batch is rejected and the frozen model head is used instead. All thresholds are fixed globally rather than tuned per dataset. They also compare against k-NN on exactly the same extracted features, deliberately as a strong nonparametric baseline.

Why This Matters

Impact on research. The paper reframes test-time adaptation for settings where nothing about the model can be changed, showing that a closed-form statistical transport is a viable substitute for gradient updates. It also makes a methodological point that is easy to overlook: averaging TTA results across architectures can mix methods that are genuinely adapting with methods that have no eligible parameters and are silently inert. And it argues that an adaptation "safety" certificate should be reported together with the mechanism it was derived for, since the same statistic that helps one mechanism can nearly disable another.

Real-world applications:

• Deploying a third-party or vendor-supplied model behind an API where weights and gradients are not exposed, but unlabeled target data is available at inference.
• Inference-only accelerators and edge devices without backward-pass support or with tight memory budgets, where optimizer state is not affordable.
• Memory-constrained systems where keeping a source feature bank is infeasible but class moments are acceptable.
• Monitoring and abstaining: systems that need an explicit signal to fall back to the frozen model when the observed batch cannot support the adapted prediction (for example, a 64-sample batch spanning 1000 classes).

Industry relevance. The speed and memory profile (1076 images/s, 1249 MiB, 90% of frozen-head throughput, 4.4× less peak memory than gradient methods on the reported cell) targets production inference constraints rather than benchmark-only settings. The paper also offers a concrete architectural observation for practitioner benchmarking: the standard Tent/EATA implementations evaluated here reproduce the frozen source model exactly on the four LayerNorm backbones, so reported LayerNorm TTA numbers for those configurations may describe inert rather than active adaptation.

Future Directions

• Certificates derived per mechanism. The authors state that certificates for other adaptation mechanisms should be derived and validated against the mechanisms they are intended to gate; constructing a post-update or representation-aware criterion that can rank benefit for feature-updating methods is an open problem.
• Benefit ranking rather than catastrophe detection. The raw residual-to-margin score orders gain in the wrong direction for a utility gate. Finding a statistic that separates benign and catastrophic regimes and ranks how much a non-degenerate adaptation will help is not solved here.
• Beyond the class-shared affine assumption. The method assumes a single affine shift shared across classes; the paper identifies strongly class-dependent shifts, and small or class-imbalanced target batches, as failure regimes that remain unaddressed by a formal guarantee.
• Class composition versus distribution shift. When only a small subset of classes is present in a batch, variation in class composition can be confounded with distribution shift; disentangling the two is raised but not resolved.

Target Audience

Researchers and engineers working on test-time adaptation, domain shift robustness, and deployment-constrained inference — particularly those who cannot modify model parameters or run backward passes. It is also valuable for practitioners benchmarking TTA across modern architectures, because of its analysis of normalization-dependent baseline behavior across BatchNorm and LayerNorm backbones, and for anyone designing or auditing selective-adaptation and risk-monitoring criteria, given the paper's emphasis on stating precisely what a certificate does and does not justify.

Authors’ abstract

Test-time adaptation (TTA) typically assumes that model parameters can be updated at inference time. This assumption is restrictive for inference-only accelerators, frozen or third-party models, and memory-constrained deployments, and standard BatchNorm-based TTA configurations may also become inactive on architectures without BatchNorm. We study adaptation when the learned model must remain frozen. We introduce CASTER, a gradient-free method that stores source class statistics in a discriminative subspace, estimates a class-shared affine transformation from target-batch moments, and analytically transports the source class distributions before classification. CASTER requires no backward pass, optimizer state, or stored source feature bank. Across four backbones and seven datasets, it outperforms k-NN on identical frozen features in 27 of 28 backbone-dataset settings while retaining a median of 18x less state. Affine transport is not always reliable. On ImageNet-C, where batches contain only 64 samples for 1000 classes, unconditional transport loses 21.2 top-1 points. We therefore introduce an empirical residual-to-margin transportability certificate. Across 307 evaluation cells, every transport losing more than 10 points has certificate value above 3.9, although benign and destructive regimes are not perfectly separated. Gating converts an average $-3.35$-point effect of unconditional transport into a +1.69-point gain, and performance remains within 0.3 points of the best threshold over a broad threshold range. Finally, we show that this certificate is mechanism-specific: when applied to Tent, it accepts only $4.3\%$ of updates and preserves 0.6% of Tent's available gain. These results position CASTER as a lightweight adaptation mechanism for frozen-model deployment, together with an explicit account of when its safety signal is informative and when it is not.