Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation

AI summary

Overview

• Research area: Machine learning systems, specifically the evaluation of online test-time adaptation (TTA) under real-world latency constraints.
• Technical level: Advanced (the framework is accessible, but the paper presupposes familiarity with TTA, domain shift, and system latency modelling).
• Scope: The paper introduces Tempora, a framework of temporal scenarios, evaluation protocols, and time-contingent utility metrics for measuring whether TTA methods deliver usable predictions under deployment pressure rather than assuming unbounded processing time.

What This Paper Is About

Conventional TTA benchmarks report mean accuracy while quietly assuming a model has unlimited time to adapt before the next input arrives. In real deployments, a correct prediction that lands after a user closes an app or a robot commits to an action is worthless. The paper argues that accuracy and timeliness must be evaluated jointly, and builds a framework to do so across three distinct temporal regimes.

Key Contributions

1. Tempora framework. A three-part structure — temporal scenarios (deployment contexts), evaluation protocols (measurement procedures), and time-contingent utility metrics (accuracy-latency quantification) — grounded in physical units (milliseconds) rather than model-relative FLOPs or latency proxies used by prior work such as Alfarra et al. (2024).

2. Three utility archetypes. ❶ Discrete utility for asynchronous streams with hard deadlines, ❷ continuous utility for interactive settings where value decays hyperbolically with latency, and ❸ amortised utility for budget-constrained deployments where only total adaptation overhead is capped.

3. Decomposable utility diagnostics. Each metric is split into interpretable components (availability α versus served accuracy ā_served; mean accuracy ā versus responsiveness κ̄; adapted versus frozen accuracy), letting the authors isolate why rankings change, not just whether they do.

4. Empirical demonstration of rank instability. Nine Fully TTA methods evaluated across 16 temporal scenarios on 15 corruptions of ImageNet-C with ResNet-50, with an extended suite (11 methods total, adding SPA and ZeroSIAM) on ViT-B/16 across ImageNet-C/R/V2 in the appendix.

Main Findings

• Rank instability persists. Offline rankings do not predict rankings under temporal pressure. The abstract reports this across 750+ temporal evaluations spanning diverse datasets, models, and hardware platforms; the main ImageNet-C/ResNet-50 study analyses 240 temporal evaluations.

• The offline winner loses under pressure. CMF is the majority offline winner (first across 10 of 15 corruptions) yet achieves first rank in only 29 of 240 temporal evaluations (12.1%). It loses in 211 cases (87.9%), yields 15% utility on average to the winner, and underperforms standard inference in 79 cases (32.9%).

• A different method becomes the temporal winner. ETA, which wins none of the offline corruptions, is the majority temporal winner with 103 of 240 evaluations (42.9%). It mainly wins in mildly relaxed settings (35% ≤ ρ ≤ 70%, T ≥ 100 ms, 8 ≤ B ≤ 16 s).

• Extrinsic overhead dominates. Per-batch latency decomposition shows intrinsic overhead (modified forward passes) is negligible, ranging 0–1.2 ms across methods, while extrinsic overhead from backpropagation runs 56–154 ms for gradient-based methods, producing 2.4× to 4.9× slowdowns. CMF, the offline leader, reaches a 4.01× slowdown and SAR 4.89×.

• Instability tracks corruption type. At T = 50 ms, Spearman correlation with offline ranking ranges from r_s = −0.75 (brightness) to r_s = 0.12 (impulse noise). At ρ = 100%, it ranges from r_s = −0.71 (frost) to r_s = 0.66 (contrast).

• Relaxing pressure restores offline rankings. Under discrete evaluation, mean correlation rises from r_s = −0.19 at ρ = 100% to r_s = 0.97 at ρ = 25%. Under amortised evaluation, it rises from r_s = −0.51 at B = 2 s to r_s = 0.44 at B = 4 s.

• Availability ceiling causes failure mode one ("computational insolvency"). At ρ = 100%, ETA reaches 45.6% served accuracy but serves only 41% of batches, giving 18.7% utility; AdaBN's lower 31.7% served accuracy with 97.2% availability yields 30.8% utility, a 12.1% utility gain. To match AdaBN, ETA would need 75.1% served accuracy and SAR 149.5%.

• Multiplicative latency penalties cause failure mode two. At T = 50 ms, ETA's 48.4% accuracy is discounted by 85% to 7.2% utility, while AdaBN's 90% responsiveness preserves most of 31.7% accuracy for 28.4% utility. ETA would require 189% accuracy to close the gap. On brightness at T = 50 ms, where baseline accuracy is already 59.1%, standard inference outperforms all gradient-based methods.

• Harmful adaptation causes failure mode three. At B = 1 s, gradient-based methods exhaust their budget within the first ~20 batches and freeze for the remaining 97%. All but one collapse to 0.1% frozen accuracy, dragging utility below 1%, roughly 20× worse than standard inference.

• SHOT-IM is the outlier. Despite higher per-batch overhead than ETA (79.8 ms versus 56.6 ms), SHOT-IM maintains 32.2% frozen accuracy matching its adapted accuracy, the highest utility among gradient-based methods. The paper attributes this to retention of source running statistics.

• No method dominates. The highest-utility method varies with the shift and the temporal pressure, and columns in the analysis show no consistent winner within any single scenario.

Methodology in Plain English

The researchers treat inference as a stream of batches arriving over time and build a wall-clock simulator around a single-model pipeline. For each batch they measure two kinds of overhead: intrinsic overhead that delays the prediction itself, and extrinsic overhead that stalls the pipeline before the next batch can be picked up. They then apply three different rules for how the environment interacts with that pipeline.

In the discrete setting, batches arrive at fixed intervals (an external rate the model cannot control) and any batch arriving while the pipeline is busy is lost, so utility equals the fraction of batches served multiplied by the accuracy on those batches. In the continuous setting, a "greedy user" sends the next batch as soon as the previous prediction returns, so nothing is skipped but late predictions lose value according to a hyperbolic discount that halves at a human-computer interaction threshold T. In the amortised setting, the model adapts freely until a cumulative overhead budget B is exhausted, after which it freezes and inference continues without updates.

They test nine Fully TTA methods — AdaBN, LAME, NEO, Tent, ETA, SHOT-IM, CMF, DeYO, and SAR (the first three gradient-free, the last six gradient-based) — plus standard non-adaptive inference as a baseline. Evaluation is episodic on ImageNet-C at severity level 5, with methods reset between corruptions and batches sampled i.i.d. within each, using an ImageNet-pretrained ResNet-50, batch size 64, on an Nvidia

Authors’ abstract

Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples. This flexibility suits real deployments, yet conventional evaluations unrealistically assume unbounded processing time, overlooking the accuracy-latency trade-off. As ML increasingly underpins latency-sensitive and user-facing use-cases, temporal pressure constrains the viability of adaptable inference; predictions arriving too late to act on are futile. We introduce Tempora, a framework for evaluating TTA under this pressure. It consists of temporal scenarios that model deployment constraints, evaluation protocols that operationalise measurement, and time-contingent utility metrics that quantify the accuracy-latency trade-off. We instantiate the framework with three such metrics: (1) discrete utility for asynchronous streams with hard deadlines, (2) continuous utility for interactive settings where value decays with latency, and (3) amortised utility for budget-constrained deployments. By applying Tempora to 11 TTA methods, we find that rank instability persists across 750+ temporal evaluations spanning diverse datasets, models, and hardware platforms; i.e., conventional rankings do not predict rankings under temporal pressure. The highest-utility method varies with the shift and temporal pressure, with no clear winner. By enabling systematic evaluation across diverse temporal constraints for the first time, Tempora reveals when and why rankings change, offering practitioners a lens for method selection and researchers a target for deployable adaptation. Code: https://github.com/sudotensor/tempora.