Robustness, Stress Tests, and Out-of-Distribution Evaluation

A vision model may tolerate Gaussian noise and still fail under a different camera, a different compression pipeline, or a different season. Another model may resist one adversarial attack while staying wide open to a changed threat model. Every robustness claim needs a specified perturbation, environment, severity, and success criterion.

Corruption robustness and adversarial robustness are two different axes. Someone has measured how separately they move. ImageNet-C scores classifiers on the corruptions ordinary images arrive with: a benchmark that “standardizes and expands the corruption robustness topic”. A companion dataset, ImageNet-P, covers “robustness to common perturbations”. Hendrycks and Dietterich built both in 2019. The suite “evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations”. Across it they report “negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers”. AlexNet to ResNet is years of accuracy gains. The two axes moved separately.

Medicine supplies the measured version of “a different camera”. Pneumonia-screening CNNs were trained on 158,323 chest radiographs from three hospital systems. Then they were tested at a site whose images the networks had never seen. Inside its own hospitals the jointly trained model scored an AUC of 0.931 (95% CI 0.927–0.936). At the independent one it scored 0.815 (95% CI 0.745–0.885, P = 0.001). Zech and colleagues put the pattern in one line: “In 3 / 5 natural comparisons, performance on chest x-rays from outside hospitals was significantly lower”.

The mechanism is the part worth keeping. The same networks, the paper reports, “robustly identified hospital system and department within a hospital”. Part of what the model had learned was where the x-ray came from. That feature is perfectly predictive inside the tested envelope and worthless one hospital over.

Different shifts require different tests to be built. Each row below has a measured instance behind it rather than a category name.

Some shifts can only be found in the field. WILDS curates 10 datasets of shifts that occur naturally in deployment: across hospitals, camera traps, and time and location in satellite imagery. Koh, Sagawa and colleagues published it in 2021. The result was the same on every one of the ten. “On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice.” The methods built to close the gap did not close it.

Domain shift has a price list. Epic's proprietary Sepsis Model was validated from outside on 38,455 hospitalisations of 27,697 patients at Michigan Medicine, between December 2018 and October 2019. Epic had reported an AUC of 0.76–0.83. The model scored 0.63 (95% CI 0.62–0.64). It alerted on 18% of all hospitalisations. It never identified 1,709 of 2,552 septic patients — 67% of them. Wong and colleagues wrote the conclusion: “The ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.”

A second group repeated the exercise at a different institution. The same model was measured across 145,885 encounters at two Harris Health county emergency departments in 2023. Sensitivity within a six-hour window was 14.7%. Those findings, Ostermayer and his co-authors wrote, “align with previously published external validations for the ESPMv1 and slightly worse than Epic's published analysis”. One deployed model, two independent sites, two collapses.

Both are useful when claims stay bounded, and the boundaries have themselves been measured.

Hardening against a synthetic corruption does not transfer to the real world. That has been measured at scale. In 2020 Taori and colleagues ran 204 ImageNet models through 213 different test conditions, putting both kinds of test on the same models: “Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift.” The main exception they found was training on larger and more diverse data. A model that survives added blur has not thereby been shown to survive a new site.

The adversarial column has a measured limit of its own. Nine non-certified white-box defenses were published at ICLR 2018, each already evaluated by its own authors. Athalye and colleagues re-attacked them without changing what the attacker was allowed to do: “In a case study, examining non-certified white-box-secure defenses at ICLR 2018, we find obfuscated gradients are a common occurrence, with 7 of 9 defenses relying on obfuscated gradients. Our new attacks successfully circumvent 6 completely, and 1 partially, in the original threat model each paper considers.” The threat model did not move. Only the quality of the search did.

Word error rate swept against signal-to-noise ratio, for Whisper and 14 LibriSpeech-trained models, under white noise and pub noise: the curves cross. Radford and colleagues ran that sweep in 2023. Their appendix on “Robustness to Additive Noise” reports: “There are many models that outperform our zero-shot performance under low noise (40 dB SNR), which is unsurprising given those models are trained primarily on LibriSpeech, but all models quickly degrade as the noise becomes more intensive, performing worse than the Whisper model under additive pub noise of SNR below 10 dB.”

A single endpoint would have picked the wrong system. On LibriSpeech test-clean, wav2vec 2.0 Large without a language model and Whisper Large V2 tie at 2.7 WER. In distribution they are indistinguishable. Across the other datasets of the paper's Table 1 the zero-shot model averages 12.8 WER against 29.3, a 55.2% average relative error reduction. The tie is the control. The curve is the finding.

Real incidents combine factors: an older device, low light, network delay, and a new user population. Testing one variable at a time may miss the interactions.

On 2 October 2023 in San Francisco, another vehicle's collision left a pedestrian low on the ground in the path of a Cruise driverless vehicle. No single-factor test had held that compound. Cruise filed a recall report with NHTSA on 7 November 2023, covering 950 driverless units. Under “Description of the Defect” it says what the Collision Detection Subsystem then did: “The Cruise ADS inaccurately characterized the collision as a lateral collision and commanded the AV to attempt to pull over out of traffic, pulling the individual forward, rather than remaining stationary.”

Two details in that filing are the lesson. Cruise's own review of the scenario took about three weeks “due to the rarity of the circumstances”. The rarity that delays an investigation is the same rarity that kept the case out of the test suite. And the regulator had already acted. On 24 October 2023 the California Department of Motor Vehicles suspended Cruise's deployment and driverless testing permits, on the ground that “the Department determines the manufacturer's vehicles are not safe for the public's operation”.

Use risk-based combination tests, and avoid a combinatorial explosion by prioritizing credible high-consequence scenarios. Record which combinations remain unevaluated. That list is where the recall gets written.

Link each test to a mechanism, a severity, and an action. Every failure above was found by someone who ran a test the builders had not run: a new hospital, a rebuilt test set, a stronger attack, a rarer collision.