Robustness, OOD, Calibration, and Interpretability

Robustness improves when the test names the shift precisely. Four different failures get filed under one complaint — the model did badly on a strange image — and each has its own test, its own owner and its own repair.

Corruption is the family with a fixed vocabulary, and ImageNet-C supplies it. Hendrycks and Dietterich built the benchmark in 2019. “The ImageNet-C benchmark consists of 15 diverse corruption types applied to validation images of ImageNet”, drawn from four groups: noise, blur, weather and digital. “Each corruption type has five levels of severity”. Four further corruptions — speckle noise, spatter, Gaussian blur and saturate — are held back as validation corruptions, so a method tuned on the first fifteen cannot quietly claim the whole benchmark. Image quality degrades; the label is meant to survive.

Domain shift moves the population, the site or the device, and it carries a measured cost. Pneumonia-screening CNNs trained on 158,323 chest radiographs from three hospital systems reached AUC 0.931 (95% CI 0.927–0.936) inside those hospitals. At the external site the best pooled model fell to 0.815 (95% CI 0.745–0.885, P = 0.001). Zech and colleagues reported the pattern in PLOS Medicine in 2018: “Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons”. The mechanism sat in the pixels. From the same paper: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” A network that can read the site off the film can predict disease burden by site. That works right up to the moment the site changes.

Out-of-distribution input is different again. The image falls outside the intended semantic or acquisition scope, so the right output may be a rejection rather than a label. That boundary is a product decision, not a fact about the world.

Adversarial input has an author. In 2018 a team of nine put black and white stickers on a real stop sign, under a method they called Robust Physical Perturbations (RP2). Their abstract: “With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier.” The classifiers attacked were LISA-CNN, 91% accurate on the LISA test set, and GTSRB-CNN, 95.7% on GTSRB. NIST's adversarial machine learning taxonomy files the same work under “Physically realizable attacks”: “The attack is applied to evade a road sign detection classifier by physically applying black and white stickers to the road signs.” Severity levels do not describe that attacker. No amount of blur testing anticipates him.

If predictions assigned 80% confidence are correct about 80% of the time, they are calibrated for that population and event. Calibration can change across classes, sites, thresholds and shifts. A softmax maximum is not automatically the probability that the product got it right.

Modern networks do not arrive calibrated. Guo and colleagues measured that across architectures in 2017, and their finding is that “modern neural networks, unlike those from a decade ago, are poorly calibrated”. The causes are ordinary architecture decisions rather than exotic ones: they report that “depth, width, weight decay, and Batch Normalization are important factors influencing calibration”. Nothing in a training run pushes the confidence number toward honesty on its own.

Then hold that against deployment. Ovadia and colleagues benchmarked the standard predictive-uncertainty methods under dataset shift in 2019 and concluded: “We find that traditional post-hoc calibration does indeed fall short, as do several other previous methods. However, some methods that marginalize over models give surprisingly strong results across a broad spectrum of tasks.” A calibration curve fitted on the validation split is a statement about the validation split. Recheck it under deployment conditions, or do not quote it there.

No single score or detector covers all failures. Robustness is assembled in layers, each with an owner.

Input controls come first: quality checks, sensor validation, format constraints and provenance. It is the cheapest place to stop an image the model was never meant to see. Model evidence is the second layer — task performance, uncertainty, calibration and sensitivity to shift, measured on data that fitted nothing.

The third layer is decision policy, where thresholds, abstention, fallback and rate limits live, and it has a cleared precedent. The FDA authorised IDx-DR on 11 April 2018 as the first autonomous AI diagnostic device. It does not only report more than mild DR (mtmDR) detected or not detected. Insufficient image quality is a third output the device is authorised to give. In the pivotal trial, 900 enrolled participants at 10 primary care sites, 38 of them — 4% — received it. Abstention there is a designed output with a regulatory record behind it, not a cut-off improvised on top of a score.

Human workflow is the fourth layer: review interfaces, escalation, contestation, incident response. It absorbs whatever the third layer declines, so it has to be sized for that traffic before the threshold is set. Monitoring and update closes the loop with drift detection, label collection, retraining gates and rollback.

Use multiple probes and compare them with controlled interventions. Each method answers a narrow question, and the questions do not overlap as much as the pictures suggest.

Tests should connect to ownership and action. For medical devices that connection is not advice — it is published regulatory text. On 27 October 2021 the FDA, Health Canada and the UK's MHRA jointly published ten Guiding Principles for Good Machine Learning Practice for Medical Device Development. Three of them map directly onto the steps below.

Step 1, enumerate shifts: list devices, environments, populations, corruptions, attacks and unknowns, including the classes you intend to exclude.

Step 2, create controlled suites: vary severity while preserving labels where that is justified, in the ImageNet-C manner.

Step 3, test real transfer. Principle 4, “Training Data Sets Are Independent of Test Sets”, states: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” Principle 8 goes on to require that performance be demonstrated “independently of the training data set” under clinically relevant conditions. A held-out random split from the same hospitals satisfies neither.

Step 4, tune selective policies: calibrate confidence, abstention, fallback and review capacity together. Measure the abstained slice, not only its size.

Step 5, connect monitoring. Principle 10 reads: “Deployed models have the capability to be monitored in 'real world' use with a focus on maintained or improved safety and performance”, and it names “dataset drift” among the degradation risks to be controlled. Metrics, label collection, alerts, incident playbooks and rollback are what that capability is made of.

The detector must work across weather, countries, cameras, night scenes, construction zones and intentional sign modifications. Every item below has a published precedent to copy from.