Image Classification Systems

Can a convolutional network tell you which hospital a chest radiograph came from, using nothing but the image? Zech and colleagues asked that in PLOS Medicine in 2018. The answer was yes. The network assigned 99.95% of NIH images to the correct source system — 22,050 of 22,062 — and 99.98% of Mount Sinai images, 8,386 of 8,388.

That cue has a price. Their best pneumonia model, trained on pooled MSH and NIH data, scored AUC 0.931 (95% CI 0.927-0.936) on internal data. At Indiana University, a site absent from training, it scored 0.815 (95% CI 0.745-0.885). P = 0.001.

The case now opens a 2020 survey of shortcut learning in deep neural networks, by Geirhos and colleagues: “Worse yet, a machine classifier successfully detected pneumonia from X-ray scans of a number of hospitals, but its performance was surprisingly low for scans from novel hospitals: The model had unexpectedly learned to identify particular hospital systems with near-perfect accuracy (e.g. by detecting a hospital-specific metal token on the scan, see Figure 1).”

The network solved a stable statistical problem. It did not solve the intended clinical one. Correct labels did not prevent shortcut learning.

A production classifier combines model outputs with rules about uncertainty and action. Steps 3 and 4 are where most of this lesson lives. A score becomes a probability only after someone calibrates it on a named population. It becomes a decision only after someone writes down what happens when the evidence is thin.

These settings need different targets and output policies. The reject option is the one teams treat as an optional nicety. A regulator has already treated it as part of the authorised product.

IDx-DR ships three answers, not two: more than mild DR detected, more than mild DR not detected, or insufficient quality. The FDA authorised it that way in April 2018, as a new Class II device type, under submission DEN180001. The pivotal study ran 900 patients at 10 primary-care sites. Of the 857 participants with a completed reading-centre grading, 38 (4%) received the insufficient-quality output — imageability 96.1% (95% CI 94.0-96.8%). Observed sensitivity was 87.4% (95% CI 81.9%-92.9%) and observed specificity 89.5% (95% CI 86.9%-93.1%).

The figure worth carrying out of the decision summary is the one about the cases the system refused: “In the 38 participants with AI system insufficient image quality, the prevalence of mtmDR was 10/38 (26%), comparable to the mtmDR prevalence in the fully analyzable dataset.” In the analysable set the rate was 23.8%, 198 of 819. So the declined cases were not the low-risk residue that abstention is usually imagined to collect. They carried the disease at the same rate as the cases the system decided. An abstention is a referral with a queue behind it, not a discard.

The sponsor's own report of the same trial gives sensitivity 87.2% (95% CI 81.8-91.2%), specificity 90.7% (95% CI 88.3-92.7%) and imageability 96.1% (95% CI 94.6-97.3%), against pre-specified thresholds of >85% and >82.5%. Near the regulator's numbers, and not identical to them. Which document you are quoting is part of the claim.

Classes should correspond to decisions the system can support from available pixels. If experts need metadata or history to distinguish two labels, an image-only model may face an impossible target.

Class definitions should cover “other,” uncertain, mixed, damaged, and out-of-scope cases. The ontology must also specify whether labels describe objects, scenes, states, or causes.

In 2020 a five-author team went back to the ImageNet labels themselves. They set out to “develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set”, then reassessed recent classifiers with the new labels and “find their gains to be substantially smaller than those reported on the original labels”. The second finding is the structural one, and it is not about any model. The original labels were no longer the best predictors of the independently collected set. A label scheme can stop being the thing worth predicting, and no backbone notices.

Use interventions that alter suspected context while preserving the intended object or attribute. A clinical classifier was put through exactly that test in JAMA Dermatology in August 2019, by Winkler and colleagues.

The edit was one variable on 130 melanocytic lesions: the same lesions imaged with and without standard surgical ink markings. On benign nevi the mean melanoma probability rose from 0.16 (95% CI 0.10-0.22) to 0.54 (P < .001). Specificity fell from 84.1% (95% CI 76.0%-89.8%) to 45.8% (95% CI 36.7%-55.2%). The Key Points box states it plainly: “Skin markings by standard surgical ink markers were associated with a significant reduction in the specificity of a convolutional neural network by increasing the melanoma probability scores, consequently increasing the false-positive rate of benign nevi by approximately 40%.”

The repair step is measured too. Cropping the marking out of the frame restored specificity to 97.2% (95% CI 92.1%-99.0%) and brought the mean score down to 0.03. A counterfactual edit that breaks a model, and reversed, fixes it, is the strongest evidence available about what the model is using. Stronger than any saliency picture over the same lesion.

An independent group at the Medical University of Vienna ran the test in the other direction. They inserted artefacts such as skin markings and rulers into HAM10000 images, and measured AUPRC losses of 0.030 for ResNet-34, 0.045 for Faster R-CNN and 0.011 for Mask R-CNN. The size of the shortcut depends on the architecture. So the intervention has to be run against the model you are actually shipping.

A 0.9 score is meaningful only after specifying the class event, model version, input population, and calibration procedure. Scores can become miscalibrated after shift or threshold changes.

Calibration should be evaluated on held-out data and on relevant slices. Temperature scaling or another post-processing method may help, but it cannot fix missing classes or unsupported evidence.

The problem was named at ICML in 2017, by Guo and three colleagues: “We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated.” They report that “depth, width, weight decay, and Batch Normalization are important factors influencing calibration” — none of which is a knob anyone turns in order to fix confidence. On most datasets “temperature scaling”, which they describe as “a single-parameter variant of Platt Scaling”, is “surprisingly effective at calibrating predictions”. One parameter was enough on most of the datasets they tried. That says the miscalibration was systematic rather than buried deep in the network.

Then the finding moved. Minderer and seven co-authors re-ran the question on newer image classifiers in 2021. They found the most recent models, notably those not using convolutions, among the best calibrated. The previous generation's trends — calibration decaying with model size, and under distribution shift — were less pronounced in recent architectures. A statement about how confident networks are was true of the networks that were measured, and stopped being true four years later. This lesson's own claim, that confidence is conditional on a model version, is demonstrated by the calibration literature reversing itself rather than asserted at you.

Any model review should connect taxonomy, evidence, metrics, and operations. Each item below has a number attached to it somewhere in this lesson, which is the standard to hold a review to.