Classification: Categories, Scores, and Decisions

The categories can relate to one another in different ways, changing targets, losses, metrics, and interfaces.

The multilabel case has a public benchmark at scale. CheXpert, released in 2019, is described by its authors in two sentences of the abstract: “We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation.” Fourteen observations, any number of which can be true of one image. Each one is recorded as positive, negative or uncertain rather than forced to a binary. The label set itself carries an abstention value. A separate team confirmed both the counts and that extra category in Scientific Data later the same year.

The ordinal case has a national example, with an abstention rule written into it. Ofqual's 2020 standardisation model produced an ordered calculated grade for every A level candidate in England. Of 718,276 centre assessment grades it adjusted 39.1% downwards, left 58.7% unchanged and raised 2.2%. Where the evidence was too thin it declined to standardise at all: “students in small cohorts received their CAG, since the statistical evidence was not sufficiently reliable to allow standardisation”. On 17 August 2020 the policy was withdrawn and every candidate received the higher of the two grades. An ordered target, an explicit insufficient-evidence branch, and a live demonstration that ordinal decisions are contested on the ladder, not on a single cut.

The binary case is the narrowest of the four, and its whole quality lives in one bit. The Copenhagen model was more sensitive than the dispatchers, 85.0% against 77.5%, and far less precise: positive predictive value 17.8% against 55.8%. Both numbers are properties of that single arrest-or-not output. Neither is visible in the other.

Many classifiers produce scores for each class. A product then applies thresholds, capacity limits, policies, and fallback rules to convert those scores into action.

A score between zero and one is not automatically a calibrated probability. And even a calibrated probability does not dictate the correct threshold, because consequences and resources differ by setting.

That went to a large-scale test in 2019. Ovadia and eight co-authors took image classifiers on MNIST, CIFAR-10 and ImageNet. They added an LSTM on the 20 Newsgroups text corpus. They also modelled the 37-million-example Criteo ad-click dataset. Each model was given temperature scaling, the standard post-hoc fix, which rescales a model's scores using a held-out validation set. Then they fed the models progressively corrupted inputs. Their finding: “while calibrating on the validation set leads to well-calibrated predictions on the test set, it does not guarantee calibration on shifted data.” On inputs entirely outside the training distribution, most methods still returned low-entropy, high-confidence answers. In the paper's own phrase, they are “confidently wrong about completely OOD data.” The score kept its shape after the world stopped matching it.

One threshold does not yield one error rate either. NIST's face recognition vendor test ran 189 mostly commercial algorithms from 99 developers over 18.27 million images of 8.49 million people, and published in December 2019. Holding the operating threshold fixed, the executive summary reports: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” False positives were highest for West and East African and East Asian faces in application photographs, and highest for American Indians in domestic mugshots. One score, one threshold, and error rates two orders of magnitude apart depending on who was standing in front of the camera. Slice-level reporting is therefore part of setting the threshold, not a fairness appendix written afterwards.

The two-threshold design is not a teaching device. A commercial risk-prediction algorithm running on a live patient population has exactly two cuts on its score. Patients above the 97th percentile are automatically identified for enrolment in the health system's care management programme. Patients above the 55th percentile are referred to their primary care physician, who is given contextual data and asked to consider whether the patient would benefit from enrolment. Below that, nothing happens. Obermeyer and three colleagues dissected the whole arrangement in Science on 25 October 2019.

Both cuts sit on one score, so a defect in the score is inherited by both. At the 97th-percentile threshold, Black patients carried 26.3% more chronic illnesses than White patients — 4.8 versus 3.8 distinct conditions. The automatic branch was demanding more sickness from some patients than from others to fire at all. “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” On the same day the paper appeared, the New York State Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group about the Impact Pro algorithm. A threshold is a policy, and a policy can be asked to justify itself.

The low-threshold end of that trade has its own measured case. The Copenhagen model sat there: 85.0% sensitivity bought at a positive predictive value of 17.8%, which is roughly five alerts that were not arrests for every one that was. A two-threshold policy exists to avoid paying that price uniformly. It spends human capacity selectively and separates uncertainty from danger. What it cannot do is escape a distortion in the score both of its cuts are drawn on.

The same word—classification—covers several distinct product designs, and the difference is visible in the shape of the label, not in the model.

A useful classification specification contains more than class names, and Michigan has measured what the missing parts cost.

The state's MiDAS system decided unemployment fraud with no human in the loop. The US Court of Appeals for the Sixth Circuit put it plainly in Cahoo v. SAS Analytics: “From October 2013 to August 2015, MiDAS exclusively determined whether claimants engaged in fraud—no human being took part in this process”. The same opinion records that “the Michigan Auditor General reviewed over 22,000 of MiDAS' fraud determinations and found that 93% of them did not actually involve fraud”.

The agency then reviewed 62,784 penalty cases itself, and announced the result on 11 August 2017. “Of those cases, 40,195 were originally resolved by way of computer program based on available information. As part of the review, 85 percent of these original fraud findings were reversed.” The other 22,589 cases had been flagged by the computer and then examined by a human investigator, and 44% of those were reversed. One classifier, two review paths, and the reversal rate roughly doubled on the path where nobody looked. That is the price of steps 2 to 4 below, written in reversals rather than in principle.

Two of the six steps have their own public numbers. Protecting unknowns is what IDx-DR does when it returns no grade for an image below quality, and what Ofqual did when it left the smallest cohorts unstandardised. Testing by slice is what the NIST face test did at one fixed threshold, where false positive rates varied across demographic groups by factors of 10 to beyond 100. A single threshold that is several different policies depending on who walks up to it.

A classifier can identify patterns associated with a category while remaining wrong about cause, intent, or mechanism. An image label does not explain why a defect formed. A risk class does not establish why a person will act. The commercial risk algorithm above sorted patients accurately on the quantity it was fitted to, and that quantity was not the one the care programme existed to act on.

Claims must match the evidence. The target and the evaluation are what set the level, and classification claims should not exceed it. Causal questions require different evidence, developed later in this path.