Multiclass, Ordinal, and Thresholded Decisions

The output encoding should reflect whether classes are exclusive, ordered, overlapping, or deferrable.

That structure is usually inherited from an editorial rule rather than chosen at modelling time. RCV1 is the Reuters newswire benchmark NIST has distributed since autumn 2004. It holds about 810,000 English-language stories dated 20 August 1996 to 19 August 1997. The people who coded them worked to a written policy: at least one Topic code and one Region code per story, and no cap at all on how many codes a story could carry. The corpus paper reports 804,414 documents in the corrected RCV1-v2 and 103 assignable Topic codes. Of the Hierarchy Policy it says: “In contrast to some coding systems, there was no limit on the number of codes with the same parent that could be applied.”

So the “Document topics” column below is multilabel for a documented reason. A human coding policy declined to bound the number of labels per story. And 103 codes over roughly 810,000 stories is the shape that policy produced. Read the labelling rule before choosing the output head.

A multinomial model assigns one score per class and normalizes them together, so probabilities sum to one. Changing evidence for one class affects the relative probabilities of all classes.

One-vs-rest trains separate binary problems. The approach can be practical, but its raw outputs are not automatically a coherent multiclass distribution.

Whether that incoherence costs accuracy is contested, not settled. In 2004 Rifkin and Klautau published a defence of one-vs-all classification, and its abstract opens by picking a fight: “Our main thesis is that a simple “one-vs-all” scheme is as accurate as any other approach, assuming that the underlying binary classifiers are well-tuned regularized classifiers such as support vector machines. This thesis is interesting in that it disagrees with a large body of recent published work on multiclass classification.” They name three earlier papers their results disagree with. One is Hsu and Lin's 2002 comparison of five multiclass SVM schemes, whose abstract concludes the opposite: “Our experiments indicate that the ‘one-against-one’ and DAG methods are more suitable for practical use than the other methods.”

Two peer-reviewed head-to-head comparisons, opposite verdicts, the later one naming the earlier as wrong. Accuracy will not settle the decomposition choice for you. The probability structure will. Decide first whether downstream code needs one coherent distribution over classes. That requirement holds whichever paper turns out to be right about the error rates.

Treating grades as integers imposes equal spacing. The label design may not support it.

One ordered vocabulary has been written into federal law, which makes it easy to inspect. Every mammography report must carry one of seven named overall final assessment categories: “Negative”, “Benign”, “Probably Benign”, “Suspicious”, “Highly Suggestive of Malignancy”, “Known Biopsy-Proven Malignancy”, “Post-Procedure Mammogram for Marker Placement”. The list is 21 CFR 900.12, as amended by FDA's Mammography Quality Standards Act final rule of 10 March 2023 and enforced from 10 September 2024. The rank is unmistakable. The spacing is nowhere defined. Nothing in the rule says the step from “Probably Benign” to “Suspicious” equals the step from “Suspicious” to “Highly Suggestive of Malignancy”, and no arithmetic on the seven positions is authorised by the text.

The rule also makes abstention a first-class output rather than a modelling preference. Where no final assessment can be assigned, two Incomplete statements are supplied instead: “In cases where no final assessment category can be assigned due to incomplete work-up, one of the following classification statements shall be assigned as an assessment and reasons why no final assessment can be made shall be stated by the interpreting physician.” When the reason was that prior mammograms were needed, a follow-up report carrying a real assessment is required within 30 calendar days. The cut points, the ordered categories and the deferral route are all decisions someone made explicitly and wrote down.

A convenient integer or one-hot encoding can silently introduce the wrong assumptions.

The benchmarks practitioners train on have often settled these questions in advance, in writing. CheXpert states its own structure in its 2019 abstract: “We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation.” MIMIC-CXR, published the same year, holds 377,110 images from 227,835 studies of 65,379 patients over 2011-2016. It uses the same 14-finding scheme with an added uncertain category. Fourteen observations recorded per image, plus an explicit uncertainty state, is not one winner among fourteen.

The policy should preserve the target structure and allow operational constraints.

Steps 4 and 5 have a regulator-reviewed worked example. IDx-DR is a classifier licensed to abstain, and FDA's De Novo decision summary of 12 January 2018 records three possible outputs rather than two: “During the study, the IDx-DR operator made sure that for every participant, a final IDx-DR output of more than mild DR (mtmDR) detected (defined below), more than mild DR not detected, or insufficient quality, was obtained.”

The numbers behind those outputs are reported next to each other. The pivotal trial enrolled 900 participants at 10 sites, of whom 892 completed all procedures. Of those, 40 exams could not be graded by the reading centre and 33 received an insufficient-quality output. That leaves 819 fully analyzable, an analyzable fraction of 92% (95% CI 90%-93%). On that accepted set the device showed observed sensitivity 87.4% (95% CI, 81.9% - 92.9%) and observed specificity 89.5% (95% CI, 86.9%-93.1%). The reference standard was ETDRS level 35 or higher and/or DME.

Read those two accuracy figures with the coverage figure attached. They describe only the 92% of exams the system agreed to answer. The abstention rate is 33 of 892, and it is printed rather than absorbed. That is what steps 4 and 5 look like when someone has to defend them: an acceptance rule with a named third output, and a coverage number reported alongside the accuracy it makes possible.

Different classes can have different prevalence, review cost, and harm, and a single maximum-probability rule may overload one queue or under-detect a rare class.

One fixed cutoff does not treat everyone alike, and NIST measured by how much. NISTIR 8280, its 2019 study of demographic effects in face recognition, ran 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. Across demographics, false positive rates often vary by factors of 10 to beyond 100 times, while false negatives vary often by factors below 3. One error type, one threshold, a spread of two orders of magnitude — and the error type that spreads is not the one most reports look at. The executive summary says so directly: “As most systems are configured with a fixed threshold, it is necessary to report both false negative and false positive rates for each demographic group at that threshold. This is rarely done - most reports are concerned only with false negatives.” A GAO review published on 22 April 2024 went over the literature independently. False-positive performance differentials have decreased, it found, but differences remain.

Class-specific thresholds and capacity constraints should be selected on held-out evidence. Monitor them as prevalence changes, and monitor both error rates per group, not the convenient one.

The arithmetic behind that choice was published in 2001, by Charles Elkan. His rule is that “an example should be predicted to have the class that leads to the lowest expected cost, where the expectation is computed using the conditional probability of each class given the example”. Compute that expectation from the cost matrix and the estimated class probabilities and you get “some target threshold p* that in general is different from 0.5”. His Theorem 1 offers the rebalancing route: “To make a target probability threshold p* correspond to a given probability threshold p, the number of negative examples in the training set should be multiplied by …”. His own recommendation runs the other way — “to learn a classifier from the training set as given, and then to compute optimal decisions explicitly using the probability estimates given by the classifier”.

Multiclass confusion matrices expose substitution patterns. Ordinal problems may use weighted errors that penalize distant mistakes more strongly. Multilabel tasks need per-label and example-level measures: 103 Topic codes in RCV1-v2, or 14 observations per chest radiograph in CheXpert and MIMIC-CXR, cannot be summarised by a single winner-take-all score.

Report aggregate results with class prevalence, slice behavior, and coverage under abstention. Two reporting habits above are worth copying literally. IDx-DR's sensitivity of 87.4% travels with the analyzable fraction of 92% that defines it. And NISTIR 8280 insists on “both false negative and false positive rates for each demographic group at that threshold”, because reporting one of the two hid a spread of 10 to beyond 100 times in the other.