Accuracy, Balanced Accuracy, and Averaging

Sepsis occurred in 2,552 of 38,455 hospitalizations, across 27,697 patients. That is 7%. The figure comes from an external validation of the Epic Sepsis Model, published by Wong and colleagues in JAMA Internal Medicine in 2021. A base rate that low settles what a bare accuracy figure can be worth here, before anyone fits a model.

Accuracy gives equal weight to each evaluated observation. It does not give equal weight to each class, person, site, or type of consequence. When classes are balanced and errors have similar costs, accuracy can be clear and useful. Under imbalance or asymmetric stakes, it may mostly describe the dominant class.

Two teams once agreed on the aggregate and disagreed about everything underneath it. ProPublica analysed COMPAS scores for Broward County defendants in May 2016 and found the tool correctly predicted recidivism 61 percent of the time overall. The errors underneath that number were not alike. “In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways,” the reporters wrote. Among defendants who did not re-offend, 44.9% of black defendants had been labelled higher risk, against 23.5% of white defendants. Among those who did re-offend, 47.7% of white defendants had been labelled lower risk, against 28.0% of black defendants. A rejoinder in Federal Probation re-analysed the same data later that year and reported AUCs of .71 overall, .69 for white and .70 for black defendants, with no significant difference by race. Both readings are arithmetically correct. The aggregate is level across the two groups and the composition of the errors is not. Only one of those two facts fits inside a single number.

Balanced accuracy has rivals, and they do not always agree. A 2021 paper in BioData Mining set it beside the Matthews correlation coefficient, along with bookmaker informedness and markedness, and found cases where the measures disagree. A coefficient close to +1, the authors note, “means having high values for all the other confusion matrix metrics.” “The same cannot be said for balanced accuracy,” they write. They also describe “three exceptions where BM can be more appropriate”, the first of them “analyzing classifications where dataset prevalence is unrepresentative”. High on one summary measure is not high on all of them.

Aggregation choices encode different priorities, and which one you used is a reporting fact rather than a matter of taste. An international standard treats it that way. ISO/IEC TS 4213:2022, on assessing machine learning classification performance, gives averaging a normative clause of its own: 6.4.3, “Macro-average, weighted-average and micro-average”. Its introduction says why a reader needs that clause at all: “If a classifier is susceptible to majority class classification, grossly unbalanced training data can overstate accuracy by representing the prior probabilities of the majority class.” Three of the four policies below are named and numbered in a published international standard. Report a bare average without saying which one produced it and you have dropped something the standard counts as part of the result.

It asks whether each reference class is recognized, not whether every row is equally common. Compute the recall for each class against that class's own support. Average the recalls without using prevalence as the weight. Report the class counts and the uncertainty beside the average. Then look at the confusion matrix, because the same balanced figure can sit on top of very different error patterns.

Macro averaging across classes does not guarantee fairness across people, sites, or time. A model can recognize each label adequately overall while failing one hospital or one language.

The largest published demonstration of this is a government evaluation. NIST ran 18.27 million images of 8.49 million people through 189 algorithms from 99 developers and published the result in December 2019 as Face Recognition Vendor Test Part 3: Demographic Effects. The finding in the executive summary is stated as a range, not as an average: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” One class of error moved by up to two orders of magnitude between groups, inside evaluations that also produce a single headline number. The U.S. Government Accountability Office restated the result in the plainest possible terms: white males had the lowest false positive rate and black females the highest.

Use class averages for class questions and slice reports for population questions. Keep the aggregation dimension explicit in charts and tables.

Write the weighting question in plain language, then answer it in five moves. Name the atomic event: decide whether one row, one label decision, one class or one user receives equal weight. Inspect prevalence, measuring both the support in the test sample and the expected deployment mix — the 7% sepsis rate at Michigan Medicine and the emergency-department rate are different problems for the same model. Match the harm model, deciding whether rare classes or repeated entities deserve separate weight. Report the components: publish the per-class and per-slice values beside the aggregate, and say which averaging policy produced the aggregate, since clause 6.4.3 of ISO/IEC TS 4213:2022 gives macro, weighted and micro averaging three distinct names. Finally, stress the weights by recomputing under plausible future prevalence or policy priorities.