Skip to content
AI.info

How machines learn

Metrics, Thresholds, and the Cost of Different Errors

Connect task metrics, score thresholds, error costs, calibration, and operational capacity without turning one number into the entire evaluation.

By the end you can

Key idea

A 99.9% accurate fraud detector that catches nothing

If fraud occurs in 0.1% of transactions, a system that predicts “legitimate” every time achieves 99.9% accuracy and zero investigative value.

Accuracy counts all correct cases equally. On rare-event tasks, it can be dominated by the majority class and hide complete failure on the cases that motivated the project.

US screening mammography runs the same arithmetic in public, on purpose. Roughly 10% of women are recalled after a screening exam. Only about 0.5% of those tested have cancer. So about 9.5% of tested women have a false-positive exam, and only about one recall in twenty finds a cancer. Repeat the exam and the count compounds. “Approximately 50% of women screened annually for 10 years in the United States experience a false-positive exam; of these, 7% to 17% will undergo biopsies,” the National Cancer Institute reports in its section on the harms of screening.

The surveillance data says the same thing from the other direction. Hubbard and colleagues reported from the Breast Cancer Surveillance Consortium in 2011 that more than half of women have at least one false-positive recall after ten years of annual screening, and 7 to 9 percent a false-positive biopsy recommendation. Screening every two years instead of every year cuts false-positive recalls by about one third.

Read that last sentence twice. Nothing about the test changes — only how often it is run — and the false-positive burden falls by roughly a third. The programme is not broken. It is a deliberate operating point on a rare event. Its per-exam error rate and its per-decade false-positive rate are two different numbers describing one system. Which of them counts as “the accuracy” is a decision, not a measurement.

A metric is useful only when its averaging reflects the decisions and errors you care about.

Case

Cardiff 2017: 2,470 alerts, and 2,297 of them wrong

South Wales Police published the arithmetic. Its live facial-recognition system ran at the 2017 UEFA Champions League final in Cardiff, in front of a crowd reported at about 170,000. The system produced 2,470 alerts against a watchlist. Of those, 2,297 — 92 percent — were false positives. The force replied that “no system is 100 percent accurate” and that no arrest was “based on an incorrect match”. Both statements are true. Neither touches the mechanism. When the target is rare and the population is large, a system with an unremarkable error rate still spends most of its alerts on innocent people.

Nor were those alerts spread evenly across the crowd. NIST measured that in December 2019, at scale: 18.27 million images of 8.49 million people, run through 189 mostly commercial algorithms from 99 developers. Under the heading “WHAT WE FOUND” in the executive summary, the report says: “Our main result is that false positive differentials are much larger than those related to false negatives and exist broadly, across many, but not all, algorithms tested.” Across demographic groups, false positive rates often vary by factors of 10 to beyond 100 times. False negatives vary often by factors below 3.

The U.S. Commission on Civil Rights repeated the finding in September 2024. African Americans and Asian American, Native Hawaiian and Pacific Islanders face false positive rates “of 10 to 100 times more than white individuals”, it wrote. Then came the sentence that matters most here: “false positives and false negative rates are determined by a cutoff threshold set for the algorithm by the user, and this threshold will often vary depending on the intended use for the FRT algorithm.”

So the 92 percent was not a fixed property of the software. It was the visible face of a cutoff somebody chose, applied to a population in which almost nobody was on the watchlist, by an algorithm whose error was never a single number to begin with.

The Court of Appeal arrived at the same place down a different road. In August 2020 it held, in R (Bridges) v Chief Constable of South Wales Police, that the force's use of the technology was unlawful on three of five grounds: the interference with Article 8 rights was not “in accordance with the law”, the data protection impact assessment was inadequate, and the force had not complied with the public sector equality duty by investigating whether the software was biased on grounds of race or sex. The court's third ground and the NIST result are one observation. One is written as a duty, the other as a measurement.

Figure

When the target is rare and the population is large, a system with an unremarkable error rate still spends almost every alert on an innocent person.

Visual

Five layers between prediction and outcome

Evaluation becomes clearer when these layers are named separately. A model score is not a decision. A decision is not an error pattern. An error pattern is not an operational effect, and none of them is the outcome the project was funded to change.

Cardiff shows what collapsing them costs. The vendor's error rate, the operator's chosen cutoff, the size of the crowd and the number of innocent people stopped are four different quantities that arrive in one headline figure. Only the first is a property of the model, and it is the one least able to predict the last.

FigureProcess · 5 steps
  1. 1

    Model score

    A numeric output such as risk, probability estimate, relevance, or forecast.

  2. 2

    Threshold or policy

    A rule converts scores into actions, queues, or categories.

  3. 3

    Error pattern

    The policy creates false positives, false negatives, and other task-specific misses.

  4. 4

    Operational effect

    Review volume, latency, interventions, and user experience change.

  5. 5

    Real outcome

    Cost, safety, service quality, or another product result is observed later.

Comparison

0.42 against 0.22, 0.70 against 0.69: one dataset, two verdicts

The definitions are simple; choosing between them is a product decision. The choice can carry the whole of an argument. One dataset can support opposite conclusions depending on which measure is quoted.

On 23 May 2016 ProPublica published its Machine Bias investigation. It analysed more than 7,000 COMPAS risk scores from Broward County, Florida for 2013-2014, and reported that black defendants were falsely flagged as future criminals at almost twice the rate of white defendants. Flores and two colleagues re-analysed the same data in Federal Probation in September 2016. They reproduced the gap exactly: a false positive rate of 0.42 for Black versus 0.22 for White defendants, and false negative rates of 0.28 versus 0.50. They also reported near-identical discrimination — an AUC of 0.70 for Black and 0.69 for White defendants — against rearrest base rates of 52% and 39%. Their conclusion was flat: “In all instances, we failed to find evidence of predictive bias by race in the COMPAS.”

Both parties held the same scores and the same outcomes. One conditioned on the true outcome and asked how often each group was wrongly flagged. The other asked how well the score ranked risk within each group, and got 0.70 against 0.69. Neither number is an error. They are answers to different questions, and no amount of care about definitions will merge them into one. Choosing precision, recall, specificity or a summary score is choosing which of those questions your organisation is going to be held to.

FigureComparison · 4 columns

Precision

Among cases predicted positive, how many are truly positive?

  • Matters when actions are costly
  • Drops when many false alarms occur
  • Depends on prevalence
  • Example: alerts worth investigator time

Recall

Among true positives, how many did the system find?

  • Matters when misses are costly
  • Can rise by lowering the threshold
  • Does not describe alert quality alone
  • Example: dangerous defects detected

Specificity

Among true negatives, how many were correctly left negative?

  • Describes control of false positives
  • Useful with recall in screening
  • May look high under extreme imbalance
  • Example: legitimate messages left unblocked

F1 and summaries

Combine selected aspects into one number.

  • Convenient for comparison
  • Embed a trade-off choice
  • Can hide threshold and slice behavior
  • Not a substitute for costs

The model and decision policy are separate levers

A classifier may return a continuous score. Raising the threshold usually reduces positive actions, increasing precision in some settings while decreasing recall. Lowering it generally does the opposite.

The best threshold depends on prevalence, capacity, costs, and the intervention it triggers. It may reasonably differ from one workflow stage to the next, though group-specific thresholds require careful legal, ethical, and operational review.

This is why the U.S. Commission on Civil Rights took care to write that false positive and false negative rates “are determined by a cutoff threshold set for the algorithm by the user”. The vendor ships a score. The deployment picks the number that turns that score into an arrest, an alert or a denial. Most public arguments about a model's errors are arguments about that number, held under the impression that they are arguments about the model.

Example

18% of admissions alerted, 12% of those alerts real

Metrics have to respect how many cases can actually be acted upon. A deployed system will tell you what happens when they do not.

The Epic Sepsis Model was checked by somebody other than its vendor. Wong and colleagues ran it against 38,455 hospitalizations of 27,697 patients at Michigan Medicine, between December 2018 and October 2019, and published in JAMA Internal Medicine in June 2021. The area under the ROC curve was 0.63 (95% CI, 0.62-0.64). At the operating point in use, sensitivity was 33%, specificity 83% and positive predictive value 12%. The abstract puts the cost in one sentence: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

Hold the two halves together, because either alone reads as a different system. Alerts fired on 18% of all hospitalizations. Of those alerts, 12% marked a patient who had sepsis. Meanwhile 1,709 of 2,552 sepsis patients — 67% — arrived without an alert at all. Specificity of 83% sounds respectable right up to the moment it is multiplied by 38,455 hospitalizations and handed to a ward. That is the shape of a capacity failure. The queue was large, its quality was low, and the misses were not bought back in exchange. A metric suite that reported only the AUC of 0.63 would have described none of it.

  • Alert volume against capacity: 6,971 alerts across 38,455 hospitalizations, 18% of every admission. That is the load the clinicians actually absorb, and no ranking metric contains it.
  • Queue quality: a positive predictive value of 12% says what a clinician finds on opening an alert. Where the queue is capped by hand instead of by a score cutoff, precision among the top 500 of a ranked list answers the same question.
  • Coverage at the same operating point: sensitivity of 33%, or 1,709 of 2,552 sepsis patients missed. That is what the queue costs on the other side, and it must be quoted in the same breath as the precision.
  • Score distribution: a cutoff of an ESM score of 6 or higher is a choice. Whether the cases just above and just below it differ meaningfully is a separate question from the model's AUC of 0.63.
  • Slices: report the populations, wards and conditions separately. One averaged rate hides which groups vanish from the alerted set entirely — the failure the Court of Appeal faulted South Wales Police for never testing.
  • Time-to-label and operational tracking: sepsis is confirmed after the fact, so alert quality can only be measured on a delay. The queue must still be processed inside the window in which the intervention does any good.

Key idea

A score of 0.8 may not mean an 80% chance

Some models produce values that rank risk well without matching observed frequencies. Calibration is the check. It asks whether cases assigned similar probability estimates occur at corresponding rates.

Calibration matters when scores drive expected cost, resource planning, or communication. A later evaluation path will cover it in depth; here, do not interpret every bounded score as a trustworthy probability.

Push test data further and further from the training distribution and models do not grow less sure. They grow more sure and wrong. Ovadia and colleagues showed this at NeurIPS 2019, across images, text, online advertising and genomics. “The models are not becoming increasingly unsure with shift, but instead are becoming confidently wrong,” Snoek and Nado wrote in Google's summary of the work in January 2020. They added that “improving calibration and accuracy on an in-distribution test set often does not translate to improved calibration on shifted data”. Minderer and colleagues revisited the question at NeurIPS 2021 and found the decay less pronounced in the most recent architectures. So measure calibration on the model you intend to ship. Do not assume either answer.

There is a harder version of the same gap. It is a theorem rather than an empirical finding. Kleinberg and two co-authors proved in 2017 that the fairness conditions people argue about cannot all hold at once, and their abstract opens with the result: “We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously.” Chouldechova reached the same collision independently from the other side in October 2016, showing that it arises whenever prevalence differs across groups.

That is why the COMPAS dispute two sections above could not be settled by better engineering. With rearrest base rates of 52% and 39%, a score that is calibrated within each group and a score with equal error rates across groups are not the same object. A group-specific threshold does not repair that. It chooses which error to move.

Ranking quality and probability accuracy are different properties.

Analogy

An analogy: an airport security gate

The sensitivity of an airport security gate can be turned up or down. A stricter setting catches more questionable items. It also sends more harmless bags for inspection, consuming staff and delaying passengers.

A model threshold creates a similar trade-off between misses and false alarms. Scores may be miscalibrated, groups may differ, and the intervention itself can alter future behavior. The analogy holds in one further respect worth keeping: nobody sets the gate by asking how accurate it is. They set it by asking how many bags the inspection line can open before the flights start leaving without their passengers.

Steps

Choose metrics from the decision backward

A compact metric suite is better than a dashboard of unrelated numbers.

Three regulators have already written most of this list down. On 27 October 2021 the FDA, Health Canada and the UK's MHRA jointly identified ten guiding principles for good machine learning practice in medical device development. They are explicitly not requirements. The document says the principles “can inform the development of Good Machine Learning Practice” and are “intended to lay the foundation” for it. What they contain is nonetheless the same sequence as the steps above, arrived at independently.

Principle 6, “Model Design Is Tailored to the Available Data and Reflects the Intended Use of the Device”, says the clinical benefits and risks related to the product are “well understood, used to derive clinically meaningful performance goals for testing”. That is steps 2 and 4: error costs first, measures chosen to expose them. Principle 9, “Users Are Provided Clear, Essential Information”, says users are given ready access to the “performance of the model for appropriate subgroups”, the “characteristics of the data used to train and test the model”, and “known limitations”. That is step 5, and the reason a single headline number is not a report. Principle 7, “Focus Is Placed on the Performance of the Human-AI Team”, is step 6: “Where the model has a 'human in the loop,' human factors considerations and the human interpretability of the model outputs are addressed with emphasis on the performance of the Human-AI team, rather than just the performance of the model in isolation.”

That last principle names the unit of evaluation, and it is not the model. It is the model together with the person who acts on its output — the clinician deciding whether to open the alert, the investigator working the queue, the officer approaching someone in a crowd of 170,000.

FigureProcess · 6 steps

  1. 1. Name the action

    What happens to positive, negative, ranked, or numeric outputs?

  2. 2. List error costs

    Who is affected by misses, false alarms, and magnitude errors?

  3. 3. Add capacity

    How many cases can be reviewed, served, or intervened upon?

  4. 4. Choose core measures

    Select metrics that expose the relevant trade-off.

  5. 5. Add slices and uncertainty

    Report variability and important populations or conditions.

  6. 6. Connect to outcomes

    Validate that metric gains improve the workflow and not only offline numbers.

Report enough context to make the number interpretable

A metric report should include the dataset and the period, the split policy, prevalence, sample count, threshold, confidence or variation, and the relevant slices, since a metric compared across different datasets or different thresholds can be meaningless.

Avoid selecting the metric after seeing which one makes the model look best. Define decision-relevant measures before final evaluation.

One model, two screening programmes, two different reports. McKinney and colleagues evaluated a breast-cancer screening system in Nature in January 2020, on a curated, representative UK dataset of 25,856 women and an enriched US dataset of 3,097. The absolute reduction in false positives was 1.2 percent in the UK and 5.7 percent in the US. In false negatives it was 2.7 percent and 9.4 percent. The model was the same across all four numbers. The populations, the prevalence and the reading practice were not.

The period and the comparison baseline fail in the same way, and Google Flu Trends is the case where both did, in public, for years. The model was tuned to reproduce CDC influenza-like-illness reports — an excellent fit to the thing it was fitted to. Then it drifted. GFT “missed high for 100 out of 108 weeks starting with August 2011”, four researchers reported in Science on 14 March 2014, and even three-week-old CDC data projected current flu prevalence better than GFT did. “In short, the initial version of GFT was part flu detector, part winter detector,” they wrote.

The same divergence had already been found independently, a year earlier. Olson and colleagues reported in PLOS Computational Biology on 17 October 2013 that GFT estimates of epidemic intensity ran to 268% of ILI surveillance nationally, 208% regionally and 296% locally in 2012/2013. The original model had erred the other way during the spring 2009 pandemic wave, giving estimates 30-40 fold lower than surveillance. The same system was 30-40 fold low in one season and nearly triple in another.

That is why a metric quoted without its dataset, its period, its operating point and its comparison baseline is not yet a fact about a model. It is a fact about one measurement, and the four numbers it omits are the ones that decide whether the measurement transfers.

Key takeaways