Precision, Recall, Specificity, and F-Scores

Recall asks about actual positives. Precision asks about positive predictions. Specificity asks about actual negatives. Each denominator picks out a different population of cases. The difference matters enough that US federal law writes the denominator into the definition.

Every certified mammography facility in the country is told exactly what to compute: “(i) Positive predictive value—percent of patients with positive mammograms who are diagnosed with breast cancer within 1 year of the date of the mammographic examination.” That is Title 21 of the Code of Federal Regulations, §900.12(f)(1)(i).

Notice what the rule fixes. The denominator is patients with positive mammograms, not patients with cancer. The window is one year from the date of the examination. The same paragraph requires cancer detection rate and recall rate alongside it, computed individually for each interpreting physician and collectively, at least once every 12 months. A facility that quietly swaps in a different denominator is not making a stylistic choice about reporting. It is failing a licensing condition.

The names are easier to remember as plain questions: “How many urgent cases did we catch?” and “How many alerts were worth reviewing?” The familiar summaries also carry a hidden bias. Powers argued in 2011 that “Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic”. Under them, he warned, “a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures”. His remedy: state the chance level first.

The same confusion matrix supports several views. One national benchmark shows how far apart those views land when they are computed from identical data.

The Breast Cancer Surveillance Consortium reported on 1,682,504 digital screening mammograms performed between 2007 and 2013 on 792,808 women, read by 359 radiologists at 95 facilities. Published in Radiology in 2017, the Results sentence reads: “sensitivity, 86.9% (95% CI: 86.3%, 87.6%); specificity, 88.9% (95% CI: 88.8%, 88.9%); false-negative rate per 1000 screens, 0.8 (95% CI: 0.7, 0.8); positive predictive value (PPV) 1, 4.4% (95% CI: 4.3%, 4.5%)”.

Recall of 86.9%, specificity of 88.9%, PPV1 of 4.4%. Nothing changed about the radiologists between the first number and the third. Nothing changed about the images. What changed is which crowd sits in the denominator.

Recall divides by the cancers, and most of them were caught: only 0.8 false negatives per 1,000 screens. Specificity divides by the women without cancer, and nearly nine in ten of them were correctly left alone. PPV1 divides by the positive readings, and there the arithmetic is unforgiving. The abnormal interpretation rate was 11.6%; the cancer detection rate was 5.1 per 1,000 screens. The great majority of positive predictions have no cancer behind them to find. Negative predictive value divides by the readings called normal, a pool so dominated by healthy women that it flatters any test. The F-score divides by nothing in particular. It is a weighted blend of two of the five, and which two is a choice.

One matrix, five questions, five answers. A reader who is told only that “the model is 86.9% accurate at finding cancer” will never reconstruct them.

A deployed clinical alert converts the two rates into hours of human attention, and one of them has been measured in public.

Epic's proprietary sepsis prediction model was externally validated at Michigan Medicine across 27,697 patients and 38,455 hospitalizations, of which 2,552 (7%) had sepsis. The model's area under the curve was 0.63. At Epic's own recommended alert threshold of 6, the authors report: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” Wong and colleagues published that in JAMA Internal Medicine in 2021.

One sentence, two denominators. They point at two different budgets: the patients the system was built to protect, and the clinicians who must answer every alert it raises.

F-scores summarize precision and recall but omit other dimensions. The two anchored examples above show exactly what falls outside the formula.

Feed the screening benchmark to F1 and it blends recall of 86.9% with PPV1 of 4.4% into a single number. The 88.9% specificity never enters the calculation at all, because F1 ignores true negatives — and that specificity is the reason the overwhelming majority of the 792,808 women went home without a callback. Feed the sepsis model to F1 and neither the 18% alert volume nor the clinician hours behind it appears anywhere in the result.

F-beta lets you tilt the blend. Beta above one weights recall; beta below one weights precision. What it cannot do is price a missed sepsis case against a wasted review, or tell you whether the score behind the threshold is a calibrated probability. Choosing a beta does not answer those questions. It only records which of the two visible errors you have decided to mind more.

For multiclass tasks, each class can be treated as positive against the rest. That produces class-specific precision and recall, followed by macro, micro, or weighted aggregation.

A macro F1 can improve while the most severe class worsens. Keep critical class metrics and confusion patterns visible beside the aggregate.

Thresholding for F1 has a closed-form answer. Lipton and colleagues set out in 2014 to “derive the relationship between the best achievable F1 score and the decision-making threshold that achieves this optimum”, and the result is compact: “if the classifier outputs are well-calibrated conditional probabilities, then the optimal threshold is half the optimal F1 score”.

The degenerate case is the warning about what F1 rewards. A completely uninformative classifier is optimally handled by labelling every example positive. That scores respectably on F1 and conveys nothing.

Their case study predicted “26,853 labels for Medline documents”. That is a multilabel setting, in which per-class thresholds, not a single global cutoff, are what the theory actually governs.

Do not select a threshold from one curve label alone. Sweep thresholds and record precision, recall, specificity, alert volume and capacity at each candidate. Attach severity, so that high-cost misses stay separate from low-value false alerts. Estimate uncertainty on the correct analysis unit, with enough positive support. Compare slices. Only then select a policy: threshold, abstention, tiers, or queue limits.

Two of those steps have regulators standing behind them, which is worth knowing when the table is contested.

On uncertainty, FDA's 2007 guidance on reporting results from studies evaluating diagnostic tests is explicit: “FDA recommends you report measures of diagnostic accuracy (sensitivity and specificity pairs, positive and negative likelihood ratio pairs) or measures of agreement (percent positive agreement and percent negative agreement) and their two-sided 95 percent confidence intervals.” It asks for both forms of the number, the fraction (e.g., 490/500) as well as the percentage (e.g., 98.0%), so a reader can see how much data stands behind the rate. Its appendix says of sensitivity and specificity that they “each provide distinct and equally important information, and FDA recommends they be presented together”. The document is non-binding guidance rather than a requirement. That is precisely why the habit of publishing one rate, with no interval and no partner, survives.

On slices, the evidence is quantitative. NIST's Face Recognition Vendor Test processed 18.27 million images of 8.49 million people through 189 algorithms from 99 developers, and reported on demographic effects in December 2019. Its executive summary states: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” False negatives, by contrast, varied by factors usually below 3. The report also criticises the common convention of quoting a false negative rate at one fixed false positive rate. Holding the false positive rate constant across the whole population is exactly what conceals excursions of that size within it. A single operating point in a summary table is an average over groups whose error rates differ by two orders of magnitude.