ROC, Precision–Recall Curves, AUC, and Discrimination

ROC AUC averages ranking behavior across false-positive rates that may never be operationally acceptable. Average precision emphasizes positive retrieval across recall increments, but its baseline depends on prevalence. Curves are valuable because they show tradeoffs across thresholds. They do not select the threshold or determine which tradeoffs matter.

What the area means is old and exact. It was fixed in Radiology in 1982, by Hanley and McNeil: “the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject”. That is a statement about the ordering of pairs of cases. Nothing else. It says nothing about which false-positive rates a hospital can afford.

The rest of this lesson is a sequence of places where that gap decided the outcome. A sepsis model ran across 38,455 hospitalizations with an area under the ROC curve of 0.63, and alerted on 18% of them. A simulated biomarker held an AUC above 0.9 while its precision fell below 0.2. A diagnostic device was authorised on a prespecified pair of sensitivity and specificity numbers, and on no curve at all.

Each curve uses a different view of the confusion matrix. Which of the two deserves the weight under class imbalance is an open dispute rather than settled practice.

On imbalanced data, the precision–recall plot is the more informative one. That was Saito and Rehmsmeier's argument in PLOS ONE in 2015. A NeurIPS paper in 2024 contested exactly that. AUPRC is not generally superior, McDermott and colleagues showed, and it can unduly favour improvements in subpopulations with more frequent positive labels — a fairness consequence sitting inside what looks like a plotting preference. Then they went looking for where the belief came from: “Prompted by these insights, we conduct a review of over 1.5 million scientific papers to understand the origin of this invalid claim, finding that it is often made without citation, misattributed to papers that do not argue this point, and aggressively over-generalized from source arguments.”

So read the two columns below as two views of one ordering, not as a ranking of the two plots.

One way to read it is pairwise ranking. The 1982 paper that gave the area its meaning also named the estimator: “this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic”.

That quantity was written out explicitly at NIPS in 2004. Cortes and Mohri gave it as the Wilcoxon-Mann-Whitney statistic A = (sum over m positives and n negatives of 1[x_i > y_j]) / mn. They used Hanley's variance formula as the baseline their confidence intervals were compared against. The equivalence is what gives AUC a standard error and its sample-size rules.

It also fixes the limits of the number. Every pair of one positive and one negative counts once, wherever on the curve it falls. So a false-positive rate no service could staff contributes exactly as much to the average as one it operates at every day.

Epic's proprietary Sepsis Model was validated by somebody other than its vendor, and the numbers came apart. The external validation ran at Michigan Medicine and appeared in JAMA Internal Medicine in June 2021. Wong and colleagues covered 27,697 patients and 38,455 hospitalizations, 2,552 of them with sepsis — 7%. The model's area under the ROC curve was 0.63 (95% CI, 0.62-0.64). Epic had reported 0.76-0.83.

That single figure is one line of the story. The queue is the rest of it: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

Global AUC can be replaced or supplemented by targeted summaries. A regulator did precisely that, in public, and the paperwork is the clearest worked example available.

IDx-DR is a retinal diagnostic software device. When the FDA authorised it in January 2018, the feasible region had been fixed in advance: prespecified performance thresholds of 85.0% sensitivity and 82.5% specificity. The pivotal study enrolled 900 participants at 10 primary care sites. It returned observed sensitivity 87.4% and specificity 89.5%. Enrichment-corrected, that was 87.2% (95% CI 81.8-91.2%) and 90.7% (95% CI 88.3-92.7%), with imageability 96.1% (819/852) and positive predictive value 72.7% (173/238). Abràmoff and colleagues reported the same trial in npj Digital Medicine.

The special controls attached to the classification then say what evidence has to travel with the device: “Clinical performance testing must evaluate sensitivity, specificity, positive predictive value, and negative predictive value for each endpoint reported for the indicated disease or condition across the range of available device outcomes.” No area under any curve appears in that list. The steps below are the same discipline applied to a model nobody is required to file.

Ranking and calibration come apart: a model can rank cases well while producing unreliable probabilities. Another model can be calibrated on average yet offer weak discrimination. Neither property alone specifies utility.

The two plots were tested against each other on imbalanced data in PLOS ONE in 2015. Saito and Rehmsmeier found that the visual reading of a ROC plot “can be deceptive with respect to conclusions about the reliability of classification performance.” The cause they identify is “an intuitive but wrong interpretation of specificity.” Precision–recall plots hold up better because they “evaluate the fraction of true positives among positive predictions.” The two plots are drawn from the same ordering and still tell different stories.

That finding is itself contested. The NeurIPS 2024 paper from McDermott and colleagues, circulated first as a preprint in January 2024, rejects the general claim that AUPRC is the better summary under imbalance. Treat the dispute as the lesson rather than the embarrassment. Which plot to trust is an argument to be made about a specific deployment, not a rule to be inherited.

Report ranking, calibration, and operating performance separately, so one strength does not conceal another weakness. And note what a regulator asked of IDx-DR when it had to commit: sensitivity, specificity, positive predictive value and negative predictive value, at one operating point, prespecified.