Imbalanced Data, Costs, and Rare Events

A one-percent event can be the only outcome that matters, or it can be too noisy to automate, and class prevalence alone does not determine the right metric, threshold, or learning intervention.

Start with false-negative harm, false-positive workload, outcome delay, label quality, and the population eligible for action.

Combining them blindly can double-count the same preference. That double-counting is not a matter of taste. It was proved.

Weighting and resampling are one lever with two handles. Charles Elkan showed why in 2001. His Theorem 1 fixes what has to change if a learner whose threshold is p0 is to act as though its threshold were the target p*: multiply the number of negative training examples by (p*/(1-p*))((1-p0)/p0). That is one factor, and there are two ways to apply it. “Theorem 1 does not say in what way the number of negative examples should be changed. If a learning algorithm can use weights on training examples, then the weight of each negative example can be set to the factor given by the theorem. Otherwise, we must do oversampling or undersampling.”

Set the class weights, then rebalance the training set as well, and the same preference has been expressed twice, at a strength nobody chose. Elkan’s own recommendation is to pull neither handle. His abstract advises “to learn a classifier from the training set as given, and then to compute optimal decisions explicitly using the probability estimates given by the classifier”.

Why are there so few knobs? Because prevalence and cost reach the decision through a single quantity. A 2010 paper in the Journal of Machine Learning Research shows that “the risk minimization problem is uniquely defined by the ratio” of prior times cost, so that “even though the priors and costs may vary, as long as this ratio stays constant, the optimization problem is unchanged”. Four adjustments aimed at one quantity will collide. Only the threshold policy acts after the probabilities are formed rather than inside them.

Even a classifier with strong sensitivity and specificity can produce many false alerts when the event is rare. The chain below is short enough to run on the back of an envelope, and the section that follows shows a screening programme in which it was run on 53,454 people.

The clearest rare-event queue on record is a screening programme, not a simulation. The National Lung Screening Trial randomised 53,454 participants at 33 US centres, and published its primary results in the New England Journal of Medicine in 2011. Across three screening rounds, 24.2% of the low-dose CT screens came back positive. The Results section says what that queue contained: “A total of 96.4% of the positive screening results in the low-dose CT group and 94.5% in the radiography group were false positive results.”

And the programme works. The same screening cut lung-cancer mortality by 20.0%, 247 against 309 deaths per 100,000 person-years. The U.S. Preventive Services Task Force reviewed the trial and recommends the screening. Both things are true of one system: an alert queue in which 96.4% of the alerts were false, and a fifth fewer lung-cancer deaths. Low precision is not by itself a verdict on a rare-event system. It is one input to a decision about whether the burden it creates is worth the events it catches. That decision belongs to the people who carry the burden.

The arithmetic that produces such a queue is not difficult. It is also not reliably done. A 2014 study in JAMA Internal Medicine put one question to 61 respondents — 24 attending physicians, 26 house officers, 10 medical students and 1 retired physician: “If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person's symptoms or signs?” The answer is about 2%. What came back: “In our study, 14 of 61 respondents (23%) gave a correct response, not significantly different from the 11 of 60 correct responses (18%) in the Casscells study”. Twenty-seven of the 61, 44%, answered 95% — the false-positive rate read backwards as if it were the answer. Steven D. Stovitz repeated the exercise in 2020. He found “Eleven out of 51 (22%) answered question 1 correctly” and “14/48 (29%) answered question 2 correctly”, with “only 3 of the respondents answered both questions correctly”. Nearly four decades separate the 1978 Casscells study from that replication. The number barely moved. That is why the base-rate calculation is a section of this lesson and not a footnote to it.

Oversampling can expose the learner to more minority cases, while undersampling can reduce majority cost and redundancy, and synthetic methods interpolate or generate examples under assumptions about minority geometry.

Probability outputs often need correction or calibration because the sampled prevalence differs from deployment.

How much damage the uncorrected version does has been measured, and the measurements are unkind to the corrections themselves. A 2022 simulation study in the Journal of the American Medical Informatics Association simulated training sets of 2,500 and 5,000 with event fractions of 0.30, 0.10 and 0.01. Its result: “The use of random undersampling, random oversampling, or SMOTE yielded poorly calibrated models: the probability to belong to the minority class was strongly overestimated.” Discrimination did not improve to pay for it. AUC was unchanged. The conclusion goes further than the finding: “Outcome imbalance is not a problem in itself, imbalance correction may even worsen model performance.” The same group then extended the experiment to machine-learning algorithms, in work later published in Statistics in Medicine, and found that “In all simulation scenarios, prediction models developed without a correction for class imbalance consistently had equal or better calibration performance than prediction models developed with a correction for class imbalance.” Not on average, and not in most settings. In all of them.

Where a deliberate undersample is unavoidable, the correction is not vague. It has a closed form. Gary King and Langche Zeng studied “binary dependent variables with dozens to thousands of times fewer ones … than zeros”, in a 2001 paper in Political Analysis. They report that “popular statistical procedures, such as logistic regression, can sharply underestimate the probability of rare events”. In such data, ones and zeros are often sampled at different rates. That is the design behind every deliberate undersample. Under it the slope estimates stay consistent; the intercept does not. The corrected intercept is β̂0 − ln[((1 − τ)/τ)(ȳ/(1 − ȳ))], where τ is the fraction of ones in the population and ȳ the fraction in the sample. That expression, they note, “equals β̂0 only in randomly selected cross-sectional data”. Rebalance the training set, skip the line, and every probability the model prints refers to a population that does not exist.

Treat modeling and operations as one thresholded workflow. Two of its steps rest on published evidence rather than on preference.

Step 3, the choice of metric, was settled as a theorem. Jesse Davis and Mark Goadrich proved it in 2006: “We show that a deep connection exists between ROC space and PR space, such that a curve dominates in ROC space if and only if it dominates in PR space.” Domination carries over. Visibility does not. Their introduction says why: “ROC curves can present an overly optimistic view of an algorithm's performance if there is a large skew in the class distribution.” Saito and Rehmsmeier measured that in 2015. They ran the same classifiers twice — once on a balanced set of 1,000 positives and 1,000 negatives, then on an imbalanced set of 1,000 positives and 10,000 negatives — and concluded that ROC “requires special caution when used with imbalanced datasets”. Ten negatives per positive is a mild skew next to most rare events.

Step 6, monitoring realized yield, has a deployed example large enough to argue from. Wong and colleagues validated the Epic Sepsis Model externally across 38,455 hospitalisations of 27,697 patients, sepsis occurring in 2,552 (7%). The model’s AUC was 0.63. Their report, in JAMA Internal Medicine in 2021: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” Both failures in one deployment: most events missed, and alerts on 18% of all hospitalised patients. An independent validation in two county emergency departments covered 145,885 emergency-department encounters in 2023. It measured “a sensitivity of 14.7%, specificity of 95.3%, positive predictive value of 7.6%, and negative predictive value of 97.7%”, and cited the earlier “sensitivity of 33% and PPV of 12% for ESPMv1 using standard alerting thresholds”. None of those numbers is visible from an AUC. Every one of them is what step 6 exists to see.

Fraud tactics, disease waves, maintenance campaigns, and review criteria can change both event rate and observed labels. A stable ranking metric may coexist with deteriorating precision because prevalence moved.

Monitor base rates, score distributions, outcome maturity, and policy changes separately.

A moved prevalence does not always require a retrain. A 2002 paper in Neural Computation took the case “that a classifier is trained on a data set that does not reflect the true a priori probabilities of the target classes on real-world data”. Its procedure adjusts the trained classifier’s outputs “with respect to these new a priori probabilities without having to refit the model, even when these probabilities are not known in advance”. It is “a straightforward instance of the expectation-maximization (EM) algorithm”. The authors show that it maximises the likelihood of the new data. Estimates of the new prevalence arrive as a by-product. They also describe “a statistical test that can be applied to decide if the a priori class probabilities have changed from the training set to the real-world data”. That test is the monitor this section is asking for.

A single cost matrix can hide different harms for applicants, reviewers, patients, or communities, and raising sensitivity may protect one group while increasing burdensome false positives for another.

Document stakeholders, appeal paths, and slice-specific consequences before optimizing one aggregate utility.

Where base rates differ between groups, the difficulty is not only political. It is a proved impossibility, settled in 2017. The abstract: “We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously.” The special cases are two. One is perfect prediction. The other the paper names directly: “Equal base rates. Suppose, alternately, that the two groups have the same fraction of members in the positive class”. Rare-event work is very often the work in which subgroup base rates are not equal. So calibration and balance for both classes cannot all hold at once, and a cost matrix is quietly choosing which of them to break. Alexandra Chouldechova reached the same tension from the applied side, studying recidivism prediction instruments, and demonstrated “how adherence to the criterion may lead to considerable disparate impact when recidivism prevalence differs across groups”. Naming whose cost the matrix represents is the first obligation. The second is naming which fairness property the chosen threshold gives up. Under unequal prevalence, one of them is going.