Imbalance, Rare Cases, and Cost-Sensitive Training

A failure event may appear in one of ten thousand records and still dominate safety or financial risk. Another rare label may be noisy, ambiguous, or irrelevant.

Training needs an explicit reason to weight one class more heavily than another. Frequency, decision cost, annotation reliability, and achievable recall are four separate things. Most of this lesson is a record of what gets measured when they are treated as one. An ovarian-cancer study where the standard imbalance corrections left discrimination untouched and drove the calibration intercept from 0.06 to −1.50. A sepsis model running in a hospital that alerted on 18% of hospitalizations while missing 67% of the sepsis patients. And a workshop paper from 2000 that put the problem in the decision threshold, before most of today's remedies existed.

These effects are common enough to require explicit checks. One of them has an exact answer rather than a rule of thumb. Multiply the number of negative examples by the right factor, and a target probability threshold becomes the learner's default threshold. Charles Elkan proved that factor in 2001, as Theorem 1 of his paper on the foundations of cost-sensitive learning. He then says what it means for weights: “if a learning algorithm can use weights on training examples, then the weight of each negative example can be set to the factor given by the theorem”. Reweighting the negatives and moving the operating point are not two independent controls. They are two ways of writing one operation.

Elkan goes further. Changing the balance of negative and positive training examples “has little effect” on standard Bayesian and decision-tree learners. His recommendation is to learn from the training set as given, and compute optimal decisions from the probability estimates.

Class weights and altered sampling change the objective distribution seen during training, and the size of that change has been measured. A 2022 study in JAMIA, by Van den Goorbergh and colleagues, applied random undersampling, random oversampling and SMOTE to an ovarian-cancer case study: 3,369 premenopausal patients, 20% malignancy prevalence, split into 2,695 training and 674 test cases. Discrimination did not move. AUROC stayed between 0.79 and 0.80 across all eight models. Calibration did move. The calibration intercept was 0.06 for standard logistic regression and 0.05 for ridge on the uncorrected data. After correction it lay between −1.32 (SMOTE) and −1.50 (random undersampling) — strong overestimation of minority-class risk. In their simulations with a 1% event fraction, the corrected models had median calibration intercepts of −4.5 or lower.

The authors' own verdict on what was bought with that damage: “Although imbalance correction improved the balance between sensitivity and specificity, similar results were obtained by shifting the probability threshold instead.” The threshold shift is free of the calibration cost. The resampling is not.

The obvious escape route has also been tested. A 2024 follow-up by Carriero and co-authors reproduced the effect across machine-learning algorithms, and reports that the induced miscalibration “was not always able to be corrected with re-calibration”. So: evaluate calibration on data with the intended class prior, choose thresholds using deployment costs, and do not read weighted-loss logits as probabilities without evidence.

A different line of work questions the counting itself. Each new sample of a class adds less than the last, argue Cui and colleagues in 2019, so the raw count should be replaced by an effective number of samples. It “can be calculated by a simple formula (1−β^n)/(1−β), where n is the number of samples and β ∈ [0,1) is a hyperparameter”. Weighting by that quantity is not the same as weighting by frequency.

The remedy should match the deficiency you actually found, and the third card below is the one with a canonical case attached to it. The selective labels problem was named and formalised in 2017, by Lakkaraju and co-authors, in one line: “we observe the outcome of whether a defendant fails to return for their court appearance only if the human judge decides to release the defendant on bail”. The label exists only where the human said yes.

They evaluated their “contraction” technique on three real datasets. A bail dataset of about 9,000 defendants decided by 18 judges, with an evaluation subset of 4,800 defendants and 9 judges. A medical dataset of 60,000 coughing or wheezing patients. And about 50,000 insurance requests decided by 74 managers. Each has the same structure of unobserved outcomes. The stakes are not small. The companion bail analysis, an NBER working paper from the same group in February 2017, reports a policy simulation in which “crime can be reduced by up to 24.8% with no change in jailing rates, or jail populations can be reduced by 42.0% with no increase in crime rates”. No sampling scheme and no class weight touches this deficiency. Auditing the selection process does.

Start with evidence quality and end with the decision workflow. Two orderings in the sequence below are load-bearing rather than tidy. Baselines and calibration are measured before the objective is touched, because in the ovarian-cancer study AUROC alone would have reported that nothing had happened. And the threshold is tried before the training distribution is altered, because “similar results were obtained by shifting the probability threshold instead”.

Several controls can be combined, but each changes a different layer, and the earliest clear statement of the problem already argued that the decisive layer is the last one. A 2000 AAAI workshop paper by Foster Provost identifies the two assumptions standard learners carry into the objective layer: “that maximizing accuracy is the goal” and “that, in use, the classifier will operate on data drawn from the same distribution as the training data”. When 99% of the data are one class, Provost observes, a learner will be hard pressed to beat the 99% accuracy of the trivial majority-class classifier.

His conclusion is not about sampling or weights. “The bottom line is that when studying problems with imbalanced data, using the classifiers produced by standard machine learning algorithms without adjusting the output threshold may well be a critical mistake (depending on your research question).” Read the layers below with that in mind. The first three change what the model learns. The fourth changes what you do with what it learned. The fifth changes what the organization can absorb.

Use precision-recall curves, class-conditional error, ranking quality, calibration, and expected cost — and know what each of them conceals. Davis and Goadrich proved in 2006 that a curve dominates in ROC space if and only if it dominates in PR space, and warned that “algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve.” Saito and Rehmsmeier put the consequence in counts in 2015. They compared 1,000 positives with 1,000 negatives against 1,000 positives with 10,000 negatives. The ROC plots and every AUC(ROC) score were unchanged. Yet the identical ROC operating point meant 160 false positives against 500 true positives when balanced, and 1,600 false positives against the same 500 true positives at 1:10 imbalance. The precision-recall baseline itself falls from 0.5 to P/(P+N) — 0.09 at 1:10. An unchanged AUC(ROC) is therefore not evidence about the rare class.

A deployed system shows the same thing with public denominators. The Epic Sepsis Model was validated externally in 2021 by Wong and colleagues, over 38,455 hospitalizations of 27,697 patients, of which 2,552 (7%) involved sepsis. It “predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer”; the 95% CI was 0.62–0.64. At the vendor threshold of 6 or higher it alerted on 6,971 of the 38,455 hospitalizations (18%), while failing to identify 1,709 of the 2,552 sepsis patients (67%). It flagged only 183 sepsis patients (7%) who had not already received timely antibiotics. An independent validation by Ostermayer and co-authors in 2024, over 145,885 emergency-department encounters, found the model alerted in 7,183 encounters (4.9%), with sensitivity 14.7%, specificity 95.3% and positive predictive value 7.6%.

That is what a rare-event report has to contain: the denominator, the alert burden, the complementary error, and the cases themselves. Report uncertainty when rare-case counts are small. A two-point recall change may represent one case in a tiny slice rather than stable progress.