Imbalanced Data, Rare Events, and Coverage

Percentages become unstable when denominators are small. Report raw counts, exposure, label maturity, review rate, and confidence intervals alongside precision or recall.

Track how positives enter the dataset. A sudden rise may reflect better detection, a policy change, delayed backfill, or a real shift in incidence.

Slice monitoring needs minimum evidence thresholds. For the slice that looks safest of all — the one with no failures in it — the bound is arithmetic rather than a matter of judgement. Hanley and Lippman-Hand printed it in JAMA in 1983: “This "rule of three" states that if none of n patients shows the event about which we are concerned, we can be 95% confident that the chance of this event is at most three in n (ie, 3/n).” Put a monitored slice where the patients are and the number is unforgiving. Zero incidents in 200 cases is still consistent with a true rate of 1.5%. A 1995 note in the BMJ restated the same formula for surgeons as “maximum risk = 3/n (for n > 30)”. An empty cell in a slice report is a sample size, not a safety result.

The curve you read the rest of the monitoring on can reverse the ranking as well. Davis and Goadrich proved in 2006 that one curve dominates in ROC space if and only if it dominates in PR space. Then they built a domain of 20 positives against 2,000 negatives in which the areas under those same curves disagree about which model to ship. AUC-ROC is 0.813 for curve I and 0.875 for curve II, which picks curve II. On the identical two rankings they wrote: “The AUC-PR for curve I is now 0.514 due to the high ranking of over half of the positive examples, while the AUC-PR for curve II is far less at 0.038, so the direct opposite choice of curve I should be made to optimize the AUC-PR.” They also showed that interpolating linearly in PR space on a dataset of 433 positives and 56,164 negatives inflates AUC-PR from 0.031 to 0.50. That is an artefact of the drawing, not of the model.

The effect reproduces. Saito and Rehmsmeier ran it in PLOS ONE in 2015, with 1,000 positives held against 1,000 and then 10,000 negatives. On strongly imbalanced data the visual interpretability of ROC plots “can be deceptive”, owing to “an intuitive but wrong interpretation of specificity”. Precision-recall plots held up on the same imbalanced data, because they “evaluate the fraction of true positives among positive predictions”.

Preserve a rolling challenge set of verified rare cases and difficult negatives. Refresh it carefully so repeated tuning does not turn it into another training set.

Rare-event systems should preserve unknown and unexposed states when they differ from confirmed negatives. The cost of collapsing them has been measured. Elkan and Noto worked in 2008 with 2,453 labelled positive protein records from TCDB and 4,906 SwissProt records carried as “unlabeled”. Of those 4,906, 348 were in fact positives that had simply never been labelled. That is the shape of the problem in one dataset. The negative column was wrong 348 times before any model saw it.

Their result is that the damage is recoverable when the labelling is “selected completely at random”: “Under the assumption that the labeled examples are selected randomly from the positive examples, we show that a classifier trained on positive and unlabeled examples predicts probabilities that differ by only a constant factor from the true conditional probabilities of being positive.” The constant is the labelling propensity itself — p(y=1|x) = p(s=1|x)/c with c = p(s=1|y=1). A pipeline that records how examples came to be labelled can divide the bias out. One that does not, cannot.

A positive class may be rare because the event truly occurs infrequently. It may also appear rare because the system rarely observes it, reviews only selected cases, or aggregates at the wrong unit. Class imbalance affects optimization and how the metrics read. But data engineering determines which positives and negatives exist at all, and a missed join, a delayed label, or a review policy can dominate any algorithmic remedy.

Start from a number rather than an adjective. In 2021, €1.53 billion of card fraud sat on €5.40 trillion of card transactions issued in SEPA. The European Central Bank put the figures in a report on card fraud on 26 May 2023: “The total value of transactions using cards issued in SEPA amounted to €5.40 trillion in 2021, of which €1.53 billion (0.028%) was fraudulent”. The public benchmark most teams actually train on reports a different figure for what is nominally the same problem. The ULB Machine Learning Group's credit-card dataset covers two days of September 2013 European cardholder transactions. It contains 492 frauds among 284,807 transactions, or 0.172% of rows. One counts euros across a continent and a year. The other counts rows in a two-day sample. Both are called rare, and a model tuned against one prevalence is not calibrated for the other.

Negative examples also deserve scrutiny. In recommendation and link prediction, a pair with nothing between them may mean the item was never shown rather than that the user refused it. In fraud, an unreviewed transaction may be unknown rather than legitimate.

The goal is representative evidence for the decision, especially in important rare slices. Resampling can help training. But it cannot create missing coverage or reliable ground truth.

Each source requires a different intervention, and each one has a measured example elsewhere in this lesson. True outcome rarity is the European Central Bank's 0.028% of SEPA card value in 2021, and the 492 positives among 284,807 rows in the ULB benchmark. Exposure rarity is the Yahoo! Music R3 split, where 54,000 ratings collected under random assignment describe a different world from the roughly 250,000 collected by user choice. Measurement rarity is Elkan and Noto's 348 positives sitting unlabelled among 4,906 records that the pipeline called negative. Slice rarity is the 200-case slice with no observed events, where the 3/n rule still allows a true rate of 1.5%. Definition rarity is the choice between counting euros and counting rows. Both 0.028% and 0.172% mean “card fraud is rare”, on different populations in different units.

The strongest plan often combines a modest change to training with collecting better evidence. The training half is worth less than its reputation suggests. Elor and Averbuch-Elor re-evaluated balancing across 73 datasets in 2022, separating the classifiers by strength: “However, we find that balancing does not improve prediction performance for the strong ones.” The strong ones are the gradient-boosted classifiers, XGBoost and CatBoost. Balancing the training data bought them nothing. It still helped the weaker learners. So the four interventions below are not interchangeable knobs on one dial. Two of them change the optimization. One changes what the model is asked to look at. Only the last changes the evidence, and that is the only lever that survives a strong classifier.

Start with label semantics and operational slices before choosing a resampling ratio. Step one is where most of the value sits, and it is rarely a matter of opinion. The clock on outcome maturity is often set by statute. Regulation E, at 12 CFR § 1005.6(b)(3), is a 60-day clock, in the Consumer Financial Protection Bureau's words: “A consumer must report an unauthorized electronic fund transfer that appears on a periodic statement within 60 days of the financial institution's transmittal of the statement to avoid liability for subsequent transfers.” The UK window is far longer. Regulation 74(1) of the Payment Services Regulations 2017 allows the payment service user to notify an unauthorised or incorrectly executed transaction “no later than 13 months after the debit date”.

A transaction 30 days old is therefore censored under both rules and confirmed under neither. Write those windows into the maturity audit as dates. The difference between a confirmed negative and an unresolved row then stops being a modelling preference and becomes a fact the pipeline can compute. The remaining steps — mapping exposure and review, naming the critical slices, choosing the training intervention, and improving collection — all inherit their meaning from that first separation.