Label Delay, Censoring, and Proxy Targets

Fraud disputes, equipment failures, churn, medical outcomes and loan defaults mature over different horizons. Treat every not-yet-positive case as a negative and you have written systematic label error into the dataset. Label delay is not merely waiting for a column to arrive. It decides which examples are eligible, how the cohort is censored, and which model errors can be measured today.

In the domains where being wrong is expensive, notice who chooses the horizon. It is not the modeller. US bank regulators classify a retail loan Substandard at 90 cumulative days past due and charge off an open-end loan at 180. EU banking law fixes default at more than 90 days past due. US Regulation E gives a consumer 60 days after the statement to report a disputed electronic fund transfer, and the institution 10 business days to determine whether an error occurred — or up to 45 days if it credits the account first. The 2008 Minnesota US Senate election was certified on 5 January 2009 and affirmed by the Minnesota Supreme Court on 30 June 2009, 238 days after polling day.

A sound protocol separates four things: the event being predicted, the observation window, the outcome horizon, and the date when the label is mature enough for use.

A usable label depends on the business event and on the protocol chosen for maturity. For one common event these five intervals are not a diagram. They are a statute: US Regulation E's "Procedures for resolving errors", for a disputed electronic fund transfer.

Scoring time is the transfer itself. The outcome window runs while the customer can still object: the consumer's notice of error must reach the institution no later than 60 days after the statement on which the error first appears. Initial observation is the institution's determination, which it has 10 business days to make. The correction period is what the rule buys when 10 business days are not enough. The institution may take up to 45 days, and that 45 days becomes 90 days for transfers not initiated within a state, point-of-sale debit card transactions, or accounts less than 30 days old. The maturity date is when that investigation closes.

A model that reads the ledger on day 12 is not reading an outcome. It is reading the position of a clock the rules wound. Every one of those numbers is a boundary a training cohort can be cut on the wrong side of.

Collapsing them into one value can make evaluation look stronger than it is, and the provisional state is the one most often mistaken for a resolved one. Regulation E shows why, because it requires the placeholder in writing. An institution that takes the longer investigation window must be one that “Provisionally credits the consumer's account in the amount of the alleged error (including interest where applicable) within 10 business days of receiving the error notice.” That money is in the account and it is not a finding. A later determination can reverse it.

So a snapshot taken between day 10 and day 45 shows credited accounts that include confirmed errors, disputes still open, and disputes that will be denied. Three states wearing one value.

The remaining distinctions are of the same kind. An observed negative has run its declared window to the end. A right-censored case has not, and is common near the dataset cutoff simply because the cutoff is recent. Missing follow-up means the observation system stopped watching before maturity. That is worse than censoring whenever it stopped watching selectively, because then the missingness carries information about the outcome it hid.

The same temporal pattern appears under different operational names. In several of these domains the horizon is already written down by someone other than the team building the model.

The procedure should be executable from a declared dataset cutoff. One industry has already had this contract adopted by two regulatory bodies. ICH E9(R1), the addendum on estimands and sensitivity analysis in clinical trials, was adopted in November 2019. The EMA's CHMP adopted it in January 2020, in effect from July 2020.

It requires the target of estimation to be specified in advance, through five attributes: the treatment condition, the population, the variable or endpoint, the handling of intercurrent events, and a population-level summary. The first three are steps 1 to 3 here — name the subject and the decision, set the horizon, define maturity. The fourth is step 4, classifying the unresolved cases. The addendum defines exactly what makes them unresolved: “Intercurrent events are events occurring after treatment initiation that affect either the interpretation or the existence of the measurements associated with the clinical question of interest.” It then lists five strategies for handling them.

The ordering is the point. The handling of the events that destroy or distort the outcome is written down before the data are collected. It is not chosen afterwards by an analyst who can already see which choice flatters the metric. Steps 5 and 6 are what an operational system adds on top. Report maturity and follow-up coverage by time period and population. Keep the provisional labels beside the mature ones, so revision can be measured rather than absorbed.

Recent predictions may lack mature outcomes, so a current performance dashboard can overrepresent older traffic. The clocks in this lesson make the size of the gap checkable rather than rhetorical. A payments dashboard read 30 days after scoring cannot see a dispute the consumer still has until 60 days after the statement to file, nor the institution's determination, which may take 45 or 90 days after that. A credit cohort cut before day 180 has not observed the open-end outcome at all, and one cut before day 120 has not observed the closed-end one. Both cuts will happily report a positive rate.

So report the age distribution of evaluated cases and the coverage of mature labels beside every metric. When revisions are common, retain prediction-time scores and label versions. That supports cohort backfill without rewriting the historical decision context: the difference between the 18 November count and the 5 January certification is only measurable if both were kept. For survival or time-to-event problems, methods that model censoring may use more of what is known than excluding every unresolved example, but their assumptions still need domain review.