Prediction Time, Unit of Analysis, and Outcome Horizon

Each of these joins executes successfully, and each one changes the statistical population the model is actually fitted to. How much it changes has been measured.

Take three-dimensional brain MRI. Cut it into two-dimensional slices, then split the slices instead of splitting the subjects. Different slices of the same brain now land in training and in test. Yagis and colleagues did exactly that and reported the result in Scientific Reports in 2021: “slice-level CV erroneously boosted the average slice level accuracy on the test set by 30% on Open Access Series of Imaging Studies (OASIS), 29% on Alzheimer's Disease Neuroimaging Initiative (ADNI), 48% on Parkinson's Progression Markers Initiative (PPMI) and 55% on a local de-novo PD Versilia dataset.” Each of the three public cohorts used 200 subjects, 100 cases and 100 controls. The local de-novo Parkinson's dataset from Versilia had 34.

Nothing was wrong with the model, the features, or the code. The row was the wrong thing, and thirty to fifty-five points of accuracy came out of that alone.

Nor is this an isolated slip. Wen and colleagues surveyed convolutional-network papers on classification of Alzheimer's disease for Medical Image Analysis in 2020. They concluded that “more than half of the surveyed papers may have suffered from data leakage” from splits that put the same subject in train and test. A grain error is the most reproducible mistake in applied machine learning because it never announces itself. The query returns rows. The metrics improve. The improvement is the error.

Every training row makes an assertion: this subject existed in this state at this time. If the assertion is vague, the model learns from an unstable target.

The unit of analysis is the thing one row represents. It may be a transaction, customer-day, machine-hour, hospital encounter, product-week, image, document, or user-item impression.

The correct unit follows the decision. A model that approves one transaction needs transaction-level examples. A model that schedules store inventory may require product-store-week examples.

Many data problems begin when sources use different grains. Joining order lines to orders, device pings to sessions, or repeated laboratory measurements to encounters can silently multiply records.

Leakage has now been counted across the sciences. Sayash Kapoor and Arvind Narayanan surveyed reviews of machine-learning-based science in 2023 and found “17 fields where leakage has been found, collectively affecting 294 papers”. They set out “a detailed taxonomy of eight types of leakage, ranging from textbook errors to open research problems”. Much of what that taxonomy catches is a grain or a time mistake rather than a modelling one. In civil war prediction, once the errors were corrected, they report that “complex ML models do not perform substantively better than decades-old LR models”. Two hundred and ninety-four papers is not a warning about carelessness. It is a description of what happens when the row is defined after the join rather than before it.

Selecting a grain is not a question of how the table is laid out. It changes labels, how far one row leans on the next, and operational use.

An event-level row turns one observed action or measurement into one example. It is the natural choice for transaction scoring and click prediction: score each card payment at authorization. It can also create many correlated rows for a single entity, and every feature it uses has to have existed before the event it describes.

An entity snapshot represents one entity at a scheduled prediction time — predict customer churn every Monday. It is natural for churn, risk, and account prioritization, and it requires a clear history window and cutoff. Repeated snapshots of the same entity are not independent of one another, so they need grouped or temporal evaluation rather than a random split.

An episode-level row makes a bounded sequence the unit: a visit, a session, a trip, a claim, a support conversation classified after closure. Episode boundaries are frequently inferred rather than observed. A late event can reopen or alter an episode the dataset has already labelled.

These three are not interchangeable descriptions of the same data. Each defines a different population, with different dependence between rows and a different meaning for the same metric. The choice is the first modelling decision a team makes, and usually the only one nobody records.

The same person or asset can appear at several useful levels, and those levels should not be mixed accidentally. An entity is a persistent subject: a customer, device, merchant, patient, or machine. An episode is a bounded period inside that subject's life — a session, visit, claim, loan, or maintenance cycle. An event is an occurrence with a timestamp, such as a click, payment, reading, or diagnosis. A windowed example is a modelling snapshot that aggregates permitted history before a prediction time.

Collapsing one level into another is not a tidiness concern about joins. It is a named error with a counted prevalence. Kahan and colleagues reviewed 100 cluster randomised trials in 2016. Of the 99 that reported the number of clusters, 64 — 65% — were at risk of an inflated type I error rate. In 14 of them, no clustering-aware analysis was reported at all. In 50, an individual-level analysis had been run on fewer than 40 clusters with no correction. That is roughly two trials in three, in a literature with editors, statisticians and reviewers, making the same mistake a one-to-many join makes silently.

The standards text is explicit about the mechanism. The Cochrane Handbook for Systematic Reviews of Interventions puts it this way: “If the clustering is ignored and cluster-randomized trials are analysed as if individuals had been randomized, resulting confidence intervals will be artificially narrow and P values will be artificially small.” Artificially narrow intervals and artificially small p-values are precisely what a duplicated-label join produces in a model report. Apparent precision, manufactured by counting one subject many times.

The reporting is no better than the analysis. Richardson and colleagues examined 50 reviews covering 201 cluster-randomised trials and found that only 8 of 33 — 24% — reported the method of cluster adjustment. When the nesting is not written down, a reader cannot tell whether it was handled.

One short page agreed up front prevents months of teams disagreeing inside transformation code. Name the subject: the persistent entity and any secondary entity involved in the decision. Fix the decision moment: when the prediction is requested, and which clock establishes that time. Bound the history: the earliest and latest observations allowed in the feature window. Define the outcome: the event, the horizon, and the censoring rule used to assign the target. Declare uniqueness: the key that must identify exactly one example, tested continuously.

The second step is the one that gets skipped, and a widely deployed model shows what skipping it costs. The Epic Sepsis Model runs at hundreds of US hospitals. Wong and colleagues validated it externally on 27,697 patients and 38,455 hospitalizations at Michigan Medicine. Reporting in JAMA Internal Medicine in 2021, they found an area under the curve of 0.63 (95% CI 0.62–0.64), against the developer's reported 0.73–0.83.

The timing result is more damaging than the discrimination result. Ostermayer and colleagues examined 145,885 emergency-department encounters and reported in 2024 that “Providers were alerted with a median lead time of 0 minutes (80% CI, −6 hours and 42 minutes to 12 hours and 0 minutes).” Half the alerts carried no lead time at all. The reported 80% interval around that median runs from a lag of six hours and 42 minutes to a warning of twelve hours.

A prediction that arrives at the moment of the decision it was meant to precede is not an early warning. It is a description of something already happening. That is what an unfixed prediction time produces in the field. If the specification does not say which clock defines the decision moment, and which observations are permitted strictly before it, the cutoff drifts onto the decision itself. The model then reports what a clinician can already see.

Ask whether two engineers would produce the same row count from the written specification. If not, the grain or boundaries remain ambiguous.

Check whether each serving request maps to exactly one training-example definition. A mismatch means the learner is solving a related but different problem. That question is not house style. It is Guiding Principle 4 of the ten in Good Machine Learning Practice for Medical Device Development, issued jointly on 27 October 2021 by the US Food and Drug Administration, Health Canada and the UK Medicines and Healthcare products Regulatory Agency. The principle is titled “Training Data Sets Are Independent of Test Sets”, and it reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.” Three named sources of dependence — patient, acquisition, site — and the verb is addressed, not assumed absent. The document is non-binding guidance rather than a requirement. But three regulators wrote the same sentence down together, which is a stronger endorsement than most engineering conventions have.

Measure examples per entity and per episode. A few entities dominating the count can reveal duplicate joins, heavy users, repeated snapshots, or operational processes that determine who gets measured.

Finally, inspect a small sample as timelines rather than rows. Timelines make cutoff violations, incomplete horizons, and episode-boundary mistakes easier to see, because the thing that goes wrong in all of them is an ordering, and an ordering is invisible in a row preview.