Skip to content
AI.info

Classical machine learning

The Tabular Data Contract

Turn operational records into point-in-time-correct learning examples with explicit identity, timing, type, and dependence rules.

By the end you can

Example

A row is a claim about the world

Tabular learning starts with one question, and it sounds simpler than it is: what does one row mean at the instant a prediction is made?

Some industries have been forced to answer it in writing. The answer is never a sentence. It is a specification. Article 178(1)(b) of Regulation (EU) No 575/2013 — the Capital Requirements Regulation, of 26 June 2013 — does not ask a bank to record when a loan goes bad. It puts an obligor in default when more than 90 days past due on any material credit obligation. An event, a threshold, a clock, all fixed before anyone opens a notebook.

Where the question is left implicit, the cost turns up in the literature. Kapoor and Narayanan counted leakage in 294 papers, spread over 17 fields, in eight distinct forms. The question does not disappear when nobody writes the answer down. It just gets answered by the data.

  • Loan application: one row should represent one application at a declared decision time, not an applicant merged with later repayment information — and the label is the legal one, more than 90 days past due on a material credit obligation.
  • Equipment maintenance: one row may represent a machine-day, while the target describes failure within the following seven days.
  • Customer retention: repeated monthly rows for one customer require grouped and temporal evaluation rather than random row shuffling.
  • Fraud screening: transaction features must be available before authorization, even if richer fields appear in the warehouse minutes later.
  • Clinical triage: measurements from the encounter must be separated from test results obtained after the intended intervention point; Epic's sepsis model, deployed across hundreds of US hospitals, scored an AUC of 0.63 when Michigan Medicine validated it externally.

Entity, event, observation, and prediction time are different objects

An entity is a persistent object such as a patient or a machine. An event is something that happens to it. An observation is a recorded measurement. A training row is none of those: it is an analytical construction, assembled afterwards.

Confusing these levels creates duplicate evidence, wrong sample sizes, and leakage. The dataset contract should state how events become rows, and which timestamps govern availability.

The INFORMS 2010 Data Mining Challenge shows what happens when nobody governs the timestamps. Entrants worked from intraday trading data. Every five minutes they had to say whether a target stock would rise or fall over the following 60 minutes. Almost 900 people took part. The winning entry scored an AUC of 0.99, and about 30 groups passed 0.9.

Kaufman, Rosset and Perlich reconstructed why the next year. The explanatory variables carried values recorded after the prediction moment. Publicly available price series could also be used to work out which anonymised stock was being predicted. The organisers had seen part of it coming: they ran two leaderboards, one allowing future information and one forbidding it. In the end they conceded that verifying which entries had stayed on the right side of that line was impossible.

The task was specified precisely. The row was not.

Visual

The time contract behind one supervised row

A valid row uses only information available by the cutoff, and defines the target in a later outcome window. That is usually presented as good practice. In one industry it is enforceable text.

Article 178(1)(b) of Regulation (EU) No 575/2013 puts an obligor in default when more than 90 days past due on any material credit obligation. That single clause fixes three things: the outcome window of more than 90 days, the materiality threshold below which the count does not start, and the event whose clock is authoritative.

What the clause does not fix is when the label may flip back. The European Banking Authority supplied that. Its guidelines require a bank to “consider that no trigger of default continues to apply to a previously defaulted exposure, where at least 3 months have passed since the moment that the conditions referred to in Articles 178(1)(b) and 178(3) of Regulation (EU) No 575/2013 cease to be met”. The same document states that “These guidelines apply from 1 January 2021”, and it carves “technical past due situations” out of the day count altogether.

Read that as a data contract and every item on this timeline is already in it: a history window, a cutoff, an outcome window with a stated length, an exclusion rule for rows that only look like events, and a three-month probation before the label is allowed to flip back. That probation is the maturity date. Paragraph 5 of Article 178 governs the return to non-defaulted status, and until the probation has run, the label is not trustworthy.

The feature cutoff has an institution of its own, kept by two Federal Reserve Banks. The analogy section below describes it.

FigureTimeline · 5 stops

  1. History window

    Past events are aggregated under explicit lookback rules.

  2. Feature cutoff

    The latest instant from which information may enter the row.

  3. Decision point

    The moment the system would produce its output in operation.

  4. Outcome window

    A future interval in which the label is observed.

  5. Maturity date

    The earliest time when the row has a complete, trustworthy target.

Comparison

Fields that look similar but carry different risks

Column names rarely reveal whether the data is valid for learning. How each field was produced does. The sharpest demonstration involves an identifier that was never a column at all.

A 2018 study in PLoS Medicine assembled 158,323 chest radiographs from three hospital systems: 112,120 from NIH covering 30,805 patients, 42,396 from Mount Sinai covering 12,904, and 3,807 from Indiana University covering 3,683. Pneumonia was distributed very differently across the three, and Zech and his co-authors put the consequence in one sentence: “The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset.”

Knowing only which hospital sent the image was worth 0.861 of the diagnostic task. And convolutional networks could recover that fact almost perfectly from the pixels: the source hospital for 99.95% of NIH radiographs (22,050 of 22,062) and 99.98% of Mount Sinai radiographs (8,386 of 8,388). The best internal model reached an AUC of 0.931 and fell to 0.815 when tested externally.

An identifier does not have to be stored in a key column to become a memorisation shortcut. It only has to be reconstructible.

The same mechanism turned up in a different disease. A group at the University of Washington reported in Nature Machine Intelligence that COVID-19 detectors on chest radiographs select shortcuts over signal. The Agency for Healthcare Research and Quality's PSNet summarises it in one line: “This study evaluates diagnostic 'shortcuts' learned by AI systems in detecting COVID-19 in chest radiographs.”

FigureComparison · 4 columns

Snapshot feature

A value known at the prediction cutoff.

  • Account balance at 09:00
  • Sensor average through midnight
  • Current inventory quantity
  • Safe when historical reconstruction is correct

Retrospective feature

A value computed after the outcome began to unfold.

  • Chargeback status added later
  • Final repair code
  • Discharge diagnosis
  • Often leaks future knowledge

Stable attribute

A slowly changing property tied to an entity.

  • Machine model
  • Region
  • Product category
  • Needs effective-date handling

Identifier

A key used for joining or grouping.

  • Customer ID
  • Device serial number
  • Hospital encounter ID
  • Can become a memorization shortcut

Steps

Write the row contract before extracting data

A short written contract prevents months of downstream confusion. Two of its five steps have already been written, for particular industries, by people with enforcement powers. That is the best evidence they are not stylistic preferences.

Step 3, the target horizon, has a finished worked example in EU bank capital law. The outcome window is more than 90 days past due, under Article 178(1)(b) of Regulation (EU) No 575/2013. There is a materiality threshold. There is an exclusion for technical past due situations. And there is a three-month probation, under paragraph 71(a) of EBA/GL/2016/07, before a defaulted exposure may be labelled non-defaulted again. Outcome, window, censoring, maturity — all four, in force.

Step 4, identity rules, is guiding principle 4 of the 10 Guiding Principles for Good Machine Learning Practice for Medical Device Development, issued jointly on 27 October 2021 by the US Food and Drug Administration, Health Canada and the UK's Medicines and Healthcare products Regulatory Agency: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.”

Three regulators name the three dependence sources explicitly: patient, data acquisition, site. That list is exactly what the two medical failures in this lesson violate — site, in the chest radiographs above, and patient, in the brain MRI splits below. Deduplication, repeated entities, joins and group boundaries are what step 4 asks a team to write down. The wording to copy already exists.

FigureProcess · 5 steps

  1. 1. Define the unit

    State exactly what one prediction and one row represent.

  2. 2. Fix the cutoff

    Declare when features stop and which clocks are authoritative.

  3. 3. Define the target horizon

    Specify outcome, window, censoring, and maturity.

  4. 4. Set identity rules

    Document deduplication, repeated entities, joins, and group boundaries.

  5. 5. Record exclusions

    Explain which rows disappear and whether exclusion depends on future information.

Analogy

A dataset as a stack of sealed case files

Each training row is a case file sealed at the prediction time. Reviewers may open only documents stamped before that seal, then return later to attach the observed outcome.

Case files stay as they were filed. Source systems do not: they revise old records and run on delayed clocks. So a correct pipeline must track event time, processing time, and revision history explicitly.

Two central banks pay to keep those seals intact, and they do it because the alternative changed a published conclusion. Macroeconomic series are revised after publication, so the value stored today is not the value that was knowable at the decision date. Athanasios Orphanides measured what that costs, in the American Economic Review in 2001: “Using Taylor's rule as an example, I demonstrate that real-time policy recommendations differ considerably from those obtained with ex post revised data.” His abstract concludes that “reliance on the information actually available to policy makers in real time is essential for the analysis of monetary policy rules”.

So the archives exist. The Federal Reserve Bank of Philadelphia maintains the Real-Time Data Set for Macroeconomists — “vintages, or snapshots, of time series of major macroeconomic variables”. The Federal Reserve Bank of St. Louis independently runs ALFRED, the ArchivaL Federal Reserve Economic Data, which “allows you to retrieve each economic data release (vintage) that was available on a specific date in history”.

Point-in-time correctness is not a metaphor a data scientist reaches for. It is a line item in two institutional budgets.

A valid row recreates what could have been known, not what the warehouse knows today.

Key idea

Rows are often not independent

Repeated customers, households, sites, or devices can make a dataset look much larger than its independent evidence. Random splits then place related patterns on both sides of the line.

Group-aware or time-aware evaluation is a property of the data contract, not an optional metric trick.

A 2021 paper in Scientific Reports put a number on the wrong split unit. Nine authors cut 3D brain MRI volumes into 2D slices, then cross-validated at the slice level, so that slices from one patient sat on both sides of the split. Doing so “erroneously boosted the average slice level accuracy on the test set by 30% on Open Access Series of Imaging Studies (OASIS), 29% on Alzheimer’s Disease Neuroimaging Initiative (ADNI), 48% on Parkinson’s Progression Markers Initiative (PPMI) and 55% on a local de-novo PD Versilia dataset”.

The control is the sharpest part of the paper. They took an OASIS-derived set whose labels had been assigned at random — a dataset in which there is, by construction, nothing to learn. The slice-level split still reported “about 96% of (erroneous) accuracy”. The subject-level split reported the 50% a randomised experiment must produce.

Between 96 and 50 lies no signal at all. Only the same patient, appearing twice.

Count independent decision contexts, not merely table rows.

Types are semantics, not storage formats

The storage type is not the meaning. An integer column can be a count, a category, a rank, a duration, an identifier, or a code. Treat every integer as a continuous number and you invent distances and trends that mean nothing.

The contract should describe units, valid ranges, missingness meanings, category evolution, and whether zero is an observation or a placeholder.

A day count shows how much definition one integer needs. The 90 days of Article 178(1)(b) run only against a material credit obligation, and paragraphs 23–24 of EBA/GL/2016/07 exclude technical past due situations from the count entirely. Two databases can both store an integer named days_past_due and mean different things by it.

The model sees encodings; the team must preserve the meaning behind them.

Key idea

A polished table can still be invalid evidence

Clean schemas and complete columns do not prove point-in-time correctness. A perfectly formatted feature can still contain future information, or describe a population that is excluded from deployment.

Epic's proprietary sepsis prediction model is the case that reached the bedside. It was deployed across hundreds of US hospitals, on tables that passed every check a hospital data warehouse applies. Michigan Medicine then validated it externally across 27,697 patients and 38,455 hospitalizations containing 2,552 sepsis cases. Wong and colleagues reported the result in JAMA Internal Medicine: “The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64).” Epic's own documentation reported 0.76–0.83. The paper concludes that “the ESM has poor discrimination and calibration in predicting the onset of sepsis”.

A second institution repeated the exercise on the same shipped model. Writing in JAMIA Open, six authors covered 145,885 encounters at two county emergency departments in 2023 and reported “a sensitivity of 14.7%, specificity of 95.3%, positive predictive value of 7.6%, and negative predictive value of 97.7%” within a 6-hour sepsis window. Their conclusion: “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED.”

Validate how each field was produced, not only whether it passes a type check. Formatting is not evidence. Neither is a vendor's retrospective number.

Data quality begins with provenance and timing before it reaches missing-value counts.

Case

Anonymised node numbers were re-identified from a second crawl of Flickr

The IJCNN 2011 Social Network Challenge shipped a table that passed every format check and was still not valid evidence. Kaggle released a graph of 1,133,547 nodes and 7,237,983 edges, with user identities replaced by numeric IDs, plus a test set of 8,960 candidate edges, half real and half fabricated.

Arvind Narayanan and two colleagues learned from the contest forum that the graph had been crawled from Flickr. So they crawled Flickr themselves — 9,124,801 nodes and 163,579,517 directed edges — and matched the two graphs onto each other: “we were able to successfully de-anonymize 64.7% of the test edge-set”. They trained a random forest on standard link-prediction features plus the de-anonymised portion and won with a test AUC of 0.981. The best entry that actually predicted links reached 0.969.

Deleting the identifier column removed nothing. The shape of the table was the identifier.

Figure

The margin that won the contest is a hairline; what bought it was a second crawl an order of magnitude larger than the released table. Narayanan, Shi and Rubinstein at the IJCNN 2011 Social Network Challenge; the 0.012 gap, the recovered edge count and the two size ratios are derived from the challenge’s own counts.

Position

Leakage is settled by the definition of a row, not by an audit of the columns

The hygiene a beginner is taught is a blocklist. Drop the identifier, drop the fields that were obviously written after the outcome, run the schema check, proceed. Every case in this lesson defeats it, and in none of them would a longer blocklist have helped.

Kaggle had already removed the identities from the IJCNN 2011 graph. The node numbers were random and the columns were clean. The winning team still recovered 64.7% of the test edges by crawling Flickr themselves and matching one graph onto the other, which was worth a test AUC of 0.981 against the 0.969 of the best entry that only predicted links. In the brain-MRI benchmarks every column was a legitimate measurement of the patient it came from; the leak was that slices of one subject sat on both sides of the split, worth between 29% and 55% of apparent accuracy depending on the cohort, and 96% on labels assigned at random where the honest answer was 50%. In the chest radiographs there was no hospital column to drop, and the site was still worth an AUC of 0.861 on its own. The INFORMS organisers understood their own leak well enough to run two separate leaderboards for it, and still conceded that verifying which entries had stayed on the right side was impossible.

Nor is this a collection of famous accidents. Kapoor and Narayanan surveyed machine-learning-based science and report that “we find 17 fields where leakage has been found, collectively affecting 294 papers”, sorted into “a detailed taxonomy of eight types of leakage”. They then reproduced one literature end to end — civil-war onset prediction — narrowing 124 papers to the 12 that both used a train-test split and shared complete code and data. Four of those 12 had errors. The four were “exactly the 4 papers that claimed superior performance of complex ML models over baseline LR models”. Their verdict: “When the errors are corrected, complex ML models do not perform substantively better than decades-old LR models.”

Yu Wang had already reached the same conclusion independently about the flagship paper of that literature, Muchlinski and colleagues (2016). Writing in Political Analysis, he found that “the real AUC as presented in Figure 2 in the original article is 0.97 rather than 0.91, and the model is trained with the entirety of the dataset”. Nothing on the page of the original paper showed a reader which of the two numbers they were looking at.

So the question a data review has to answer first is not which columns are dangerous. It is this: what does one row represent, at which instant was it sealed, and which other rows describe the same entity. Answer those three and most of the column list follows from them. Leave them unwritten and there is nothing for a blocklist to be a list of.

No blocklist is long enough to repair a row nobody defined.

Key takeaways