Units, Features, and Preprocessing for Unsupervised Data

Each example below produces structure. The structure may be technically real and analytically misleading. The fourth one is not a hypothetical. It has a report, a date and a wreck.

The Mars Climate Orbiter launched on 11 December 1998 and was lost on 23 September 1999. NASA's mishap investigation board reported on 10 November 1999. The executive summary of its Phase I Report gives the root cause: “The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,” used in trajectory models.”

Thruster impulse data were passed in pound-force seconds where newton-seconds were expected. That understates the trajectory effect by exactly the 4.45 conversion factor. The estimated first periapsis came out at 57 km against a planned 226 km. Two years later, in 2002, the U.S. General Accounting Office described the same finding from the outside: “A NASA mishap investigation team determined that the loss was due to the mistaken use of English rather than metric units in the navigation software.”

No column was empty. No value was malformed. Nothing in the numbers themselves showed that the two sides meant different things by them. That is exactly the failure a distance function cannot see either.

An operations team clusters a table. Some rows represent vehicles, others represent daily summaries, several represent repair visits. The algorithm returns crisp groups. The groups mainly separate row types.

Unsupervised methods are especially vulnerable to this mistake, because no target exposes the mismatch. Before distance has meaning, every observation must answer the same analytical question.

Five choices determine whether rows are comparable before any algorithm runs.

A stored zero can hide several operationally different states, and the difference between them carries measurable information.

Every patient treated at two Boston hospitals over one year, 2005–2006 — 669,452 people — across 272 laboratory test types. Agniel, Kohane and Weber published what that material showed in the BMJ in 2018, and their abstract states the finding: “The presence of a laboratory test order, regardless of any other information about the test result, has a significant association (P<0.001) with the odds of survival in 233 of 272 (86%) tests.” The timing of when a test was ordered beat the result of the test itself as a predictor of three-year survival in 118 of 174 tests, 68% of them.

An independent group at a different institution, writing in the American Journal of Epidemiology, named the same mechanism informed presence bias. Appearing in the record is itself informative about the subject's state, rather than a neutral fact.

So a cell that says nothing was measured is not a cell that says zero. In 86% of these tests it was carrying prognostic signal of its own. Imputing it away deletes that signal and invents a measurement that never happened.

The pipeline should make every feature reproducible. It should make every missing value interpretable. Step 2 is the one that gets waved through.

The rate at which one entity becomes several unlinked rows has been measured on real record systems. The Patient Identification and Matching Final Report of 2014, prepared for the Office of the National Coordinator for Health Information Technology, gives one organisation's numbers: “For example, Kaiser Permanente (which has 17 instances of Epic across its regions) reported a match rate of greater than 90 percent within each instance; that rate fell to around 50 percent to 60 percent when sharing between regions using a separate instance of Epic or with outside Epic partners.” Same organisation, same software vendor, seventeen installations. Roughly half the cross-region records failed to find their twin.

Five years later the U.S. Government Accountability Office reported the same order of magnitude: “A 2014 study found that as few as 50 percent of records are accurately matched when organizations exchange information.”

Auditing cardinality before aggregation therefore means assuming, until you have counted, that a large share of the rows describing one entity across sources are not linked to each other. The centroid you compute is weighted by that failure.

Rows and columns do not arrive neutral. Aggregation chooses which events matter, missing-value rules define equivalence, and transformations decide which differences survive. Kaufman and colleagues call leakage “one of the top ten data mining mistakes”. Their definition is “the introduction of information about the data mining target” that should not have been available. Nothing in it depends on having labels. A scaler fitted on the whole timeline meets the same description.

The reason to version the contract alongside the algorithm is that defensible choices, made by competent people on one fixed dataset, do not converge. Twenty-nine teams — 61 analysts in all — were given the same football dataset. Their “estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31”. On identical data, “Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship”. Overall the 29 analyses used 21 unique combinations of covariates.

A different consortium in a different field reproduced the phenomenon. Seventy independent teams were given one fMRI dataset and nine pre-specified hypotheses, and they produced 70 different workflows. The report in Nature says it plainly: “The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data.”

A trustworthy unsupervised pipeline therefore versions the dataset contract alongside the algorithm. When structure changes, analysts can investigate the data definition before blaming the model. On this evidence the data definition, not the model, is where the spread lives.