Examples, Observations, and the Unit of Experience

Operational systems store events for billing, logging, support, and compliance. A training example is a deliberate construction. It combines selected information around a prediction unit and time.

The Human Activity Recognition dataset this lesson comes back to twice has every step of that construction on the record. Thirty volunteers aged 19 to 48 each wore a waist-mounted Samsung Galaxy S II. The phone sampled its accelerometer and gyroscope at 50 Hz. The signals were median- and Butterworth-filtered, and then cut. “The time signals were then sampled in fixed-width sliding windows of 2.56 sec and 50% overlap between them”, says the 2013 paper that released the data. Each window was summarised into 561 features. The output is 10,299 records — what Saeb and colleagues, using the same dataset, describe as “an average of 343 records per subject”. On that dataset the multiclass SVM reached “an overall accuracy of 96% for the test data composed of 2947 patterns”.

Read the pipeline backwards and the point is hard to miss. Not one of those 10,299 rows was an event the phone recorded. Every one of them is a decision: this sampling rate, this filter, this window length, this overlap, these 561 numbers. And because consecutive windows overlap by 50%, adjacent rows literally share half of the raw signal underneath them.

One history can be cut into examples several valid ways, and someone chooses which. Netflix made that choice in public and wrote down the reason. Its own 2007 paper describes the release: “over 100 million ratings (and their dates) from over 480 thousand randomly-chosen, anonymous subscribers on nearly 18 thousand movie titles”, collected between October 1998 and December 2005. Those subscribers had years of viewing history apiece — 209 rows per subscriber on average. What went into a row, and where the held-out set was cut, were two separate decisions. Neither was the default.

If one patient contributes ten visits, one machine contributes hourly readings, or one user creates thousands of clicks, those rows are not independent in the ordinary sense. They share identity, environment, history, and often near-duplicate information.

How much identity does a handful of rows carry? Netflix released the Prize data on 2 October 2006 with names replaced by numbers. Narayanan and Shmatikov then measured what was left in it. With 8 movie ratings, of which 2 may be completely wrong, and dates that may have a 14-day error, 99% of records are uniquely identified in the dataset. Two ratings with dates accurate to 3 days pin down 68%. The identifier column had been removed. The rows were still the identifier.

On 12 March 2010 the FTC closed its inquiry into that release. Maneesha Mithal, of its Division of Privacy and Identity Protection, wrote the closing letter. Netflix had agreed to suspend the planned Prize 2 release and, if it ever released such data again, to do so only to researchers under contractual limits. The letter records what had already gone out: “The Prize 1 data set represented the movies rated by over 480,000 Netflix customers and the date each rating was given.”

A model does not need an identifier column either, and the leak has a name and a number. A 2019 paper in npj Digital Medicine calls it identity confounding: with a record-wise split, “the relationship between feature data and disease labels learned by the classifier is confounded by the identity of the subjects”. Its authors ran a permutation test on Parkinson's voice recordings from 22 participants in the mPower study. Shuffle the disease labels until nothing but identity survives, then see what a classifier still scores: “The identity confounding issue was particularly strong for the classifier built from the voice data collected from a subset of 22 participants from the mPower study (Fig. 2c) where the permutation null distribution (which captures the identity confounding) was centered at a 0.95 AUC score.” A model that could not possibly have been detecting disease still scored 0.95. It was recognising who was speaking.

This matters when splitting data and interpreting metrics. Randomly placing records from the same entity in training and test sets can make the task easier than deployment, especially when the model can recognize the entity indirectly — and 0.95 is how easy that recognition can be.

Time-based examples need three separate intervals. Combining them carelessly is a common source of leakage and mislabeled cases.

Outcome horizons are often fixed by policy rather than discovered in the data. Medicare’s Hospital Readmissions Reduction Program was created under the Affordable Care Act. It is set out in section 1886(q) of the Social Security Act. It counts “unplanned readmissions that happen within 30 days of discharge from the index (that is, initial) admission”. CMS began reducing payments to hospitals with excess readmissions for discharges from 1 October 2012. The reduction is “capped at 3 percent (that is, a payment adjustment factor of 0.97)” of a hospital’s Medicare inpatient payments. Six conditions and procedures are counted. They are heart attack, chronic obstructive pulmonary disease, heart failure, pneumonia, coronary artery bypass graft surgery, and elective hip or knee replacement. Nothing in a patient’s chart says thirty days. The statute does not say it either. It defines a readmission as one occurring “within a time period specified by the Secretary”, and offers thirty only as an example. The number is an administrative choice made downstream of the law. Every model built to anticipate that penalty inherits it along with the penalty.

Healthcare is not a special case, and the pattern is not confined to one country's statute book. Banking supervisors fix the outcome clock for credit risk the same way. The Basel Framework's minimum requirements for the IRB approach say a default has occurred when the bank judges the obligor unlikely to pay, or when “The obligor is past due more than 90 days on any material credit obligation to the banking group”. A footnote then admits outright that the threshold is somebody's choice rather than a property of borrowers: “In the case of retail and public sector entity (PSE) obligations, for the 90 days figure, a supervisor may substitute a figure up to 180 days for different products, as it considers appropriate to local conditions.” The EU wrote the same rule into Article 178(1)(b) of Regulation (EU) No 575/2013, carrying the same 90-to-180-day discretion for exposures secured by residential property. The European Banking Authority then spent 29 pages, in 2017, specifying how to count the days — down to the rule that where a law suspends repayment “the counting of days past due should also be suspended during that period”. A model that predicts default predicts a label three bodies defined, on a clock a fourth document says when to stop.

Where no regulator sets the horizon, someone still sets it, and the defensible practice is to say why in writing. Netflix's held-out set, above, is that practice: a per-user time cutoff, adopted because the deployed system had to predict future ratings from past ones, and stated as such in the paper that released the data.

A ten-minute review can reveal mismatches that later appear as mysterious model failures. Name the row, trace its sources, mark the cutoff, identify its relatives, and confirm that live inference can build the same record.

Run those five questions against a dataset whose construction is documented and the answers are short. In the Human Activity Recognition data, one example is one 2.56-second window of filtered accelerometer and gyroscope signal from one volunteer, summarised into 561 features. Its sources are a 50 Hz sensor stream and a filtering-and-windowing pipeline. Its relatives are the adjacent windows sharing half their raw signal, and the 343 windows a subject contributes on average. In the Netflix Prize data, one example is one subscriber's rating of one movie. The cutoff was placed per user, at that subscriber's 9 most recent ratings, not per row. Its relatives are the 209 ratings the average subscriber contributed. Neither review takes long. Both of them change the split before anything is trained.

Sliding windows can turn one long sequence into thousands of examples. This may help training. It also creates overlapping records whose features and targets share much of the same underlying data.

The Human Activity Recognition dataset shows the multiplication with both numbers printed. Thirty volunteers, one waist-mounted phone each, and 10,299 rows: an average of 343 records per subject, with each row sharing half its raw signal with the row beside it. MIT-BIH multiplies harder still — 47 subjects, roughly 110,000 annotations, about 2,340 rows per person. Neither dataset is small. Both have a sample size in the dozens.

Count both rows and distinct entities, episodes, or time periods, because a dataset with one million windows from fifty machines is a very different evidence base from one million windows from fifty thousand machines.