Data Profiling and Statistical Baselines

Always profile important slices and intersections, not only the full table. An aggregate can hold perfectly still while a subpopulation inside it collapses.

The largest published demonstration of this is a government test rather than an argument. In December 2019 the National Institute of Standards and Technology published NISTIR 8280: 18.27 million images of 8.49 million people, run through 189 algorithms from 99 developers. The executive summary section headed “WHAT WE FOUND” states the result without hedging. “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” False positive rates were highest for West and East African and East Asian people, and lowest for Eastern Europeans, on the same high-quality application photos. One reported error rate, two orders of magnitude between the slices inside it.

The U.S. Government Accountability Office restated the same measurement in its own words: “In verification algorithms, false positive rates for white males and black females varied by factors of 10 to more than 100, meaning the lowest-performing algorithm could be over 100 times more accurate on white male faces than on black female faces”. It confirmed the scale of the test too: “NIST tested 189 mostly commercial algorithms from 99 developers”. Nothing in a single global accuracy figure carries that information. Neither does a one-axis breakdown. The intersectional profile is what exposed it.

The same shape recurs in ordinary pipelines. Profile each of these axes on its own, and then their intersections:

A dataset can be complete but stale, valid but unrepresentative, unique but temporally leaked. It can be statistically stable while one critical region disappears. Calling all of these “quality” hides the mechanism.

Profiling summarizes what is present before the team decides what should be present. It measures types, ranges, categories, quantiles, missingness, cardinality, duplicates, time coverage, and relationships.

Validation compares data against declared expectations. Anomaly detection looks for unusual departures. Observability connects those signals to lineage, incidents, ownership, and response.

The purpose of profiling is not a decorative dashboard. It is to discover assumptions, expose population structure, and decide which failures deserve enforceable controls.

The cost of skipping this work has been measured. Sambasivan and colleagues reported in 2021 on evidence drawn “from interviews with 53 AI practitioners in India, East and West African countries, and USA”. They found data cascades “pervasive (92% prevalence), invisible, delayed, but often avoidable”. Ninety-two percent is the prevalence. The word to notice is delayed. The damage surfaced far downstream from the data step that caused it, which is why the profile is the cheapest place to stop a cascade and the model evaluation is the most expensive.

Validity, completeness, consistency, timeliness, representativeness and historical correctness are separate properties. One can pass while another fails.

Between 25 September and 2 October 2020, Public Health England lost 15,841 positive COVID-19 cases from its reported figures. Every row that survived was valid, internally consistent and delivered on time. The population was simply incomplete. Michael Brodie, the Interim Chief Executive of Public Health England, set out the mechanism on GOV.UK on 4 October 2020: “A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards. After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases.” The stated cause was a file boundary — “some files containing positive test results exceeded the maximum file size”. Over 75% of the loss, 11,968 cases, was concentrated in the last three days, 30 September to 2 October.

Completeness failed alone, and the consequence was quantified afterwards. Fetzer and Graeber treated the incident as a natural experiment in PNAS: “Because of a coding error involving spreadsheet data used by the health authorities, a total of 15,841 COVID-19 cases (around 20% of all cases) failed to have timely contact tracing”. Cases that were properly traced were associated with 63% fewer subsequent infections and 66% fewer subsequent COVID-19-related deaths over six weeks.

A composite quality score would have rated this dataset healthy. Five dimensions passing, one silently failing at a file-size limit, and roughly a fifth of the population gone.

Organize the report around likely failure modes and consumers, not around every statistic a tool can compute: confirm grain and keys, profile fields and time, compare slices, examine relationships, and convert findings into controls.

Step four — examine relationships, cross-field invariants, target associations and suspicious shortcuts — is the step teams treat as optional. It is where a defect can sit that no distribution check will ever raise.

One commercial risk-prediction algorithm, from a class of tools applied to roughly 200 million people in the United States each year, shows what that looks like. Obermeyer and colleagues dissected it in Science in 2019. The feature distributions were unremarkable. The target was the defect. The algorithm predicted health-care cost rather than illness, so Black patients at the same risk score were considerably sicker than white patients at that score. The abstract states the size of it in one sentence: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.”

Regulators named the same mechanism on the same date. The New York State Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group Incorporated about that algorithm, Optum’s Impact Pro: “By relying on historic spending to triage and diagnose current patients, your algorithm appears to inherently prioritize white patients who have had greater access to healthcare than black patients.”

A profile that stopped at types, ranges and missingness would have passed this dataset without comment. Step four is for profiling how the target itself associates with slices — the label, not only the features. It belongs in the profile, not in a fairness review held after the model ships.

Compare current data with several references: the previous period, the same seasonal period, the training snapshot, serving traffic, and a known-good release. Each comparison answers a different question. A day-over-day reference catches sudden pipeline breaks but may alert on weekends. A year-over-year reference captures seasonality but can miss recent product changes.

Google Flu Trends is what a baseline looks like when it is fitted once and never re-compared. The model was built against CDC influenza-like-illness reports, then drifted away from them in one direction for years. Lazer and colleagues counted the drift in Science in 2014: “GFT also missed by a very large margin in the 2011–2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011”. A hundred misses out of a hundred and eight is not noise. The errors were autocorrelated and seasonal, the signature of a stale reference rather than of a surprising world. Three-week-old CDC data projected current prevalence better than the model did.

An independent reassessment in PLOS Computational Biology found the same failure at national, regional and New York City scales: “We identified substantial flaws in the original and updated GFT models at all three geographic scales, including completely missing the first wave of the 2009 influenza A/H1N1 pandemic, and greatly overestimating the intensity of the A/H3N2 epidemic during the 2012/2013 season.”

Slice references need adequate sample size and stable definitions. It is possible to state the size at which a slice stops being trustworthy, instead of leaving it to judgement. Ofqual did so for its 2020 grade standardisation model in England: “Where schools and colleges had a relatively small cohort for a subject – fewer than 15 students when looking across the current entry and the historical data – the standardisation model put more weight on the CAGs”, the centre assessment grades supplied by teachers. Its interim report of 13 August 2020 gives the reason in its executive summary: “However, there is no statistical model that can reliably predict grades for particularly small groups of students. We have therefore used the most reliable evidence available, which is the CAGs.” The regulator set its statistical threshold at 5, below which no significance test could establish a real difference. It also reported that 96.4% of final calculated grades were within one grade of the submitted centre assessment grade, an aggregate that looked reassuring.

The Office for Statistics Regulation later identified where the checking had stopped: “For output data, the regulators undertook a wide range of analysis, largely at an aggregate level. There was limited human review of outputs of the models at an individual level prior to results day.”

Store the reference version and comparison method with every alert. An anomaly without a named baseline cannot be reproduced or interpreted. A baseline that is never re-compared can be wrong in the same direction for a hundred weeks running.