Data Quality Rules, Constraints, and Release Gates

A quality program needs several response levels. Hard failure is costly, and silence is dangerous.

The service-objective column is not an abstraction invented for teaching. Banking supervisors wrote it into a standard. BCBS 239, issued by the Basel Committee in January 2013, sets out fourteen numbered principles. Principle 3 covers accuracy and integrity, Principle 4 completeness, Principle 5 timeliness. G-SIBs designated in 2011 or 2012 had until January 2016 to comply.

Five years after publication, and two years past that deadline, the European Central Bank went to look. It reviewed 25 significant institutions and reported, in May 2018: “Thus far, none of those significant institutions – some of which are classified as global systemically important banks – have fully implemented the BCBS 239 principles.”

The supervisor attributed the weaknesses mainly to unclear responsibility and accountability for data quality. Note what that diagnosis is not. It is not that the thresholds were wrong, or that the banks lacked metrics. Twenty-five of the largest institutions in Europe had a numbered standard demanding accuracy, completeness and timeliness, and zero of them had it working. A target with no owner and no prescribed response is an observation wearing the costume of a control.

“Null rate below five percent” is incomplete. The rule should name the field, population, window, reference, severity, owner, and response when it fails.

Rules should distinguish producer defects from legitimate population change. A holiday can alter order volume without breaking the pipeline. A stable row count can hide duplicated events.

So the output of validation has to carry evidence: violating examples, affected slices, first-seen time, lineage, and the prior successful version.

Amazon built exactly that and published the design in 2018. Deequ offers “a declarative API, which combines common quality constraints with user-defined validation code, and thereby enables ‘unit tests’ for data”. It runs the workload “by translating it to aggregation queries on Apache Spark”, so the check executes at the size of the table rather than at the size of a sample.

Deequ also retains its historic quality metrics, which is the precondition for anomaly detection over the series. Without the stored history there is no reference. Without a reference, a threshold is a guess about yesterday.

Higher layers catch failures that lower layers cannot see.

Google runs this stack at scale and published how in 2019. Its data validation system is “deployed in production as an integral part of TFX” and “used by hundreds of product teams” to “continuously monitor and validate several petabytes of production data per day”. The paper names the hard part as pipelines that must “soldier on in the face of unexpected patterns, schema-free data, or training/serving skew”. Scale changes which checks are affordable.

Field semantics — the second layer — carries the most expensive documented failure of the five. NASA lost the Mars Climate Orbiter at Mars on 23 September 1999. The mishap board’s report states: “The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,” used in trajectory models.” Thruster impulse data had been delivered in pound-seconds where the software interface documentation specified Newton-seconds. The small-forces values feeding orbit determination were low by a factor of 4.45. At insertion the trajectory ran approximately 170 kilometres lower than planned, and post-hoc estimates put periapsis at 57 km against a survivable minimum of 80 km. The U.S. General Accounting Office recorded the same cause independently: “The loss was due to the mistaken use of English rather than metric units in the navigation software.”

Every value in that file parsed. Every value was typed, and every value fell inside a plausible numeric range. The encoding layer had nothing to report, because no rule compared the delivered unit against the declared one. The FDA states the general form of the defect in one sentence about laboratory records: “the number ‘23’ is meaningless without metadata, such as an indication of the unit ‘mg.’” The values were fine. What was missing was what they meant.

Population fitness sits at the top because it is the layer at which every record can be individually valid and the dataset still wrong. Three researchers audited the test sets of ten of the most widely used vision, NLP and audio benchmarks in 2021. Their estimate: “we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set”. Human crowdworkers confirmed 51% of the algorithmically flagged candidates as genuine errors.

The consequence is a reversed decision, not just a worse number. On corrected labels ResNet-18 overtakes ResNet-50, once the prevalence of originally mislabelled test examples rises by just 6 percentage points. A separate group re-annotated the same ImageNet validation set and landed in the same place: “we find the original ImageNet labels to no longer be the best predictors of this independently-collected set”. No row-level validator can see any of this. No individual row is malformed.

These examples show why row-level schema validation is necessary but insufficient. Three of the five are documented failures with a public record and a cost attached. In each case the structural checks were correct.

Start with an incident, or with a decision the team could not accept, then design the narrowest control that would have caught it.

Take a documented one. In October 2020 England found out that thousands of COVID-19 cases had never reached its contact tracers. Fetzer and Graeber open their study of it with the whole incident in two sentences: “Between September 25 and October 2, 2020, a total of 15,841 COVID-19 cases in England (around 15 to 20% of all cases) were not immediately referred to the contact tracing system due to a data processing error. Case information was truncated from an Excel spreadsheet after the row limit had been reached, which was discovered on October 3.” Public Health England’s own statement of 4 October 2020 gives the same count, with 11,968 of the cases falling between 30 September and 2 October, and attributes the loss to files exceeding the maximum size handled by the data load process.

Run the six steps against it. Step 1 is the bad outcome, not the anomalous metric: contacts who were never traced, not a dip in a load chart. Step 2 is the earliest observable signal, and here it is unusually clean. The defect becomes detectable at the load boundary itself, where submitted rows can be reconciled against loaded rows. Step 3 fixes scope: per file, per load run. No seasonality argument is available, because a submitted case is either loaded or it is not. Step 4 chooses the response, and this one is an invariant rather than a guardrail. A fixed-capacity container silently dropping records means the data cannot represent a valid state, so the release blocks or quarantines. Step 5 attaches the diagnostics that make the block actionable: the file, the run, the count of rows submitted against rows loaded, the owner.

Step 6, reviewing precision, is where the case earns its place. A row-count reconciliation raises almost no false alarms, so the only real question is what the missed incident cost. Fetzer and Graeber estimate the delayed contact tracing was associated with more than 125,000 additional infections and over 1,500 additional COVID-19 deaths. Their peer-reviewed version in PNAS gives roughly 127,000–185,000 additional infections and 1,500–2,000 additional deaths. That is the price of one comparison of two integers that nobody had written down as a rule.

Quality controls accumulate as systems evolve. Without review, teams inherit obsolete thresholds, duplicate alerts, and exceptions that nobody understands. Version rules with the data contract, record suppressions and overrides, and evaluate which incidents each rule actually detects. Retire controls whose premise no longer holds. A release report should show rule version, result, affected population, action taken, and unresolved waivers. That history is part of the dataset’s evidence.

Two medicines regulators already require this discipline as a condition of releasing a batch, and they arrived at it independently. The FDA defines data integrity as completeness, consistency and accuracy under the ALCOA attributes — attributable, legible, contemporaneous, original, accurate — and refuses to let the obligation stop at the moment of capture. Its December 2018 guidance is explicit: “Data integrity is critical throughout the CGMP data life cycle, including in the creation, modification, processing, maintenance, archival, retrieval, transmission, and disposition of data after the record’s retention period ends.” Questions 7 and 8 of the same guidance cover audit-trail review, which is the release report’s ancestor.

The MHRA states the ownership half. Its March 2018 guidance requires that “Data governance should address data ownership and accountability throughout the lifecycle, and consider the design, operation and monitoring of processes/systems to comply with the principles of data integrity including control over intentional and unintentional changes to data.” It carries the same ALCOA attributes.

Ownership, lifecycle, controlled change: that is also what BCBS 239 asked of banks from January 2013. By the ECB’s May 2018 count, none of the 25 significant institutions reviewed had fully implemented it, and the supervisor blamed unclear responsibility and accountability for data quality rather than any missing metric. Two industries, two regulators each, one finding. The hard part of a quality rule is not the threshold. It is the name attached to it, and the record of what happened when it fired.