Data Validation and Release Gates

A metric can be theoretically useful and still do nothing, because the alert carrying it has no context and no action attached. The cost of that is measured, not guessed. Hospital prescribing software warns clinicians about dangerous drug combinations. A 2006 review of 17 such studies by van der Sijs and colleagues, in the Journal of the American Medical Informatics Association, opened with the number: “Drug safety alerts are overridden by clinicians in 49% to 96% of cases.” The review named the conditions that produce the overriding: low specificity, unclear information content, unnecessary workflow disruptions. Those are the same three properties an under-designed data-quality alert has.

The decay itself was measured next. Ancker and colleagues examined 1,266,325 alerts, of which 326,203 were drug-drug or drug-allergy interaction alerts, across 430,803 encounters, and published the result in 2017. The likelihood of accepting a reminder fell by 30% for each additional reminder in the same encounter. It fell by 10% for each five-percentage-point rise in the share of reminders that were repeats. Attention is a budget with a decay curve. Every uninformative page spends part of it.

Validation should run in three places: near the sources, after major transformations, and before a version becomes available to consumers. Each place catches a different failure, and each costs a different amount to recover from. A release gate combines evidence from several checks and records the exceptions it accepted. It should identify exactly which dataset version and which consumer population the decision covers.

What a missing row-accounting invariant costs at the load boundary is on the public record. Public Health England disclosed on 4 October 2020: “A technical issue was identified overnight on Friday 2 October in the data load process that transfers COVID-19 positive lab results into reporting dashboards. After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases.” The cause was mundane: “some files containing positive test results exceeded the maximum file size that takes these data files and loads then into central systems”. Nothing crashed. The dashboards published on schedule, and they were wrong by 15,841 positive results.

The downstream cost was later measured in infections rather than rows. Findlater and colleagues counted 15,861 case records that failed to import from SGSS into the CTAS contact-tracing tool, and published the analysis in BMJ Open in 2023. Among non-household contacts, the secondary attack rate was 7.9% (95% CI 6.5–9.2) in the delayed group, against 5.9% (95% CI 5.3–6.6) in the comparison group. Reconciling the count that leaves the source system against the count that enters the published table is a cheap check at that boundary. Nothing further downstream could recover the week.

Google turned this judgement into a scored checklist. Breck and colleagues published the ML Test Score in 2017, presenting “28 specific tests and monitoring needs, drawn from experience with a wide range of production ML systems”. The tests fall into four groups: data, model, ML infrastructure and monitoring. A release gate is where some of those tests stop being advisory and become binding.

Earlier controls reduce blast radius. Later controls reveal consumer-specific failures. Source and contract checks validate payloads, schemas and identifiers near generation. Pipeline invariants check row accounting, joins, timestamps and uniqueness. Dataset statistics watch distributions, missingness, coverage, freshness and important slices. Consumer parity compares training, serving and logged feature behaviour for the same contract. Outcome impact connects data incidents to model quality, decisions, users and operational queues. A file-size limit that silently truncates a load is invisible to the first layer and obvious to the second. That is precisely why the layers are not substitutes.

Response should reflect confidence, reversibility and consumer harm, rather than one universal threshold. This split is not a taxonomy invented for a lesson. Amazon built it into Deequ, its data-validation system, and published how Deequ works in 2018. On its use inside the company, the paper reports: “Additionally, data producers can leverage our system to halt their data publishing pipelines when they encounter cases of data anomalies.”

The worked example puts a number on the boundary between the two middle dispositions below. The count of newly added records raises a warning once it moves more than three standard deviations from the running mean. Beyond four, it throws an error. Same measurement, same slice, two different release consequences, separated by one standard deviation of confidence. The shipped AWS library, awslabs/deequ, exposes exactly that pair as CheckLevel.Warning and CheckLevel.Error, attached to constraints such as isComplete, isUnique and isContainedIn. The severity choice is written into the check definition, not argued about after the alert fires.

A rule should state scope, reference, threshold, severity, owner, consumer impact, exception policy, and remediation. “Alert if distribution changes” is not operationally complete.

Use slices that map to real risk, but control combinatorial explosion. Prioritize protected groups, high-volume segments, high-consequence workflows, source versions, and known weak points.

Evaluate alert precision and recall using incident history and synthetic fault injection. Missing a failure and paging on harmless noise are both measurable outcomes.

Banking supervisors have written this list into a standard. BCBS 239, the Basel Committee's Principles for effective risk data aggregation and risk reporting, dates from January 2013 and sets out 14 principles. The third is “Accuracy and Integrity – A bank should be able to generate accurate and reliable risk data… Data should be aggregated on a largely automated basis so as to minimise the probability of errors”. Paragraph 40 states the operational half: “Supervisors expect banks to measure and monitor the accuracy of data and to develop appropriate escalation channels and action plans to be in place to rectify poor data quality.” Measurement, escalation channel, action plan — the same fields a well-formed rule carries.

Which half organisations actually fail is also on the record. In May 2018 the European Central Bank reviewed 25 significant institutions and concluded: “Thus far, none of those significant institutions – some of which are classified as global systemically important banks – have fully implemented the BCBS 239 principles.” The review named the main weakness as “a lack of clarity regarding responsibility and accountability for data quality”. The failing field is the owner, not the threshold.

Retire or revise rules that no longer represent the product. Monitoring definitions need versioning because normal behavior, sources, and consumer tolerance evolve.

The review should separate correctness invariants from lower-confidence change signals. Freeze the candidate, run layered checks, review exceptions against the prior approved release, choose a disposition, and record accountability.

The last step is the one teams treat as paperwork. It is also the one two regulators on two continents have written down in nearly identical terms. FDA's 21 CFR 11.10(e) requires “Use of secure, computer-generated, time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records. Record changes shall not obscure previously recorded information.” The MHRA imposes the same non-obscuring requirement in its data integrity guidance, revised in March 2018: “An audit trail provides for secure recording of life-cycle details such as creation, additions, deletions or alterations of information in a record, either paper or electronic, without obscuring or overwriting the original record.” It also requires the trail to capture the who, what, when and why of the action: reviewer, change, timestamp, rationale.

Storing reviewer, rationale, waivers, expiry and follow-up evidence is therefore not a documentation preference. In regulated industries it is the control. And the specific thing both texts forbid is the one a silent threshold edit does: overwriting the record of what the check used to say.