ML data engineering
Data Observability, SLIs, and Incident Response
Operate data products with consumer-centered reliability signals, proportionate responses, lineage, and recovery discipline.
By the end you can
- Design data SLIs and SLOs around consumer evidence
- Choose fail, degrade, quarantine, warn, or observe responses
- Trace and recover from data incidents using lineage and release evidence
- Evaluate alert quality and postmortem effectiveness
Example
Every job was green and 15,841 positive cases were missing
Public Health England's loader moved positive COVID-19 lab results into the reporting dashboards and into the contact-tracing system. It finished its runs cleanly while it was dropping records. When a file exceeded the maximum size the loader could handle, the records past that limit were silently discarded. No job failed. No error surfaced. The daily case figures published to the country were simply short.
The admission came on 4 October 2020. “After rapid investigation, we have identified that 15,841 cases between 25 September and 2 October were not included in the reported daily COVID-19 cases.” That was Michael Brodie, Interim Chief Executive of Public Health England.
Nothing in the pipeline's own telemetry could have said this. The one signal that would have fired compares what consumers received against what the laboratories produced.
- The load step reported success. Records beyond the loader's maximum file size were dropped in silence, so there was no failed run for anyone to investigate.
- Counts stayed plausible because the missing rows were never counted anywhere. Between 25 September and 2 October 2020, 15,841 positive cases were absent from the reported daily figures.
- The consequence was not a wrong chart. Those cases were not referred for contact tracing, and the backlog was only transferred by 1am on 3 October 2020.
- Every producer-side indicator was healthy from 25 September to 2 October 2020. The consumer-centred one — results loaded against results issued — was the indicator nobody was watching.
- The cost landed downstream. Fetzer and Graeber estimated the tracing delay was associated with more than 125,000 additional infections and over 1,500 additional COVID-19 deaths.
A data SLI should describe what the consumer receives
A service-level indicator is a measured property: the fraction of examples delivered before a deadline, the age of a critical feature, the rate of valid point-in-time joins. The objective sets an acceptable target over a window. The error budget sets how far short the team may fall before priorities change. Choose indicators tied to what a wrong decision costs. Pipeline duration is useful, but it is not a substitute for feature age, record-count reconciliation, or label coverage.
Public Health England's loader would have passed any indicator built on job status for the whole of 25 September to 2 October 2020. It would have failed on the first day against an indicator that reconciled positive results issued by laboratories with positive results referred for tracing. That is the difference between a dashboard and a control.
Measure the consumer’s evidence, not merely the producer’s job status.
Comparison
Fail, degrade, quarantine, warn, or observe?
The response depends on what is at stake, on confidence, on reversibility, and on the fallback available. Each of these is a decision somebody made in advance about what the system should do when evidence is bad. The PHE loader made none of them. An oversized file was neither rejected, nor quarantined, nor flagged for review: the surplus records were dropped and the run was marked successful. A pipeline with no designed response to bad evidence has not chosen to observe. It has chosen not to notice.
Fail closed
Block the release or decision when evidence violates a critical correctness invariant.
- Best for leakage or identity corruption
- High availability cost
- Needs clear ownership
- Requires tested recovery
Degrade safely
Use a simpler feature set, older approved snapshot, rule, or abstention path.
- Preserves continuity
- Changes decision behavior
- Needs separate telemetry
- Must be evaluated beforehand
Quarantine
Publish or retain affected data separately while investigation proceeds.
- Limits contamination
- Supports forensic review
- Can delay consumers
- Needs explicit release criteria
Warn and observe
Record a lower-confidence deviation without immediate interruption.
- Appropriate for exploratory drift
- Builds historical context
- Risks alert fatigue
- Needs review and retirement
Steps
Run a data incident from detection to prevention
The incident process should preserve the decision context and the affected releases. Containment is usually the fast part. Public Health England bounded the impact at 15,841 cases and had the backlog transferred by 1am on 3 October 2020. What could not be recalled by rebuilding a table were the decisions already taken on the bad evidence — here, the contacts not traced during those days. Restatement repairs the data. It does not repair what consumers did with it. Bounding impact means naming the consumers and the decisions, not only the rows.
1. Contain
Stop publication, freeze versions, or activate the approved fallback.
2. Bound impact
Identify consumers, models, decisions, time range, entities, and populations affected.
3. Reconstruct lineage
Trace the first bad source, job, configuration, and release through run metadata.
4. Repair and restate
Correct data, rebuild affected products, and mark superseded versions.
5. Validate recovery
Compare restored outputs with known-good evidence and consumer behavior.
6. Prevent recurrence
Add the earliest reliable control, update contracts, and test the failure path.
Case
GitLab restored from a six-hour-old copy that had been loaded into staging
On 31 January 2017 GitLab lost production data from the GitLab.com database, and wrote its incident notes in public while it worked. Five backup and replication mechanisms were meant to make that survivable. The notes record what each one was actually doing. pg_dump was failing silently on a PostgreSQL 9.2/9.6 version mismatch, and the e-mails reporting that failure were rejected by the receiving mail server because DMARC was not enabled for the cronjob e-mails. The S3 bucket was empty. Azure disk snapshots were enabled for the NFS server but not for the database servers. Replication was fragile. LVM snapshots were taken only every 24 hours.
GitLab's own summary is one sentence: “So in other words, out of five backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a six-hour-old backup.”
What they restored from was a six-hour-old LVM snapshot of production that had been loaded into staging. Roughly 5,000 projects, 5,000 comments and 700 new user accounts were permanently lost, on GitLab's own best estimate. The follow-up postmortem of 10 February 2017 asks “Why was the backup procedure not tested on a regular basis?” and answers “Because there was no ownership, as a result nobody was responsible for testing this procedure.”
Note where the failure sits. The backups were configured, documented, and drawn on a diagram; five of them. A recovery path that is never exercised is not a recovery path, and an unowned one is never exercised.
Key idea
An alert without a decision is accumulated noise
Many teams add a threshold after every incident. The result can be hundreds of low-context alerts whose owners no longer understand the original failure. Measure alert precision, acknowledgement time, action taken, and incidents missed. Consolidate correlated signals, and remove controls that no longer change a decision. A quiet dashboard can be dangerous. A permanently red dashboard is functionally quiet as well.
The monitoring layer fails in its own ways too, and it fails hardest when it shares a dependency with the thing it monitors. On 28 February 2017 Amazon S3 was disrupted in the Northern Virginia (US-EAST-1) region. A mistyped input to a routine capacity-removal command took down the index and placement subsystems, which had not been completely restarted in the larger regions “for many years”. Full recovery ran to 1:54 PM PST.
Meanwhile the instrument customers were watching said nothing. AWS's own post-event summary: “From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3.” A status page that goes down with the service, and a restart path unexercised for years, are both monitoring artefacts that were never operated as products.
Monitor the monitoring system as an operational product.
Case
Two and a half million alarms in a month, and ninety-eight events when one was missed
Hospitals have counted both halves of this problem. Over 31 days, every physiologic monitor alarm for 461 adult ICU patients at UCSF was recorded: 2,558,760 unique alarms — 1,154,201 arrhythmia, 612,927 parameter and 791,632 technical. Of those, 381,560 were audible. That is a burden of 187 audible alarms per bed per day.
Nurse scientists then annotated a sample. Drew and colleagues report the hit rate in one line: “88.8% of the 12,671 annotated arrhythmia alarms were false positives.” Even among the true ones the news was mixed. Of the 168 true ventricular tachycardia alarms, 93% were too brief to warrant treatment.
That is the denominator. The numerator is what happens when the alarm that mattered is one of the two million. The Joint Commission reported 98 alarm-related events between January 2009 and June 2012, of which 80 resulted in death, 13 in permanent loss of function, and five in unexpected additional care or extended stay. Its 2013 alert estimates that “between 85 and 99 percent of alarm signals do not require clinical intervention”. Reporting to that database is voluntary. Those 98 are the events somebody chose to report, not an incidence rate.
Figure
Position
An alert nobody acts on is a control the team has already lost
A threshold is the cheapest thing to add after an incident, and adding one feels like prevention. It becomes prevention only when somebody reads the result. A securities regulator has put a number on the gap.
On the morning of 1 August 2012 a component of Knight Capital Americas LLC's systems was failing, and the systems said so. Between 8:01 a.m. and the 9:30 a.m. open they sent 97 automated “BNET reject” e-mails naming the failing component to staff. The SEC's order of 16 October 2013 records what became of them: “Knight’s system sent 97 of these e-mail messages to a group of Knight personnel before the 9:30 a.m. market open. Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.” Knight lost more than $460 million in roughly 45 minutes and paid a $12,000,000 penalty. The warning was generated, delivered, timely, and specific about which component was failing. It was not a control, because nothing in the design made it something anyone read.
Why channels stop being read has been measured elsewhere. In the UCSF intensive-care study, “88.8% of the 12,671 annotated arrhythmia alarms were false positives.” The Joint Commission's 2013 alert estimates that “between 85 and 99 percent of alarm signals do not require clinical intervention”, and reports 98 alarm-related events between January 2009 and June 2012, of which 80 resulted in death, 13 in permanent loss of function, and five in unexpected additional care or extended stay. Those are bedside alarms, the reporting is voluntary, and none of the percentages transfer to a warehouse. The mechanism transfers. A channel that almost never calls for action stops being read, and once it has stopped being read it makes little difference whether it fires. That is why the count of alerts is the wrong measure, and precision, acknowledgement time, action taken and incidents missed are the right ones.
GitLab's incident is the same failure in other clothes. Five mechanisms on the diagram, “out of five backup/replication techniques deployed none are working reliably or set up in the first place”, a restore from a six-hour-old backup, and a postmortem that explains the untested procedure with “Because there was no ownership, as a result nobody was responsible for testing this procedure.” An unread alert and an untested recovery path fail identically. Both appear in the design. Neither appears in the incident.
An alert earns its place by changing a decision, not by existing.
Analogy
Triage logic applies to data incidents
Telemetry plays the part of vital signs, quality checks the part of diagnostic tests, and lineage the patient history. No single measurement establishes the diagnosis. Triage exists precisely because the signals never stop arriving — 187 audible alarms per bed per day in the UCSF count. Severity and available evidence decide whether to observe, isolate, intervene, or escalate. Signal volume does not. The response can change as more information arrives. One difference is scale: a data incident reaches many downstream consumers at once, and the remedy is often rebuilding history rather than treating a single patient.
Signals become useful when they support prioritization, containment, and explanation.
A postmortem should improve the system map, not only assign a root cause
Many incidents have several contributing factors: permissive schema evolution, missing lineage, ambiguous ownership, untested fallback, and a monitor placed after the wrong boundary. Document the timeline, the detection gap, the decision impact, why existing controls failed, and the earliest point where the defect could have been caught. Separate the triggering event from the systemic conditions.
The model for that separation is not a data pipeline but a spacecraft. The Mars Climate Orbiter Mishap Investigation Board reported on 10 November 1999 and named a single root cause: English rather than metric units in the ground software file SM_FORCES. Alongside it the board listed eight separate contributing causes, among them inadequate verification and validation of ground software, inadequate communication between project elements, understaffing and inadequate training.
Its framing of why the root cause is the less interesting half is the sentence worth copying: “The Board recognizes that mistakes occur on spacecraft projects. However, sufficient processes are usually in place on projects to catch these mistakes before they become critical to mission success. Unfortunately for MCO, the root cause was not caught by the processes in-place in the MCO project.”
The orbiter had cost $75 million to develop, on the later account of the U.S. General Accounting Office, and was destroyed on 23 September 1999. That the same office returned to the subject in 2002, under the title “NASA: Better Mechanisms Needed for Sharing Lessons Learned”, is the second lesson: corrective actions count only where there is evidence they were carried into the next mission. A new dashboard panel prevents nothing until it has an owner, a threshold rationale, and a tested response.
A strong postmortem changes contracts, controls, and recovery behavior—not just the incident narrative.
Visual
A data control room needs service, quality, and consequence signals
Timeliness, validity, representativeness, and safety to consume are four different questions, and no single metric answers them. The layers stay separate because failures cross them without warning. The PHE loader was healthy at the pipeline layer while the consumer-impact layer was short 15,841 cases. AWS could not update per-service status on its own Service Health Dashboard until 11:37AM PST, because the console reporting the impact depended on the service that was down. A control room whose consequence signals inherit the dependencies of the systems they describe will be green at exactly the moment it matters.
- 01
Pipeline health
Run state, backlog, retries, resource saturation, and dependency availability.
- 02
Delivery service
Freshness, completeness, latency, and successful publication for each consumer.
- 03
Data semantics
Schema, invariants, distributions, joins, labels, and point-in-time checks.
- 04
Consumer impact
Models, reports, decisions, and populations affected by the data product.
- 05
Incident evidence
Lineage, release version, examples, ownership, timeline, and remediation status.
Observability is useful when a signal leads to an accountable decision.
Key takeaways
- Data observability connects pipeline health, delivery service, semantic quality, consumer impact and incident evidence. PHE's loader was healthy at the first layer while 15,841 positive cases were missing at the last.
- Consumer-centred SLIs measure the evidence received at decision time, not producer job status. No PHE job failed, and the dropped cases went unreferred for tracing until the backlog was transferred by 1am on 3 October 2020.
- Fail, degrade, quarantine, warn and observe are distinct responses, chosen by consequence and confidence. Silently discarding the overflow of an oversized file is none of them.
- Incident response should contain, bound impact, trace lineage, restate data, validate recovery and prevent recurrence. GitLab could do it only from a six-hour-old backup, with roughly 5,000 projects, 5,000 comments and 700 new user accounts permanently lost.
- An alert is a control only when the design makes someone read it. Knight personnel received 97 “BNET reject” e-mails before the 9:30 a.m. open, and Knight lost more than $460 million in roughly 45 minutes.
- Postmortems should improve contracts, system maps, controls and recovery paths rather than stop at one root cause. The Mars Climate Orbiter board named one root cause and eight contributing causes.