Skip to content
AI.info

MLOps

ML Incident Response, Rollback, and Disaster Recovery

Prepare detection, triage, containment, rollback, compensation, communication, and recovery for ML-specific incidents.

By the end you can

Example

Knight's rollback put the defective code on seven more servers

On 1 August 2012, 212 parent orders at Knight Capital Americas turned into over 4 million executions. They covered 154 stocks and more than 397 million shares. The whole thing took approximately 45 minutes and left a $460 million realised loss. The SEC set the sequence out in a settled order in 2013.

  • Bad deployment: The new RLP code reached seven of the eight servers. The eighth still carried the defective legacy Power Peg code, and orders routed to it began firing.
  • Technical rollback: Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly — reversing the one artefact that had just changed.
  • The rollback widened the harm: The uninstall activated the Power Peg code sitting on those seven servers too. Additional incoming parent orders kept executing. The SEC records this at paragraph 27 of its order.
  • Scale while it ran: Approximately 45 minutes produced over 4 million executions in 154 stocks and more than 397 million shares. The SEC found a $460 million realised loss. A Chicago Fed Letter by Carol Clark reports the $440 million pre-tax figure Knight itself announced.
  • Missing controls: Knight had no written deployment procedures and no second technician review — paragraph 26 of the same order. Which is why the correct response is to pause execution, preserve the decision envelope, restore policy, and reconcile the actions already sent downstream.

The first question is not “Which metric moved?”

A pricing model begins issuing implausible discounts. Engineers debate whether the model drifted while orders continue to close. The urgent question is different: which decisions are still being made, and how do we stop further harm? At Knight Capital Americas the entire event lasted approximately 45 minutes. That is less time than most teams spend agreeing on a root cause.

ML incidents combine software state, data state, learned behavior, policy, and downstream action. So contain the decision pathway first. Preserve evidence and diagnose after.

Key idea

Two federal agencies make evidence preservation a step of its own

Ad hoc fixes, retraining, queue deletion, and configuration changes can overwrite the state you need to understand the event. Where safety allows, preserve first: exact versions, effective times, sampled inputs, decision records, and downstream consequences.

This ordering is codified, not merely advisable. NIST SP 800-61r3, published in April 2025, makes “Incident data and metadata are collected, and their integrity and provenance are preserved” a High-priority element. Its recommendation R1 under RS.AN-07 reads: “Collect and retain evidence from an incident in accordance with the organization's evidence preservation procedures and data retention policies, and consider factors such as the possibility of prosecution, and the cost of retaining the data and the hardware and software needed to access the data in the future.” RS.AN-06 covers the integrity and provenance of incident records. RS.MI-01 and RS.MI-02 make containment a distinct step that precedes eradication.

CISA's federal incident and vulnerability response playbooks, from 2021, sequence the same thing operationally. Step 3a is “Collect and preserve the data necessary for incident verification, categorization, prioritization, mitigation, reporting, attribution, and as potential evidence”. Step 7c requires evidence collection “prior to eradication”.

Evidence retention still follows privacy and security policy. Incident urgency does not authorize uncontrolled copying of sensitive data.

Contain quickly, but change evidence-bearing state deliberately.

Case

The EU puts a 15-day clock on a serious incident

For high-risk systems in the EU, the reporting clock is written into law. A provider must report a serious incident to market surveillance authorities “immediately after the provider has established a causal link between the AI system and the serious incident or the reasonable likelihood of such a link, and, in any event, not later than 15 days”. That is Article 73 of the EU AI Act. Two shorter clocks sit inside it. Article 73(3) gives “not later than two days” for a widespread infringement, and Article 73(4) gives “not later than 10 days” where a person has died.

Article 73(6) then constrains the diagnosis itself. The provider “shall not perform any investigation which involves altering the AI system concerned in a way which may affect any subsequent evaluation of the causes of the incident, prior to informing the competent authorities of such action”. The instinct to fix first and explain later is, for these systems, unlawful.

Visual

The incident command sequence

Stop the harm first. Work out why second. That is the order NIST SP 800-61r3 encodes when it makes containment a distinct step ahead of eradication, and the order CISA's playbooks encode when they put data collection and preservation ahead of it.

FigureProcess · 5 steps
  1. 1

    Detect and declare

    Name severity, affected decisions, population, and incident commander.

  2. 2

    Contain

    Pause, abstain, switch fallback, block actions, or reduce exposure.

  3. 3

    Preserve evidence

    Freeze logs, traces, versions, inputs, queues, and external state.

  4. 4

    Diagnose and recover

    Test hypotheses, repair the right layer, reconcile, and validate restoration.

  5. 5

    Learn and prevent

    Write a blameless review, assign controls, and verify completion.

Rollback has several layers, and reversing one can be the wrong move

Traffic can return to a previous service while corrupted features remain in a store. A threshold can be restored while queued decisions created under the bad policy continue to execute. Runbooks should list every reversible layer: route, model, feature view, source snapshot, policy, queue, and downstream state. Recovery may require rollback plus reconciliation plus compensation.

Knight Capital's responders did what almost anyone would do under time pressure. They reversed the artefact that had just changed. The SEC's order records the result at paragraph 27, under the heading “Incident Response”: “In one of its attempts to address the problem, Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly. This action worsened the problem, causing additional incoming parent orders to activate the Power Peg code that was present on those servers, similar to what had already occurred on the eighth server.”

The uninstall was a correct rollback of one layer. It moved the system into a configuration nobody had tested: seven servers freshly returned to defective legacy code, with orders still arriving. A rollback target is only known-good as a whole configuration. Knight had no written deployment procedures and no second technician review, so nobody could say what that whole configuration was.

Comparison

Rollback, compensation, and disaster recovery solve different problems

Using one word for all recovery creates dangerous assumptions, and the distance between the three is measurable.

What the fix does not undo. CrowdStrike's defective Channel File 291 content was withdrawn within hours on 19 July 2024. Delta Air Lines was still paying for it a quarter later. In the September-quarter results it filed with the SEC in October 2024, Delta reported 7,000 flight cancellations over five days and wrote: “The direct revenue impact of the incident was approximately $380 million, primarily driven by refunding customers for cancelled flights and providing customer compensation in the form of cash and SkyMiles. The non-fuel expense impact was $170 million, primarily due to customer expense reimbursements and crew-related costs.” The legal complexity is not hypothetical either. On 23 July 2024, while the outage was still being worked through, the US Department of Transportation opened an investigation into Delta's handling of passengers — confirmed in a statement by Ranking Member Rick Larsen of the US House Committee on Transportation and Infrastructure, which also notes that passengers are entitled to compensation under the FAA Reauthorization Act of 2024. Reconciliation and compensation are a separate workstream from the rollback that ended the technical fault. Slower, and more expensive.

Why the alternate control plane is the whole point. On 4 October 2021 a command issued during routine backbone maintenance took down all the connections in Meta's backbone network. The responders then lost the means to fix it. Santosh Janardhan, Head of Infrastructure at Meta, wrote the next day that “The total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this.” Primary and out-of-band network access were both gone. Engineers had to be sent on site. From outside, Celso Martinho and Tom Strickx of Cloudflare put a clock on it: routing changes peaked at around 15:40 UTC, then “At 15:58 UTC we noticed that Facebook had stopped announcing the routes to their DNS prefixes”, resolvers worldwide handled 30x more queries than usual, and BGP activity returned at around 21:00 UTC. Disaster recovery is the case where the tools you would use to recover are themselves inside the blast radius.

FigureComparison · 3 columns

Rollback

Return future execution to a known-good technical configuration.

  • Changes model, route, policy, or feature version
  • Usually fast
  • Does not undo past actions
  • Requires a tested target

Reconciliation and compensation

Repair external state and address consequences already created.

  • Find affected decisions
  • Reverse or offset actions where possible
  • May require user communication
  • Can be legally or operationally complex

Disaster recovery

Restore service and evidence after major infrastructure or regional loss.

  • Uses backups and alternate control planes
  • Needs recovery-time and recovery-point objectives
  • Tests dependency restoration
  • May operate in degraded mode

Steps

Write an ML incident runbook with a stated tolerance

The runbook should be executable by an on-call team that did not build the model. Its severity definitions should be written before the incident, not argued during one.

UK financial regulators require exactly that, in advance and in time. The Financial Conduct Authority defines an impact tolerance as “the first point at which a disruption to an important business service would cause intolerable levels of harm to consumers or risk to market integrity”. Its 2021 policy statement PS21/3 also requires mapping and scenario testing against severe but plausible scenarios. The rules commenced on 31 March 2022, with full compliance required by 31 March 2025.

The Bank of England's Prudential Regulation Authority adds two constraints in Supervisory Statement SS1/21. An impact tolerance “must, in all cases, include a time-based metric to measure the tolerable level of disruption”. And there is no escape hatch: “The PRA expects impact tolerances to be set on the assumption that a disruption will occur. Firms should not consider the cause or probability of disruption when setting their impact tolerances.” A recovery objective argued down on the grounds that the failure is unlikely is not an objective.

FigureProcess · 5 steps

  1. 1. Define severity by decision harm

    Include scale, reversibility, affected populations, and ongoing action.

  2. 2. List immediate containment

    Pause, fallback, abstain, isolate sources, or block downstream execution.

  3. 3. Preserve attribution

    Capture model, features, policy, route, queue, and external state.

  4. 4. Provide layered recovery

    Specify rollback targets, reconciliation queries, and disaster procedures.

  5. 5. Close with verified actions

    Test fixes, monitor recurrence, assign owners, and confirm prevention work.

Blameless is a design decision, and aviation has fifty years of data on it

The review should explain how normal incentives, interfaces, and controls allowed the failure. It should not search for one careless person. Every action item needs an owner, a deadline, a verification method, and a priority.

The strongest evidence that removing the sanction is what makes the reports arrive is not an argument but a programme. NASA's Aviation Safety Reporting System is deliberately voluntary, confidential and non-punitive. Its own program briefing, with data through December 2025, records 2,321,050 incident reports received between April 1976 and December 2025, 8,069 safety alert messages issued, and 7,650 search requests.

That flow rests on an immunity the regulator itself gave up. FAA Advisory Circular 00-46F, under “Use Restrictions”, states that “The FAA will not use any reports submitted to NASA under the ASRS (or information derived therefrom) in any enforcement action, except information concerning criminal offenses or accidents that are covered under paragraphs 10.1.1 and 10.1.2.” The same circular records that “There has been no breach of confidentiality of the ASRS under NASA management”, and waives civil penalty and certificate suspension for inadvertent violations reported within 10 days. The same footing is set in 14 CFR 91.25.

A runbook that has never been rehearsed is a hypothesis. Conduct game days with realistic data, policy, dependency, and queue failures.

Case

The catastrophic failures began with errors the code had signalled

Big distributed systems rarely fall over out of nowhere. They fall over on an error the software had already noticed and then mishandled. Yuan and colleagues measured that at OSDI in 2014, reading “198 randomly selected, user-reported failures” across Cassandra, HBase, HDFS, Hadoop MapReduce and Redis.

They report three things. That “almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software”. That “in 58% of the catastrophic failures, the underlying faults could easily have been detected through simple testing of error handling code”. And that, of all the failures they studied, “77% of the failures can be reproduced by a unit test”.

The denominators differ. The first two figures share the catastrophic-failure population; the third covers every failure in the sample. So the numbers are not subtracted from one another. What they establish jointly is that the software usually knew something had gone wrong before the catastrophe. The code that was supposed to act on that knowledge had never been exercised.

Figure

Failures that began with an error the code had already signalled, and the clock the AI Act starts once the link is established.

Position

A rollback that has never been executed is a diagram, not a capability

Nobody discovers on the worst night of the quarter that the recovery path works. They discover the opposite. Rollback appears in this lesson with a precondition that reads like a footnote and is not one: it requires a tested target. Knight Capital shows the shape of the surprise. The uninstall was the recovery action, performed by people trying to end the incident, and it activated the Power Peg code on the seven servers that had until that moment been fine.

CrowdStrike wrote the missing control down afterwards. Its own external root cause analysis of Channel File 291 traces the 19 July 2024 global Windows crash to a mismatch: a template type defining 21 input fields, integration code supplying only 20. A Content Validator logic error and a missing runtime bounds check sat alongside it. Finding 6 of six, “Template Instances should have staged deployment”, states: “New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected.” That is a description of a capability the company did not have on the day. David Weston of Microsoft estimated on 20 July 2024 that the update affected 8.5 million Windows devices — the blast radius of shipping to every ring at once, with no rehearsed way back.

The OSDI study cited above was measuring something adjacent, and that is worth keeping straight. Yuan and colleagues were studying error handlers, not rollback procedures, when they found that “almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software”. But error handlers and rollback paths share the property that decides this argument. Neither runs on a good day. Ordinary traffic never exercises either, and both are believed in until the hour they are needed. Route, model, feature view, source snapshot, policy, queue and downstream state may each require reversing separately. A game day is how a team finds out which of them it can actually reverse — before a regulator, a filing, or 8.5 million devices tells it.

Ordinary traffic never exercises the recovery path. Until a rehearsal does, the runbook is a hypothesis.

Key takeaways