Continuous Delivery and Progressive Release

A fraud model receives five percent of transactions. The question is what five percent buys, and the base rate answers it.

Card fraud is rare, and European supervisors measure how rare. The European Banking Authority and the European Central Bank put the number in the executive summary of their 2024 Report on Payment Fraud, published in August 2024: “In relative terms, the highest fraud rates were observed for card payments, where fraud accounted for 0.031% of total card payments in value and 0.015% of total card payments in volume terms in H1 2023”.

The standard public benchmark is denser and still thin. It holds two days of September 2013 transactions by European cardholders, assembled by the ULB Machine Learning Group and Worldline. The dataset record states the split: “This dataset present transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions… the positive class (frauds) account for 0.172% of all transactions.”

At that rate, a canary carrying five percent of the traffic would contain roughly 25 fraudulent transactions. None of them is confirmed on the day the decision is made. The canary is small in the only dimension the promotion decision depends on, and large in every dimension the team can see immediately.

A candidate passes offline gates and receives ten percent of traffic. Service latency remains healthy. The candidate, though, routes twice as many cases to a manual queue that is already full.

Progressive delivery is useful only when exposure is paired with the right evidence and stop rules. Traffic percentage alone does not define safety. A release plan has to settle four things: the mode, the evidence, the recovery path and the record. The sections below take each of them from organisations that published what happened when one was missing.

Each mode observes a different part of production behavior. Shadow sends copied requests to a candidate without using its decisions. Canary uses the candidate for a bounded fraction of real decisions. Blue-green prepares a complete parallel environment and switches a route. Rolling replaces serving instances gradually within one environment.

The blue-green promise is the switch that can be switched back. It is worth exactly what the return path is worth.

TSB migrated 5.2 million customers to the Proteo4UK platform in a single event over the weekend of 20-22 April 2018. There was no way back. The Financial Conduct Authority's Final Notice of 20 December 2022 records the method as “a 'big bang' or single operation data migration approach, under which all data would be migrated in a single main migration event weekend”, and heads a section of its findings "Inability to revert to the LBG Platform": “Migration off the LBG IT Platform to Proteo4UK was irreversible, creating risk if any major issues arose.”

Major issues arose. Disruption lasted until December 2018. TSB paid £32.7m in redress. In December 2022 the FCA fined it £29,750,000 and the Prudential Regulation Authority £18,900,000 — £48.65m in total — for failing to organise and control the migration. The Bank of England's announcement of the second fine records that “All of TSB's branches and a significant proportion of its 5.2 million customers were affected”. Blue-green with no way back is not blue-green. It is one deployment with a rehearsal.

Evidence at one layer can pass while evidence at the next fails, and the layers do not arrive at the same speed.

Service evidence is latency, availability, resource use, errors and dependency health. It is fast to observe, it supports overload rollback, and every release needs it. It does not prove the model is useful.

Behavior evidence is scores, calibration proxies, slices, abstention and output contracts. It detects immediate prediction shifts, and it can run on shadow labels or heuristics. It needs a candidate-versus-control comparison, and it may not measure true outcomes.

Workflow evidence is queue size, reviewer time, override rate and downstream errors. It captures operational burden and often responds quickly. It has to be tied to the decision unit.

Outcome evidence is delayed labels, business effect, harm, appeals and long-term behavior. It is closest to product value. It needs causal and selection care, and it may govern final promotion.

The asymmetry is the point. Service evidence is computed over every request in the canary. The outcome evidence in a fraud canary is computed over the handful of cases the 0.172% base rate implies, weeks after the traffic moved.

Continuous delivery keeps an approved candidate ready for controlled release. Continuous deployment automatically moves changes into production when gates pass. The difference matters when someone still has to approve the change, wait for the right moment, or coordinate with another team.

ML releases can include model, feature logic, calibration, threshold, routing, and fallback. Whoever promotes or rolls back should say which of those changed together, and whether old and new components can coexist.

Rolling updates have documented defaults. Both rolling-update parameters of a Kubernetes Deployment default to 25%. The v1.36 API reference says maxSurge is “The maximum number of pods that can be scheduled above the desired number of pods… Defaults to 25%”, and maxUnavailable is “The maximum number of pods that can be unavailable during the update… Defaults to 25%”. Red Hat records the same two values for OpenShift. A cluster on the defaults runs a mixed fleet for part of every rollout.

Exposure ladders are published too. Meta's web release goes to employees first, then 2 percent of production, then everyone. Chuck Rossi described the ladder in 2017: “Each release is rolled out to 100 percent of production in a tiered fashion over a few hours, so we can stop the push if we find any problems.” Feature exposure is kept separate from code deployment by Gatekeeper — “If we do find a problem, we can simply switch the gatekeeper off rather than revert back to a previous version or fix forward.”

The mobile path is slower for a reason. In a 2016 paper, Rossi and five colleagues record an Android rollout to “typically first only 20%, then 50%, and finally 100% of the population… with periodic stability checks after every increase in the rollout percentage”, spread over three days. On mobile, “hot-fixes and roll-backs are largely unacceptable”.

Hours on the web, three days on Android, same company. The ladder is set by how fast the organisation can get the change back, not by how fast it can push it out.

The plan should name exposure, evidence, timing, and recovery before traffic moves. Freeze the candidate — bind model, features, policy, runtime, and evaluation evidence. Choose the exposure unit — route by user, device, account, region, or request without contamination. Define gates and clocks — set service, behavior, workflow, and delayed-outcome thresholds. Automate stop and rollback — clear triggers, owner authority, known-good targets. Reconcile consequences — track side effects, compensation, and post-release learning.

CrowdStrike wrote the fourth step out after skipping it. Its own root cause analysis of the 19 July 2024 Channel File 291 failure, published on 6 August 2024, lists staged deployment as mitigation number 6: “New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected.” The finding is stated plainly — “Findings: Each Template Instance should be deployed in a staged rollout.” — and the analysis adds bake-in time between rings. The Rapid Response Content that failed went out without any of it. On 20 July 2024 David Weston of Microsoft published the scale: “We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines.”

The lesson for step 1 is sharper than the lesson for step 4. What shipped was content, not a code release — an artefact that travelled on its own channel, outside the ladder the code went down. If the freeze does not bind every asset that can change behavior in production, the stop rule has nothing to stop.

Each stage claims that the candidate is acceptable under a larger scope and stronger evidence. Record that scope, the unresolved uncertainty, and the target for immediate recovery.

Three regulators have turned that recommendation into a filing requirement. FDA's final guidance on Predetermined Change Control Plans for AI-enabled device software functions was announced on 4 December 2024: “FDA reviews the PCCP as part of a marketing submission for a device to ensure the continued safety and effectiveness of the device without necessitating additional marketing submissions for implementing each modification described in the PCCP.” FDA, Health Canada and the MHRA had already published five guiding principles for PCCPs on 24 October 2023. Principle 1, "Focused and bounded", reads: “A PCCP describes specific changes that a manufacturer intends to implement. Such changes are limited to modifications within the intended use or intended purpose of the original MLMD.” The same principle requires “mechanisms to detect and revert or stop implementation of a change that fails to meet specified performance criteria”.

That is the release record as an obligation. The scope of change, the evidence that will validate it, and the rule for stopping it, written down and authorised before anything moves. Do not let an alias move erase the experiment. Persist allocation, exact versions, decision units, and observed outcomes.