Idempotency, Caching, Backfills, and Recovery

A monthly feature backfill is rerun after a partial outage. The repaired job duplicates rows in one table, recomputes labels using current policy, and triggers thousands of downstream alerts.

The problem is not that the pipeline failed. The problem is that nobody had defined what running the same job twice should mean. Recovery has to be designed as part of the workflow contract.

That question has already been answered, expensively and in public, by people who wrote down what happened. A High Court judgment answered it. So did an order of the U.S. Securities and Exchange Commission, two published postmortems, and the Code of Federal Regulations. The rest of this lesson is built on those documents.

A training task reuses a cached dataset because code and source tables are unchanged.

The web hit this failure first and wrote the fix into a standard. RFC 9111, the HTTP caching standard published in 2022, says what a cache is allowed to match on: “The "cache key" is the information a cache uses to choose a response and is composed from, at a minimum, the request method and target URI used to retrieve the stored response.” The load-bearing words are *at a minimum*.

A second rule adds a secondary key. A cache MUST NOT reuse a stored response carrying a Vary header field without revalidation unless every presented request header field nominated by that Vary value matches the request that caused the response to be stored. MDN Web Docs describes the same mechanism in its own words, saying that Vary “causes the cache to be keyed based on a composite of the response URL and the Accept-Language request header — rather than being based just on the response URL”.

A pipeline fingerprint that stops at code and source tables is a primary key with no secondary key at all.

Both process historical events, but the intended semantics may differ.

Historical replay reproduces what the system would have known and done at the original time. It runs on the historical code and policy, preserves point-in-time inputs, and supports incident reconstruction. It may deliberately reproduce an old bug — that is the point of it. A corrective backfill recomputes the same history under a declared new rule. It produces a new dataset version, requires downstream impact review, and must never masquerade as original knowledge. Operational reconciliation touches neither past. It compares intended with actual external state and repairs the difference, closing duplicate tickets and restoring one canonical record instead of repeating the work blindly.

Choosing between the three is a claim about which past you are representing. Write the claim down before the run. Nothing in the output distinguishes them afterwards.

An operation is idempotent when repeating it with the same identity produces the same externally visible result. The identity may include entity, time window, code version, source snapshot, and policy version.

A job can be idempotent for artifact creation but not for email, payment, or ticket creation. Those effects need deduplication, transactional boundaries, or compensating actions.

HTTP wrote the definition down first, and it is worth borrowing. RFC 9110, published in 2022, calls a method idempotent “if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request”. Of the methods it defines, PUT, DELETE and the safe methods qualify. The spec is careful about scope. A server may still log each request, and that logging is a side effect the guarantee never covered.

What a misjudged scope costs is on the court record. Horizon Online repeated the last part of the Pouch Delivery script when the Post Log On script had not closed down. A subpostmaster at Dalmellington scanned £8,000 from her core branch into her outreach branch. The transfer replicated four times. Her receipts showed £32,000, and her branch a discrepancy of £24,000. The High Court set all of this out in Bates v Post Office in 2019. The scan was idempotent as an act of data entry and ruinous as an accounting side effect, and the shortfall landed on the operator rather than on the software.

Nor was it a one-off. The judgment records what the supplier found when it looked: “When investigating this Fujitsu found 112 occurrences affecting 88 different branches in the previous 5 years.” The court counted 65 incidents between February 2010 and January 2011, then 6, 9, 7, 9 and 16 across 2011 to 2015. Fixes were applied in April 2010, January 2011 and January 2016. The duplicated side effect outlived the first two of them. That is what an unspecified repeat looks like when the repeated thing is money.

Different failures call for different operational responses. Retry, resume, reconcile, compensate, and rebuild-or-rollback are not a ladder to climb in order. Each one assumes something the others do not.

The last assumes the most: that trusted inputs still exist. GitLab found out on 31 January 2017 what happens when that assumption has never been tested. An engineer wiped the wrong data directory. The Register reported 300GB of live production data deleted, and that “Just 4.5GB remained by the time he canceled the rm -rf command”. Then the recovery paths were tried one at a time. The pg_dump job had silently been running pg_dump 9.2 against a PostgreSQL 9.6 cluster, so it produced nothing and the S3 bucket was empty. The cron error mail that would have reported this was being rejected by DMARC. GitLab's own postmortem is blunt about the result: “This means we were never aware of the backups failing, until it was too late.”

What restored the site was an LVM snapshot. An engineer had taken it by hand roughly six hours earlier, for load testing. It was not a disaster-recovery mechanism. It was an accident. “The last potentially viable backup was taken six hours beforehand”, The Register noted. Data written between 17:20 and 00:00 UTC was lost: roughly 5,000 projects, 5,000 comments and 700 new user accounts. Four of the five ways out were untested or misconfigured. The fifth was luck.

Treat a backfill as a release with its own blast radius. The two steps that fail hardest are the second and the fifth: how much history you meant to touch, and what stops the run.

Bounding the scope is often a single boolean. In the Apache Airflow 3.3.1 documentation, a DAG with a start_date and a non-asset schedule defines a series of data intervals, and “If you set catchup=True in the Dag, the scheduler will kick off a Dag Run for any data interval that has not been run since the last data interval (or has been cleared).” The shipped default is the opposite. The Airflow config scheduler.catchup_by_default is False, so only the latest interval runs. Astronomer's documentation states the same default — “By default, catchup is set to False” — and warns that a DAG running every 5 minutes with a start date of one year ago and catchup set to True will schedule numerous DAG runs at once. One flag is the difference between a bounded scope and every interval since the start date.

The stop condition in step 5 is what stands between a partial deployment and an unbounded one. During a staged deployment beginning 27 July 2012, a technician did not copy Knight Capital's new RLP code to one of the eight SMARS servers. Dormant "Power Peg" code was left running on it, and its cumulative-quantity counter had been moved back in 2005. The SEC's 2013 order against Knight Capital Americas describes what that server then did: “Because the cumulative quantity function had been moved, this server continuously sent child orders, in rapid sequence, for each incoming parent order without regard to the number of share executions Knight had already received from trading centers.”

On 1 August 2012, 212 incoming parent orders reached that eighth server. They produced millions of child orders: 4 million executions in 154 stocks, more than 397 million shares in approximately 45 minutes, an approximately $3.5 billion net long and $3.15 billion net short position, and a $460 million realised loss. Knight paid a $12,000,000 civil money penalty. In October 2012 the Federal Reserve Bank of Chicago published the controls that would have bounded it, among them order-rate limits and a kill switch. Two hundred and twelve orders, no stop condition.

The workflow must know what operation it is repeating, which inputs define sameness, and what external state already exists. Without that memory, retries become guesses.

Test recovery by interrupting work deliberately. Waiting for a production outage to reveal the semantics is not a test.

On 28 February 2017 Amazon Web Services learned which of those two it had been doing. By its own account, an authorised team member following an established playbook ran a capacity-removal command in the US-EAST-1 region. One input was entered incorrectly and a larger set of servers was removed than intended. That took down the index subsystem — which holds the metadata and location of all S3 objects in the region — and the placement subsystem. Both required a full restart, and AWS explained why that took as long as it did: “While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.”

The index subsystem began serving GET, LIST and DELETE at 12:26 PM PST and was fully recovered at 1:18 PM PST. The placement subsystem finished recovering at 1:54 PM PST. The recovery path existed and was designed for. It had simply never been exercised at the scale it now had to run, so the restart itself became the outage. NPR filed the story as Amazon And The $150 Million Typo, with the cost estimate attributed to Wall Street Journal reporting. The typo is the least interesting part of it.

Defaults decide how often a step repeats, whether or not anyone chose them. In the Kubernetes v1.36 documentation a Job's .spec.backoffLimit “is set by default to 6”, and failed Pods are recreated “with an exponential back-off delay (10s, 20s, 40s ...) capped at six minutes”. Red Hat's OpenShift documentation says the same property “defaults to six” and repeats the same cap. Nobody chose that number for your workload. Six attempts at a step that is not idempotent are six chances to duplicate it.