Deadlocks, Cascading Errors, and Conflict Resolution

On 19 October 2025 at 11:48 PM PDT, the Northern Virginia (US-EAST-1) Region of Amazon Web Services began a disruption. It did not end until 2:20 PM PDT the following day. The post-event summary traces the trigger to a latent race condition between two of the three redundant DynamoDB DNS Enactors. One Enactor had been delayed. When it ran, it applied a much older plan over the newer one, because its own staleness check had itself gone stale. The second Enactor's clean-up then deleted that plan. Every IP address for dynamodb.us-east-1.amazonaws.com was removed. Two automated components, each doing the job it was written to do, and nothing in the system owning the conflict between them.

What happened next is why this belongs in a lesson on coordination rather than one on DNS. “Additionally, because the active plan was deleted, the system was left in an inconsistent state that prevented subsequent plan updates from being applied by any DNS Enactors. This situation ultimately required manual operator intervention to correct.” — Amazon Web Services. The redundancy was real. There were three Enactors. None of the three could undo what two of them had done together. The repair path led out of the automated system entirely.

Multi-agent systems can deadlock, livelock, duplicate work, overwrite state, or propagate one incorrect artifact through many downstream agents. That list is no longer a practitioner's intuition. It is a taxonomy built from annotated evidence, published in 2025 and presented at NeurIPS. The method is stated in the abstract: “We develop MAST through rigorous analysis of 150 traces, guided closely by expert human annotators and validated by high inter-annotator agreement (kappa = 0.88). This process identifies 14 unique modes, clustered into 3 categories: (i) system design issues, (ii) inter-agent misalignment, and (iii) task verification.” — Cemri and twelve co-authors.

Alongside the first Multi-Agent System Failure Taxonomy they released MAST-Data: 1,600+ annotated traces collected across 7 popular multi-agent frameworks. Read the three categories again and notice where two of them live. Not inside a model's reasoning. In how the system was designed, and in whether anyone checked the work.

The habit of reading a coordination fault as an individual one is much older than agents, and its most studied instance cost lives. Six massive radiation overdoses were delivered by the Therac-25 between June 1985 and January 1987. The mechanism was concurrency, not reasoning. The software allowed concurrent access to shared memory with no real synchronisation and used non-atomic test-and-set on shared variables. Race conditions between the treatment task and the keyboard handler played an important part in the accidents. For a long stretch the fault was located anywhere but there. The manufacturer wrote to the Yakima hospital that the damage “could not have been produced by any malfunction of the Therac-25 or by any operator error.” The investigation published in IEEE Computer in 1993 reached the conclusion to carry into any dependency graph: “Most accidents are system accidents; that is, they stem from complex interactions between various components and activities. To attribute a single cause to an accident is usually a serious mistake.” — Leveson and Turner.

Conflict resolution therefore requires explicit priorities, ownership, timeout policy, evidence standards, and a mechanism to pause the system when goals cannot be reconciled. None of those five live in a prompt.

A stuck system is diagnosed from the Wait-for graph, the Ownership map, the Health signal, and the Conflict rule. The Conflict rule and Containment should each keep their own owner and their own test.

Two of these views are not aspirational. They ship with documented defaults, in databases most teams already run. PostgreSQL 17 does not check for a cycle on every blocked lock, because the check is expensive. It waits first. The documentation describes deadlock_timeout as “This is the amount of time to wait on a lock before checking to see if there is a deadlock condition”, and sets the value: “The default is one second (1s)”. That is the whole design in one parameter. Waiting is normal. Waiting past a threshold is a question, and the question is asked of the graph rather than of the waiter.

The Conflict rule needs a verdict as well as a detector. MySQL 8.4's InnoDB supplies one: “When deadlock detection is enabled (the default), InnoDB automatically detects transaction deadlocks and rolls back a transaction or transactions to break the deadlock.” The victim is chosen deterministically rather than argued about, by preferring small transactions as measured by the rows inserted, updated or deleted. Detection itself can be turned off through innodb_deadlock_detect. Copy the shape: a threshold before you look, a graph to look at, a deterministic tie-break, and an owner of the abort.

Deadlock, Livelock, and Cascade look different from outside. One stops, one spins, one spreads. All three are cheapest to handle before the run burns through its budget, which is why the detection bar is that deadlocks and livelocks are detected before budgets are exhausted.

Livelock is the one teams argue about, so take a dated instance. DynamoDB recovered at 2:25 AM PDT on 20 October 2025, and EC2's DropletWorkflow Manager (DWFM) began re-establishing droplet leases. The work could not complete before the leases timed out, and further attempts queued behind the work already failing. DWFM was fully busy and going nowhere: “At this point, DWFM had entered a state of congestive collapse and was unable to make forward progress in recovering droplet leases.” — Amazon Web Services. A heartbeat would have reported DWFM alive for every minute of it. Only a progress metric separates that state from health: leases actually re-established, not attempts made. What ended it was less work rather than more capacity. Engineers throttled incoming work and began selective host restarts at 4:14 AM, and all leases in the Region were re-established by 5:28 AM.

A cascade is the harder case. It keeps making progress while the error spreads, because each agent accepts the label the one upstream attached instead of checking it.

Cycles and cascades are cheap to cause deliberately and expensive to meet by surprise. Drill them on a real workflow. Feed one agent a confidently mislabeled input and watch how far it travels. Error correlation grows when agents trust upstream labels instead of evidence, and a drill is what makes that visible.

The drill is not an improvisation you have to justify from first principles. It is a named discipline with a stated method. Netflix's Traffic and Chaos Team defined it in IEEE Software in 2016: “Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in its capability to withstand turbulent conditions in production.” — Basiri and six colleagues. They set out four principles for designing the experiments: build a hypothesis around steady state behavior, vary real-world events, run experiments in production, automate experiments to run continuously. The first is the one most agent teams skip. Without a written statement of what steady state looks like, the drill produces anecdotes instead of a verdict.

Their reason for injecting error conditions rather than happy-path inputs is the finding already in this lesson, which they cite in their own paper: “A recent study reported that 92% of catastrophic system failures were the result of incorrect handling of non-fatal errors”. Then write down what your run showed about detection: how much budget was left when the cycle was caught, and whether anything caught it at all.

Treat coordination failures as first-class incidents. They require graph, state, and ownership observability beyond model traces. On-call engineers usually meet the cascade before they meet its cause: a stream of wrong answers that all trace back to one bad label nobody rechecked.

The cost of learning this late is on the record. On 1 August 2012 a repurposed flag at Knight Capital Americas activated years-dead “Power Peg” code that had been left on one of eight SMARS servers. 212 incoming parent orders produced over 4 million executions in 154 stocks, for more than 397 million shares, in about 45 minutes. The loss was over $460 million. The SEC's order of 16 October 2013 sets out what happened. The system had said so in advance: 97 automated “BNET reject” e-mails naming the fault were sent before the 9:30 a.m. open. They were not designed as alerts and were generally not reviewed. Machine evidence with no owner is not detection.

Then the containment attempt made it worse. “In one of its attempts to address the problem, Knight uninstalled the new RLP code from the seven servers where it had been deployed correctly. This action worsened the problem, causing additional incoming parent orders to activate the Power Peg code that was present on those servers, similar to what had already occurred on the eighth server.” A rollback is an action taken under time pressure by people reading incomplete state. It can widen the blast radius as easily as close it. The Commission imposed a $12,000,000 civil penalty.

Containment is working when the failure is caught while there is still budget left, when the containment action itself has been rehearsed against the dependency graph, and when the observability views are enough to say which agent stalled, which pair are waiting on each other, and where the bad label entered.