State Machines, Policies, and Explicit Orchestration

On 1 August 2012 a new deployment at Knight Capital repurposed a flag. The flag woke dormant “Power Peg” routing code that Knight had stopped using in 2003 and left on one of eight SMARS servers. The retired path started routing and did not stop. In about 45 minutes, 212 incoming parent orders produced 4 million executions in 154 stocks for more than 397 million shares. The loss was $460 million.

The authoritative state existed. It was simply held somewhere the acting component could not read. The SEC's order puts the split in one sentence: “Although one part of Knight’s order handling system recognized that the parent orders had been filled, this information was not communicated to SMARS.” SMARS was not confused about the state of the task. It had no state of the task to be confused about. So it went on filling an obligation that was already discharged. On 16 October 2013 the SEC ordered Knight Capital Americas LLC to pay a $12 million penalty for violating the Market Access Rule.

An agent that reads its lifecycle out of a transcript is in the same position as SMARS. The finished state may be perfectly well known to the system and still not be known to the component that can still act.

Conversation-led control, state-machine control, and a workflow language disagree about where the truth of a task lives. The choice is working when the authoritative task status exists outside the transcript. Anyone arriving late can then ask the system what state the task is in instead of inferring it from messages. It has failed when a stale message can resurrect work that policy already terminated.

The third column is not a hypothetical. Its properties are published rather than argued. AWS Step Functions models a workflow as a state machine, and the documentation prints the contract in numbers: “Standard workflows have exactly-once workflow execution and can run for up to one year.” Express workflows take the opposite side of the same trade: at-least-once execution, up to five minutes. One year against five minutes is what “strong long-running support” actually measures.

Exactly-once against at-least-once matters more for this lesson. At-least-once is a documented delivery property. Duplicate and late events are therefore a design input for the guard, not an unlucky edge case to be surprised by in production.

Underneath the service, the Amazon States Language defines the machine as a JSON object. It terminates only on a Terminal State — Succeed, Fail, or an End State. The operational overhead in the third column is the price of that. Someone has to write the states down.

A state machine defines valid task states and the events that permit transitions between them. It can encode approvals, timers, retries, cancellation, escalation, and terminal conditions outside the model.

That the durable record should outrank the transcript is not merely a preference. For at least one production orchestrator the equivalence has been proved. A 2021 paper, Durable Functions: Semantics for Stateful Serverless, formalises Azure Durable Functions. It shows that a record-replay execution model is equivalent to the fault-free high-level model. Record-replay persists progress as a logged history rather than as a language-runtime checkpoint. The abstract states the two problems it takes on: “Next, we demystify how the DF runtime can (1) execute in a distributed unreliable serverless environment with compute-storage separation, yet still conform to the fault-free high-level model, and (2) persist execution progress without requiring checkpointing support by the language runtime.”

Notice what that architecture makes impossible. The logged history is not a summary of what happened, kept alongside the real state. It is the only authority, because the code is re-executed from it. Nothing can be true of the workflow that is not in the record.

The model can still decide among valid transitions or produce artifacts. It should not infer the authoritative lifecycle state from a transcript when durable state exists.

State, event, guard, transition: four words, and they are not this lesson's coinage. The W3C fixed them in State Chart XML (SCXML), a Recommendation since 1 September 2015. The specification puts the whole model in one sentence: “Transitions between states are triggered by events and conditionalized via guard conditions.”

In that same specification the guard is not a concept but an attribute you can look up. It is called cond. It sits on the <transition> element. It is Required: false, its type is a Boolean expression, and its default value is 'true'. The documentation describes it as “The guard condition for this transition. See 3.13 Selecting and Executing Transitions for details.”

The default is the line to read twice. A transition whose guard nobody wrote is not an unguarded transition awaiting a decision. It is a transition whose guard evaluates to true. That is exactly how a late event gets accepted.

The formalism behind the notation is older. Statecharts arrived in 1987, after three years of applying the formalism to the specification of one particularly complex system. David Harel's abstract says precisely what they add: “Our diagrams, which we call statecharts, extend conventional state-transition diagrams with essentially three elements, dealing, respectively, with the notions of hierarchy, concurrency and communication.” Hierarchy, concurrency and communication are the three things a flat transition table cannot express. They are also the three that an agent lifecycle with nested waits, parallel branches and cross-task signals requires. Transitions and terminal states still need separate owners and separate tests.

Model the full lifecycle of one long-running task, including its terminal states and the events that must be refused after them. Then test it with a message that arrives after a terminal state. See whether the model quietly restarts the work. SMARS kept routing for about 45 minutes because nothing it could read said the parent orders had been filled.

Borrow the shape instead of inventing one. The Amazon States Language ends a machine only at a Terminal State — Succeed, Fail, or an End State. A2A separates the four states that end a task from the two it labels interrupted. SCXML gives every transition a cond that defaults to 'true', so the states you decline to guard are the ones that will accept anything. Lamport's rule supplies the version: each message carries the clock value it was sent with, and the receiver advances past it.

The lifecycle is good enough when every component that needs to know the state of the task can read it from the record rather than replay the conversation.

Use state machines for obligations and lifecycle. Use models for judgments that cannot be encoded as fixed transitions. When a late message arrives, the engineer who owns the orchestration checks two things. Would accepting it restart work the policy has already ended? And does the state it claims agree with the state the system holds outside the conversation?

The agent protocols reached the same conclusion as the workflow engines, and wrote it into a machine-readable file. The Agent2Agent (A2A) protocol reached version 1.0.0 on 12 March 2026, and 1.0.1 on 28 May 2026. Its TaskState enum has nine values, and the specification does not treat them as a flat list. TASK_STATE_COMPLETED, TASK_STATE_FAILED, TASK_STATE_CANCELED and TASK_STATE_REJECTED are marked terminal states. TASK_STATE_INPUT_REQUIRED and TASK_STATE_AUTH_REQUIRED are labelled interrupted states — a task that has stopped moving but has not ended. The protocol file carries the distinction itself, in the comment on TASK_STATE_COMPLETED: “Indicates that a task has finished successfully. This is a terminal state.”

That is what makes refusal possible. A state named terminal in an enum is something a receiver can check an arriving event against and decline. A state that is only implied by the last few messages gives it nothing to decline with, and the work resumes.