Durable State, Provenance, Replay, and Recovery

Durable state records the canonical task lifecycle; an event log records how it changed.

Provenance links model, prompt, tools, policies, artifacts, identities, approvals, and environment versions to each decision. Four records carry a run: the state snapshot, the event record, the artifact lineage, the execution provenance. None of the four is a matter of local taste. Each has a published definition, a byte-level specification, a protocol, or a legal floor behind it. The rest of this lesson names them.

Replay can reconstruct decision inputs and simulate transitions. It cannot always repeat nondeterministic models or reverse external side effects. Recovery must reconcile the present world.

Reconstructing a run means holding four records at once: a State snapshot, an Event record, an Artifact lineage, and an Execution provenance.

The last two are not improvised categories. Provenance is “the chronology of the origin, development, ownership, location, and changes to a system or system component and associated data”. That definition is official, and NIST wrote it into its Secure Software Development Framework in February 2022. It carries a second clause: provenance “may also include personnel and processes used to interact with or make modifications to the system, component, or associated data”. That clause is the one agent teams skip. The people and the processes, not only the versions.

A numbered task hangs off the definition. Task PS.3.2: “Collect, safeguard, maintain, and share provenance data for all components of each software release (e.g., in a software bill of materials [SBOM]).” The implementation examples under it include protecting the integrity of the provenance data, and updating that data every time any of the software's components are updated. A live obligation, not a release-day artifact. The framework is a set of recommended practices, not a mandate. That is precisely why a team has to decide who owns each box.

Execution provenance and the Recovery procedure should not share an owner or a test. The record of what stood behind a decision and the procedure that puts the world back are answerable to different failures.

The hardest forensic failure is not a missing log. It is a log that attributes an action to the wrong actor.

Fujitsu could insert, inject, edit or delete the branch transaction data in the Post Office's Horizon system. It could implement fixes affecting that data, or rebuild it. All of it was possible without the subpostmaster's knowledge or consent. The High Court found this in December 2019, in the Horizon Issues judgment in Bates v Post Office. An injected transaction, the judgment records, “would look as though the SPM had done it”.

The second half of the failure is that the use of those privileged powers was itself unrecorded. Fraser J wrote: “I have taken into account, in reaching these conclusions, not only the evidence of bugs, errors and defects, but also the lack of records or logs at Fujitsu over time over the use and controls of the powerful access roles permitted to Fujitsu personnel.” There was a full record of what the system did. There was no record of who had been allowed to change it.

The consequences were not internal. The Court of Appeal relied on these findings in 2021, in Hamilton v Post Office. Post Office Limited conceded that in all but three of the 42 cases, material failures of investigation and disclosure meant 39 appellants did not have a fair trial. An agent that can act with elevated credentials on a customer's behalf inherits exactly this shape of risk. The action is logged against the principal. The privileged path that produced it is not logged at all.

The first test is whether an operator can explain which evidence and policy produced each action. Three more follow, and each is checked on its own: that artifacts and state changes retain lineage across handoffs and retries, that recovery avoids replaying already-committed side effects, and that re-evaluation can compare versions using the same recorded task distribution. Run the last two under repeated or adversarial cases. A run that passes once may not pass on the retry.

Signal 3 is the one with a written protocol behind it, so it can be tested rather than believed. The mechanism is one value travelling with the request: “An idempotency key is a unique value generated by the client which the resource uses to recognize subsequent retries of the same request.” That is the Idempotency-Key HTTP header field, specified in a 2025 IETF draft by Jena and Dalal. Where a request is retried after the original completed, the resource SHOULD respond with the result of the previously completed operation, success or error. If the original is still in flight, it SHOULD return a conflict error.

Stripe's API implements exactly this. It saves the status code and body of the first request made for a given idempotency key — keys of up to 255 characters, prunable after at least 24 hours — and returns that same saved result to later retries, including 500 errors. So the test for Signal 3 is concrete. Retry a completed action with its key. Check that the saved response comes back instead of a second execution.

An event schema written after the first incident is always missing the field that incident needed. Write one now. Design it around the fact that reproducible inputs do not guarantee reproducible outcomes: record what the model saw, what policy was in force, and what the run actually did, so that two runs from the same input can be told apart afterwards. Those are the fields an operator needs when the explanation is demanded later.

The first step, assigning run identity, does not need inventing. It is standardised down to the byte, and two independent bodies specify the same shape. One id names the whole run: “This is the ID of the whole trace forest and is used to uniquely identify a distributed trace through a system. It is represented as a 16-byte array, for example, 4bf92f3577b34da6a3ce929d0e0e4736.” That is the trace-id carried in the traceparent header, defined by W3C Trace Context in 2021. It is rendered as 32 lowercase hex digits, all-zero values forbidden. The span id, parent-id, is an 8-byte array rendered as 16 hex digits. The OpenTelemetry tracing API specification requires the identical shapes: a valid TraceId is a 16-byte array and a valid SpanId an 8-byte array, each with at least one non-zero byte.

Adopt those shapes and the identity step is finished before the argument starts. What remains genuinely yours to design is what hangs off the identifier. The decision inputs: selected context, tools, budgets, policy state. The outputs and checks: proposals, validations, executions, verification results. The checkpoints, taken after consequential transitions and before long waits. And the recovery drill that resumes after crashes, expired credentials, changed models, and external drift.

Traceability is not only for audits. It is the foundation for debugging, regression evaluation, recovery, and safe model upgrades.

For one class of system it is also a legal duty with a number attached. “High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system.” That is Article 12(1) of the EU's Artificial Intelligence Act, adopted on 13 June 2024 and published in the Official Journal on 12 July 2024. Article 12(2) ties that logging to the “traceability of the functioning” of the system. Article 19(1) requires providers to keep those logs for a period appropriate to the intended purpose of at least six months. The floor is automatic, lifetime-long recording and a six-month retention. Not a design preference a sprint can defer.

The failure to keep in view is that the same inputs can still produce a different run. A durable system therefore has to record what happened rather than assume it can be recomputed. When something goes wrong, an operator should be able to open the record and say which evidence the agent had and which policy applied at the moment it acted.