Long-Horizon Work, Checkpoints, and Resumability

Long-horizon agents need durable task state, artifact checkpoints, event logs, progress summaries, and restart logic. Context is a temporary working set. It is not the canonical record of the project.

The condition that record is designed against has been counted. Nine years of failure records at Los Alamos National Laboratory cover 22 HPC systems, 4,750 machines and 24,101 processors. They contain roughly 23,000 failures — every one that required a system administrator's attention. Schroeder and Gibson analysed them in 2006 and reported: “We find for example that average failure rates differ wildly across systems, ranging from 20–1000 failures per year, and that time between failures is modeled well by a Weibull distribution with decreasing hazard rate.” Mean repair time ran from under an hour to more than a day. At 20 to 1,000 failures a year, with repair sometimes measured in days, a crash is not the exception a design tolerates. It is the schedule the design is written to.

The horizon being designed for is a measured quantity too. In March 2025 METR proposed the 50%-task-completion time horizon: the time human experts typically take on tasks a model completes with 50% success. METR timed domain experts on RE-Bench, HCAST and 66 novel shorter tasks. The headline example was Claude 3.7 Sonnet, at around 50 minutes. A later version of the paper replaces it with o3, at around 110 minutes. The abstract closes: “Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024.” That is the unit of uninterrupted work actually on offer. A multi-day task is that unit repeated many times. Everything that has to survive between repetitions has to be written down somewhere else.

A resumable system can reconstruct the next valid action from persisted state after a process crash, model upgrade, user pause, or context reset.

A 16K-GPU cluster ran Llama 3 405B pre-training. For 54 days of it, someone counted every time the job stopped. Meta's Llama 3 technical report, published in 2024, records the total: “During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator-initiated operations like configuration or dataset updates.”

The other 419 were unexpected. About 78% of those were attributed to confirmed or suspected hardware issues. GPU issues alone accounted for 58.7%.

The outcome is the part worth studying. The team achieved higher than 90% effective training time. Across all 466 interruptions, significant manual intervention was required only three times. None of that depended on a running process remembering anything. The process stopped 466 times in 54 days. Recovery was carried by automation and saved state, not by anyone — or any context window — remembering what had happened.

Progress that survives a crash lives in four records, governed by a fifth thing that is not a record at all.

The Task ledger holds goals, constraints, decisions, open questions, and current status. The Artifact checkpoint holds versioned files, patches, datasets, or reports produced so far. The Event history holds actions, results, approvals, failures, and budget consumption. The 466-interruption count in the Llama 3 report is exactly this record, kept where the interrupted process could not erase it. The Resume summary is a compact, evidence-linked briefing for the next model call.

The fifth item is the Recovery policy: the rules for verifying state before work continues. Whoever writes the Resume summary should not also own the Recovery policy. The summary asserts what was true when it was written. The policy decides whether that is still true. Each of the five needs its own test.

Resumability shows up first: the next model call can continue from durable evidence instead of a full transcript. There is a measured reason to prefer a compact Resume summary to replaying everything that was said. A 2025 paper isolated execution by handing models the knowledge and the plan outright, so that nothing but the length of the run was being tested. Even with an error-free history, model turn accuracy at turn 100 falls below its initial value. Putting the model's own errors back in front of it makes that worse. In the authors' words: “More interestingly, as we increase the rate of injected errors into the context, accuracy at turn 100 consistently degrades further.” They call the effect self-conditioning. It is distinct from long-context degradation, scaling model size does not remove it, and thinking mitigates it. A transcript carrying the agent's own earlier mistakes is not a neutral record of the run. It is an input that lowers the accuracy of the next step.

Three further signals need their own tests. Whether artifacts and decisions remain attributable across model or runtime changes is checked separately, on its own record. Then, under repeated or adversarial cases, test whether resume logic detects external state drift before committing new actions — the stale-entity-tag case, expecting a 412 rather than a silent overwrite. And test whether cancellation and retirement can clean up partial work and credentials.

A workflow that already runs for hours is the right place to design a resumable task record. Test it by restarting from a checkpoint written before the world changed underneath it, and watch what the agent does with the stale picture.

Separate facts from hypotheses: persist verified state, assumptions, and failed approaches distinctly. Replaying a history in which the failed approaches look like findings is the self-conditioning case. Version artifacts: record hashes, branches, resource versions, and ownership. The entity tag of RFC 9110 is the minimal version of this — one opaque string per resource, carried so the next write can be refused.

Create checkpoints: write progress after meaningful state transitions, not only at the end. How often is a computed quantity, not a matter of taste. The optimum compute interval is the square root of 2·δ·M, where δ is the time to write a checkpoint file and M is the system's mean time to interrupt (MTTI). That is Young's first-order estimate, recorded in a 2006 paper by Daly, at Los Alamos, on the optimum checkpoint interval for restart dumps. It opens on the problem it is solving: “This paper examines methods of approximating the optimum checkpoint restart strategy for minimizing application run time on a system exhibiting Poisson single component failures.” Daly then derives a perturbation solution whose first three terms guarantee, in his words, that the relative error in total problem-solution time “never exceeds 0.2% of the exact solution time”. The paper also shows that restart time R makes no contribution to the optimum, though the first-order model treats it as a contributor. So two measurable numbers set the cadence: what a checkpoint costs to write, and how long the system runs between interruptions. How long recovery takes does not enter it.

Build a resume protocol: revalidate permissions, external versions, budgets, and pending actions before the first side effect, in that order. Test interruption: stop the process at several stages and recover without duplicate effects. The Llama 3 run supplies the target — 466 stops and three rescues.

Checkpointing is both a reliability mechanism and a governance mechanism. It creates review points where humans, policies, or new evidence can change the course. A run that checkpoints only at the end offers a person exactly one control, which is to kill it.

At every restart, a reviewer asks the same two questions. Does this checkpoint still describe the world as it is now, or is it resuming into a world that has moved on? That is the question Chubby answers with a 12-second lease and a jeopardy signal, and RFC 9110 with a strong entity-tag comparison and a 412. The second question: does the checkpoint hold enough saved evidence for the next call to continue from, without the whole transcript being fed back in? That transcript is not free. Its injected errors measurably degrade accuracy by turn 100.

Both questions are answerable only against durable records. Neither can be answered by a longer context window.