Summarization, Compaction, and Context Drift

Twelve simulated patients. Five consecutive one-to-one handovers each. At every cycle, a score for how much clinical information survived. Three handover methods went down the same chain. The results were published in 2007, in the Annals of the Royal College of Surgeons of England. Verbal-only handover retained 2.5% of the patient information after five cycles. Verbal handover with note-taking retained 85.5%. A printed handout containing all patient information retained 99%.

No single step in the verbal chain looks like a failure. Each person hears something coherent and passes on something coherent. The record is destroyed anyway, because every cycle summarises the previous cycle rather than the patient.

That is recursive compaction with the source out of reach. It is the exact shape of the agent failure this lesson is about. Requested and approved are different states, and that difference does not survive five relays, because each turn reads the previous turn's account of the one before it. The agent then acts on a record that reads complete and is 2.5% intact.

Recursive summary, Summary from sources, and Structured compaction differ mainly in what they summarize from. That difference has been measured rather than asserted. The measurement is the handover experiment, and the three columns map onto its three arms.

Recursive summary is the verbal-only arm. It is cheap, and its "Compounds drift" line is a quantity, not an adjective: 2.5% after five cycles. Summary from sources is the printed handout — the full record re-issued each time rather than relayed. That costs more per cycle. It is also why it can be corrected, and why it ends at 99%. Structured compaction is the note-taking arm, made mandatory and typed: a written carrier for the fields that must not be lost. That is what lifts 2.5% to 85.5%.

The 2007 abstract states the headline flatly: "After five handover cycles, only 2.5% of patient information was retained using the verbal-only handover method".

Whichever you pick, the same standard applies. The compacted context has to preserve every material constraint and every pending commitment. A compaction that reads well but quietly drops a caveat has failed. Nothing in the summary will look wrong.

Compaction reduces context by replacing detailed history with structured state, summaries, and references. It is necessary for long tasks. But every compression chooses what to preserve and what to discard, and the loss does not wait for the second generation to appear.

One 2020 study measured a single, non-recursive pass. It covered 2,500 document–summary pairs on XSum: 500 documents across 5 systems. The judgements came from three trained annotators, fluent in English and put through two pilot studies rather than recruited as crowd workers. All three marked at least one word as a hallucination in 73.1%–79.3% of the model summaries. They did the same in 76.9% of the human-written gold summaries. Over 90% of the extrinsic hallucinations were erroneous. The abstract does not hedge: "Our human annotators found substantial amounts of hallucinated content in all model generated summaries."

The same paper explains why "the summary reads well" is not a defence. ROUGE and BERTScore correlated with faithfulness far less well than textual-entailment measures did. The metric that rewards fluent output is the one that misses unfaithful output.

So summaries accumulate drift across generations, and each generation starts from a pass that was already unreliable at one hop. Critical facts, permissions, open questions, and negative evidence have to remain structured or linked to source events. That link is the only thing an entailment check has left to run against.

Compaction stays safe only while four things stay separate. The Immutable source: raw events, artifacts, and tool results retained outside context. The Structured state: canonical status for facts, commitments, approvals, and blockers. The Narrative summary, written for model orientation. And the Source references that permit drill-down and verification. A Refresh policy governs the whole arrangement, saying when summaries are rebuilt after material change. The owner of the Source references should not be the owner of the Refresh policy. The tests should be separate too.

That separation is not hypothetical. Shipped platform defaults already encode a version of it. Anthropic's context-editing beta, strategy clear_tool_uses_20250919, activates at a default trigger of 100,000 input tokens and keeps only the 3 most recent tool use/result pairs. What it keeps and what it clears is the whole lesson: “By default, only the tool results are cleared while keeping Claude's original tool calls visible, so Claude can see what operations were performed even after the results are removed.” That sentence is Amazon's, from the Bedrock user guide. Bedrock documents the identical strategy name, the identical 100,000-token and 3-tool-use defaults, and the same clear_tool_inputs default of False. The Claude platform documentation lists the same defaults, alongside a second strategy, clear_thinking_20251015.

Read that split against the map. The narrative of what was done — the tool call — is deliberately retained. The evidence that it worked — the tool result — is deliberately discarded. The default is defensible for one reason. Context editing is applied server-side before the prompt reaches Claude, while the client keeps the full history. The Immutable source still exists, off-context, reachable. The separate compaction beta makes the same bet at a higher threshold: a default trigger of 150,000 input tokens, a minimum of 50,000, and then every content block prior to the compaction block is dropped. If your architecture has no record outside the window, these defaults are not editing your context. They are deleting your only copy.

Adversarial omissions are what a compaction test set is for. Fill the set with cases where compression tends to remove caveats before removing conclusions. Run the compactor over them. Compare what came out against what went in. Whatever the summary dropped is the material the next turn will act without. Two published precedents tell you what the artifact looks like, and neither of them is a reading exercise.

The first is the field version of "choose canonical forms". Nine hospitals replaced free-text sign-out with a prespecified element list — nine written elements and five oral ones — under a handoff program called I-PASS. The New England Journal of Medicine published the result in 2014: “In 10,740 patient admissions, the medical-error rate decreased by 23% from the preintervention period to the postintervention period (24.5 vs. 18.8 per 100 admissions, P<0.001), and the rate of preventable adverse events decreased by 30% (4.7 vs. 3.3 events per 100 admissions, P<0.001).” Inclusion of the key elements rose significantly on every one of the 14 comparisons, P<0.001. Handoff duration did not change significantly: 2.4 vs. 2.5 minutes per patient. Typed fields, prespecified. Success measured downstream by error rate, not by how the handoff read.

The second precedent is the shape of the evaluation itself. SummEval benchmarked 23 summarization models on CNN/DailyMail and released the human judgments alongside them: “we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations”. Its argument is that the lack of consensus on evaluation protocols is itself what blocks progress. Which is the argument for writing your protocol down before you need it.

The five steps follow from those two precedents. Identify protected information: list constraints, approvals, blockers, failed attempts, and unresolved conflicts, the way I-PASS prespecified its elements. Choose canonical forms: store critical status in fields rather than free text. Generate summaries from evidence rather than repeatedly summarizing prior summaries — the 99% arm, not the 2.5% arm. Create omission tests: check whether qualifiers, negative findings, and pending actions survive. Measure downstream behavior: run the agent with full and compacted context on the same decisions. That, and not the readability of the summary, is the I-PASS endpoint.

Treat summaries as derived artifacts. They should never be the sole record of permissions, executed actions, or unresolved risk. The engineer who writes the compactor is the one who has to answer for what it drops. That engineer keeps the original record reachable rather than trusting the summary to stand in for it.

For high-risk systems this has stopped being advice. The EU AI Act — Regulation (EU) 2024/1689 — states in Article 12(1): “High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system.” Article 19(1) attaches a duration to it. Providers must keep those logs “for a period appropriate to the intended purpose of the high-risk AI system, of at least six months”, to the extent the logs are under their control. A compactor that leaves no pointer back to that log is not a design smell in such a system. It is a compliance failure with a number on it.

The honest test of a compactor is not whether the summary reads well. It is whether every constraint and every open commitment from the original is still there afterwards. And whether the log the regulation requires is still reachable from the line of prose that replaced it.