Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers

AI summary

Overview

Research area: AI agent infrastructure — specifically observability and context management for long-running autonomous agents (LLM agents, agent traces, streaming state models).

Technical level: Advanced. The paper is an architecture-and-measurement paper with detailed protocols, cost accounting, statistical tests, and benchmark design; the prose is readable but the density of experimental detail assumes familiarity with agent systems and evaluation practice.

Scope (1 sentence): The paper proposes a single "live trace model" — an append-only event ledger that is folded into typed run state and compiled into two views — and measures it against deterministic ground truth for both the human observer monitoring a run and the agent folding its own trace back into a bounded context.

What This Paper Is About

A long-horizon agent continuously produces one artifact: its trace. That trace outgrows both of its consumers — the human trying to monitor what the run is doing mid-flight, and the agent itself, whose context window cannot hold the history it must reason over. Today these two consumers are served by two separate systems built over the same stream: observability tooling on one side and context management on the other. The paper models the trace once, folds it into typed state, and compiles a separate view for each consumer from that single substrate, then measures both legs against deterministic ground truth.

Key Contributions

1. An architecture (tracelab): an incremental, resumable, cache-aware parse of an agent's stream, organized as four layers — an append-only typed event ledger with byte-offset resume, a single-pass fold producing RunState, versioned derived nodes with a validity lifecycle and hindsight re-parsing, and per-consumer compiled views (an observer HTML page and a compact worker text block). A curator loop re-materializes the worker's view from the worker's own recorded trace every K = 5 steps as deployed.

2. Eleven requirements on the fold, each surfaced by a live failure during a sequential development ladder and pinned by a regression test or confirmation rerun — plus a measured boundary: an order-sensitive task family (alternating-sign chains) on which the fold's aggregates do not apply and it ceases to help.

3. COMPREHEND, a live-run comprehension benchmark with mechanically generated questions and deterministic grading, whose corpus is fully regenerable from code: a seeded generator rebuilds twelve realistic sessions byte-identically, so the instrument is auditable without releasing real data.

4. CONTINUE, a workbench isolating context policy under matched adversity (pre-committed error schedules, retry/repeat separation, costed protocol failures), with a family of controls including a scratchpad, a calculator tool, retrieval, observation masking, summarization in two configurations, and a masked-history-plus-notes hybrid.

Main Findings

• Observer reads get dramatically cheaper and more accurate with the compiled view. On 12 real transcripts (112 MB; 70 questions per condition), Sonnet 5 scored 0.871 accuracy on the compiled view using 57K input tokens and $0.42, versus 0.479 on a capped raw tail using 779K tokens and $2.37, and 0.621 on a flat log at 479K tokens and $1.59. Haiku 4.5 scored 0.850 on the view (43K tokens, $0.08) versus 0.476 on the raw tail (652K tokens, $0.53) and 0.522 on the flat log (372K tokens, $0.32). The abstract summarizes this as approximately 14x and 15x fewer input tokens by reader and 5–7x lower cost, with accuracy 0.85–0.87 versus 0.48.

• The condition gap exceeds the reader gap. Within any condition, the two readers differ by at most 0.10, while view-versus-raw is approximately 0.37–0.39. The smaller reader on the compiled view (0.850, $0.08) beats the frontier reader on the raw tail (0.479, $2.37); the paper notes the resulting approximately 30x cost ratio conflates model-tier pricing with the representation change, so the same-reader 5–7x figure is the appropriate summary.

• Confidence intervals are disjoint. Transcript-level bootstrap 95% CIs from an independent full rerun: view 0.86 [0.78, 0.93] versus raw 0.51 [0.42, 0.61] for Sonnet 5; 0.81 [0.74, 0.88] versus 0.46 [0.39, 0.54] for Haiku 4.5. Rerun cell-by-cell deltas span 0.013–0.051.

• The view's advantage concentrates in aggregation. Whole-run aggregation questions collapse without the view (files: 0.13–0.20 raw versus approximately 1.0 view). The dangling-calls question scores 0.917 in every condition because eleven of twelve transcripts share the majority "none" answer. Macro-averaging over the remaining five question types widens view versus raw to 0.850 versus 0.385 (Sonnet 5) and 0.815 versus 0.382 (Haiku 4.5). The latest-ask question still scores 0.833 from the raw tail because recency survives truncation.

• Capping the raw trace is the binding constraint. The reading budget is 100K characters (57.8K tokens; 1.73 characters/token on escape-dense JSONL). The median transcript is approximately 11 MB, or approximately 6M tokens at that ratio — beyond the context window of any model in the deployed stack, which is the infeasibility the system targets.

• Agent leg: a clean crossover on 120-link sequential-dependency tasks. Under the final protocol (shipped renderer, no injected errors, identical seeds, n = 30 per arm), the curated view scored 30/30 at $1.59/run (cached), the scratchpad 30/30 at $0.97/run (cached), and flat full context 8/30 at $7.13/run (uncached). Paired on shared seeds, the curated arm has 22 sole successes and 0 sole failures (exact McNemar, two-sided, p approximately 5 × 10⁻⁷).

• Full context's cost and failure are separate problems. On the sequential-dependency workbench, a full-context worker's cumulative billed input reaches 2.37M tokens at 120 links (per-call context approximately 33K tokens and growing) while success falls to 7/30 under development-era injected-error schedules (8/30 under the clean protocol). The uncached flat arm bills $7.49 per run; the cached conversational variant costs approximately $1.06 yet fails entirely (0/10).

• Every mechanism carrying the running statistic in per-step state succeeds. At 120 links in the development-era grid: curated fold 25/30 ($1.93), cached scratchpad 26/30 ($1.04 mean at n = 30), masked-history-plus-notes hybrid 10/10 (approximately $2.98), calculator tool 10/10 ($14.88). Failures include plain full context 7/30, conversational-form full context 0/10, retrieval 0/10, and word-capped summarization 0/3 at 30 and 60 links. The dividing line is whether the per-step context carries the task's needed statistic explicitly — not external state as a category, since the retrieval arm stores the entire trace externally and still fails. The paper treats this as a hypothesis with one forward confirmation (the hybrid), not an established regularity.

• Paired analysis at 120 links. Of 30 shared seeds: 19 curated-only successes, 1 full-only (seed 11), 6 in both, 4 in neither (exact McNemar, two-sided, p approximately 4 × 10⁻⁵; unpaired two-sided Fisher p approximately 6 × 10⁻⁶ reported for continuity). The extension-only cells (16/20 vs 4/20, Fisher p approximately 3.6 × 10⁻⁴) confirm the effect independently of the triggering sample; excluding the three development seeds gives 22/27 vs 6/27.

• A prompting-only instruction is itself a first-order treatment. One instruction to emit a per-step note field moves full context from 7/30 to 26/30.

• A two-arm decomposition separates accuracy from cost. Holding computation equal and varying boundedness, the compiled view plus full raw history scores 10/10 at $7.63/run. Holding boundedness equal and varying computation, the same bounded view with aggregates stripped scores 4/10 at $2.14/run, its failures being end-stage arithmetic errors (two runs wrote partial-prefix sums, scored 0.81 and 0.78). In these n = 10 cells, the deterministic aggregate carries the accuracy and boundedness carries the cost.

• A minimal alternative sets a lower bound. A fifteen-line regex tally over the worker's own past observations, prepended to the last five raw steps with no ledger, fold, or view, scores 10/10 at $0.67/run — the cheapest cell reported. The paper concedes that on this family a minimal artifact is sufficient, and that everything the fold carries beyond it is justified by the second consumer and the audit trail rather than by chain accuracy.

• Requirement 11 was surfaced by failure analysis. The curated arm's five 120-link misses all had the exactly correct total displayed (the fold was correct in 30/30 runs), and the worker then added one already-included delta (+37/+78/+79/+31/+71). After adding a coverage stamp to each aggregate line, the five failing seeds recovered 5/5 and five passing controls held 5/5. A full-grid rerun with the shipped renderer under injected schedules scores 29/30 ($1.77/run). The curated sequence is reported as three separate, never-pooled cells: 25/30, 29/30, 30/30.

• Boundary of applicability. On alternating-sign chains, where the k-th file's delta enters with sign (−1)^(k+1) so per-key sums are uninformative by construction, the curated view scores 3/10 at 60 links against full context's 6/10 (two-sided Fisher p approximately 0.37), and both arms score 0/10 at 120. Worker-side notes adapt: the cached scratchpad scores 5/5 at 60 and 9/10 at 120 on the same family. The limitation is specific to fixed harness-side aggregates, not to external state in general.

• The summarization cap was catastrophic, not merely weak. A rolling summary word-capped at approximately 400 words scored 0/3 at both tested horizons; removing the cap yields 3/5 and 4/5 at 30 and 60 links. At 120 links an n = 10 cell confirms degradation (3/10, at approximately $2.84/run including the summarizer).

• Cache behavior is determined by structure, not intent. A cache breakpoint on a rebuilt monolithic prefix produced zero cache reads and cost more than no caching ($2.06 vs $2.03), because lookup occurs only at the new request's breakpoints. With caching, full context at 60 links drops from $2.03 to $0.374 (5.4x, approximately 501K reads/run), below the curated arm's cached $0.66 — yet full context remains the less reliable arm (6/10 vs 8/10 uncached at that horizon; 1/3 in the cached conversational cell). The curator's refresh cadence acts as a cost parameter: $0.66, $0.61, and $0.55 at the intervals reported before the text truncates.

• Accounting hazard. The per-block format repeats message-level usage on every record; without deduplication by API message id, token accounting inflates by up to 3.49x. Corpus-wide inflation ranged roughly 2–3.5x, and the cost ratio tracks the token ratio.

• Ingest is cheap. 104 MB across the ten largest transcripts parses into 16,737 events in 0.4 s on one Apple silicon performance core (a single warm-cache, single-threaded timing, described as indicative rather than a benchmark).

• Floor checks pass. DETECT's rule-based detectors achieve P = 1.0 / R = 1.0 with zero false positives across 48 labeled pathology instances in 30 synthetic traces. FIDELITY's incremental fold matches a mutation-tested independent oracle on 8/8 bookkeeping fields, and a from-scratch recount reproduces every per-key fact-plus-aggregate sum on five 120-link traces with zero mismatches. Tool-call/result matching links 3669/3671 pairs = 99.95% on the real corpus.

• Results are labeled descriptive. Because the task family, the mechanism, and the extension decision were developed adaptively against the generator, the paper calls its p-values descriptive rather than confirmatory, and states that the confirmatory experiment — a frozen system on externally authored tasks — has not been run.

Methodology in Plain English

The researchers take the raw stream an agent emits and parse it once, rather than building two unrelated systems over it. First, each content block becomes a typed event in an append-only ledger, with a byte offset serving as a resume token because the observed writer only appends at the tail. Content-bearing records carry a SHA-256 fingerprint of the original text, so referenced material is tamper-evident even when only an excerpt is inlined. Malformed lines are quarantined and counted.

Second, a single-pass fold reduces that event stream into typed state: execution position, counters the SDK does not expose, files touched, and "facts" extracted from tool results and keyed by source with occurrence identity. The fact store is bounded, and eviction is aggregate-preserving — evicted numbers fold into per-key running counts and sums, so the tracked statistics survive exactly. Third, versioned derived nodes (per-turn episode digests) carry a validity lifecycle, because an interrupted turn is only recognizable once the next user turn arrives with calls dangling; the parser revises the past by superseding versions rather than rewriting them.

Fourth, both views compile from that one state: an HTML observer page with goal, live frontier, anomaly badges, stats, episode drill-downs, and provenance links; and a compact text worker view. A curator loop refreshes the worker view from the worker's own recorded trace every K = 5 steps.

Evaluation avoids LLM judges entirely. Every score is computed against deterministic ground truth — constructed (synthetic pathologies, pre-committed schedules), computed by an independent oracle, or mechanically derived from the ledger. The LLM reader in COMPREHEND is not a judge but the measured consumer of the representation under test, graded by exact match, substring, or set-F1. For the agent leg, arms share worker model, tasks, seeds, and pre-committed per-call-index error schedules, with success verified against final environment state; retries are distinguished from repeats and protocol failures are recorded and costed. Cache costs were measured with actual cache-read and cache-creation token accounting.

Why This Matters

Impact on research. The paper argues that trace handling for long-horizon agents has been split across two research literatures that rarely meet: observability tooling, which aggregates spans at the transport layer rather than maintaining a semantic model of what a run is doing, and context compression, an active but contested family where reported effect sizes conflict and independently measured compaction effects are small. Its closest neighbors — VISTA's proprioceptive working-memory dashboard, ESAA's strict event sourcing for project governance, PROJECTMEM's cross-session projection, LangGraph's single-log-many-views pattern — each supply part of the picture. The single-fold-serving-both-consumers architecture, measured for both consumers, targets that remaining intersection. The paper also contributes instruments (COMPREHEND, CONTINUE) designed so that their corpora are regenerable from code and their grading is deterministic, in contrast with LLM-judged agent evaluation.

Real-world applications:

• Monitoring production agent runs, where an operator needs to know mid-flight what has settled and what remains pending without reading a 112 MB transcript.
• Cost control for long-horizon deployments, where a view's compact per-call context (approximately 6K tokens at 120 links versus approximately 33K and growing) determines whether a task is economically viable.
• Regression testing and incident review: the refusal-tolerant batching, verbatim validation, and aggregate-coverage requirements arose from specific live failures and persist as audit trails.
• Systems serving both a human dashboard and the agent itself from one substrate, avoiding duplicated parsing pipelines and inconsistent state.

Industry relevance. The authors are at Salesforce AI Research and release code, benchmarks, a regenerable synthetic corpus, and all workbench traces. The paper is candid that its headline agent-side result is narrower than a single crossover suggests: the fold's remaining value over cheaper alternatives is deterministic auditability and serving the observer from the same state. The deconfounding arms, the $0.67 minimal accumulator, and the reverse inference that the fold can underperform raw history on order-sensitive tasks are all reported as boundaries, which makes the work

Authors’ abstract

A long-horizon agent's trace outgrows both of its consumers: the human observer monitoring the run, and the agent itself, whose bounded context the trace must be folded back into. We present a live trace model, an append-only event ledger folded incrementally into typed run state and compiled into per-consumer views, and evaluate it for both consumers against deterministic ground truth. For the observer side, evaluated with an LLM reader as proxy, the compiled view answers monitoring questions using approximately 14x and 15x fewer input tokens (by reader) and at 5-7x lower cost than a budget-capped single-call reading of the raw trace, with higher accuracy (0.85-0.87 versus 0.48). Because the questions were co-designed with the view schema, we treat the token and cost reduction, conditional on schema coverage, as the transferable result. For the agent, on 120-link sequential-dependency tasks, mechanisms that maintain the task's running statistic in per-step state succeed where full-context prompting fails (30/30 versus 8/30 under a clean protocol, n=30, labeled descriptive owing to benchmark-system co-development); a prompt-level scratchpad matches the fold's accuracy at lower cost, and a two-arm decomposition attributes the fold's accuracy to its deterministic aggregate and its cost advantage to its compactness. The fold's remaining value over cheaper alternatives is deterministic auditability and serving the observer from the same state. We derive eleven candidate requirements for trace folding from observed failures and delimit them with an order-sensitive task family on which the fold ceases to help. Code, benchmarks, a regenerable synthetic corpus, and all workbench traces are released.