Does an Agent's History Tell You When Compaction Will Hurt? A Modest, Bounded Effect on the TRACE Paired-Replay Corpus

AI summary

Overview

Research area: Context management and process-level evaluation of long-horizon LLM agents (context compaction, paired counterfactual replay, learned triggering policies), published in the Interpreting Agent Behavior (IAB) workshop track at arXiv (2610.08722v1 [cs.AI], 06 Oct 2026).

Technical level: Advanced. The paper assumes familiarity with AUROC, bootstrap confidence intervals, preregistration gates, held-out cross-validation, and agent harness internals such as context windows and token budgets.

Scope: A preregistered, gate-based empirical study asking whether the pre-boundary history of an agent predicts whether a given context compaction will harm the agent's next actions, evaluated on TRACE's released corpus of 590 harness-triggered AppWorld compaction boundaries.

What This Paper Is About

Long-horizon agent harnesses compact their context on a global rule, usually a token budget, without regard to what the agent was doing at that moment. The authors ask whether the agent's recent behaviour predicts when compaction will hurt, using TRACE's paired-replay corpus in which each boundary is replayed from a re-executed prefix state both under the pre-compaction context (PRE) and under the generated summary (POST). The goal is to test both a pre-specified "placement" hypothesis and, failing that, whether a history-based trigger policy can beat random selection at matched retention of compaction opportunities.

Key Contributions

1. An internally prespecified placement test contrasting boundaries whose prefix contains a write against read-only boundaries, paired with a post hoc diagnostic showing that the naive "has-written" label largely measures trajectory phase rather than anything about placement.
2. A task-held-out risk ranking of boundaries and an opportunity-retention frontier evaluated against global ON/OFF and the random-rule expectation at matched retention, which the source release did not report.
3. A benchmark of two fixed learners (an L2 logistic regression and a shallow gradient-boosted classifier over seventeen engineered summaries of the released observables) against a split-half replicate yardstick, under a frozen extension protocol with post-freeze additions dated in Appendix A.
4. A statement of what corpora should ship (ordered actions, token counts, summary text, more replays) so that the question of whether a trigger beats a token-budget rule at matched retention can actually be answered.

Main Findings

• The prespecified placement contrast is a wide null. On the primary outcome dU@3, post-write minus read-only is −0.062 [−0.209, +0.079], half-width 0.144. The frozen informativeness criterion (interval excluding zero, or half-width at most 0.10) was not met. A stem-clustered post hoc sensitivity gives [−0.238, +0.100].

• The naive "has-written" label measures trajectory phase. Of the 494 write-prefixed boundaries, 368 have no write beyond login or session calls and 126 contain a state change to task-relevant application data. Mean prefix steps are 7.9 (no write), 15.1 (login-only write) and 22.1 (substantive write), and mean compaction index is 1.22, 1.89 and 3.14 respectively.

• Post hoc relabelling collapses under controls. Excluding login/session endpoints from the write set yields an uncontrolled contrast of −0.22 that falls to −0.004 [−0.19, +0.18] once task fixed effects, log prefix steps and a first-compaction indicator are added — consistent with confounding by the measured phase variables.

• Policy triggers beat the random-rule expectation on count but not decisively. At ≥80% retained, the two-feature logistic avoids 0.206 [0.156, 0.256] of positive-burden boundaries at 0.837 retained, a paired advantage of +0.044 [+0.017, +0.070] over the random-rule expectation (a post hoc interval). By burden mass the logistic policy has no advantage: −0.001 [−0.034, +0.035].

• Interpretable policies alone sit close to random. Across Table 2's original-label cells, held-out AUROC for the interpretable policies is 0.404–0.635 (0.55–0.62 on the primary target), and the seven policies sit two to seven points above the random-rule expectation on count (only the stump's interval crosses zero) and zero to eleven points on burden mass.

• The best trigger avoids 21% of harmful boundaries while keeping 84%. The best frozen, interpretable trigger achieves this; its advantage over the random-rule expectation holds on count but not on burden mass, and that comparison is post hoc.

• Learned extension triggers reach AUROC 0.66, short of a replicate of the same boundary. The boosted classifier on seventeen history features reaches held-out AUROC 0.66 [0.61, 0.70] (0.64 on the replicate's own label). At 80% retained it blocks 0.247 [0.217, 0.277] of positive-burden boundaries and 0.238 [0.201, 0.284] of positive burden mass against a random-rule expectation of 0.200 (+0.047 [+0.019, +0.079] by count). The 144-feature representations reach AUROC 0.66 [0.61, 0.71]; hashed observation text or exact calls did not improve held-out AUROC point estimates (0.61–0.65).

• A replicate of the same boundary is a stronger predictor than any policy. Splitting each boundary's PRE and POST rollouts into disjoint halves (200 random splits) gives AUROC 0.69–0.72 at k=3 (0.72 [0.69, 0.75] under the frozen extension protocol; 0.74 for first compactions, 0.68 for later). The half–half correlation is 0.40, the task-level share of variance is near zero, and three-class agreement is 0.55 (κ 0.28), meaning the target itself is noisy.

• First compactions carry more burden. Under task fixed effects the first-compaction indicator carries +0.274 [+0.084, +0.474] under the original label and +0.273 [+0.091, +0.466] under the post hoc label. The tree's leaf for a short history ending in a near-empty observation reaches 0.89–0.93 in four of five folds on its fitted rows (0.71 in the fifth; 28–36 boundaries per fold, mostly first compactions).

• Gate outcomes. G1 (reproduction) met on Python 3.12; G2 (taxonomy coverage) met as amended — the frozen hand-audit clause was never performed and was replaced by a map-blind LLM-labeler audit (119/119, no threshold); G3 (informative heterogeneity answer) not met; G4 (informative policy answer) met with half-width 0.050 against a criterion of ≤0.15; G5 (layer distinctness from the source) met.

• One illustrative boundary. For task 6ea6792_2 (easy), a first compaction after seven steps with dU@3 of 1.8 (five times the cohort mean of 0.35), all five PRE rollouts avoid blocked or refetch actions in their first three actions, while under POST four rollouts hit a NameError and re-issue a prefix call and the fifth re-reads an already-fetched API list.

Methodology in Plain English

The authors take TRACE's released cohort as-is: 590 compaction boundaries drawn from 275 of 294 trajectories across 147 AppWorld tasks, with the agent and the compressor being the same model (minimax-m3) running a deliberately tight 4,096-token window, and all boundaries triggered by the harness rather than chosen by the agent.

At each boundary, the recorded code prefix is re-executed into a fresh environment to rebuild the state, and then the agent runs at most five actions either from the PRE rendering (the full raw prefix at first compactions; the previous summary plus the raw suffix later) or the POST rendering (the recorded summary plus the retained last turn). Five samples per arm are used at first compactions and three at later ones. This yields a paired counterfactual outcome per boundary, which task-success metrics cannot provide.

Burden is measured in three channels over the first min(k, L) actions, for horizons k in {1, 3, 5}: blocked (the action errored under AppWorld's native error contract), refetch (the exact action signature had already been seen), and union. The paired delta is mean(POST) minus mean(PRE), giving dE, dR and dU. dU at k=3 is the preregistered primary outcome, with 344 of 590 boundaries positive. Intervals are 95% percentile bootstraps clustered over tasks, with 10,000 draws for preregistered analyses and 2,000 for extensions.

For the label test, each AppWorld endpoint is mapped to read or write via the HTTP method in its decorator, covering all 119 observed functions; the map is cross-checked by an agent-authored semantic table (43/43 non-GET agreement) and a map-blind audit by two LLM labelers of different model families (119/119, and relabelling moves 0 of 590 boundaries).

For the policy half, triggers score each boundary from pre-boundary observables and block compaction at or above a threshold. Because both arms are observed for every boundary, a policy's value is a plug-in sum over logged arms rather than an off-policy estimate. Boundaries are ranked by out-of-fold risk score, and the authors trace a frontier of positive-burden boundaries blocked against equal-weight compaction opportunities retained, with the random-rule expectation as the diagonal y = 1 − x. The replicate yardstick comes from splitting each boundary's rollouts into disjoint halves so one half's paired delta predicts the sign of the other's.

Why This Matters

Impact on research. The paper supplies a rigorous negative-ish result about a widely assumed design lever. It shows that the released pre-boundary observables barely separate harmful from harmless compactions, that a plausible-sounding label ("the agent had written something") is really a proxy for how long the trajectory has run, and that the outcome is moderately reproducible (replicate AUROC 0.72, r 0.40) yet only weakly predictable from history. It also introduces a reusable evaluative structure — paired replay, burden channels, a retention frontier against the random-rule expectation, and a replicate yardstick as an upper reference — while being candid that the final comparison against a token-budget rule at matched retention cannot be evaluated on this release.

Real-world applications.

• Coding-agent scaffolds and other long-horizon harnesses that currently compact on a fixed token threshold or structural cap can use the frontier framing, and the finding that this trigger does not clearly work, when deciding whether to invest in history-conditioned compaction scheduling.
• Teams shipping agent memory or summarization middleware can reuse the paired PRE/POST replay design to measure whether a given summary hurt a specific step rather than only measuring end-task success.
• Benchmarks and dataset releases for agent evaluation can adopt the paper's checklist of what a corpus should ship: ordered actions, token counts, summary text and more replays.
• Risk and reliability teams can use the phase-confound diagnostic as a template for auditing any behavioural label that turns out to correlate with trajectory length before treating it as a meaningful signal.

Industry relevance. Both authors are at Salesforce AI Research and the work was funded by Salesforce, so the question of when to compact is directly tied to the cost and quality of deployed long-horizon agents. The paper notes it is the first/later stratification and process-level evaluation framing — judging compaction at the boundary by the agent's next actions rather than only by task success — that motivates the study. The released observables being coarse (no token counts, no summary text, no ordered actions) limits what any automated trigger can do on this corpus, which is itself an argument for richer instrumentation in production harnesses.

Future Directions

• Ship the missing data: ordered action sequences, per-boundary token counts, compression ratios and summary text, so that token-budget rules can be compared against history-based triggers at matched retention, which the current release does not permit.
• Collect more replays per arm. The release provides three to five replays per arm, and the authors report no re-execution of a reach-k sensitivity, which they list as future work.
• Extend beyond the current scope: the study covers one agent model, one compressor, the same model serving both roles, one deliberately tight 4,096-token window, and harness-triggered boundaries on AppWorld only, leaving agent-invoked compression out of scope.
• Resolve whether the weak signal is an availability problem or an attenuation problem. With three-class sign reliability of only 0.55 (κ 0.28), the authors state that the same numbers also support an attenuation reading in which the target itself is too noisy for history to predict better.
• Evaluate on a deployable fitted policy rather than pooled fold models, since the reported frontiers are pooled over fold models (5 or 100) rather than a single fitted policy that could be shipped.

Target Audience

Researchers and engineers working on agent context management, memory and harness design; evaluation scientists interested in process-level, counterfactual measurement of agent behaviour and in preregistered methodology with dated amendments; and machine-learning practitioners who build risk–coverage or retention frontiers and need a careful example of how to distinguish a real predictive signal from a trajectory-phase confound.

Authors’ abstract

Many long-horizon agents compact their context on a global rule, usually a token budget, blind to what the agent was doing. We ask whether the agent's recent behaviour predicts when a compaction will hurt. TRACE's public corpus of 590 harness-triggered AppWorld compaction boundaries replays each boundary from a re-executed prefix state under the pre-compaction context and under the summary, and records the burden of the next actions: calls that error or repeat a call already made. We find that pre-boundary history predicts post-compaction harm only weakly. An internally prespecified contrast by prefix placement is a wide null, and the naive "has-written" label behind it turns out to measure trajectory phase. The best extension-protocol trigger reaches held-out AUROC 0.66 (0.64 on the replicate's own label) against a same-boundary replicate of 0.72; the best frozen, interpretable trigger avoids 21% of harmful (positive-burden) boundaries while keeping 84% of compaction opportunities, and exceeds the random-rule expectation on count but not on burden mass (a post hoc comparison). Whether the best trigger beats a token-budget rule at matched retention cannot be evaluated on the release. We state what corpora should ship to answer it.