In-Model Context Compression with Agentic Soft Tokens for Long-Horizon LLM Agents

AI summary

Overview

Research area: Natural Language Processing — context compression and memory management for long-horizon LLM agents.

Technical level: Advanced. The paper combines a systems-level inference mechanism (KV-cache reuse across agent steps) with formal analysis (KL-divergence bounds, attention-pair accounting, attention-mass reweighting).

Scope: A training-free, in-model compression method, Agentic Soft-Token Compression (ASTC), that uses a hybrid Gated Delta Network (GDN)–attention model's own recurrent state to decide which context to compact into soft tokens while preserving exact values as hard tokens.

What This Paper Is About

Long-horizon LLM agents accumulate ever-growing interaction histories — instructions, tool outputs, code, and environment feedback — which raise prefill and attention cost, enlarge the KV cache, and bury decision-relevant evidence among stale text. Existing fixes train external compressors or context-management agents to rewrite history as shorter hard-token sequences, which is lossy and may not match what the acting model has actually absorbed. The paper asks whether the acting model itself, mid-forward-pass, can decide what is already absorbed and carry that information forward in its own representation space, without any compressor training.

Key Contributions

1. Problem framing: The authors identify the core difficulty of long-horizon compression as reducing context cost without losing future-relevant evidence and without diverging from the acting LLM's evolving task representation — arguing that compressibility depends on what the model has absorbed, not on length, repetition, or semantic similarity.

2. The ASTC method: A training-free procedure that reads native GDN recurrent dynamics (a gated delta update) at a compression boundary after early layers have processed the full context, keeps exact values and the action boundary as hard tokens under a protection mask, and mean-pools eligible positions into agentic soft tokens with a compact, reusable KV cache.

3. Theory: Bounds on trained-compressor objective mismatch (gap of at most 2ε + ξ between the surrogate optimum and the true next-token distortion), conditions under which ASTC's soft tokens have lower next-token distortion than any equal-length hard-token representation, an amortized attention-pair reduction theorem, and a lemma on how merging shifts downstream attention mass.

4. Empirical evaluation: On AppWorld, OfficeBench, and 8-Objective QA with a frozen Qwen3.5-9B acting model, ASTC matches or exceeds uncompressed accuracy with up to 50.68% fewer end-to-end Attention Pairs, and achieves the highest accuracy with the lowest Attention Pairs per successful task — a 15.4×–103.7× advantage over trained compressors.

Main Findings

• Accuracy with less work (AppWorld): 33.33% accuracy versus 32.14% for the uncompressed Base, while Decode KV, Attention Pairs, and generated tokens fall to 81.55%, 80.70%, and 94.83% of Base — the only evaluated method that improves accuracy over Base while reducing all three workload measures.
• Near-parity at much lower cost (OfficeBench): 47.37% accuracy versus 48.42% for Base, with Decode KV at 54.52% and Attention Pairs at 49.32% of Base — the source of the headline 50.68% Attention Pair reduction.
• Best answer quality (8-Objective QA): EM 30.56 and F1 38.46 versus Base's 28.28 and 37.34, with generated tokens at 86.55% of Base; Attention Pairs are 98.51%, i.e. roughly unchanged.
• Large margin over trained compressors: Prompted compression reaches 25.00% on AppWorld but uses more Decode KV (143.59%) and more generated tokens (104.25%) than Base; ACON UT and UTCO reach 29.17% and 30.36% with Attention Pairs at 1,064.69% and 1,130.67% of Base; LLMLingua-2 reaches 7.14%; LCLM reaches 6.55% and answers none of the 8-Objective QA tasks correctly (EM and F1 of 0.00) despite the lowest measured workload.
• Cross-model transfer: On Qwen3.8-27B, accuracy moves from 78.57% to 77.98% with Attention Pairs at 64.43% of Base (35.57% fewer) but generated tokens rising 3.82%; on Kimi-Linear-48B-A3B, accuracy rises from 1.47% to 4.41% with Decode KV at 84.01% and Attention Pairs at 88.76% of Base.
• Selector ablation: Swapping the GDN-based selector for hidden-state reconstruction error ("Hidden-only") or random scores ("Random"), while keeping protection and grouping fixed, gives ASTC the strongest joint success-versus-workload outcome on AppWorld — though paired bootstrap uncertainty makes the success advantage suggestive rather than conclusive.
• Persistence of savings: In a fixed-continuation replay that permits at most one commit, conditional action-token distributions stay close to Base while Attention Pair savings recur at later actions, supporting amortization; a separate autonomous one-commit run has somewhat lower success than archived Base and full ASTC.
• Wider compression scopes: Compressing assistant history yields 31.55% accuracy with Decode KV, Attention Pairs, and generated tokens at 11.16%, 3.54%, and 7.76% reductions; compressing full context yields 30.95% accuracy with reductions of 20.50%, 19.68%, and 9.19% respectively. The introduction separately reports matching uncompressed accuracy with 24.52% fewer Decode KV interactions under full-history compression.

Methodology in Plain English

The agent's model is a hybrid that interleaves Gated Delta Network layers (a recurrent, associative-memory-style mechanism with a state matrix) and ordinary attention layers. Early layers read the entire new chunk of context as usual; at a chosen layer boundary, ASTC inspects the GDN state tensors that the model computed anyway.

Two tests decide whether a position inside a tool observation can be compacted. The first, χ, compares the incoming value against what the recurrent state already predicts — a small ratio means the model has effectively already absorbed that content. The second, ψ, checks whether the observation-end state can still recover that value using a fixed, untrained random projection (a readout basis) — guarding against a value that looks predictable locally but is lost later. A position is eligible only if both tests pass and it is not covered by the protection mask, which shields recognized exact values such as paths, identifiers, and checksums, plus the action boundary.

Eligible positions are then divided into at most M causal groups, trimmed of outliers, and averaged in FP32 before being cast back to the model's hidden precision; a group is only merged if its post-cast relative error stays under a threshold. Merged groups become soft tokens placed at the group's latest logical position, and a stable sort restores causal order. A per-observation commit gate requires a minimum downstream token–layer reduction, expressed with a budget B, a remaining-layer count, and a fraction ρ of eligible positions. Because the shortened sequence is cached, later decoding steps and later agent calls reuse the compact cache, so a single decision keeps paying off. Everything is training-free: the model is frozen, and the paper reports that their earlier attempt to train an attention–MLP compressor with on-policy distillation was both expensive and weak.

Why This Matters

For research, the paper argues against the prevailing pattern of training a separate compressor, showing formally that a surrogate objective can drift from the acting model's true next-token distortion and empirically that "model-taste gap" costs both accuracy and compute. It also reframes compression as a state-coverage question that a hybrid model can already answer internally.

Real-world applications:

• Coding and repository-repair agents, where long tool outputs must be compressed but file paths, identifiers, and checksums must stay byte-exact.
• Enterprise workflow automation such as the OfficeBench setting, where agents execute multi-step office tasks across many tool calls.
• Tool-calling assistants and API orchestration, where arguments and endpoints must survive compression verbatim.
• Long-running personal or customer-support agents that need a persistent, cheap memory of earlier decisions and observations.

Industry relevance is direct: the savings are measured as end-to-end Attention Pairs, Decode KV interactions, and generated tokens relative to an uncompressed baseline, which are the quantities that drive serving cost and latency for agentic products. A training-free method that slots into an existing frozen model avoids a separate compressor training pipeline and its maintenance burden.

Future Directions

• Reducing reliance on GDN: the selector depends on hybrid GDN–attention dynamics; extending model-native selection to architectures without a recurrent state is left open.
• Strengthening exact-value protection: the paper notes the protection mask "may miss some exact fields," so more reliable detection of identifiers and numerics is a clear next step.
• Tightening the empirical story: the one-commit analysis is descriptive, with differing archive versions, and the selector-control advantage is described as suggestive rather than conclusive; larger, better-controlled closed-loop studies would settle both.
• Controlling decode-side and generation effects: on Qwen3.8-27B, generated tokens increase by 3.82% even as Attention Pairs drop 35.57%, and on 8-Objective QA Attention Pairs are essentially unchanged at 98.51% — understanding where savings do and do not materialize remains open.

Target Audience

Researchers and engineers working on LLM agents, long-context inference, and KV-cache or memory-efficiency systems; readers interested in soft-token and latent compression methods; and practitioners deploying multi-step tool-using agents who need to cut serving cost without retraining a compressor. The formal sections assume comfort with KL divergence, Lipschitz-style perturbation bounds, and attention mechanics, while the method and results sections are accessible to agent-systems engineers.

Authors’ abstract

Long-horizon LLM agents play an important role in solving complex and sustained tasks, with ever-growing interaction contexts. However, these accumulated contexts will raise substantial computational cost and obscure key decision-relevant evidence. Existing methods typically train external compressors or context-management agents to rewrite context as shorter hard-token sequences, but such external selection may not align with the LLM's evolving representation and is necessarily lossy. In this paper, we introduce Agentic Soft-Token Compression (ASTC), a training-free compression method using soft tokens within LLM agent for long-horizon tasks. Specifically, ASTC uses the model's native recurrent dynamics to determine the already absorbed information, leveraging the commonly-used hybrid Gated Delta Network (GDN)-attention. It then carries this information forward as agentic soft tokens while retaining exact values as hard tokens. The resulting compact sequence reduces later-layer computation and yields reusable KV caches, extending the savings across the agent's trajectory. Extensive experiments on long-horizon tasks show that ASTC matches or exceeds the uncompressed agent's accuracy with up to 50.68% fewer end-to-end Attention Pairs, while trained compressors require 15.4 to 103.7 times more Attention Pairs per successful task. These findings highlight agentic soft tokens as a model-native way to carry absorbed context forward as the task representation evolves.