Tools Need Not Speak in Text: Latent Context Compression for Agents

AI summary

Overview

Research area: Artificial Intelligence — LLM agents, tool-use, and context compression (arXiv:2611.06500v1 [cs.AI], 2026).

Technical level: Advanced. The paper assumes familiarity with the ReAct agent loop, token embeddings, KL-divergence training objectives, and LoRA fine-tuning.

Scope: This paper proposes Coca-Cola, a continuous latent compression model that replaces raw text tool observations with a small number of learned latent tokens inside an LLM agent's input, trained by aligning next-action distributions rather than reconstructing text.

What This Paper Is About

LLM agents that use tools such as web search keep appending full raw tool outputs to their context, and these outputs quickly dominate the prompt — the authors report tool observations reaching 96.7% of total tokens by Step 4 — which raises cost and distracts the model from the few details that actually matter for the next decision. Prior fixes (token pruning, LLM summarization) still produce discrete text, so their compression ratio is bounded. Coca-Cola instead compresses each tool observation into a handful of continuous latent vectors injected directly into the agent's embedding space, keeping only the information needed to choose the next action.

Key Contributions

1. Reframing the tool–agent interface. The authors argue, to their knowledge for the first time, that tools need not convey information to agents through explicit text tokens, and they compress tool outputs into latent tokens instead.
2. The Coca-Cola model. A compressor consisting of a context-aware encoder (a small language model conditioned on the task instruction, the current-step reasoning trajectory, and the tool observation) plus a latent adapter that maps the result into the target agent's input embedding dimension, delimited by <latent_obs> boundary markers.
3. Action-Preserving Training. Instead of reconstructing the original text, training aligns the agent's next-action distribution under full and compressed observations with a KL divergence term, plus a cross-entropy term against the recorded next action; only the encoder's LoRA parameters and the adapter are updated while the agent stays frozen.
4. A capacity argument for latent tokens. A finite-precision codebook analysis gives text capacity as n·log₂|V| versus latent capacity m·d_eff·log₂B, so latent tokens can carry more information per input position when d_eff·log₂B > log₂|V|.

Main Findings

• Large token savings: Coca-Cola reduces total token consumption by 70.01%–89.24% relative to the uncompressed (Vanilla) agent across four QA benchmarks and two model families. On NQ with the Qwen models, average input tokens fall from 14,362.12 to 1,766.53 (87.70%); on 2Wiki with the GLM models, from 31,693.47 to 3,003.38 (90.52%).
• Faster end-to-end runtime: Runtime drops by 24.45%–32.80% with the Qwen models (NQ: 96.19 s to 64.64 s, 32.80%) and by 41.61%–55.97% with the GLM models (2Wiki: 106.37 s to 46.83 s, 55.97%). Coca-Cola has the lowest average execution time in every dataset and model configuration tested.
• Compression does not hurt accuracy — it improves it: Coca-Cola beats the Vanilla agent on both EM and F1 in every evaluated setting. On TriviaQA with Qwen, F1 rises from 62.87% to 67.07% at an 83.68% compression rate; on 2Wiki with GLM, F1 rises from 31.29% to 36.20% at an 89.24% compression rate.
• Best against compression baselines: Coca-Cola achieves the highest score in 13 of the 16 dataset–metric pairs and the best EM and F1 simultaneously in six of the eight model–dataset combinations, including all four GLM settings. On TriviaQA with GLM, F1 is 59.41% versus 49.29% for the strongest baseline, LLMSummary; on HotpotQA with Qwen, F1 is 33.75% versus 31.21% for ICAE.
• Two acknowledged exceptions: With the Qwen models, ICAE scores higher EM and F1 on TriviaQA, and LongLLMLingua a higher F1 on 2Wiki — but both compress far less (66.92% vs. 83.68% and 41.96% vs. 70.01%).
• Action-preserving beats reconstruction: Against ICAE, which uses the identical architecture but a text-reconstruction objective, Coca-Cola achieves a higher compression rate in all eight settings (e.g., 89.24% vs. 74.12% on 2Wiki with GLM).
• More latent tokens help, with a trade-off: Sweeping m ∈ {8, 16, 32, 64, 128} on NQ with GLM, F1 climbs steadily from 26.34% to 31.45%, while compression stays between 78.83% and 82.56% for m ≤ 64 and falls to 73.73% at m = 128, which is the adopted default.
• Context conditioning matters: Removing the reasoning state, the task goal, or both lowers EM by 2.05, 3.48, and 5.16 points respectively (Full: 23.50% EM at 73.73% compression). Even the Observation-only variant still beats all compression baselines on NQ with GLM (18.34% EM vs. at most 17.00%; 79.94% compression vs. at most 74.62%).
• Case study: On HotpotQA #377, the Vanilla Model confuses two pieces of evidence in the search results and answers incorrectly even after an extra search, whereas Coca-Cola isolates the question-relevant evidence and answers correctly with fewer searches and lower context overhead.

Methodology in Plain English

The starting point is the standard ReAct loop: the agent thinks, calls a tool, receives an observation, and repeats. Normally that observation is pasted into the prompt as text. Coca-Cola inserts a small trained model at that point.

That compressor reads three things: the original task question, the agent's reasoning text from the current step, and the raw tool output. A lightweight adapter then turns the encoder's internal states into a fixed number of vectors (128 by default) sized to match the target agent's embedding dimension, and these vectors are spliced into the agent's input right where the text observation used to sit, bookended by special markers.

Training copies the agent's decision twice for each stored trajectory. One pass sees the full observation and produces a "teacher" distribution over the next action; the other sees only the latent tokens and produces a "student" distribution. The compressor is trained to make the student match the teacher (KL divergence) and also to predict the action the agent actually took (cross-entropy). Because latent vectors are continuous and differentiable, gradients flow back through the frozen agent into the compressor. Only LoRA parameters on the encoder and the adapter are updated — the agent itself never changes — and no step ever requires reproducing the original text.

Evaluation uses two model pairs (Qwen3.5-35B-A3B with Qwen3.5-4B, and GLM-4.7-Flash with GLM-Z1-9B) on NQ, TriviaQA, 2WikiMultiHopQA, and HotpotQA, comparing against an uncompressed agent, LongLLMLingua, LLMSummary, and a re-purposed ICAE. Training data came from 4,188 distinct tasks yielding 4,363 tool observation–action pairs, split 10% for validation (3,927 training / 436 validation pairs); LoRA rank 16, scaling 32, dropout 0.05; paged 8-bit AdamW with learning rate 1.5×10⁻⁴ decaying to 5×10⁻⁵; λ_KL = 0.3 and λ_CE = 1.0, with the KL weight warmed up over the first 20% of steps. Inference retrieves up to 5 results per tool call and allows at most 5 tool calls per task, on 4 NVIDIA A100-SXM4 GPUs (80 GB each).

Why This Matters

The work moves agent context management from "shorten the text" to "don't use text for this at all," and shows that a compressor optimized for the decision the agent will make can outperform one optimized to preserve the original wording — at compression ratios that text-based methods do not reach. The improvement in accuracy alongside a 70%–89% token cut is the distinctive result: compression here is not purely a cost-saving trade-off.

Real-world applications:

• Long-horizon research and browsing agents that make many search or retrieval calls per task, where raw page snippets would otherwise dominate the window.
• Enterprise assistants wired to databases, ticketing systems, or document stores that return verbose payloads the model only needs to act on, not read aloud.
• Cost and latency reduction for production agents billed per token and evaluated on response time.
• Multi-agent or tool-heavy pipelines (the paper cites latent-based agent communication as related work) where passing compact vectors between components could replace text serialization.

Industry relevance: The method is tool-agnostic — the paper notes there is no search-specific design, so the compressor conditions only on instruction, trajectory, and observation. Because the agent backbone is frozen and only LoRA parameters plus a small adapter are trained, it is an add-on to an existing deployed model rather than a retraining effort. Code is released at github.com/THU-Agent/Coca-cola.

Future Directions

• Extending beyond web search: the paper argues the design transfers to other tools without architectural change, but only web-search QA is evaluated; non-textual tools (SQL results, structured APIs, code execution output) remain untested.
• Interpreting the latent tokens: the paper reports the theoretical capacity of latent representations but does not report what individual latent tokens encode, or how the 128-token budget should be set for other task types beyond the NQ sweep.
• Closing the remaining baseline gaps: the two Qwen exceptions (ICAE on TriviaQA, LongLLMLingua on 2Wiki) suggest the action-alignment objective could be strengthened or combined with a text-level signal.
• Transfer and composition: whether one trained compressor generalizes across agent backbones, and whether latent tokens can serve simultaneously for memory and inter-agent communication, are open questions the related-work discussion raises but does not answer.

Target Audience

Researchers and engineers working on LLM agents, tool-use pipelines, and context-efficiency methods, particularly those already familiar with ReAct-style loops, embedding-space interventions, and distillation objectives. It is also relevant to practitioners running tool-augmented agents at scale who care about token cost and latency, and to students interested in latent/continuous representations as an alternative to text-level prompting. The paper is not beginner-friendly: the theoretical capacity analysis and the KL-plus-cross-entropy training setup assume comfort with representation learning and distribution matching.

Authors’ abstract

Large language model (LLM) agents typically append raw tool outputs to their context. Over multi-step interactions, long and noisy observations incur substantial context overhead and dilute the agent's attention to task-relevant information. Existing token pruning and summarization methods remain constrained by discrete text representations, limiting the achievable compression ratios. In this work, we challenge the prevailing assumption that agents must consume tool observations as explicit text tokens. To this end, we propose Coca-Cola, a continuous latent context compression model for LLM agents. Coca-Cola compresses tool observations into latent tokens and embeds them into the input embedding space of the agent. We further introduce Action-Preserving Training, which aligns the agent's next-action distributions under full and compressed observations. This trains the compression model to learn action-sufficient latent tokens that retain decision-critical information. Experiments with web-search agents on four QA benchmarks and two model families show that Coca-Cola reduces the agent's token consumption by 70.01%-89.24% and end-to-end runtime by 24.45%-55.97%, while consistently improving task performance. These results suggest that Coca-Cola offers a promising approach for context compression in agents. Our code is available at https://github.com/THU-Agent/Coca-cola.