LatentPress: Context Compression Beyond Text and Vision

AI summary

Overview

Research area: Context compression and memory representations for large language models (LLM efficiency, long-context QA, conversational memory).

Technical level: Advanced. The paper assumes familiarity with transformer internals (input-embedding layers, KV-cache, soft tokens, forward-KL distillation) and with benchmark conventions in long-context evaluation.

Scope: The paper proposes and evaluates a machine-facing context interface — LatentPress — that writes conversational histories and long documents into continuous soft tokens read directly by a frozen decoder, measuring accuracy, write cost, read cost, and trainable footprint on LongMemEval and LongBench-QA.

What This Paper Is About

Long-running assistants and document-QA systems accumulate far more context than they can afford to re-read, and the compressed form of that context is almost always human-readable text (summaries, OCR reconstructions) even when the only consumer is a language model. The authors ask whether long context can instead be written into a compact sequence of continuous vectors that a frozen LLM consumes directly through its input-embedding interface, with no text reconstruction at inference. LatentPress answers this by separating context handling into a small trained "Write" component and a frozen "Read" component, and testing that interface on both multi-turn dialogue histories and long documents.

Key Contributions

1. A direct-read soft-token interface. LatentPress writes context into continuous memory tokens that are injected into a frozen decoder's inputs_embeds alongside the embedded question, so the decoder decodes the answer directly with no intermediate text-reconstruction step at inference.

2. A reader-matched writer with a very small trainable footprint. The writer reuses two frozen decoder layers (deep-copied, L=2) plus a single linear, identity-initialized adapter, training only 4.196M–26.220M parameters (12.849M for Qwen2.5-7B, 16.781M for Qwen3-8B, 4.196M for Qwen3-1.7B, 26.220M for Qwen2.5-14B) — roughly 0.1% of decoder parameters — while the decoder stays frozen.

3. Variable, structure-aware compression. For conversational memory the authors apply a hand-specified role-based schedule (k_user = 1, so user turns bypass the writer losslessly, and k_assistant in {8, 16, 32}), producing emergent conversation-level ratios of 4.62×, 6.27×, and 7.70×; for unstructured documents they use uniform pooling with a single factor.

4. A two-axis efficiency measurement plus two transfer settings. The paper quantifies write cost (43 ms per conversation) and read cost (0.43–0.49 s per example, 5.0–9.2× faster than raw context), and validates transfer both zero-shot from UltraChat to LongMemEval and from LongMemEval-derived QA to unseen LongBench document domains.

Main Findings

• Compressed soft memory matches uncompressed oracle evidence on LongMemEval. On the frozen Qwen2.5-7B-Instruct reader, LatentPress scores 0.476, 0.478, and 0.504 at 4.62×, 6.27×, and 7.70× compression, compared with 0.490 for uncompressed oracle evidence. The authors note more tokens are not automatically better on this task.

• Text summaries and OCR compression are clearly weaker. Text summarization reaches 0.184 at 12.06× compression. DeepSeek-OCR degrades from 0.426 at 2.33× to 0.390 at 5.97× and 0.312 at 9.34×. ICAE scores 0.452 at 4.12×, 0.318 at 8.96×, and 0.174 at 17.28×. A uniform-rate variant of the same LatentPress writer stays 0.06–0.12 lower over the same range.

• The frontier generalizes across three readers. Repeating the zero-shot comparison on Qwen2.5-7B, Qwen3-8B, and Qwen3-1.7B (a 4.7× range in scale) with one compressor head per reader, LatentPress beats uniform pooling by +0.34 to +0.45 overall accuracy at matched compression. On Qwen3-1.7B it leads the visual baseline 0.434 vs. 0.264; on Qwen3-8B it trails OCR at the lowest compression (0.506 vs. 0.542) but overtakes it as compression grows. Text summarization remains the weakest baseline on every reader (0.184, 0.348, 0.106).

• Answer-bearing facts survive because user turns are kept lossless. On the single-session-user category, uncompressed evidence scores 0.946 and LatentPress scores 0.938 / 0.891 / 0.938 across the three rates.

• Cross-domain transfer to LongBench-QA is only partly successful. Training on LongMemEval-derived QA, Qwen2.5-7B exceeds its own raw baseline at 4× (45.13 vs. 43.80) but falls below at higher rates (40.69 and 32.94); Qwen3-8B's best configuration is 4× (32.79 vs. 30.80). Accuracy declines as compression grows across all three readers, and part of the Qwen3-8B drop at higher rates is a formatting pathology rather than semantic loss.

• In-domain adaptation exceeds raw context at mild compression but not at 16×. With training on LongBench-QA training splits, Qwen2.5-14B rises from raw 47.93 to 57.99 at 4× and 52.18 at 8× before falling to 40.30 at 16×; Qwen2.5-7B goes from 43.80 to 49.06 at 4×, 43.77 at 8×, and 37.78 at 16×; Qwen3-8B goes from 30.80 to 39.62 at 4×, 36.93 at 8×, and 26.12 at 16×.

• Writing is near-real-time. With batches of eight on LongMemEval using a Qwen3-8B backbone in bfloat16 on one NVIDIA H100 80GB GPU, LatentPress takes 43 ms per conversation. Batched DeepSeek-OCR takes 844–1056 ms (~934 ms average, about 22× longer), text summarization 407–645 ms (9–15× longer), and ICAE 350–700 ms (8–15× longer).

• Reading from the compressed prefix is much faster. On 30 warm-loaded LongBench-QA examples at the f8 operating point, LatentPress takes 0.43–0.49 s per example versus 2.44–4.14 s for raw full context and 2.71–4.34 s for cached DeepSeek-OCR at base_size=640 — 5.0–9.2× faster than raw and 5.5–9.4× faster than cached OCR. Job-level wall-clock is 6.0–13.7× shorter than the cold-cache OCR pipeline.

Methodology in Plain English

The authors split context handling into two roles. A writer takes text — dialogue turns or document chunks — and produces a short sequence of continuous vectors. The writer is built from two layers copied from the frozen decoder itself (so it "matches" the reader's representation space) plus one trainable linear adapter initialized to the identity, which means the writer starts out behaving like raw token embeddings and drifts only as training requires. A reader is the frozen LLM; it receives the writer's vectors concatenated with the embedded question and decodes an answer directly. Nothing ever decodes the vectors back into text.

Training has two signals: a reconstruction term that teaches the compressed context to recover target tokens, and a forward-KL term that distills the frozen decoder's full-context next-token distribution into the writer, with λ = 1.0. Training uses 2,000 UltraChat conversations (text only, no QA labels), 400 chunks of up to 2,048 tokens, batch size 1, AdamW at 1×10⁻⁴, and 1,000 steps.

For conversational memory, compression is allocated by turn role: user turns get k = 1 (kept lossless) and assistant turns get k in {8, 16, 32}, so the overall ratio emerges from the role and length mixture rather than being preset. For documents, a single uniform pooling factor of 4, 8, or 16 is used. Evaluation uses the oracle-evidence setting of LongMemEval (500 questions, each paired only with its ground-truth evidence sessions, so retrieval is idealized away) and LongBench-QA English across six subsets (narrativeqa, qasper, multifieldqa_en, hotpotqa, 2wikimqa, musique). Answers are generated greedily, and scoring uses an Llama-3.1-70B-Instruct judge on LongMemEval and the official LongBench F1 implementation. LatentPress results are reported as mean ± std over five seeds.

Why This Matters

The paper argues that a model's stored or compressed context need not be human-readable, and shows a concrete interface where it isn't. If compressed context can be written in 43 ms and read 5–9× faster while preserving or improving accuracy, the economics of long-context deployment change: you stop paying for reconstruction pipelines and long prompts.

Impact on research: It reframes context compression as a Write/Read interface problem rather than a summarization or retrieval problem, and provides a controlled comparison against soft-token (ICAE), pruning-like, and visual (DeepSeek-OCR) alternatives under a shared frozen reader. It also isolates representation from retrieval by using oracle evidence.

Real-world applications:

• Long-running assistants and agents that accumulate instructions, dialogue, plans, tool calls, and environment feedback, and need fast access to a small answer-relevant part of it.
• Document QA over long inputs, where domain-adapted writers can beat raw-context reading at 4–8× compression.
• Latency-sensitive serving, where a compressed prefix cuts per-example inference from 2.44–4.14 s to 0.43–0.49 s.
• Storage-constrained memory systems that can hold a compact continuous state instead of full histories, with read costs 5.5–9.4× below a cached OCR route.

Industry relevance: The trainable footprint (~0.1% of decoder parameters) means the decoder itself is never modified, which fits deployments where a production LLM is fixed and only adapters are trained. The write stage being a single forward pass rather than an autoregressive generation or OCR reconstruction directly reduces preprocessing cost, and the reported end-to-end job time being 6.0–13.7× shorter than a cold-cache OCR pipeline speaks to pipeline-level, not just per-query, savings.

Future Directions

1. Learned, dynamic compression rates. The role-based and uniform rates are explicitly described as simple hand-specified heuristics; the authors propose a learned policy that chooses the per-segment rate, potentially optimized with reinforcement learning against downstream answer reward under a latency or memory budget.

2. Learned token-wise fusion. The general framework permits a learned, importance-weighted fusion H of literal and contextual features; the paper uses a lightweight instantiation and leaves the learned version to future work.

3. Integration with full memory systems. LatentPress isolates compression and reading using oracle evidence, leaving retrieval, memory updates, conflict resolution, and reflection — the concerns of systems like Generative Agents, MemoryBank, MemGPT, and Mem0 — to pipelines built on top of the interface. The longer, distractor-laden LongMemEval-S/M haystacks were not evaluated because they exceed the history lengths the compressor was trained on.

4. Writers for more readers and non-text context. The authors propose training writers for additional readers and for context such as tool, multimodal, or embodied traces.

Target Audience

Researchers working on LLM efficiency and long-context methods; engineers building agent memory, document-QA, or retrieval-plus-reading pipelines; practitioners who need to cut inference latency or preprocessing cost while keeping a production decoder frozen; and readers interested in the broader question of non-text, machine-facing representations of context. Some background in transformer internals and benchmark methodology is assumed, so it is most valuable to readers at the graduate or applied-research level.

Authors’ abstract

Compressed context is usually carried as human-readable text or as rendered images that must be decoded, even when its consumer is a language model. We introduce LatentPress, which writes conversational histories and long documents into a third representation: continuous memory tokens that a frozen decoder reads directly through its input-embedding interface, with no text reconstruction at inference. A small reader-matched writer compresses $4$-$16\times$ while training only an adapter (4.2M-26.2M parameters, $\sim\!0.1\%$ of the decoder). On LongMemEval, LatentPress reaches $0.504$ accuracy at $7.70\times$ compression versus $0.490$ for uncompressed evidence, outperforming text summaries (0.184) and OCR-based compression (0.426 to 0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at $4$-$8\times$ compression, while $16\times$ trails raw. Writing takes 43ms per conversation, roughly an order of magnitude faster than text summarization or OCR reconstruction, and reading is $5$-$9\times$ faster than raw context or cached OCR. We validate the interface under two transfer settings, zero-shot from UltraChat to LongMemEval memory QA and from LongMemEval-derived QA to unseen LongBench document domains, establishing direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation of the experiments could be found at: https://github.com/xuyd16ai/context_softtoken_compress .