REMORY: Learning Residual Memory for Context Compaction

AI summary

Overview

Research area: Natural Language Processing — specifically long-horizon LLM agents, context compaction, and learned memory representations.

Technical level: Advanced. The paper assumes familiarity with transformer language models, cross-attention, KL-divergence training objectives, and agent harnesses, though its motivations are explained through human-memory analogies that make the framing accessible.

Scope: The paper proposes Remory, a small trainable memory network that generates a bounded sequence of continuous "soft memory tokens" conditioned on a text summary and trained through a frozen LLM, and evaluates it on the SummHay attribution benchmark plus four agent benchmarks with Qwen3.8-27B and GLM-5.3-Flash.

What This Paper Is About

Long-horizon agents must compact their history to keep working inside a finite context window, and the standard practice is to replace earlier interaction with a textual summary. The problem is that preserving the main thread of a task does not guarantee the summary will support every subsequent decision, and which earlier context matters can change as the task unfolds. Remory's goal is to keep a readable summary as a checkpoint while adding a learned, bounded sequence of soft memory tokens that help a frozen LLM approximate the continuation it would have produced with the full history.

Key Contributions

1. A residual memory formulation. The paper defines memory as a sequence of vectors produced by a network conditioned on the summary and appended after it, which the authors describe as an analogue of a residual connection along the sequence dimension. Memory is bounded (up to 4,096 slots) and each block of up to 1,024 source positions produces 64 soft tokens.

2. A two-stage training procedure through a frozen actor. Stage I (reconstruction) teaches the memory network to preserve source information from MMFineReason-Full-2.3M text fields; Stage II (summary-conditioned continuation) trains memory to close the remaining prediction gap using reasoning and coding trajectories, with the memory encoder never seeing the future response. Only the memory parameters are updated, via student-to-teacher reverse KL over the full vocabulary and complete response.

3. Benchmark evidence on source attribution. On SummHay (all 92 queries from ten collections), residual memory raises citation F1 by 4.04 points and joint score by 3.55 over summary alone, with a coverage change of 0.40, using 5,024 mean input positions versus 96,872 for raw context (5.2%).

4. End-to-end agent results and behavior changes. Qwen3.8-27B and GLM-5.3-Flash both improve on Terminal-Bench 2.1 and BrowseComp and show fewer repeated tool outputs and tool errors; Qwen also improves on AutomationBench and JobBench, and a judge prefers residual next actions after compaction 70.7% of the time.

Main Findings

• SummHay attribution gain: Raw context scores 72.46 coverage / 56.60 citation F1 / 43.41 joint; summary alone scores 67.55 / 56.98 / 39.84; summary++ (equal extra text budget) scores 67.52 / 56.94 / 39.78; summary plus Remory scores 67.95 / 61.03 / 43.39. A paired collection bootstrap gives a 95% interval of [+0.74, +6.21] for the joint-score gain. On the 553 insights covered by both summary conditions, citation F1 rises from 57.31 to 61.41.

• Equal text budget does not match learned memory: The text supplement using the same additional-position budget stays close to summary alone (39.78 vs 39.84 joint score), while residual memory approaches the raw-context joint score of 43.41.

• Terminal-Bench 2.1 and BrowseComp gains (one run per configuration): Qwen3.8-27B goes from 71.9 to 76.4 on Terminal-Bench and 74.0 to 77.0 on BrowseComp; GLM-5.3-Flash goes from 84.3 to 87.6 and 84.9 to 89.0. Qwen gains 4.5 and 3.0 points; GLM gains 3.3 and 4.1.

• Fewer repeats and errors: Across the four actor–benchmark pairs, repeated tool outputs fall by 17.9–29.8% and tool errors by 25.3–49.7%. For example, Qwen BrowseComp repeats drop from 25,801 to 18,105 and errors from 25,244 to 12,698.

• Workflow benchmark gains for Qwen (five runs): AutomationBench rises from 35.5 ± 1.3 to 45.3 ± 1.0 (+9.8 points) and JobBench from 33.4 ± 1.25 to 41.0 ± 1.41 (+7.6 points).

• Cost: Generation cost falls in every reported comparison except JobBench, which rises from $3.09 to $3.21 per task. Estimates price actor and summary tokens at September 2026 uncached provider rates, counting residual positions as input, and exclude source-feature encoding, memory-network execution, and tools. Total execution cost is not reported.

• First action after compaction: Among 488 paired next actions from 100 sampled Qwen BrowseComp trajectories, residual actions receive 345 wins (70.7%), 21 ties (4.3%), and 122 losses (25.0%). This measures judged proposed actions, not executed outcomes.

• Model sizes: The Remory networks have 1.94B and 1.502B trainable parameters for Qwen3.8-27B (DFlash) and GLM-5.3-Flash (DFlash-2 with grouped dynamic convolutions) respectively, with the actors left frozen. GLM-5.3-Flash is described as a 320B-parameter model.

• Live Trading case study: At the September 23 cutoff, settled ROI was −11.86% without residual and +5.11% with it, against an initial $20,000 bankroll, with logged generation costs of $303.81 and $170.30. Compactions fell from 56 to 45 and responses from 5,336 to 3,013; per 1,000 responses compactions rose from 10.5 to 14.9, and mean live context fell from 105,372 to 101,185 positions. The two trajectories differ in market selection and timing, limiting return comparisons.

• Cell segmentation case study: In a Terminal-Bench task where a refactor deleted helper functions before compaction, the summary retained the known error and backup identity but omitted the exact edit commands; with summary and memory the actor reread both files, restored the helpers, and passed all nine verifier tests. The paper notes this does not establish which omitted details came from memory.

Methodology in Plain English

The authors keep the existing agent workflow — a text summary produced at a compaction boundary — and add a small network alongside it. That network reads the history and the new summary and emits a fixed, bounded set of continuous vectors that are placed after the summary in the actor's input. Because these vectors are conditioned on the summary and appended after it, the paper calls them residual memory.

The memory network is trained against a frozen actor so that the actor's weights never change. In Stage I, a teacher sees a source passage and an instruction to repeat it, while the student sees only the soft memory plus the same instruction, and both predict the source text. In Stage II, the teacher sees the full interaction history and the student sees the fixed summary plus memory built from that history; both predict the same subsequent assistant response. Training minimizes reverse KL divergence from student to teacher over the full vocabulary and complete response, with weights that mildly emphasize earlier positions, updating only the memory parameters. The number of source positions the memory has to cover is reduced hierarchically, so memory is capped at 4,096 slots while source order is preserved.

Evaluation holds everything else constant: the same actor, query, decoding, and base summary are used across compacted conditions on SummHay, and the agent experiments fix weights, the Codex CLI harness, prompts, tools, decoding, budgets, and compaction policy so that only residual memory varies. Qwen3.8-27B and GLM-5.3-Flash use 262,144-position and one-million-position context windows respectively, and the intervention adds up to 4,096 residual slots at compaction.

Why This Matters

Impact on research. The paper reframes compaction as a prediction-gap problem: rather than trying to enumerate what future decisions will need, it trains a supplement to reduce the mismatch between a summary-conditioned continuation and a full-history continuation. It also provides a controlled within-actor comparison showing that learned continuous tokens outperform allocating the same extra input positions to generated text on attribution, which is a useful data point for the broader prompt-compression and context-management literature.

Real-world applications:

• Long-running coding agents that compact repeatedly and then need to resume edits accurately, as in the cell-segmentation repair case.
• Deep-research and web-browsing agents that must keep track of sources and attribute claims to evidence, which is what SummHay measures.
• Business workflow and professional-task automation, where AutomationBench and JobBench gains of 9.8 and 7.6 points were observed for Qwen.
• Continuous autonomous operation such as the paper's paper-trading setup, where fewer compactions and smaller live contexts accompanied lower generation spend.

Industry relevance. Developers of coding CLIs and agent harnesses (the paper cites /compact in Claude Code and Codex CLI) could add a learned memory channel at the compaction boundary while keeping their existing summary and archive-retrieval interfaces. Cost matters commercially: generation cost per task dropped in nearly every comparison, though the figures exclude the cost of running the memory network itself.

Future Directions

• Closing the coverage gap. Residual memory approaches but does not reach the raw-context joint score (43.39 vs 43.41) with lower coverage, so whether larger memory budgets or better encoders can raise coverage remains open.
• Attributing credit inside the case studies. The paper explicitly notes that the cell-segmentation repair does not establish which omitted details came from memory, and that the first-action preference study judges proposed actions rather than their outcomes — both invite follow-up measurement.
• Extending to more actors and longer horizons. Results cover Qwen3.8-27B and the 320B-parameter GLM-5.3-Flash; whether the residual formulation transfers to other architectures and to compactions beyond those tested is not reported.
• Accounting for true system cost. Since the reported costs exclude source-feature encoding and memory-network execution, a full systems-level cost analysis of the approach relative to summary-only compaction is not yet available.

Target Audience

Researchers and engineers working on long-context language models, agent memory systems, and context management for autonomous agents will benefit most. The paper is also relevant to practitioners building coding or research agents on top of existing compaction workflows, and to readers interested in learned continuous representations as a complement to text summarization and retrieval.

Authors’ abstract

Long-horizon agents compact their history to continue within a finite context window, but a textual summary alone may not support every subsequent decision. We introduce REMORY, a neural memory network that supplements the summary with a bounded sequence of soft memory tokens. Given the history and summary, the network learns to generate tokens that help a frozen LLM approximate the continuation it would produce with the full history. The tokens are conditioned on the summary and appended after it, forming an analogue of a residual connection along the sequence dimension. On SummHay, REMORY improves source attribution at nearly unchanged insight coverage and approaches the full-context joint score using only 5.2% of the input positions. Across long-horizon agent benchmarks, Qwen3.8-27B and GLM-5.3-Flash show consistent gains with residual memory. Both models also exhibit substantially fewer repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1.