Towards In-Parameter Memory Augmentation for Large Language Models

AI summary

Overview

• Research area: Natural Language Processing — memory augmentation for large language models, specifically representing reusable memory inside model parameters rather than as text in the context window.
• Technical level: Intermediate. The paper is a survey and taxonomy; it assumes familiarity with transformer blocks, attention, FFN layers, prompt/prefix tuning, LoRA adapters, and KV caches, but it explains the organizing framework in plain terms.
• Scope: A focused survey and two-axis taxonomy of "in-parameter memory" (IPM) methods that plug a memory-bearing parameter object into an LLM's forward pass at deployment, covering the formulation, the method landscape, property comparisons, and open directions.

What This Paper Is About

LLMs and LLM-based agents increasingly need knowledge acquired after pretraining — new documents, domain constraints, user preferences, and experience from agentic interactions. Today this is usually done with in-context learning (ICL), which feeds retrieved text into the prompt; that approach is inspectable and easy to revise, but it consumes context capacity and its compute and latency grow with the number of retrieved tokens, and long contexts can exhaust a finite window. This survey maps the alternative: in-parameter memory, where a compressed memory object is stored in parameters, adapters, or parameter-like objects and composed into the forward pass at inference time.

Key Contributions

1. A two-axis taxonomy. The authors organize in-parameter memory methods along (1) Parameter Placement — where the memory is composed, split into Embedding layers, Attention layers, FFN layers, or Hybrid when two or more are used — and (2) Parameter Acquisition Time — whether the memory object is formed during deployment (online) or before it (offline).
2. An explicit operator formulation. Any IPM method is characterized by a memory object φ, an acquisition operator A that produces it, and an augmentation (composition) operator C that plugs it into the forward pass, with acquisition and augmentation kept conceptually distinct.
3. Explicit scope boundaries. The paper applies three tests — is the memory parametric, is it produced by an independent acquisition operator A, and is it augmented at deployment as a distinct object — to place neighboring paradigms inside or outside the survey.
4. A curated landscape plus discussion of open problems. The survey walks through representative methods per class, compares them by task purpose, materialization, persistence, and qualitative write/read cost, and flags open directions in interference, safety, co-design with ICL, and recursive self-improvement.

Main Findings

• Two dimensions with four classes each side. Placement has four classes: Embedding, Attention, FFN, and Hybrid. Acquisition time has two: online and offline. Table 2 gives operational criteria and typical memory objects for each class.
• Placement is defined by where C reads φ, not where A lives. The authors stress that a hypernetwork may be trained offline while the LoRA it emits for a deployment context is acquired online, so acquisition time is about the deployed object, not the training of the producer.
• The three scope tests. Pluggable PEFT passes all three tests and is included. Full-parameter SFT fails the "augmented at deployment" test because the update is merged into θ before serving, leaving no distinct object for C. ICL-based RAG fails all three. An unmodified KV cache passes "parametric" and "augmented at deployment" but fails independent acquisition, because its entries are a byproduct of prefilling discrete tokens.
• Token-derived KV caches are excluded by design. The authors classify an unmodified token-derived KV cache as token memory rather than in-parameter memory, since it is a byproduct of prefilling rather than an object produced by an independent acquisition operator A. Encoded KV banks produced by an encoder remain in scope.
• Typical memory objects by class. Embedding-side: soft prompts, soft tokens, memory tables. Attention-side: encoded KV banks, adapters, fast weights, online neural memory, soft prompts. FFN-side: adapters, memory tables, online neural memory, fast weights. Hybrid: adapters, online neural memory.
• Two forms of acquisition. One-shot acquisition maps a source K to an IPM object (gradient optimization over continuous vectors, one-pass hypernetworks, knowledge encoders emitting KV pairs). Stateful acquisition rewrites the current object from new evidence during deployment (recurrent write rules such as the delta-rule and surprise-driven updates). Update frequency varies: per token, per segment, or on demand.
• Online versus offline cost profile. Online rows are almost always per-request or per-token and ephemeral or session-scoped: generated adapters and soft prompts pay a medium-cost one-pass write for a fresh object, while recurrent fast weights and online neural memory pay a low-cost per-token write to keep a mutable state. Offline rows are almost always pre-deployment and persistent: a high-cost one-time write freezes knowledge into soft prompts, KV banks, or memory tables, which are then read at low cost.
• The same object type behaves differently by acquisition time. Adapters are cheap when generated online but expensive when trained offline. Encoded KV banks are composed per request in CEPE and M+, but precomputed in KBLaM and AtlasKV.
• Trade-offs by axis. Online acquisition adapts immediately but pays higher write-time compute; offline acquisition is more stable and validated but cannot absorb a new stream without an offline training process. Embedding-side objects are cheapest to compose but influence the model only indirectly; attention-side objects are content-addressable but coupled to head structure and attention budget; FFN-side objects write where factual associations are stored but are read position-wise with no content-based selection among entries, and sequential writes to shared weights interfere; Hybrid stacks are the most expressive but their behavior cannot be traced to a single component.
• ICL still wins where exactness matters. The authors state that ICL and ICL-based RAG remain preferable for tasks needing exact quotation, provenance, targeted deletion, or human revision — trivial for a text store, hard once text is compressed and abstracted into φ.
• No numerical benchmarks are reported. Write cost, read cost, and retention are qualitative bands (L/M/H) inferred from mechanisms, because the source papers rarely share a benchmark or comparable FLOPs or wall-clock measurements.

Methodology in Plain English

This is a survey, not an empirical study, so the "method" is literature organization and conceptual framing. The authors first define what counts as in-parameter memory using three tests (parametric form, independent acquisition operator, composed at deployment), and they use these tests to decide which neighboring paradigms — ICL-based RAG, pluggable PEFT, full-parameter SFT, unmodified KV caches — are inside or outside the boundary. They then describe the memory mechanism as two separate steps: an acquisition operator A that produces a memory object φ (either in one shot from a source, or statefully by rewriting φ from new evidence during serving), and an augmentation operator C that composes φ into the forward pass at a chosen site. Methods that pass the scope tests are then located on two axes: where C reads φ (Embedding, Attention, FFN, or Hybrid) and when A forms it (online or offline). The audited collection of representative methods is presented as a taxonomy tree (Figure 3) and in comparison tables (Table 3 records placement, memory object, and acquisition operator per method; Table 4 records task purpose, materialization, persistence, and qualitative write/read cost). The comparison discussion is qualitative because the underlying papers do not report comparable measurements.

Why This Matters

• Research impact: The paper names and formalizes a shift already underway from ICL-based memory toward in-parameter memory, and it provides shared vocabulary (memory object φ, acquisition operator A, augmentation operator C, and the two taxonomy axes) so that otherwise very different methods — soft prompts, encoded KV banks, hypernetwork-generated adapters, fast weights, memory tables — can be compared on common ground.
• Efficiency motivation: The paradigm is motivated by avoiding the quadratic cost of long-context prefilling and by reducing inference latency when memory is reused across queries.
• Real-world applications (from the paper's task purposes):
  • Document QA and knowledge injection: offline methods such as PRAG, PolyPRAG, KBLaM, and AtlasKV encode documents or knowledge bases into FFN-side parameters or encoded KV banks that are plugged in at inference.
  • Long-context modeling: methods such as δ-mem, Titans, ATLAS, MesaNet, qTTT, Infini-Transformer, and LaCT compress past segments into fixed-size associative state or FFN-side state.
  • Agent self-evolution and multi-agent communication: MemGen and TMEM target agent self-evolution, while C2C, Draft-KV, and LatentMem address multi-agent communication and trajectory distillation.
  • Lifelong editing and continual learning: Locas, WISE, GRACE, and Sparse Memory Finetuning target lifelong editing or continual learning by writing into FFN-side memory.
• Industry relevance: The methods surveyed map directly onto deployed concerns — serving cost per query, context window limits, the ability to update a model's knowledge after pretraining without full retraining, and adapter-based customization on a frozen base model.

Future Directions

1. Interference and forgetting. Sequential edits and recurrent delta-rule writes overwrite earlier associations, and the damage grows with the update sequence: edited weights increase in norm and disrupt general-ability directions, while a fixed state interferes along the current key. Subspace projections, norm constraints, and erase/write gates slow this collapse but either keep a growing record of past keys or gate only the current key, so conflict detection and fixed-capacity allocation over a long stream remain open.
2. Safety-aware acquisition. A single write from untrusted context can later be retrieved as trusted knowledge, and current memory systems do not record provenance. Because the write is parametric, IPM objects can implant behavior in weights, and behavioral unlearning leaves traces — leaving update budgets, provenance-conditioned write gates, and privacy-aware consolidation as open problems.
3. Co-design with ICL. ICL keeps memory as textual tokens that can represent more detailed information, while compressed IPM objects enable efficient augmentation but may not suffice for fine-grained access. The open direction is a co-design where A decides what stays in context and what enters φ, and C reads both in one forward pass.
4. Model-native recursive self-improvement. Most recursive self-improvement today lives in the agent harness, where prompts, skills, knowledge bases, and tools are revised outside the model. A model-native version would let A write useful deployment experience into φ itself, so the backbone improves in-parameter and the gap between offline training and online inference narrows.
5. Stated limitations to address. The survey only considers the IPM paradigm under text modality (excluding vision, audio, and video), omits IPM designs built into newer architectures such as Mixture of Latent Experts (MLE), Multi-head Latent Attention (MLA), and Manifold-Constrained Hyper-Connections (mHC), and offers no numerical measurements — write cost, read cost, and retention are qualitative bands inferred from mechanisms.

Target Audience

• LLM memory and retrieval researchers who need a structured map of parametric memory methods and vocabulary for positioning new work.
• Agent-system designers deciding when to keep knowledge in context versus writing it into parameters, adapters, or memory tables.
• Efficient-inference and serving engineers concerned with context-prefill cost, latency, and reuse of memory across queries.
• Continual-learning and model-editing researchers interested in where the boundary between continual learning and in-parameter memory falls (the paper's dividing line is whether φ is still augmented at deployment).
• Graduate students and practitioners with intermediate NLP background seeking a single organized entry point into the IPM literature, since the paper assumes but does not teach transformer internals, PEFT, or KV caches.

Authors’ abstract

Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience. In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length. \textbf{In-parameter memory} offers a complementary substrate: reusable memory information is represented in model parameters, adapters, or other parameter-like objects that are composed into the forward pass at inference time. This survey focuses on methods that augment LLMs with such parametric memory at deployment: a memory-bearing parameter object is plugged into the forward pass during inference, whether it is acquired before or during deployment. We organize the landscape with two orthogonal axes: \textbf{Parameter Placement}, which includes Embedding, Attention, FFN layers, or Hybrid when two or more layers are used; and \textbf{Parameter Acquisition Time}, which distinguishes methods whose memory object is acquired during deployment (online) from those acquired before it (offline). We clarify boundaries, conduct comparisons, and discuss open directions in interference, safety, co-design with ICL, and recursive self-improvement.