MemT-1.0: Towards Memory-Native Evolution

AI summary

Overview

Research area: Natural Language Processing — specifically memory foundation models, i.e., models that carry information across interactions inside their own parameters and state rather than through an external context window.

Technical level: Advanced. The paper assumes familiarity with Transformer attention, gated linear-attention/state-space updates (Gated DeltaNet-2), knowledge distillation, and mixture-of-experts routing.

Scope: The paper describes the architecture, training pipeline, data construction, infrastructure, and evaluation of MemT-1.0, a family of memory foundation models (MemT-1.0-Nano and MemT-1.0-Flash) that internalize memory as part of model computation.

What This Paper Is About

Foundation models normally "remember" only what fits in their input context, so long interactions require replaying or retrieving old text. The authors argue that memory should instead be an intrinsic capability of the model, continuously accumulated and reused across steps. MemT-1.0 is their attempt to build such a model: it keeps a per-trajectory memory state that is written to and read from during generation, is trained to imitate a full-context version of itself, and is evaluated on memory benchmarks, general capabilities, efficiency, and agentic coding tasks.

Key Contributions

1. A revised memory-native architecture. Building on the team's earlier Metis model, MemT-1.0 keeps the dynamic local memory plus trainable hyper memory structure but replaces token selection with a stable token-level update mechanism based on Gated DeltaNet-2 (GDN-2). It removes the query-key normalization vector used in Metis, extends the memory state to multiple channels, and adds a parallel memory attention branch alongside the unchanged backbone attention in selected Transformer blocks.

2. Mixture-of-Storage (MoS) for scaling memory capacity. A native memory block is expanded into E parallel memory experts with dense routing (every expert participates, unlike sparse MoE). Two separate routers — one for memory utilization, one for memory storage — share a trainable expert address matrix but learn different projections.

3. A two-stage training pipeline plus backbone preparation, with 34B tokens of data. The pipeline is continued pre-training (CPT, with cross-entropy loss) followed by post-training (with KL distillation from a frozen full-context backbone teacher). Backbone preparation post-trains DeepSeek-V4-Flash on a curated mixture before native memory is attached. The corpus comprises 20B CPT tokens and 14B post-training tokens.

4. A released model family, infrastructure, and scaling study. MemT-1.0-Nano (36B parameters, 3B activated, 10M dynamic memory) and MemT-1.0-Flash (288B parameters, 17B activated, 336M dynamic memory) are released, supported by distributed infrastructure that preserves memory isolation across independent trajectories, together with scaling studies over memory capacity and training data and a deployment study on experience-guided coding agents.

Main Findings

• Memory benchmarks: Across 11 memory benchmarks, the models outperform parametric memory baselines on average and remain competitive with retrieval-augmented methods, without accessing the original context during inference. They also largely preserve general capabilities under the noise setting. Per-benchmark scores are not included in the portion of the paper provided here.

• Inference efficiency at long history: At a history length of 128K tokens, MemT-1.0-Nano achieves approximately 1.86x end-to-end inference speedup and a 66x reduction in persistent storage relative to a full-context KV cache. The advantage grows with longer histories.

• Constant persistent state: For a fixed memory configuration, the persistent state size remains constant as interactions accumulate.

• Agentic coding gains: When retrieved procedural experience is internalized into native memory, task success and consistency across repeated runs improve on SWE-bench Verified and SWE-bench Multilingual over agents without experience, reaching performance comparable to supplying explicit textual experience.

• Backbone preparation pays off: The prepared backbone improves on all six monitored benchmarks by 6.3 points on average over the public release. At 284B total and 13B active parameters, it exceeds reported scores of GLM-5.3 Flash (320B total, 18B active) and the larger DeepSeek-V4 Pro (1.6T total, 49B active) on SWE-bench Multilingual and IFBench v2.

• Reliability versus information: Post-training mainly improves reliability, not new knowledge. Tasks the backbone never solved in eight attempts reach only 1.6–4.0% accuracy across four benchmarks after preparation, while on a separate probe set of hard knowledge questions, supplying the relevant facts in context raises accuracy from 27% to 91%. The authors conclude that the remaining gap is largely informational, which native memory is meant to supply.

• Memory capacity is a scaling dimension: Unlike general foundation models, memory foundation models add memory capacity as a critical axis, so the paper runs scaling studies over both memory capacity and training data. The specific scaling results are not reported in the excerpt provided.

Methodology in Plain English

The model is used in a sequence of steps. At each step it sees a new input plus a memory state left over from previous steps, generates a response, and then updates that memory state before the next step. The memory state starts at zero for each independent trajectory, so different users or tasks never contaminate each other.

Inside the network, selected Transformer blocks get an extra parallel branch. The original attention works exactly as before, so the backbone's existing abilities are untouched; the new "memory attention" branch reads from and writes to the memory state, and its output is added to the normal attention output. Writing to memory is done token by token with a gated update rule adapted from Gated DeltaNet-2, with an explicit stabilization step intended to keep the memory state from diverging. Reading is a query against the stored memory, passed through normalization and a gated, soft-capped activation (the cap is set to 2 by default).

To scale capacity, the memory block is split into multiple experts that are all used at once. Two routers decide, separately for reading and writing, how much each expert contributes.

Training happens in two stages. First, long documents are chopped into non-overlapping segments (2K tokens each); the model predicts tokens inside a segment while earlier segments are available only through memory, supervised with ordinary next-token cross-entropy. Second, the model is trained to match a frozen copy of the backbone that still sees the full textual history — an off-policy distillation setup where the teacher's top-64 token probabilities are matched via forward KL divergence, with responses generated by the teacher rather than the student. Only the memory parameters are trained; the backbone stays frozen.

The data for this is organized as context-query-answer triples: the context is pushed through memory, then the model must answer the query from memory alone. The corpus mixes memory-native language modeling, storage/utilization tasks such as summarization, continuation and compression, and longer-range memory tasks, partly drawn from anonymized internal MemOS interaction data. For MemT-1.0-Flash, the backbone itself was first post-trained through a three-step recipe (capability profiling, attribution by intervention, capability-conditioned synthesis) settled over five rounds of full-mixture training and more than fifty runs, producing a final mixture of 126.5K trajectories (2.4B tokens, 1.1B loss-bearing; 56.4% general, 18.0% agentic coding, 25.3% agentic working by count, with agentic trajectories accounting for about 90% of loss-bearing tokens).

Why This Matters

Research impact. The paper reframes memory as an architectural and training problem rather than a retrieval or context-window engineering problem. It offers a concrete recipe (stable token-level memory updates, dense memory experts, full-context distillation) and, importantly, evidence about what post-training can and cannot fix: reliability improves, but missing facts do not appear, which sharpens where memory systems must sit relative to retrieval.

Real-world applications.

• Long-running personal assistants and agents that must remember preferences and prior decisions across sessions without resending entire histories.
• Coding agents that accumulate procedural experience from past tasks and reuse it on new repositories, as tested on SWE-bench Verified and SWE-bench Multilingual.
• Enterprise workflow and tool-use automation where each user or job needs an isolated, persistent state.
• Cost- and storage-sensitive deployments, where the reported 66x reduction in persistent storage and 1.86x inference speedup at 128K history make long interactions practical.

Industry relevance. Constant persistent state size for a fixed memory configuration changes the economics of serving long-lived sessions: storage stops growing with interaction length, and per-trajectory state can be managed like a database record rather than a growing prompt.

Future Directions

• Closing the loop between backbone preparation and native memory. The authors explicitly leave this to future work: experience currently shapes behavior either through weights before deployment or through memory states during interaction, but not both together.
• Handling knowledge gaps. Since synthesis closed knowledge gaps only item by item, the paper leaves these mainly to retrieval and to native memory — an open question is how far native memory can substitute for retrieval on factual knowledge.
• Scaling laws for memory. The paper treats memory capacity as a scaling dimension and reports studies over capacity and data; extending these into predictive scaling guidance for future memory models is a natural next step.
• From recall to procedure. The agentic results extend memory from information recall to procedural guidance in multi-step execution; how far this generalizes to other agentic domains, and what the current limitations are, is flagged for discussion but not resolved in the excerpt provided.

Target Audience

Researchers and engineers working on long-context models, agent memory, retrieval-augmented generation, and efficient inference serving. It is most useful to readers already comfortable with Transformer internals and distillation, and to practitioners deciding whether to build memory into model weights and state versus keeping it in an external store or prompt.

Authors’ abstract

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.