External Memory and Memory-Augmented Networks

Each mechanism has a different lifetime, capacity, update rule, and governance boundary.

The second row has a measured version. Transformer-XL caches the hidden states of earlier segments and lets the next segment read them. The architecture “consists of a segment-level recurrence mechanism and a novel positional encoding scheme”. That was reported at ACL in 2019, by Dai, Yang and four co-authors. The cached states buy range. The model “learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers”. They buy speed too: it is “up to 1,800+ times faster than vanilla Transformers during evaluation”. Caching past states is a memory decision. It changes latency, not only quality.

The fourth row is a trade, and it prices storage against parameters. DeepMind's RETRO reads from a database instead of holding everything in its weights. The abstract states the deal: “With a 2 trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25× fewer parameters.” That was 2022. A second lab checked it. NVIDIA reproduced RETRO in 2023, up to 9.5B parameters over a 330B-token retrieval corpus. The result held, but not everywhere: it “largely outperforms GPT on knowledge-intensive tasks, but is on par with GPT on other tasks”. A RETRO++ variant gained +8.6 exact-match on Natural Questions. Two trillion tokens of external store bought a 25× parameter reduction. The benefit landed on knowledge-intensive work, not on everything.

A controller emits keys, strengths, gates, and erase or add vectors. Soft addressing turns those into weighted reads and writes over a memory matrix.

Coupling a network to an outside store has a date on it. The Neural Turing Machine arrived in 2014, from Graves and two co-authors, and the abstract opens with the entire idea: “We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes.” The controller reaches that memory through attentional read and write processes. The whole system is differentiable end to end.

Content-based addressing finds similar slots. Location-based mechanisms can shift or iterate through memory. The architecture learns an algorithmic access policy rather than compressing everything into one hidden state. Learning that policy turned out to be hard to reproduce. The existing open-source NTM implementations “are unstable during training and/or fail to replicate the reported performance of NTMs”, Collier and Beel found four years later. What helped was not more memory but how the memory started. Contents initialised to small constant values “converge on average 2 times faster than the next best memory contents initialization scheme”. A differentiable store hands you a capability and an optimisation problem in the same package.

Repeated access can support copying, sorting, association, and longer algorithmic behavior.

These systems solve related access problems with different operational semantics.

Accuracy on short examples can hide overwrite and addressing failures.

Step five is the one that has already failed in public. Ask a language model enough of the right questions and it will hand back pieces of its training data. “This paper demonstrates that in such settings, an adversary can perform a training data extraction attack to recover individual training examples by querying the language model,” Carlini, Tramèr and ten co-authors told the USENIX Security Symposium in 2021. The demonstration was manual. They took 1,800 candidate sequences generated by GPT-2 and inspected them by hand. The resulting set was validated against GPT-2's training data through the model's original authors. Of the 1,800 candidates, 604 were unique memorised training examples — an aggregate true-positive rate of 33.5%. Seventy-eight of them were examples of individuals' names, phone numbers, addresses and social-media accounts. Larger models were more vulnerable than smaller ones. Scaling the store scales the exposure.

That failure now has a catalogue entry. NIST's 2025 taxonomy of adversarial machine learning lists Data Extraction as identifier NISTAML.038, inside the Privacy Compromises class NISTAML.03. It extends the exposure explicitly to “sensitive information in RETRIEVAL-AUGMENTED GENERATION (RAG) databases provided in-context”. A permissions audit that checks only the retrieval filter has checked one of the two doors.

A store may preserve exact records. The controller must still formulate useful queries, combine retrieved evidence, and decide what to write. Access operations can become the new source of error.

Two measurements separate the capacity advertised from the capacity actually usable. The first one is about position. Where a needed record sits inside a long context decides whether the model uses it. On multi-document QA and on key–value retrieval, Liu, Lin and five co-authors reported in 2024: “In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.” The record was present in every one of those trials. Position alone decided whether it was used.

The second measurement counts models rather than positions. NVIDIA's RULER benchmark evaluated 17 long-context models on 13 tasks in 2024. All of them claim context sizes of 32K tokens or greater. And yet “only half of them can maintain satisfactory performance at the length of 32K”.

Measure write precision, read recall, stale-memory rate, interference, and end-task benefit. More slots do not fix a controller that addresses them poorly. Half of an advertised 32K is a capacity the access policy never delivered.