Technical Deep Dives
The Memory Problem: Why AI Systems Forget and How Persistent Memory Changes Everything
Transformers keep no state between calls, so everything people call AI memory is scaffolding: KV caches, million-token windows now billed at standard rates, retrieval, tiered paging and server-side compaction.
Gabriele Masetti ·
The architecture has no memory by design
A transformer, at inference time, is a pure function. Give it a sequence of tokens and it returns a probability distribution over the next token. Call it again with the same input and you get the same output. Nothing persists between calls unless you put it there yourself, in the input. This is not a bug that will be patched in the next release — it is the architecture working as designed.
The original 2017 transformer ("Attention Is All You Need," Vaswani et al.) was built to map sequences to sequences using self-attention, not to accumulate state across independent invocations. Every large language model since — GPT, Claude, Gemini, Llama — inherits that property. Ask ChatGPT a question on Monday and again on Tuesday, and absent an explicit memory feature layered on top, the model has no idea Monday happened.
This statelessness has a practical name in engineering circles: the context window is the entire world the model can see. Whatever isn't in that window doesn't exist for the model, no matter how important it was five minutes or five months ago. Everything people colloquially call "AI memory" — a chatbot recalling your name, an agent remembering a decision from three steps ago, a coding assistant tracking a refactor across a session — is really an engineering workaround bolted onto a system that forgets everything the instant the forward pass ends.
What actually happens during a conversation
Within a single session, models don't re-derive everything from scratch at every token — that would be prohibitively slow. Autoregressive decoding uses a KV cache: as each token is processed, the model stores the key and value vectors computed by its attention layers, and reuses them for every subsequent token instead of recomputing them. This turns generation from a quadratic-time problem into something closer to linear time per new token, and it's why a long response doesn't get proportionally slower to start each additional word.
But the KV cache is a session artifact, not memory in any durable sense. It grows with every token in the context, it dominates GPU memory usage for long-context serving, and it evaporates the moment the session ends. It is best understood as volatile scratch space that makes a single stateless computation efficient — not as a mechanism for retaining anything across sessions.
This distinction matters because it's easy to conflate "the model kept track of what I said ten minutes ago" with "the model remembers me." The former is the KV cache and the growing prompt doing their job within one bounded context. The latter requires an entirely different piece of infrastructure that has to write information somewhere outside the model and retrieve it later.
Bigger windows don't solve the problem
The obvious fix — just make the context window bigger — was pursued aggressively, and on its own terms it worked. Google's Gemini 1.5 Pro, introduced in February 2024, was the model that made a 1-million-token window ordinary; it has since dropped out of Google's model documentation altogether, superseded by the Gemini 2.5 and Gemini 3 lines. The million-token window outlived the model that announced it.
It is now a default rather than a headline. Anthropic's documentation lists a 1M-token context window on Claude Opus 4.6 and later, Claude Sonnet 4.6 and later, and its Fable and Mythos models, and states plainly that no beta header is needed: "For every model with a 1M-token context window, 1M is the default."
The price objection has gone with it. Anthropic's pricing page says Claude 4.6 and later models include the full 1M-token window at standard pricing — a 900k-token request is billed at the same per-token rate as a 9k-token one — with prompt-caching and batch discounts applying across the whole window. Attention still runs over every token you send, so latency grows with the prompt, but the explicit long-context surcharge is no longer the thing standing in the way.
If cheap context were the answer, the memory problem would be closed. It is not, for a reason no price sheet can fix: having a token in the context window does not mean the model uses it well.
Lost in the middle
This second problem was documented precisely in "Lost in the Middle: How Language Models Use Long Contexts" by Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang (published in Transactions of the Association for Computational Linguistics, 2024, after circulating as a 2023 preprint). The authors tested language models on tasks requiring them to find relevant information among a set of documents or key-value pairs, systematically varying where the relevant item sat in the input.
The result was a distinctive U-shaped performance curve: models are best at retrieving information from the very beginning or the very end of a long context, and performance degrades significantly — sometimes falling below the performance of a model given no relevant document at all — when the needed information sits in the middle.
This finding has been reinforced by the "needle in a haystack" evaluation methodology, an approach popularized by Greg Kamradt in late 2023, in which a specific fact (the "needle") is hidden at varying depths within a large block of filler text (the "haystack") and the model is asked to retrieve it. Plotting pass rate against both context length and insertion depth produces a heatmap that exposes exactly where a model's recall breaks down.
Providers now say this themselves. Anthropic's own context-window documentation warns that "as token count grows, accuracy and recall degrade, a phenomenon known as context rot," and concludes that curating what sits in context matters as much as how much room there is. The vendor with a million-token window to sell is telling customers not to fill it indiscriminately.
The upshot for anyone building on these systems: a model advertising a million-token window is not thereby guaranteed to use all million tokens equally well. Benchmarks that specifically probe effective long-context recall, rather than raw token acceptance, routinely find real-world usable context is a fraction of the advertised figure. A bigger window changes what a model can technically ingest; it does not by itself change how reliably the model reasons over everything inside that window.
Retrieval as an external memory system
If stuffing everything into context is expensive and unreliable, the alternative is to not put everything in context — and instead fetch only what's relevant, on demand. This is the core idea behind retrieval-augmented generation, introduced in "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis and coauthors at Facebook AI Research in 2020. RAG pairs a retriever, which searches an external corpus for passages relevant to a query, with a generator that conditions its output on both the query and the retrieved passages. Instead of a fixed set of facts baked into model weights during training, the model consults a document store at inference time.
The mechanism that makes retrieval practical at scale is the vector database. Text (or images, or audio) is converted into dense embedding vectors that place semantically similar content near each other in a high-dimensional space. Tools like FAISS (a library for efficient similarity search) and managed services like Pinecone let a system take a query, embed it, and pull back the nearest neighbors in that vector space — passages that are conceptually related to the query even if they don't share exact keywords.
In an agent or chatbot context, this becomes a long-term memory store: past conversations, user preferences, or documents get embedded and indexed once, and only the handful of most relevant items are retrieved and injected into the prompt for any given turn. The context window stays small and cheap; the effective memory is arbitrarily large because it lives outside the model, on disk, not inside a single forward pass.
RAG's tradeoff is that it is only as good as retrieval. If the retriever misses the relevant passage, or if relevance is ambiguous, the generator never sees the information it needed — a distinct failure mode from "lost in the middle," but one that also results in the model acting as if it doesn't know something it technically has access to.
MemGPT and the operating-system analogy
A more structured approach to agent memory came from MemGPT (introduced in the 2023 paper "MemGPT: Towards LLMs as Operating Systems" by Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, and Joseph E. Gonzalez), whose ideas now live on in the open-source Letta framework. MemGPT's central move is to treat the limited context window the way an operating system treats limited physical RAM: it uses virtual memory paging, moving information between a small, fast "main memory" (the actual context window) and a much larger, slower external store, deciding what to page in and out via function calls the model itself can issue.
Concretely, this produces a tiered memory architecture: a core memory block that sits directly in context (holding, for instance, a persona and key facts about the user, editable by the agent itself), a recall memory of full past conversation history that can be searched, and an archival memory backed by a vector store for long-term facts that don't fit in active context at all.
The agent decides, via tool calls, when to write a fact out to long-term storage and when to page something back in because it's suddenly relevant. This is a meaningfully different design from plain RAG: rather than a fixed retrieval step attached to every query, the model actively manages its own memory hierarchy, choosing what to keep close and what to archive.
Anthropic has since shipped something structurally similar as a first-party feature: a memory tool, generally available on the Claude Developer Platform without a beta flag, that lets Claude write information to files it can read back in later sessions, plus a context-editing capability that can clear stale tool results or reasoning blocks from a long-running agent's context once they're no longer needed.
A third tier has since been added below both: compaction, which runs on the server and automatically summarises the earlier part of a conversation as it nears the context limit, letting the conversation continue past it. It is in beta for Claude 4.6 and later models, and the documentation recommends pairing it with memory rather than choosing between them — "compaction keeps the active context small without client-side bookkeeping, and memory preserves the information that must survive summarization." That is the tiering argument again, with the page-out step moved server-side.
Anthropic has reported internal benchmarks combining the two showing substantial token savings and accuracy improvements on long, multi-turn agentic tasks. OpenAI has pursued a related but distinct feature on the product side: ChatGPT's memory, which launched in 2024 and was expanded through 2025 to reference broader chat history, letting the assistant carry facts and preferences across otherwise-separate conversations rather than requiring the user to restate them.
Short-term versus long-term memory in agents
Across these systems, it's useful to separate two categories that get conflated under the single word "memory." Short-term or working memory is whatever sits in the active context window for the current task — the KV cache, the running conversation, the intermediate reasoning an agent produces mid-task. It's fast, complete for whatever it holds, and gone when the session ends unless explicitly saved.
Long-term memory is everything stored outside the model between sessions: vector-indexed archives, structured memory files, user-preference records. It's durable but requires a deliberate write step (deciding what's worth keeping) and a deliberate retrieval step (deciding what's relevant right now), and both of those steps are themselves points of failure — the wrong things get saved, or the right things get saved but not found later.
This is why practical agent memory systems increasingly look less like "give the model a bigger brain" and more like systems engineering: what gets written where, what gets paged in when, what gets summarized versus kept verbatim, what gets evicted when space runs out. The tiered designs in MemGPT/Letta and in Anthropic's memory tool are explicit acknowledgments that no single storage tier — pure context, pure retrieval, pure fine-tuning — solves the problem alone.
Why this changes what's buildable
The practical consequence of solving memory, even partially, is a shift in what kind of software an LLM can plausibly sit inside. A stateless model with a bounded context window is fine for one-shot question answering but is a poor foundation for an assistant meant to work with someone over weeks — it either has to be reminded of everything at the start of every conversation, or it forgets things a human would find insulting to forget.
Persistent memory, whether via retrieval, tiered paging, or provider-level memory features, is what allows an agent to accumulate context about a specific user, project, or codebase over time rather than starting cold on every invocation.
It also reframes what "context window" means as a product decision rather than a purely architectural one. A larger window is one lever; better retrieval is another; explicit memory management by the model itself is a third. None of them is a complete substitute for the others — long context alone runs into "lost in the middle" degradation however cheaply it is billed, retrieval alone is bottlenecked by retriever quality, and self-managed paging depends on the model making good judgment calls about what to keep.
The systems getting real traction — Letta's tiered agent memory, Anthropic's memory tool paired with context editing and server-side compaction, OpenAI's cross-conversation memory in ChatGPT — are the ones combining more than one of these mechanisms rather than betting on a single fix. The stateless transformer at the core hasn't changed; what's changed is how much scaffolding now sits around it to make the system behave as if it remembers.