The Pulse
Google’s Sleep Framework Targets 10M-Token Memory for LLMs
Google Research proposes alternating active learning with offline consolidation so language models can retain new information beyond a conversation. The team reports near-perfect performance at 10 million tokens on BABILong and gains on kno
AI.info Team · Written with AI · fact-checked ·
Google Research says its “sleep” framework retained long-context knowledge up to 10 million tokens on the BABILong benchmark, using scheduled offline work to turn temporary context into model parameters. The proposal, published on October 27, 2026, gives large language models alternating “wake” periods for processing new input and “sleep” periods for consolidating what they have learned.
The idea addresses a trade-off in deployed models: information held in a prompt can disappear when a session ends, while updating model weights can make a system forget earlier capabilities. Google researchers Ali Behrouz and Vahab Mirrokni describe a cycle intended to preserve new facts and improve performance without overwriting older knowledge. Their post presents results from research that first appeared as a paper in June.
From temporary context to model parameters
During the active phase, the model takes in new information and stores it in fast, short-term components. In the offline phase, it pauses new input and works on memory consolidation and self-improvement. The Google post says the approach uses pre-allocated Mixture-of-Experts memory blocks, whose weights are updated during sleep with practice examples the model generates itself.
The paper calls the transfer process “knowledge seeding.” A smaller version of the model acts as a teacher, passing information to a larger version with newly activated capacity. The student learns from feedback on its own generated text and from an imitation-learning objective; the researchers say the older parameters remain frozen during this update to limit interference.
Two stages, with different jobs
Consolidation moves information from faster-changing memory components into more persistent ones. A second phase, which the researchers call “dreaming,” generates synthetic examples to rehearse new information and refine existing skills. The system scores candidate examples for their potential value, selects some for training and uses reinforcement learning to favor examples associated with improved task performance.
The authors describe the paper’s scope in its abstract:
“Our experiments on long-horizon, continual learning, knowledge incorporation, and few-shot generalization tasks support the importance of the sleep stage.”
Ali Behrouz, Farnoosh Hashemi and Vahab Mirrokni, authors of “Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories”
Results span long contexts and new facts
On BABILong, the paper says its Hope model, augmented with sleep-time consolidation, achieved an almost perfect score as context length scaled to 10 million tokens. The researchers also tested sequential learning on two languages that were unseen during pretraining. Their Hope-3 setup nearly recovered its single-language performance after exposure to both languages in sequence, while ordinary in-context learning lost much of its gains.
For factual knowledge incorporation, the paper reports no-context SQuAD accuracy of 48.9 in a single-passage setting and 46.2 after continued pretraining on 200 passages for its four-level Sleep model. The comparison system SEAL scored 46.7 and 43.2, respectively. In mathematical reasoning, the sleep-enhanced Qwen3-8B scored 79.2 on AIME-24, 69.0 on AIME-25 and 46.1 on HMMT-25; the table reports average scores across 16 attempts.
A research result, not continuous learning in production
The tests do not show that an existing Google product now learns continuously from its users. They evaluate a research method across distinct benchmarks and model setups, and the blog does not specify a deployment schedule, operating cost or safeguards for deciding which user-provided information should become a lasting model update. Those questions matter because offline consolidation changes model weights rather than simply storing a retrievable note.
The paper appeared on arXiv on June 2, 2026, before Google Research’s October 27 post. Its results make a case for separating active use from later model updates, but they leave practical questions about data control, update review and reliability for future work. For now, the clearest reported result is a benchmark claim: near-perfect BABILong performance at a context length of 10 million tokens.