EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning

AI summary

Overview

Research area: Memory systems for large language model (LLM) agents, specifically long-horizon conversational reasoning and structured memory organization.

Technical level: Advanced. The paper assumes familiarity with retrieval-augmented generation, embedding-based retrieval (dense, BM25, Reciprocal Rank Fusion), LLM-as-a-judge evaluation, and agentic retrieval loops.

Scope: The paper introduces EverMemOS, a three-phase memory operating system that converts dialogue streams into structured memory units (MemCells), consolidates them into thematic scenes (MemScenes) with an evolving user profile, and reconstructs query-specific context, evaluated on LoCoMo, LongMemEval, and PersonaMem-v2.

What This Paper Is About

Existing memory systems for LLM agents largely store isolated records and retrieve fragments, so agents can find relevant facts yet still fail to detect conflicts, maintain a stable user model, or reason consistently over extended interactions. EverMemOS addresses this by modeling memory as a lifecycle inspired by the biological engram: episodic trace formation, semantic consolidation, and reconstructive recollection. The goal is to turn fragmented episodic experience into coherent, stable knowledge structures that support long-horizon reasoning rather than flat record matching.

Key Contributions

1. System design: A unified, product-ready Memory Operating System for LLMs that reconceptualizes memory as a lifecycle, shifting from passive storage of records to structured organization of experience.
2. Three-phase method: A pipeline that transforms fragmented episodic experiences into coherent and stable knowledge structures: Episodic Trace Formation (MemCells), Semantic Consolidation (MemScenes plus user profile), and Reconstructive Recollection (MemScene-guided agentic retrieval under the principle of necessity and sufficiency).
3. Memory primitives: The MemCell is defined as a tuple (E, F, P, M) — Episode (a third-person narrative anchor), Atomic Facts (discrete verifiable statements), Foresight (forward-looking inferences with validity intervals [t_start, t_end]), and Metadata (timestamps and source pointers).
4. Empirical validation: State-of-the-art results on LoCoMo, LongMemEval, and PersonaMem-v2, with a reliability study of the LLM-as-a-judge protocol against human annotations (Cohen's κ > 0.89).

Main Findings

• LoCoMo, GPT-4.1-mini backbone: EverMemOS reaches 93.05% overall accuracy versus the strongest baseline Zep at 85.22%, a relative gain of 9.2%. Per category: single-hop 96.67% (+6.4%), multi-hop 91.84% (+12.1%), temporal 89.72% (+16.1%), open domain 76.04% (+1.4%). Average tokens: 2.3k for EverMemOS versus 1.4k for Zep.
• LoCoMo, GPT-4o-mini backbone: EverMemOS reaches 86.76% overall versus Zep at 81.06%, a relative gain of 7.0%. Per category: single-hop 91.08% (+3.4%), multi-hop 86.17% (+19.7%), temporal 81.93% (+10.0%), open domain 66.67% (+0.0%). Average tokens: 2.5k.
• LongMemEval: EverMemOS reaches 83.00% overall versus the strongest baseline MemOS at 77.80%, a relative gain of 6.7%. Per capability: SS-User 97.14% (+1.5%), SS-Asst 85.71% (+14.3%), SS-Pref 93.33% (−3.5%), Multi-S 73.68% (+4.3%), Knowledge Update 89.74% (+20.6%), Temporal Reasoning 77.44% (+0.0%). Token usage 2.8k versus 1.4k for MemOS.
• Complex reasoning benefits most: The largest gains appear where dispersed evidence must be integrated — LoCoMo multi-hop (+19.7%) and temporal (+10.0%), and LongMemEval knowledge update (+20.6%) — which the authors attribute to MemScene clustering presenting the solver with complete narrative context and resolving state conflicts.
• Ablation results: Removing MemScenes (flat retrieval over MemCells), removing MemCells (retrieval over raw dialogue), and removing external memory each degrade performance stepwise on LoCoMo and LongMemEval, corresponding to lost scene-level organization, lost stable semantic units, and reliance on the context window alone.
• Segmentation quality: Semantic segmentation outperforms fixed heuristics and even ground-truth session boundaries. On GPT-4.1-mini: Fixed-Message-10 88.05, Fixed-Token-512 87.55, Fixed-Token-1024 84.52, Session (Oracle) 87.66, EverMemOS default with GPT-4.1-mini 89.16 and with Qwen3-4B 89.78. On Qwen3-4B: 80.95, 80.67, 75.19, 80.63, 83.07, and 82.73 respectively. Accuracy changes across boundary-detection backbones are ≤ 0.7 points.
• Hyperparameters: Performance saturates around N = 10 MemScenes; defaults are N = 10 and K = 10 episodes. Clustering threshold τ is 0.70 on LoCoMo and 0.50 on LongMemEval, with a maximum time gap of 7 days and 30 days respectively.
• Profile study on PersonaMem v2: Adding the User Profile to episodic evidence raises overall accuracy to 53.25% versus 43.93% for episodes-only (a 9.32-point gain) and 48.30% for profile-only. In the full comparison, EverMemOS (53.25%) beats MemOS (50.72%) by 2.53 points, with MemoryOS at 40.05%, Mem0 at 43.85%, Zep at 43.40%, and MemU at 38.70%. The paper's Table 4 caption reports 5,000 questions across 9 scenarios, while the accompanying text for Table 9 states 2,447 questions across 9 scenarios.
• Accuracy can exceed recall: On LoCoMo at K = 1, recall is 65.06% but accuracy is 71.80%; 52 of 429 zero-recall questions (12.1%) are answered correctly. At K = 3, recall is 86.32%, accuracy 87.81%, and 25 of 125 zero-recall questions (20.0%) are answered correctly. Increasing K from 1 to 3 reduces zero-recall cases by 71% (429 to 125), which the authors attribute to information redundancy and non-unique evidence annotations.
• Judge reliability: Against human annotations, Cohen's κ is 0.891 on LoCoMo (95% CI [0.742, 1.000], accuracy 0.984, Pearson r 0.891) and 0.978 on LongMemEval (95% CI [0.936, 1.000], accuracy 0.992, Pearson r 0.979).
• Token cost breakdown: On LoCoMo (1,540 questions) with GPT-4.1-mini, Phase I ("add") consumes 9.42M tokens over 7,056 calls, and Phase III search+answer consumes 10.27M (~6.7k/question) over 3,557 calls; separate evaluation consumes 2.38M over 4,620 calls. With GPT-4o-mini, add is 9.34M and search+answer 9.31M (~6.0k/question).
• Rewriting frequency: The sufficiency checker triggers a second-round query rewriting for 31.0% of LoCoMo questions under GPT-4.1-mini.
• Qualitative case study: Reported cases include reconstructing a Grade-II ankle sprain during badminton, tracking waist circumference from 104 to 96 cm with stable weight, and using observed past travel failures (overcrowding, missing advance tickets) to make proactive recommendations.

Methodology in Plain English

The authors treat memory as a three-stage lifecycle rather than a lookup table.

Phase I — Episodic Trace Formation. A semantic boundary detector runs over the dialogue with a sliding window; when it detects a topic shift, accumulated turns become a raw episode history. That history is rewritten into a concise third-person Episode with resolved coreferences. From the Episode, the system extracts Atomic Facts for precise matching and generates Foresight signals with validity intervals (for example, distinguishing a temporary "flu" from a permanent "graduation"). These elements plus metadata form a MemCell.

Phase II — Semantic Consolidation. Each new MemCell is embedded and compared to the nearest MemScene centroid. If similarity exceeds a threshold, the MemCell is assimilated and the scene representation is updated incrementally; otherwise a new MemScene is created. This runs online without batch reprocessing. Scene summaries then feed an online-updated user profile holding explicit facts (including time-varying measurements) and implicit traits, with recency-aware updates and conflict tracking.

Phase III — Reconstructive Recollection. Given a query, the system fuses dense and BM25 retrieval over Atomic Facts using Reciprocal Rank Fusion, scores each MemScene by the maximum relevance among its MemCells, and selects the highest-scoring scenes. Episodes from those MemCells are pooled and re-ranked, and Foresight is filtered so only intervals where t_now lies in [t_start, t_end] survive. An LLM-based verifier then judges whether the retrieved context is sufficient; if not, a query-rewriting step supplements retrieval.

The system supports two downstream modes: Memory-Augmented Reasoning (episodes as context for benchmark evaluation) and Memory-Augmented Chat (context additionally includes the User Profile and time-valid Foresight). Retrieval uses Qwen3-Embedding-4B for dense retrieval, BM25 for sparse retrieval, and Qwen3-Reranker-4B for re-ranking, with GPT-4.1-mini (or GPT-4o-mini where specified) for reasoning and memory operations. Evaluation uses an LLM-as-a-judge protocol with GPT-4o-mini plus two auxiliary judge models averaged in a blind setting, validated against human annotators recruited via Prolific.

Why This Matters

Impact on research. The paper reframes the memory problem from storage and retrieval toward consolidation and reconstruction, arguing that many agent failures stem from poor integration rather than missing information. It also reports that semantic segmentation beats ground-truth session boundaries as a retrieval unit, and that accuracy can exceed recall against annotated evidence — findings that complicate how memory benchmarks should be scored.

Real-world applications.

• Long-running personal assistants that must track changing user constraints, such as a new medication conflicting with a previously stated preference.
• Longitudinal health and habit coaching, where measurements like waist circumference change over time and profile stability matters.
• Customer-facing email and writing support that needs consistent personal or professional persona across sessions.
• Travel or planning agents that use past failures to make proactive, time-aware recommendations.

Industry relevance. The system targets product-ready deployment with a unified runtime for storage, retrieval, filtering, and updating, and it reports token-level cost accounting per lifecycle phase. The comparison against commercial and open memory infrastructures (Zep, Mem0, MemOS, MemoryOS, MemU) under a standardized answer-generation backbone speaks directly to teams choosing a memory layer for agent products.

Future Directions

• Extending the MemCell and MemScene abstraction beyond text to multimodal or embodied settings, which the authors explicitly leave out of scope.
• Reducing latency and computational cost of LLM-mediated memory construction and retrieval through caching, batching, or asynchronous execution.
• Developing benchmarks that stress-test ultra-long timelines, since the authors state current benchmarks lack protocols for isolating performance in such regimes.
• Further investigating why accuracy can exceed recall on LoCoMo, including redundancy and non-unique evidence annotations in existing benchmark ground truth, and how that affects evaluation design.

Target Audience

Researchers and engineers building long-horizon LLM agents, memory-augmented reasoning systems, or personalization infrastructure. It is most useful to readers already comfortable with retrieval pipelines, embedding models, and agent evaluation protocols, and to practitioners deciding how to structure persistent memory for deployed conversational products.

Authors’ abstract

Large Language Models (LLMs) are increasingly deployed as long-term interactive agents, yet their limited context windows make it difficult to sustain coherent behavior over extended interactions. Existing memory systems often store isolated records and retrieve fragments, limiting their ability to consolidate evolving user states and resolve conflicts. We introduce EverMemOS, a self-organizing memory operating system that implements an engram-inspired lifecycle for computational memory. Episodic Trace Formation converts dialogue streams into MemCells that capture episodic traces, atomic facts, and time-bounded Foresight signals. Semantic Consolidation organizes MemCells into thematic MemScenes, distilling stable semantic structures and updating user profiles. Reconstructive Recollection performs MemScene-guided agentic retrieval to compose the necessary and sufficient context for downstream reasoning. Experiments on LoCoMo and LongMemEval show that EverMemOS achieves state-of-the-art performance on memory-augmented reasoning tasks. We further report a profile study on PersonaMem v2 and qualitative case studies illustrating chat-oriented capabilities such as user profiling and Foresight. Code is available at https://github.com/EverMind-AI/EverMemOS.