TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents

AI summary

Overview

Research area: Natural Language Processing / LLM-based conversational agent memory systems.

Technical level: Intermediate. The core ideas are intuitive, but the paper includes formal tree definitions, retrieval scoring formulas, and multi-benchmark evaluation protocols.

Scope: TiMem is a memory framework that organizes long conversation histories into a time-structured hierarchy so that conversational agents can retain and recall personalization information across sessions without exceeding LLM context limits.

What This Paper Is About

Long-running chat agents accumulate interaction histories that quickly outgrow an LLM's context window, and existing memory systems mostly cluster memories by semantic similarity while treating timestamps as optional metadata. This causes fragmented memories, temporally jumbled evidence, and unstable personalization as users' lives and preferences evolve. TiMem tackles this by making time the primary organizing principle: conversations are consolidated through a five-level Temporal Memory Tree (segment, session, day, week, profile), and recall is matched to how complex the user's query is.

Key Contributions

1. Temporal Memory Tree (TMT): A hierarchical memory structure that enforces explicit temporal containment (parent intervals always cover child intervals), progressive consolidation (higher levels contain fewer, more abstract nodes), and semantic consolidation via level-specific prompts.

2. A full consolidation-and-recall framework without fine-tuning: The Memory Consolidator converts raw dialogue into higher-level abstractions using instruction prompts, and the recall pipeline combines a complexity-aware planner, hierarchical propagation from leaf memories to ancestors, and a recall gating module that acts as retrieval-time forgetting.

3. State-of-the-art accuracy with reduced context: 75.30% on LoCoMo and 76.88% on LongMemEval-S (78.96% when using a stronger answer model), while cutting recalled memory length by 52.20% on LoCoMo compared to Mem0.

4. Empirical analysis of why the hierarchy helps: Ablations isolate the contributions of the planner, gating, and hierarchy depth, and UMAP manifold analysis shows higher-level memories cluster users better on LoCoMo and suppress noise on LongMemEval-S.

Main Findings

• Best overall accuracy across both benchmarks: TiMem reaches 75.30% LLJ accuracy on LoCoMo (vs. 69.24% for the strongest baseline, MemOS) and 76.88% on LongMemEval-S with gpt-4o-mini, rising to 78.96% with gpt-4o as the answer model.

• Substantial context reduction: Recalled memory drops to 511.25 tokens per query on LoCoMo, a 52.20% reduction versus Mem0's 1,070.10 tokens, while recall latency stays low (P50 of 2.35s on LoCoMo).

• Both planner and gating matter: Under a fixed recall scope, recall gating dramatically shortens contexts (e.g., 3,710 to 368 tokens on Simple queries) but can hurt accuracy when scope is too narrow. The adaptive planner produces the best accuracy-cost balance, reaching 75.30% with 511.25 tokens.

• Hierarchical recall beats flat recall: With L1-only memories, hierarchical propagation lifts LongMemEval-S accuracy from 57.40% to 72.40%, showing that ancestor memories supply necessary temporal context. Using only high-level layers (L2–L5) collapses accuracy, confirming summaries cannot replace fine-grained evidence.

• Temporal consolidation reshapes memory geometry: On LoCoMo, higher-level memories separate the 10 user groups more clearly (6.2x improvement in clustering quality), while on LongMemEval-S, consolidation reduces spatial dispersion by 50%, suppressing sampling noise.

• Finer segments are better: Increasing L1 segment size steadily degrades accuracy, from 75.30% at one turn to 65.26% at eight turns, indicating atomic evidence is valuable for downstream QA.

• Semantic-lexical balance is robust: Performance is stable across λ in [0.7, 1.0], peaking at λ = 0.9, where semantic similarity handles paraphrasing and BM25 captures exact entities.

• Modest consolidation overhead: The full five-level tree adds 25–30% more internal LLM calls versus a flat L1-only baseline, but this cost is amortized over the conversation history while sharply cutting inference tokens for the answering model.

• Causal chains improve counterfactual reasoning: A case study shows TiMem linking a user's formative experiences to a career aspiration through consolidated levels, producing a correct counterfactual answer where Mem0 returned the opposite conclusion.

Methodology in Plain English

The researchers start from the observation that human memory consolidates rapid episodic events into stable, more abstract structures over time. They translate this into a computational design:

1. Build a tree over time. Conversation turns are grouped into intervals at five scales: short segments, sessions, days, weeks, and a rolling monthly persona profile. A parent node's time span always covers its children's span, so temporal order is baked into the structure rather than inferred later.

2. Consolidate with prompts instead of training. At each level, an LLM reads the child memories, a small sliding window of recent same-level memories, and a level-specific instruction prompt, then produces a new abstracted memory. Because everything is prompt-driven, the framework plugs into different LLM backends without fine-tuning.

3. Schedule consolidation in two tiers. Level-1 segment memories are written immediately after each dialogue turn to preserve fine detail, while levels 2–5 are generated automatically when their time windows close (session end, daily, weekly, monthly).

4. Recall adaptively. A planner LLM labels each query as simple, hybrid, or complex and extracts keywords. Simple queries target factual layers plus the profile; complex queries traverse the full hierarchy. Base-level memories are selected by combining embedding cosine similarity with BM25 lexical scores, then ancestor memories at the chosen levels are pulled in.

5. Gate the results. A final LLM pass reviews candidate memories and keeps only those relevant to the query and temporally consistent, then ranks them by hierarchy level and recency. This is the framework's "forgetting" step.

6. Evaluate under a shared setup. All five baselines and TiMem use the same LLM (gpt-4o-mini), the same embedding model (Qwen3-Embedding-0.6B), and the same recall budget so comparisons isolate the memory design itself.

Why This Matters

Impact on research. Most memory systems for LLM agents treat time as metadata attached to semantically clustered content. TiMem argues that temporal containment should be a structural constraint, not a soft signal. Its ablations supply concrete evidence for that claim: hierarchical propagation of ancestors recovers 15 points of accuracy on LongMemEval-S where flat recall fails, and manifold analysis connects hierarchy depth to measurable geometric changes in embedding space. This gives the field a template for evaluating memory structure, not just memory accuracy.

Real-world applications:

• Long-term personal assistants that need to remember a user's preferences, constraints, and evolving goals across months of intermittent conversations without dragging entire histories into every prompt.

• Customer support agents handling multi-session tickets where the correct answer depends on following a chain of events rather than matching a keyword.

• Health and lifestyle coaching bots where daily routines, weekly patterns, and a stable long-term profile are naturally distinct levels of abstraction with different update frequencies.

• Enterprise knowledge companions where per-user memory trees can be isolated by design, easing privacy and compliance requirements around cross-user data leakage.

Industry relevance. The 52% reduction in recalled tokens directly translates into lower per-query inference cost and shorter latency, and the framework's model-independence means it can be deployed against whichever LLM a vendor already uses. The no-fine-tuning requirement keeps the integration effort low, and the monthly profile updates make the system practical for products where user identity is expected to drift rather than stay fixed.

Future Directions

• Storage-time forgetting. The paper explicitly notes the absence of a mechanism for selectively dropping or merging memories during storage rather than at recall time, which could further reduce tree growth for very long deployments.

• Structured high-level representations. Higher-level memories are text summaries without typed categories or knowledge graphs. Hybrid designs combining the temporal tree with entity-typed stores could capture multi-dimensional user attributes more precisely.

• Adaptive temporal boundaries. TiMem uses fixed, realistic calendrical boundaries (session, day, week, month) for reproducibility. Learning boundaries from interaction density could improve transfer to domains where activity is bursty or irregular.

• Smaller specialized middleware. Consolidation, planning, and gating all rely on general-purpose LLM calls. Distilling these into fine-tuned small models could cut latency and cost without losing functional accuracy.

• Robustness to planner errors. The discussion acknowledges that misclassified query complexity can expand or shrink the recall scope incorrectly, leaving room for more robust complexity estimation.

Target Audience

This paper is most useful to researchers and engineers building memory layers for LLM agents, particularly those working on long-horizon personalization, retrieval-augmented generation, or conversational AI infrastructure. Practitioners designing production chatbot memory will find the efficiency numbers and ablation trade-offs directly actionable, while academic readers interested in cognitive-inspired architectures will appreciate the explicit mapping from complementary learning systems theory to a working computational design. Readers should be comfortable with retrieval scoring, embedding-based similarity, and standard LLM-as-a-Judge evaluation to get the most from the experimental sections.

Authors’ abstract

Long-horizon conversational agents have to manage ever-growing interaction histories that quickly exceed the finite context windows of large language models (LLMs). Existing memory frameworks provide limited support for temporally structured information across hierarchical levels, often leading to fragmented memories and unstable long-horizon personalization. We present TiMem, a temporal--hierarchical memory framework that organizes conversations through a Temporal Memory Tree (TMT), enabling systematic memory consolidation from raw conversational observations to progressively abstracted persona representations. TiMem is characterized by three core properties: (1) temporal--hierarchical organization through TMT; (2) semantic-guided consolidation that enables memory integration across hierarchical levels without fine-tuning; and (3) complexity-aware memory recall that balances precision and efficiency across queries of varying complexity. Under a consistent evaluation setup, TiMem achieves state-of-the-art accuracy on both benchmarks, reaching 75.30% on LoCoMo and 76.88% on LongMemEval-S. It outperforms all evaluated baselines while reducing the recalled memory length by 52.20% on LoCoMo. Manifold analysis indicates clear persona separation on LoCoMo and reduced dispersion on LongMemEval-S. Overall, TiMem treats temporal continuity as a first-class organizing principle for long-horizon memory in conversational agents. The code is available at https://github.com/TiMEM-AI/timem.