Generation-Based and Emotion-Reflected Memory Update: Creating the KEEM Dataset for Better Long-Term Conversation

AI summary

Overview

Research area: Natural language processing, specifically memory management for long-term (multi-session) open-domain conversational systems, with a focus on emotion-aware memory representation.

Technical level: Intermediate. The paper assumes familiarity with multi-session chat datasets, retrieval-augmented and fusion-in-decoder dialogue models, perplexity as an evaluation metric, and the use of large language models for synthetic dataset construction.

Scope: This paper introduces the KEEM (Keep Emotional and Essential Memory) dataset, a Korean generation-based memory-update dataset built from the KMSC corpus using ChatGPT 4.0, and evaluates it against accumulation-based and operation-based memory updating.

What This Paper Is About

Long-term conversational systems need to update what they remember about a user as the user's situation changes, but existing approaches either simply accumulate every past summary (creating contradictions) or apply classification operations such as DELETE and REPLACE (erasing information that is still worth remembering, such as the fact that a user once had COVID or once traveled in Europe). The paper's goal is to replace selection/deletion-based memory management with generation-based memory management, where a model writes a new integrative memory sentence that preserves essential facts while also capturing the user's emotions and the causes of those emotions.

Key Contributions

1. A novel generation-based memory-update dataset that dynamically generates integrative memories to retain essential information, addressing the limitations of existing DELETE and REPLACE operations.
2. The Keep Emotion and Essential Memory (KEEM) dataset, described by the authors as the first of its kind to integrate both emotion and causality into memory updates, facilitating deeper empathy and understanding in conversational AI.
3. A demonstration of the dataset's quality and applicability for improving user engagement and system responsiveness in multi-session conversations, supported by human evaluation, keyword-recall analysis, conflict analysis under natural language inference, and perplexity/voting evaluation across seven long-term conversation models.
4. A generation-error analysis showing the error rate of the ChatGPT-4.0-based construction pipeline (only 6 error samples out of 70 sampled update cases, approximately 8.5%).

Main Findings

• Emotion and cause reflection improves substantially: Human evaluation of 50 sessions on a 0-to-2 scale showed the raw KMSC memory reflected emotions and their causes 35% of the time with a score of 1.18, while the KEEM memory reached 93% with a score of 1.90.
• Memory updates are accurate and increasingly needed over time: On 100 samples per session, the update-need rate was 12% for Session 1-2, 31% for Session 1-3, and 21% for Session 1-4, with update-accuracy scores of 1.75, 1.64, and 1.62 respectively on a 0-to-2 scale.
• KEEM beats the operation-based baseline in comparative evaluation: KEEM scored 1.86 versus CareCallmem's 1.60 in manual scoring, and won 76% of human votes and 81% of ChatGPT votes, against 24% and 19% for CareCallmem.
• Higher keyword recall: Using TextRank with TF-IDF and keyword sets of 5, 7, and 10, KEEM memories showed higher recall of keywords from the full session dialogues than both the accumulation method (KMSC) and the operation method (CareCallmem). The authors note they expected accumulation to win because it stores everything, and attribute its underperformance to missing information in the original KMSC summaries.
• Fewer conflicts between memory sentences: Under an NLI setup where ChatGPT 4.0 labeled sentence pairs as entailment, neutral, or contradiction, the accumulation method produced the highest conflict ratio, rising further in samples where an update occurred. KEEM and CareCallmem both showed significantly lower conflict ratios, but CareCallmem achieves this partly by deleting sentence pairs that could contradict.
• Lowest perplexity and best response voting across models: KEEM memories produced the lowest perplexity for every model tested (RAG 8.10, FiD 7.88, FiD-RAG 7.90, Llama2 7B 6.99, Llama2 13B 6.89, Llama 10.8B tuned Korean 4.56, Phi2 8B tuned Korean 4.61) versus KMSC (9.74, 9.51, 9.66, 8.70, 8.53, 5.47, 5.83) and CareCallmem (10.55, 10.49, 10.52, 10.50, 10.51, 8.13, 8.72). Korean-tuned models showed better perplexity than the others. In ChatGPT-4.0 voting, KEEM responses were selected 67% to 82% of the time depending on the model, against 15% to 26% for KMSC and 3% to 10% for CareCallmem.
• Low generation error rate: Manual review of 150 updated memories (50 each from Session 2, Session 3, and Session 4 data) found significantly low error rates in both construction phases. Of 70 sampled cases where information was actually updated, only 6 showed errors, approximately 8.5%. For comparison, the paper cites Wang et al. (2023b) reporting GPT-3 dataset-creation error rates of 8%, 21%, and 42% across fields (54% valid data) and automatic summarization error rates around 30%.
• Dataset scale: KEEM contains 2,006 episodes for Session 1-2 (61,354 utterances, 33,972 memory sentences), 1,560 episodes for Session 1-3 (70,974 utterances, 31,846 memory sentences), and 1,005 episodes for Session 1-4 (59,847 utterances, 23,148 memory sentences), compared with KMSC's 40,000 / 20,000 / 20,000 episodes.

Methodology in Plain English

The authors started from the Korean multi-session chat (KMSC) dataset from AI Hub, which covers 13 themes and provides four sessions per conversation with a summary per session, but was not designed for memory updates. They used ChatGPT 4.0 (gpt-4-1106-preview) with temperature 0.0 and top-p and top-n set to 1 for deterministic output to build two things.

First, an emotion-and-cause reflection step: given a session dialogue and its summary, ChatGPT rewrites the summary to include the user's emotions and the reasons behind them, but only when the dialogue explicitly supports them. The authors tested three prompting choices on 100 samples each — Korean versus English instructions (Korean won), a reasoning prompt listing candidate emotions such as pleasure, sadness, and happiness versus no candidates (no candidates won), and few-shot examples versus none (few-shot won). The final recipe was Korean instructions without emotion candidates, with few-shot examples.

Second, a memory-update step: given the previous memory (earlier sessions' summaries) and the current session's summary, ChatGPT generates an updated memory. The authors compared supplying the current dialogue versus the current summary (results were similar, so they used the cheaper summary) and compared a two-stage reasoning prompt that first matches summary sentences to memory sentences against direct instructions (neither outperformed direct instruction). A final verification pass feeds all session dialogues up to the current time step plus the updated memory back to ChatGPT; samples that fail verification are excluded.

Evaluation combined human judgment (five native speakers unfamiliar with either dataset's construction, scoring 0-2), keyword recall with TextRank and TF-IDF, NLI-based conflict analysis with ChatGPT 4.0 over 50 randomly sampled memories plus 50 samples where an update occurred, and downstream testing on RAG, FiD, FiD-RAG, Llama2 7B and 13B, and Korean-tuned Llama 10.8B and Phi-2 8B models. Because ordinary conversation does not prove that a model used an updated memory, the authors added a purpose-built fifth session dialogue containing at least one turn about updated information, then measured perplexity on responses to those turns and had ChatGPT 4.0 vote for the best response. All model experiments ran on one NVIDIA A100 80GB GPU.

Why This Matters

Impact on research: The paper argues that memory management in long-term conversation has been understudied, largely because relevant datasets are scarce. It reframes memory updating as a generation problem rather than a classification problem over operations, and it extends memory content beyond factual events to emotions and their causes — a combination the authors claim no prior memory-update dataset provides. It also supplies an evaluation design (a fifth diagnostic session) for testing whether a model actually applies updated memory rather than merely conversing.

Real-world applications:

• Companion and check-in chatbots for elderly users living alone, the setting that motivated the original CareCall and CareCallmem work.
• Mental health and counseling support, where recognizing a negative emotion together with its cause can enable the system to offer advice or solutions rather than superficial empathy.
• Personalized assistants that must track changing user states across sessions, such as health conditions, travel plans, or exam schedules, without keeping outdated or contradictory facts.
• Memory-efficient deployment, since deleting or revising outdated memories keeps the memory store concise instead of growing without bound.

Industry relevance: The results give product teams a concrete argument that generation-based memory update yields lower generation difficulty (lower perplexity) and more useful responses than storing everything or applying DELETE/REPLACE rules, and that the gain holds across retrieval-based, fusion-based, and instruction-tuned LLM backbones, including Korean-tuned models.

Future Directions

• Explore models that account for temporal gaps between sessions, using large language models to improve memory understanding, as the authors state in the conclusion.
• Reduce the inadvertent deletion of content unrelated to the intended update; the authors report that explicitly instructing ChatGPT not to delete unrelated content only partly solved the problem.
• Avoid the cost-driven shortcut of using the current session's summary instead of the full current-session dialogue, which risks omitting important information when the summary is insufficiently comprehensive.
• Establish relative dataset quality by comparing KEEM's generation-error rate against other datasets, something the paper notes is still needed.

Target Audience

Researchers and engineers working on long-term or multi-session dialogue systems, conversational memory architectures, and empathetic dialogue; builders of Korean-language conversational AI; and NLP practitioners interested in using large language models for dataset construction with human verification. The paper is also relevant to anyone designing evaluation protocols for whether a system genuinely uses updated memory, though readers without background in dialogue modeling or LLM-based data generation will find the experimental sections harder to follow.

Authors’ abstract

In this work, we introduce the Keep Emotional and Essential Memory (KEEM) dataset, a novel generation-based dataset designed to enhance memory updates in long-term conversational systems. Unlike existing approaches that rely on simple accumulation or operation-based methods, which often result in information conflicts and difficulties in accurately tracking a user's current state, KEEM dynamically generates integrative memories. This process not only preserves essential factual information but also incorporates emotional context and causal relationships, enabling a more nuanced understanding of user interactions. By seamlessly updating a system's memory with both emotional and essential data, our approach promotes deeper empathy and enhances the system's ability to respond meaningfully in open-domain conversations.