ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support

AI summary

Overview

Research area: Natural Language Processing — long-term dialogue memory, personalization, and emotional support conversation, positioned at the ACM Web Conference (WWW '26).

Technical level: Intermediate. The paper assumes familiarity with LLM benchmarking, retrieval-augmented generation, and multi-session dialogue evaluation metrics (F1, BERTScore, ROUGE, LLM-as-Judge), but its core argument is accessible without deep implementation knowledge.

Scope: The paper introduces EvoEmo, a synthetic multi-session emotional support dataset, and ES-MemEval, a benchmark that evaluates five long-term memory capabilities of conversational agents across question answering, summarization, and dialogue generation tasks.

What This Paper Is About

Conversational agents built on large language models work well in short exchanges but struggle when a user's information is spread across many sessions, stated implicitly, and keeps changing — exactly the situation in long-term online emotional support. Existing long-term dialogue benchmarks mostly test static, explicitly stated fact retrieval, so they do not measure whether a model can integrate, abstract, and apply evolving personal information. The paper builds a dataset and benchmark designed to test that missing capability.

Key Contributions

1. EvoEmo dataset: The first multi-session dataset specifically designed for personalized long-term emotional support, capturing fragmented, implicit user disclosures and evolving user states across 18 virtual users, 401 total sessions, and an average time span of 14.9 months per conversation.

2. ES-MemEval benchmark: A comprehensive benchmark evaluating five core long-term memory capabilities — information extraction, temporal reasoning, conflict detection, abstention, and user modeling — across three complementary tasks: question answering, summarization, and dialogue generation.

3. Task-specific evaluation sets: A QA set of 1,209 reviewed samples spanning all five capabilities, a summarization set of 125 cases (74 temporal reasoning, 51 user modeling), and 34 dialogue generation scenarios with an average of 20 turns per session.

4. Systematic empirical comparison: Evaluation of five models across three paradigms — open-source long-context models, commercial models, and their retrieval-augmented variants — yielding six documented insights into strengths and limitations of current approaches.

Main Findings

• RAG improves every task, unevenly. Mistral-24B with RAG rose from F1 15.5 to 18.8, BERTScore 47.4 to 50.4, and LLM-as-Judge 1.01 to 1.27 on QA. On summarization, the same model with RAG improved ROUGE-L from 10.9 to 21.0, event-based F1 from 26.8 to 48.1, and LLM Score from 1.45 to 2.79.

• Smaller models gain the most from retrieval. Mistral-8B's LLM-as-Judge score increased by 0.43 under RAG, indicating retrieval is especially beneficial for models with limited capacity.

• Temporal reasoning and user modeling stay hard. F1 scores seldom exceed 20.0 for these capabilities even with RAG.

• Abstention behaves differently across model types. RAG improved abstention for open-source models but reduced it for commercial ones. GPT-4o was most affected, with its abstention LLM score dropping from 1.67 to 1.30, and its abstention F1 falling from 66.7 to 12.7, suggesting retrieved content encourages overconfident answers.

• Session-level retrieval works best. Session-level RAG reached an LLM-as-Judge score of 1.27 at k=4, above the best round-level (1.20) and turn-level (1.15) settings. Recall@k exceeded 75% across all granularities, but NDCG@k peaked at only 59% to 63%, meaning coverage is good while ranking quality is not.

• Long context does not equal effective long context. Despite nominal 128K-token windows, Mistral-8B performed best at a 2K context (F1 9.8) and Mistral-24B peaked at 8K (F1 17.4). Each user's full history spans 11K to 19K tokens, so smaller models require truncation.

• Explicit memory is required for reliability. Under the No-Mem setting, dialogue generation scores remained low, and higher values largely reflected hallucinated user experiences. With Full-Hist or RAG, Mistral-24B's observation-based Weighted Score rose from 0.20 to 0.33 and 0.41 respectively.

• Personalization tracks memory; emotional support less so. LLM ratings showed a strong correlation between personalization and long-term memory scores, while emotional support scores were less sensitive to memory, indicating support can partly rely on general strategies.

• RAG narrows the open-source versus commercial gap. On summarization, both GPT-3.5-turbo + RAG and GPT-4o + RAG reached an LLM Score of 2.93, with GPT-4o + RAG achieving the highest event-based F1 at 49.4.

Methodology in Plain English

The researchers could not collect real long-term emotional support conversations at scale, so they built them synthetically. They started from real short-term support dialogues in the ESConv dataset and used them to hand-craft 18 virtual user profiles containing demographic information, social relationships, and core beliefs. For each user, GPT-4o generated an event timeline with timestamps and descriptions, then expanded it over two rounds with human review to preserve temporal and causal consistency, producing an average of 24.8 events per user.

Those profiles and timelines were then fed to GPT-4o to generate multi-turn emotional support sessions, annotated with emotion category, topic, summary, and turn-level user observations. Six annotators checked each session for consistency with the profile and across sessions.

From this dataset the team built three evaluation sets. QA questions were generated across the five memory capabilities with reference answers and supporting evidence, then reviewed by annotators. Summarization cases grouped sessions thematically and required cross-session abstraction. Dialogue scenarios specified topics with user state details. Evaluation used F1, BERTScore, and GPT-4o as a judge for QA; ROUGE, an LLM score from 0 to 5, and FActScore-inspired event-based precision, recall, and F1 for summarization; and observation-based recall/weighted accuracy plus LLM ratings on long-term memory, personalization, and emotional support for dialogue generation, with GPT-4o playing the simulated user.

Baselines covered Ministral-8B-Instruct-2410, Phi-3-Medium-128k-Instruct, and Mistral-Small-3.1-24B-Instruct-2503 (each supporting 128K tokens), plus gpt-3.5-turbo (4K) and gpt-4o (16K). Retrieval-augmented variants used the bge-m3 dense retriever over a FAISS index to fetch the top-4 full-session contexts. All experiments ran on an A100 GPU with 80GB of memory.

Why This Matters

Research impact: The paper argues that existing long-term dialogue benchmarks capture only a narrow slice of memory — static, explicit fact retrieval — and proposes a test suite for the harder problem of implicit, fragmented, and evolving user information. Its comparative table places ES-MemEval against MSC, Conversation Chronicles, DuLeMon, MemoryBank, PerLTQA, LOCOMO, LongMemEval, MADial-Bench, and DialSim, showing that ES-MemEval is the only one covering all five core memory abilities and the first targeting personalized emotional support conversation.

Real-world applications:

• Online mental health and emotional support services, where an agent must remember a user's history across weeks without inventing details.
• Customer support platforms, where long-term personalization depends on accurate recall of past interactions.
• Companion and wellness applications that track evolving user states over months.
• Memory-augmented personal assistants, where retrieval granularity and ranking quality directly affect response quality.

Industry relevance: The finding that smaller long-context models degrade with extended inputs, while RAG narrows the gap between open-source and commercial systems, gives practitioners a concrete architectural argument for combining retrieval with explicit memory rather than simply expanding context windows.

Future Directions

• Retrieval-aware calibration: RAG improved factual consistency but hurt abstention in commercial models, so methods are needed that let models judge when not to answer despite retrieved content.

• Adaptive memory granularity: Session-level retrieval performed best but was noted to introduce redundancy, raising the question of how to select granularity dynamically per query.

• Better ranking rather than just better coverage: Recall@k already exceeded 75% while NDCG@k stayed near 59% to 63%, so improving retrieval ordering may yield more gains than retrieving more.

• Hybrid memory–retrieval designs and dataset scaling: The authors point toward integrating retrieval with external memory mechanisms, and note plans to increase the number of users and sessions to improve representativeness and complexity, addressing the synthetic dataset's limited topic balance and lack of cross-cultural diversity.

Target Audience

NLP researchers working on long-term dialogue, memory, and personalization; benchmark and evaluation designers; developers building conversational agents for mental health, customer support, or companion applications; and teams deciding between long-context prompting and retrieval-augmented architectures for personalized systems.

Authors’ abstract

Large Language Models (LLMs) have shown strong potential as conversational agents. Yet, their effectiveness remains limited by deficiencies in robust long-term memory, particularly in complex, long-term web-based services such as online emotional support. However, existing long-term dialogue benchmarks primarily focus on static and explicit fact retrieval, failing to evaluate agents in critical scenarios where user information is dispersed, implicit, and continuously evolving. To address this gap, we introduce ES-MemEval, a comprehensive benchmark that systematically evaluates five core memory capabilities: information extraction, temporal reasoning, conflict detection, abstention, and user modeling, in long-term emotional support settings, covering question answering, summarization, and dialogue generation tasks. To support the benchmark, we also propose EvoEmo, a multi-session dataset for personalized long-term emotional support that captures fragmented, implicit user disclosures and evolving user states. Extensive experiments on open-source long-context, commercial, and retrieval-augmented (RAG) LLMs show that explicit long-term memory is essential for reducing hallucinations and enabling effective personalization. At the same time, RAG improves factual consistency but struggles with temporal dynamics and evolving user states. These findings highlight both the potential and limitations of current paradigms and motivate more robust integration of memory and retrieval for long-term personalized dialogue systems.