Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction

AI summary

Overview

• Research area: Natural Language Processing / dialogue systems — specifically long-term, personalized, service-oriented user-agent interaction and memory-augmented generation.
• Technical level: Advanced (assumes familiarity with retrieval-augmented generation, LLM-based data synthesis, BLEU/LLM-as-judge evaluation, and memory architectures for dialogue).
• Scope: The paper introduces PAL-Bench (a Chinese benchmark plus the PAL-Set dataset of multi-session user logs and dialogues) and H²Memory (a hierarchical, heterogeneous memory framework with retrieval-augmented generation) for personalized long-term dialogue assistants.

What This Paper Is About

Personal smart devices have made service-oriented user-agent interaction common, but existing dialogue systems treat users uniformly and fail to model subjective, individual preferences across long interaction histories. The authors build a benchmark to test whether an assistant can infer a user's real requirement from brief queries plus long-term behavioral and dialogue history, and they propose a memory framework that organizes that history so an LLM can retrieve and use it. Existing long-term dialogue benchmarks mostly cover human-human chat or only retrieve personal facts from dialogues, and they typically ignore user behavioral records (logs) — the paper targets that gap.

Key Contributions

1. PAL-Bench, described as the first Chinese benchmark for long-term user-agent interactions, built with a scalable multi-stage LLM-based synthesis pipeline plus verification and human refinement. It defines three tasks: Requirement Restatement, Solution Proposal, and Multi-turn Dialogue Interaction.
2. PAL-Set, described as the first Chinese dataset comprising multi-session user logs and dialogue histories. It contains 100 synthetic users; the introduction states each user has an average of 29 sessions, 996 behavioral logs, and 401 dialogue turns, and that an English-translated version is also provided.
3. H²Memory, a hierarchical and heterogeneous memory framework that separately models behavior logs and dialogues at both concrete and abstract levels, with update mechanisms for persona dynamics, combined with a retrieval-augmented generation strategy.
4. Validation, including a human evaluation of the PAL-Set data (average scores of 2.75 for logs and 2.67 for dialogues on a 1-to-3 matching scale) and experiments on PAL-Bench plus the external LongMemEval "single-session-preference" subset.

Main Findings

• Memory construction is verified at the data level: Synthesized logs scored 2.75 and dialogues 2.67 on a 1 (non-matching) to 3 (completely matching) human rating scale, which the authors cite as evidence of consistency with the predefined user profiles and personas.
• H²Memory leads on Requirement Restatement: In Table 2, H²Memory achieves BLEU scores of 26.67 / 12.18 / 5.68 / 3.09 (B-1 to B-4) and a GPT-4 Score of 32.54, versus the strongest baseline MemoryBank at 23.89 / 11.11 / 5.23 / 2.91 and 28.57.
• H²Memory leads on Solution Proposal: It reaches BLEU 22.24 / 8.38 / 4.39 / 2.65 and a Selection Score of 38.32, compared with MemoryBank at 20.49 / 8.12 / 4.07 / 2.34 and 29.85, and Session-level RAG at 19.66 / 7.80 / 4.00 / 2.33 and 33.78.
• Logs help by themselves: Vanilla (with log) outperforms Vanilla (w/o log) — for example, 19.71 vs 13.59 BLEU-1 and 23.00 vs 17.50 GPT-4 Score — indicating the synthesized logs are meaningfully tied to user requirements.
• Ablations show complementary memory parts, with a task-dependent twist: Removing the topic-outline memory M_T causes the largest drop in Requirement Restatement (BLEU-1 falls from 26.67 to 24.04, GPT-4 Score from 32.54 to 28.00). In Solution Proposal, removing background memory M_B hurts more than removing situation memory M_G (Selection Score 36.69 vs 37.71), which the authors interpret as preferences being more stable, high-level traits while requirements are more grounded in recent events.
• Multi-turn wins across the board: Table 3 reports Win/Tie/Lose of 478/29/319 (requirement) and 480/18/328 (preference) against Vanilla (w/o log), 447/33/346 and 452/22/352 against Vanilla (with log), 421/29/376 and 439/18/369 against RecurSum, 396/42/388 and 413/19/394 against ConditionMem, and 449/33/344 and 452/25/349 against MemoryBank. The authors acknowledge a non-negligible number of "Lose" cases and attribute it primarily to randomness in the User-LLM's utterances.
• Generalization to an English external dataset: On LongMemEval's "single-session-preference" subset, using only the dialogue components (M_T + M_P) reaches 50.00 accuracy, compared with ConditionMem at 40.00, MemoryBank at 23.33, and Vanilla and RecurSum at 10.00 each. The individual components score 40.00 (M_T only) and 46.67 (M_P only).

Methodology in Plain English

Because real long-term user data is costly to collect and privacy-constrained, the authors generate it. Using Qwen2.5-Max, they first create a fictional user profile (gender, age, Big Five personality traits, and descriptions across work, health, family, and leisure), then expand it into a persona with a months-long timeline and general requirement types paired with positive and negative preferences. Each month is expanded into 4 to 6 situation entries, each situation into a diary-style experience, and these into timestamped behavioral logs drawn from 8 predefined log types (web search, content publishing, content browsing, message sending, message receiving, schedule management, transaction record, device operation). For dialogues, they build a "framework" per situation containing topics, each with a user query, two implicit needs, the full requirement, plus 8 candidate solutions from which 2 positive and 2 negative solutions are selected based on the persona. Utterance-level templates encode dialogue actions for both sides. Automatic validation rules and regeneration, plus human verification, guard quality.

H²Memory then organizes this history into four stores. M_G turns fragmented logs into a graph: the LLM labels relations between logs (types "Caused_by" and "Follows"), producing connected subgraphs that are summarized into situation descriptions. M_B summarizes situation entries into fixed-aspect background paragraphs and is updated recursively session by session. M_T segments each dialogue into topics and extracts a schema of requirement, offered solutions with user feedback, and preference; the requirement is rewritten using retrieved situation entries. M_P clusters all extracted requirements with KMeans and has the LLM derive a requirement type and preference principle per cluster, updated as new query sessions arrive. At inference, the top k = 3 most similar entries are retrieved by cosine similarity using the "paraphrase-multilingual-mpnet-base-v2" encoder, and the retrieved bundle conditions response generation.

Evaluation uses BLEU and a GPT-4-based score for requirement restatement, BLEU plus a Selection Score for solution proposal (the assistant picks 2 positives from 8 candidates), and for multi-turn interaction a User-LLM (Qwen2.5-Max) role-plays the dataset user while an Evaluation-LLM (GPT-4-turbo) judges pairwise on requirement understanding and preference understanding. Following the FairEval framework, each pair is evaluated 6 times with different input orders, and Win–Tie–Lose counts are reported. The base model for memory construction and response generation is Qwen-Max-0428.

Why This Matters

Impact on research. The paper supplies a public benchmark, dataset, and code link for a setting — long-term user-agent interaction with both behavioral logs and dialogues — that the authors argue no prior public benchmark covers with subjective preference modeling. It also offers a memory design that treats heterogeneous history types differently rather than flattening everything into one store.

Real-world applications.

• Smart personal devices and virtual assistants that carry context across weeks or months without the user repeating themselves.
• E-commerce and shopping assistants (the paper's author affiliations include Taobao & Tmall Group of Alibaba) that infer preferences from past behavior and prior conversations.
• Health and fitness assistants using wearable/device logs (the authors show smart-band and smartphone logs as examples of synthesized behavior).
• Service or customer-support assistants that need to clarify underspecified requests and propose options aligned with an individual's known preferences.

Industry relevance. The framework is built for API-based LLMs and does not require parameter fine-tuning, and the paper reports the base model, retrieval encoder, k value, and evaluation LLMs explicitly — making the pipeline comparatively straightforward to adapt to deployed assistants. The CCF-ALIMAMA TECH Kangaroo Fund (NO. CCF-ALIMAMA OF 2024007) funded the work.

Future Directions

1. Reducing simulator noise. The authors attribute a portion of the "Lose" cases in multi-turn comparison to randomness in the User-LLM's utterances; stabilizing role-played users is an explicit open issue.
2. Scaling and diversifying the dataset. PAL-Set has 100 synthetic users and 8.4 months of history on average per the statistics table; expanding both, and adding the English version as a first-class benchmark rather than only a translation, are natural next steps.
3. Richer behavioral signals. The paper notes that prior work used app screenshots as behavioral history but that data is not public due to privacy concerns; incorporating multimodal device context is an unresolved direction.
4. Closing the preference-modeling gap. The authors argue preferences are abstract and nuanced and require modeling beyond simple fact extraction, and the ablation showing M_B's larger role in Solution Proposal than in Requirement Restatement suggests further work on how stable high-level traits should be stored and updated.

Target Audience

Researchers and engineers working on long-term dialogue, personalized response generation, memory-augmented LLMs, and dialogue benchmarks will get the most value, along with practitioners building assistant products on top of API-based LLMs. The paper is also useful to those studying LLM-based synthetic data generation and human-verified data quality pipelines. Readers should already be comfortable with retrieval-augmented generation, memory architectures, and LLM-as-judge evaluation, since the paper assumes that background rather than explaining it.

Authors’ abstract

With the rise of smart personal devices, service-oriented human-agent interactions have become increasingly prevalent. This trend highlights the need for personalized dialogue assistants that can understand user-specific traits to accurately interpret requirements and tailor responses to individual preferences. However, existing approaches often overlook the complexities of long-term interactions and fail to capture users' subjective characteristics. To address these gaps, we present PAL-Bench, a new benchmark designed to evaluate the personalization capabilities of service-oriented assistants in long-term user-agent interactions. In the absence of available real-world data, we develop a multi-step LLM-based synthesis pipeline, which is further verified and refined by human annotators. This process yields PAL-Set, the first Chinese dataset comprising multi-session user logs and dialogue histories, which serves as the foundation for PAL-Bench. Furthermore, to improve personalized service-oriented interactions, we propose H$^2$Memory, a hierarchical and heterogeneous memory framework that incorporates retrieval-augmented generation to improve personalized response generation. Comprehensive experiments on both our PAL-Bench and an external dataset demonstrate the effectiveness of the proposed memory framework.