SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

AI summary

Overview

Research area: Long-term conversational memory for large language model (LLM) agents, applied specifically to multi-party (group chat) dialogue. The paper sits in natural language processing (cs.CL) and overlaps with retrieval, structured memory, and reinforcement learning for agent components.

Technical level: Advanced. It assumes familiarity with retrieval-augmented generation, memory-graph systems (Mem0, Zep, HippoRAG, A-MEM), retrieval baselines such as BM25 and dense embedding, and policy-gradient reinforcement learning (GRPO).

One-sentence scope: The paper proposes SpeakerMem-R1, a "dual-track" memory architecture that keeps speaker-labeled verbatim messages alongside derived person-level and group-level state records, trains a small local Writer model with reinforcement learning, and evaluates the result on GroupMemBench, SocialMemBench, EverMemBench, and LoCoMo.

Paper metadata: arXiv:2609.26780v1 [cs.CL], 22 Sep 2026, licensed CC BY 4.0, by Haobo Zheng, Tan Tang, Yan Chen, Weijie Wang, and Yingcai Wu of the State Key Lab of CAD&CG, Zhejiang University. A project page and GitHub repository are listed.

What This Paper Is About

General-purpose LLM memory systems were built mainly for single-user or two-person conversations, where conversations can largely be split, compressed, indexed, and retrieved by relevance. Group chats break that assumption: they contain speaker relations, reply structure, cross-topic branches, and state revisions, so they cannot be safely flattened into a message stream and compressed in order. The paper argues that multi-party memory faces two coupled bottlenecks — message attribution (who said what, whom a statement concerns, whether information is personal or shared) and state reconstruction (recovering current or historical states from clues scattered across members, groups, and time). The goal is a memory design that preserves attribution and source-linked evidence while still reconstructing query-specific state, rather than adding more summaries or graph edges.

Key Contributions

1. A dual-track memory system (SpeakerMem-R1). It combines traceable verbatim messages (System 1, the only verbatim layer) with four derived System 2 layers organized into person-level and group-level views, intended to support attribution, scope control, and state reconstruction in multi-party dialogue.
2. A query-time evidence procedure (Anchor–Separate–Resolve–Compose). Retrieval is conditioned on compiled query constraints — rows (which PERSON/GROUP rows to address), issue/event, head-versus-full temporal mode, and source–owner scope — and the two tracks use independent retrieval budgets.
3. A locally deployable trained Writer. A Qwen2.5-3B Writer is trained with SpeakerLevenshtein and speaker-conditioned GRPO on Add/Update/Noop writing decisions, while retrieval and answering modules stay frozen.
4. An analysis framework and broad evaluation. The paper derives analysis dimensions from question requirements and recurring error patterns across three multi-party benchmarks — member coverage, information attribution, personal/group scope, term and event disambiguation, and temporal updates with multi-hop reasoning — and evaluates the system on GroupMemBench, SocialMemBench, EverMemBench, LoCoMo, and targeted ablations.

Main Findings

• Highest binary accuracy on three multi-party benchmarks. Selecting the highest-accuracy SpeakerMem-R1 configuration per benchmark gives 47.9% on GroupMemBench, 69.2% on SocialMemBench, and 61.9% on EverMemBench. The paper states these are 3.3, 12.4, and 9.4 percentage points above the best mainstream framework results evaluated on each benchmark (best baselines: 44.6% from BM25, 56.8% from A-MEM, and 52.5% from BM25, respectively).
• Best reported result on the public EverMemBench leaderboard. Under the publicly reported EverMind-AI configuration (GPT-4.1-mini for answering, Gemini-3-Flash for judging), SpeakerMem-R1 answers 1,496/2,400 questions correctly, or 62.33%, versus approximately 60.08% for EverOS and 54.75% for RippleMem. Public totals are reconstructed from rounded category results, whereas SpeakerMem-R1's uses question-level records.
• Uneven category performance exposes remaining bottlenecks. On the nine EverMemBench behavior labels, SpeakerMem-R1 scores 93.43 Single, 24.10 Multi, 35.00 Temp, 87.06 Const, 75.64 Proact, 80.22 Update, 50.57 Style, 40.83 Skill, and 43.88 Role. Strong Single, Const, Proact, and Update results contrast with weaker Multi, Skill, and Role scores.
• Near-parity with full-context input on SocialMemBench. Full context is feasible only on SocialMemBench; the 69.2% SpeakerMem-R1 result is close to the 69.4% Full-context figure. The full system obtains MeanQ/MeanN of 0.710/0.693 with a network-level 95% CI of [0.659, 0.726].
• Reinforcement learning substantially improves a small Writer. On a controlled evaluation of 305 questions over 10 held-out SocialMem networks, Qwen2.5-3B Writer-R1 reaches 68.20% ± 0.66% (208/305), up from 57.38% ± 0.33% (175/305) for SFT (epoch 10) — a gain of 10.82 percentage points, or 33 additional correct answers. The LLM writer reference (DeepSeek-V4-Flash) reaches 71.48% ± 0.66% (218/305), leaving R1 3.28 points behind and at 95.4% of the LLM writer accuracy.
• The structured tracks and views are complementary. Ablations removing the verbatim track, all four System 2 layers, only the two GROUP layers (Interaction and Insight), or only the two PERSON layers (Core and Profile) all reduce accuracy on the three benchmarks relative to the full dual-track system; exact ablation numbers are placed in an appendix table not reproduced in the provided content.
• Supplementary retrieval (ASK) helps unevenly. With ASK disabled, accuracies are 47.0 / 69.2 / 60.5% on GroupMem / SocialMem / EverMem, versus 47.9 / 64.9 / 61.9% with ASK. The paper attributes this to supplementary retrieval recovering dispersed clues on GroupMem and EverMem but adding redundant evidence on SocialMem.
• Accuracy and token-F1 are not monotonic. The full system has lower SocialMem accuracy than the ASK-disabled variant but higher token-F1 (32.7 versus 27.4), because extra people or incorrect scope can invalidate an otherwise overlapping answer. Both metrics are reported for this reason.
• LoCoMo serves as a two-person boundary test. SpeakerMem-R1 achieves 70.85% (1,407/1,986) on LoCoMo. Category accuracies are 77.88 Single-hop, 41.13 Multi-hop, 70.72 Temporal, and 40.62 Open-domain, with ALL(Non-AD) at 67.34; multi-hop and open-domain questions remain weak. The paper notes this test does not replace multi-party validation.
• Cross-model robustness is partial. In the second model block (GPT-5.6-luna), SpeakerMem-R1 posts 42.7 on GroupMem, 64.4 on SocialMem, 71.3 on EverMem multiple-choice, 35.8 on EverMem open-ended, and 60.0 on the EverMem full set. In that same block, BM25 reaches 46.2 on GroupMem and HippoRAG reaches 75.1 on EverMem multiple-choice, both above the corresponding SpeakerMem-R1 numbers.

Methodology in Plain English

Writing phase. Messages arrive with text, speaker, time, and channel. Every message is written verbatim into a System 1 "Episodic" layer with no language model involved, so wording, speaker, session, turn, timestamp, and message identifier are always preserved. A trained Writer then reads each local segment together with the channel roster and the current heads of the derived layers, and emits one of three actions: Add, Update, or Noop. Deterministic code — not the model — validates the action and fills in the coordinates (provenance, timestamps, and links). An Update does not erase history: it appends a new node, inherits the owner/source/layer coordinates of the record identified by the cited entry_id, and links old and new state via links/superseded_by. Queries can then ask for the current "head" or the complete "full" chain.

Memory structure. System 1 is a single verbatim layer (per_speaker_episodic). System 2 has four derived layers: PERSON-scoped Core (stable identity, facts, stances, recurring behavior) and Profile (observations about a person, cross-person beliefs and perceptions); GROUP-scoped Interaction (cross-speaker events, relations, decisions and outcomes) and Insight (group norms, consensus, exceptions). Each derived record is a tuple of content, source, owner, scope, event, time, state, and reference. A key distinction is source versus owner: a self-report has source = owner, while "Alice believes that Bob has agreed" has source = Alice and owner = Bob.

Query phase. Project compiles the question plus the deterministic roster into constraints: rows, issue/event, temporal mode (head or full), and source–owner scope. For example, "every person" expands all roster rows, "the final decision" selects GROUP rows, and "Alice's view of Bob" fixes source = Alice and owner = Bob. The verbatim path retrieves a default pool of n = 40 candidates, ranks them, and can Expand to neighboring messages; when evidence is insufficient and ASK is enabled, one extra query term triggers a second recalled pool that is merged and reranked before truncation to the final top-k = 10. The structured path expands PERSON/GROUP rows, retrieves records by issue within each row, and takes k = 2 records per owner row with source-k = 1; empty derived rows fall back to that person's verbatim messages, and explicit empty rows are retained when no evidence exists. The two paths have independent budgets. The four named operations — Anchor, Separate, Resolve, Compose — summarize this behavior rather than adding a separate execution stage: Anchor retains source, owner, event, time, and provenance; Separate expands rows from the roster; Resolve distinguishes issues, parallel events, and current versus historical versions; Compose organizes both evidence paths by person, relation, and update chain.

Training the Writer. Rather than using an expensive prompt-based Writer, the authors train Qwen2.5-3B on its Add/Update/Noop decisions. The reward combines local structural signals with terminal question-answering gain. SpeakerLevenshtein combines token-level F1 with a normalized sequence-matching rate and performs coordinate-consistent one-to-one matching within owner buckets, so personal records, GROUP records, and cross-person observations cannot cancel one another; the potential is a weighted macro-average over owners plus a worst-owner term, with w1 = 0.80 and w2 = 0.20. The global signal is the question-answering gain of System 1 + System 2 over System 1 alone at the same memory state, so questions already answerable from verbatim memory are not credited to the Writer. Per-position returns combine validity, memory, and QA terms with w_valid = 0.20, w_mem = 0.45, and w_QA = 0.35, discounted by γ = 0.95. The method samples G = 8 writing trajectories for the same conversation, computes group-relative advantages only at aligned writing positions, and applies clipped GRPO updates to the Writer only; positions with no effective within-group variation are skipped. In the controlled study, SFT used epoch 10 and Writer-R1 used 30 steps.

Evaluation. All main experiments, ablations, and the controlled Writer study use the same query configuration (System 1 recall-n = 40 with final top-k = 10; System 2 k = 2/source-k = 1). Each baseline system uses its official code, recommended configuration, and official prompt; the authors standardize only the data, metric definitions, judge model, and evaluation interface, retaining speaker/source metadata where supported. The primary metric is question-level binary accuracy, with supplementary token-F1 and, for SocialMem, MeanQ/MeanN rubric aggregates.

Why This Matters

Impact on research. The paper reframes multi-party memory as a problem of attribution and state reconstruction, not merely relevance retrieval, and shows empirically that this reframing produces measurable gains over mainstream frameworks on three benchmarks (up to 12.4 percentage points). It also demonstrates that a 3B-parameter Writer trained with reinforcement learning can reach 95.4% of a large LLM writer's accuracy on 305 controlled questions while keeping retrieval and answering frozen — evidence that memory writing is a trainable, separable component rather than an unavoidable prompt-engineering cost. The reported weaknesses (Multi, Skill, Role on EverMem; Multi-hop and Open-domain on LoCoMo) map out concrete unsolved problems for the field.

Real-world applications (potential, as implied by the multi-party setting rather than stated as product claims):

• Assistants embedded in workplace group chat that must track who committed to what, across topics and over long time spans.
• Social-relation and group-norm aware agents that distinguish

Authors’ abstract

Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns, how individuals perceive one another, what information is shared by the group, and how states change over time. Recent studies on multi-party dialogue benchmarks show that existing general-purpose LLM memory systems tend to lose person and group relations or struggle to integrate clues distributed across members, groups, and time. Together, these issues reveal two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, we propose $\textbf{SpeakerMem-R1}$: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. To reduce attribution and update errors during structured memory construction while enabling local deployment, we train Writer-R1 with SpeakerLevenshtein and speaker-conditioned GRPO. On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, we achieves 62.33%, the best reported result among the latest state-of-the-art frameworks. It also achieves 70.85% on all 1,986 LoCoMo questions, which we use as a two-person long-term conversation boundary test. In a controlled evaluation of 305 questions, RL raises the SFT Writer's mean accuracy from 57.38% to 68.20%. We report both binary accuracy and token-F1, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.