MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

AI summary

Overview

Research area: Natural Language Processing — long-term memory in conversational LLM systems, and the evaluation of memory benchmarks against real user experience.

Technical level: Advanced. The paper is written in accessible prose, but it relies on familiarity with memory-augmented LLM architectures (long-context prompting, RAG, summarization), benchmark design, linear mixed-effects models, Spearman rank correlations, and user-clustered bootstrap confidence intervals.

Scope: A 4-month, 40-user, 7-condition deployment study of an AI diary companion, paired with a new 72-instance benchmark (MemUse) that scores memory use in naturally cued conversation rather than in direct fact-seeking questions.

What This Paper Is About

Memory systems for conversational LLMs are conventionally judged by "Direct QA" benchmarks: ask the model a fact-seeking question about a prior conversation and check whether it recalls fact X. The authors asked whether higher Direct QA accuracy actually corresponds to higher user satisfaction in a real long-term deployment, and found that it does not. They then built a new benchmark, MemUse, drawn from real moments in the deployment where users themselves cued the system's memory, to measure something different: whether the model naturally weaves prior context into a conversational response.

Key Contributions

1. A randomized 4-month deployment (40 users, 1,872 sessions, 7 memory conditions) in which Direct QA accuracy varies from 19.7% to 70.1% across conditions, yet user satisfaction does not change.
2. MemUse, a benchmark of 72 real user-cued memory moments (62 user re-provisions plus 10 user memory probes), scored by whether the model's conversational response actually draws on the referenced memory — a measure that, unlike Direct QA, is associated with satisfaction.
3. Evidence that retrieval and integration are dissociable: the same system answers 78.8% of MemUse items correctly under Direct QA but references those facts in conversation only 7.9% of the time, a 71-point gap.
4. A release of the deployment corpus and MemUse together with all judgments and scoring prompts.

Main Findings

- Capacity raises benchmark scores but not satisfaction. Across the 7 conditions (Summary-only, LC-10/50/100%, RAG-10/50/100%), average existing-benchmark QA accuracy climbs from 19.7% (Summary) to 70.1% (LC-100%), but every condition's mean satisfaction differs from Summary-only by less than 0.06 within-user SD. Numerically, LC-100% is not better than Summary (Summary sits at +0.02 SD and LC-100% at −0.04 SD). Because all conditions include the same summary, this is a null about the marginal value of LC/RAG capacity on top of a summary, not about memory in general.

- The null is not rating noise or latency. In a session-level linear mixed-effects model, response length, response specificity (proper-noun rate in the system's reply), and cross-session continuity each independently predict higher satisfaction, while memory condition does not. On the same 29-user subset, neither mean, max, nor min total response latency — nor their TTFT counterparts — correlates with within-user z-scored rating (|ρ| < 0.06, all p ≥ .07).

- Memory moments are rare in real conversation. Detection over 1,872 session transcripts with GPT-5.4 (temperature 0), human-verified, yielded 147 true positives (11 probes, 62 re-provisions, 66 proactive recalls, 8 reactions) at 95.5% precision. These arose in only about 3.5% of sessions and 1.4% of user turns — one detected memory moment per roughly 73 user turns — against 15–24% of user turns paired with evaluation items in MemoryBank, LoCoMo, LUFY, and RealTalk.

- At memory moments, integration — not Direct QA — tracks satisfaction. On the 48 reactive memory-moment sessions (after filtering), Direct QA accuracy is uncorrelated with session-level satisfaction (ρ = +0.03), while Natural Integration is associated (ρ = +0.29, p = .046), with successful integration carrying +0.56 within-user SD higher satisfaction (user-clustered bootstrap 95% CI [+0.12, +0.98]). The two metrics also disagree on condition rankings: LC-100% ranks highest under Direct QA (78.8%) and lowest under Natural Integration (22.2%).

- Retrieval and integration are dissociable within the same model and context. MemUse Direct QA (78.8%) versus MemUse Reference (7.9%) is a 71-point gap with the same model and context on LC-100%. Per-instance, the Spearman correlation between Direct QA correctness and whether the same fact is referenced in the natural response is ρ = −0.009. Condition differentiation invisible in natural responses (22–28% Natural Integration) emerges clearly in Direct QA (45–79%).

- The dominant failure mode is a generic response. Classifying all 72 LC-100% responses: 38.9% are generic, 23.6% hallucinate recall with fabricated details, 26.4% show partial (gist-only) recall, 2.8% are honest admissions of not remembering, and only 8.3% fully integrate prior details.

- The dissociation replicates across stronger models. Running MemUse Direct QA and Natural Integration across all 7 conditions on GPT-5.5 and Gemini 3.1 Pro (judge held fixed at GPT-5.4-nano), Direct QA rises by 32–34 points across capacity for every model, while Natural Integration spans at most 8.2 points within any model and never tracks capacity. Stronger models lift the integration baseline (Natural Integration at Summary: 23.6% / 32.9% / 53.4% for GPT-4.1-mini / Gemini 3.1 Pro / GPT-5.5).

- Modern memory systems raise integration but do not close the fact-level gap. Mem0 lifts Natural Integration to 58.3% and Letta (MemGPT) to 56.9%, versus 22.2–27.8% across the seven provisioning conditions. Yet Direct QA versus Reference is 41.5% vs. 7.3% (Mem0) and 61.4% vs. 10.8% (Letta) — at least 34 and 51 points respectively are answerable on direct query but absent from the natural reply. This is an offline evaluation, so the authors make no claim that higher integration would raise satisfaction.

- The bottleneck is generation, not retrieval. Four prompt-level interventions (CoT, Cue-aware, Two-step, Query-rewrite) on GPT-4.1-mini across all 7 conditions produced no capacity-sensitivity: every integration variant spans at most 21.9 points across conditions versus Direct QA's roughly 33-point span. In the Two-step ablation, even when the extraction step names the ground-truth details, the conversational generation fails to reference them 77% of the time (37/48 cases on LC-100%).

- Proactive recall carries no symmetric satisfaction reward and a clear timing risk. Of 70 detector-flagged candidates (66 verified true positives, 64 in rating-valid sessions), 46/70 are fully grounded, 21 partial, 3 hallucinated; timing is 32 invited, 29 tangential, 9 mistimed. Grounding shows no satisfaction signal (ρ = −0.05, against the reactive baseline ρ = +0.29); neither grounding nor appropriateness predicts satisfaction (both |ρ| < 0.1). The 8 mistimed cases sit at z̄ = −0.48 SD. Users continue the recalled topic in only 30% of cases (19/64), and appropriateness does not predict whether they do (ρ = −0.02).

- Observational corroboration of the integration failure. Across all 1,270 rating-valid sessions, user word count interacts negatively with cross-session continuity (β = −0.086, p < .001): in high-continuity sessions, longer user utterances predict lower satisfaction — consistent with users forced to re-supply context the system has lost.

Methodology in Plain English

The deployment. Forty proficient English speakers (approximately half based in East Asia, the remainder from North America, Europe, the Middle East, and other regions; 37F, 3M; ages 20s–60s) interacted daily from November 2025 to February 2026 with "Luke," an AI diary companion built on GPT-4.1-mini. Users accessed it through a mobile-and-desktop web interface where they could review summaries of prior conversations before each session. They were required to interact at least one hour and write about 2,000 English words per month, on any topic. The study produced 1,872 sessions (21,575 turns), with users rating each session on a 1–7 satisfaction scale.

The seven conditions. Each session was randomly assigned to Summary-only, LC-10/50/100%, or RAG-10/50/100%. Summary-only supplies only a summary of prior conversations; every other condition adds capacity on top of that same summary. LC-k% prepends the top k% most important prior turns to the conversation history; RAG-k% retrieves the top-10 turns from a vector index over that same top-k% pool. Importance is scored by a RoBERTa model finetuned on human annotations from the LUFY dataset. Conditions were randomly rotated for balanced exposure.

The analysis sample. A predefined filtering pipeline excluded 11 users — 9 with near-constant ratings (SD ≤ 0.5 on the 1–7 scale) plus 2 flagged for template/AI-paste content — yielding 1,270 sessions from 29 users, analyzed with within-user z-scored satisfaction.

Detecting memory moments. The researchers used GPT-5.4 (temperature 0) on each transcript to find four event types via explicit linguistic signals: user memory probes, user re-provisions, system proactive recalls, and user memory reactions. Human annotators independently verified each candidate against the trigger utterance, surrounding context, and the detector's rationale. Reactive (user-cued) moments have a clear per-instance target and can be replayed across systems; proactive recalls have no canonical correct next turn, so only reactive cases were packaged as a re-runnable benchmark.

The MemUse benchmark. 62 re-provisions plus 10 probes make 72 instances. Ground-truth facts were extracted from source sessions and decomposed into 3–5 fact-seeking questions each (316 total, average 4.4 per instance), phrased in the same format existing QA benchmarks use. Three metrics are scored on the same reconstructed context: Natural Integration (the primary metric, a per-instance binary judgment by GPT-5.4-nano at temperature 0), Direct QA (each question asked literally with the same reconstructed context), and Reference (whether the facts elicited by Direct QA actually appear in the natural conversational response).

Judge validation. On a stratified 56-item subset, human–human agreement on Natural Integration was substantial (Cohen's κ = 0.57, 95% bootstrap CI [0.34, 0.78]), and the LLM judge matched humans on positive rate (LLM 51.8% vs. humans 51.8% / 55.4%). Per-question judgments showed strong three-rater agreement on the full 316-item set (Fleiss' κ = 0.65; pairwise human–human Cohen's κ = 0.70).

Modeling. All inferential models are linear mixed-effects models with user random intercepts; rank correlations are Spearman's ρ, and effect-size confidence intervals are user-clustered bootstrap.

Why This Matters

The paper argues that a widely used evaluation format may be measuring the wrong capability. If benchmark scores can rise by 50 points while user satisfaction stays flat, then leaderboard position on existing memory benchmarks is not a reliable proxy for whether a memory system helps people. The authors propose that benchmarks should score integration under user-cued conditions instead, and they release the deployment corpus, MemUse, and all judgments and scoring prompts to make that possible. The study also provides a mechanistic localization: the failure is downstream of retrieval, in the conversational generation step itself, and persists under four targeted prompt interventions and across two frontier models.

Real-world applications:

• Personal assistants and companion apps that are expected to remember a user's history, where a generic empathetic reply that ignores a user's cue can feel worse than no memory feature at all.
• Therapy bots and diary or journaling companions, where the paper's deployment setting directly applies and where mistimed proactive callbacks measurably depress satisfaction.
• Memory-layer vendors and agent frameworks (the paper evaluates Mem0 and Letta/MemGPT), who need an evaluation that distinguishes retrieval quality from integration quality.
• Long-horizon agent architectures that build in reflection or recall loops, where the paper shows proactive recall carries no symmetric satisfaction reward and a clear timing risk.

Industry relevance: Teams shipping memory features currently optimize against Direct QA accuracy, cost, and latency. This paper reports per-condition cost and latency (for example, LC-100 reaches 22.0× the Summary baseline in input-token growth and 3.27 s mean total response latency versus 1.93 s for Summary) and finds that the capacity-driven spend does not move satisfaction. The practical implication is that investment may be better directed at generation-side integration than at expanding the retrieved context.

Future Directions

1. Build and validate a graded integration metric. Natural Integration is a binary judgment; the Reference score and the failure-mode taxonomy capture partial integration but were not designed or validated as ordinal measures. A human-validated graded metric remains future work.
2. Detect implicit memory moments. MemUse relies on explicit linguistic signals, so the roughly 3.5% rate is a lower bound. Implicit moments where memory matters but is not flagged in the dialogue surface are missed by construction, and the authors note the true prevalence may be substantially higher.
3. Test whether raising integration raises satisfaction causally. The integration–satisfaction result is small-N and observational (48 memory-moment sessions across 16 users), and the Mem0/Letta comparison is offline, so the authors make no claim that their higher integration would raise satisfaction. Randomized or interventional evidence is needed.
4. Study proactive recall timing and generalize beyond the diary setting. Proactive recall has no symmetric reward and mistimed callbacks carry a negative signal; the authors also note the diary setting may not generalize to other longitudinal interaction types such as task-oriented assistants, and the sample is predominantly female (92.5%).

Target Audience

Researchers and engineers working on memory systems for conversational LLMs, long-context modeling, and retrieval augmentation; benchmark designers and evaluation researchers interested in connecting automatic metrics to human experience; product teams building personal assistants, companion apps, or memory layers who need to decide where to invest; and HCI or longitudinal-deployment researchers studying human–AI relationships over months rather than single sessions. Readers need some grounding in LLM system architecture and quantitative analysis, though the central argument is stated in plain language.

Authors’ abstract

Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.