RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

AI summary

Overview

Research area: Evaluation of long-term conversational memory and personalization in AI companion systems, with a focus on benchmarking how well a system reasons over months of real human conversation.

Technical level: Intermediate. The core arguments are conceptual and readable, while the results involve retrieval metrics (hit@k, MRR, AUROC), context ablations, and reconstruction F1 scores.

Scope: The paper releases RealCompanion, a benchmark built from ten real human–AI companion relationships containing 27,218 messages spanning up to 120 days, and uses it to measure when the past is actually needed, whether systems can detect that, and how well agent systems reconstruct a person's profile and persona.

What This Paper Is About

AI companions that talk with a person for months are expected to remember what was said, infer who the person is, and judge when earlier messages matter to the message in front of them. Existing benchmarks cannot test this honestly because they generate the person, generate the conversations, and write the questions, which settles in advance what matters and what the person is like. RealCompanion instead releases ten real relationships with an AI companion and derives the labels from the conversation itself, with every label citing the messages it rests on.

Key Contributions

1. A released corpus of real longitudinal relationships. Ten participants, 27,218 messages, spans of 36 to 120 days, released with consent and anonymized. Per participant, the release contains the conversation plus four derived files: a profile, a persona, a chat ground truth, and a question set. The two heaviest relationships hold 73% of all messages.

2. Chat labels that carry checkable reasoning traces. Every chat item records the reasoning trace that produced it, written in the same call as the decision it explains and checked stage by stage against the conversation. Each stage reads only what earlier stages wrote, so a label can be contested at the step that produced it. The derivation is done by models and audited by people, with agreement reported (Appendix D).

3. Three evaluation tracks over one corpus. The reconstruction track asks a system to write the profile and persona from the full history. The chat track scores replies to messages the participant actually sent. The question track scores answers to 3,312 authored questions across 16 categories, of which 3,235 are scoreable and 63 are marked unanswerable.

4. A formal task definition with stated corpus conditions. The paper defines the task over an arbitrary subject record and states seven conditions a corpus must meet to measure it (Appendix A), placing RealCompanion alongside prior benchmarks on five properties in Table 1.

Main Findings

• The past is rarely needed, and far away. On the natural-rate proportional stratum, 3.4% [2.5, 4.6] of participant messages carry a verified memory demand under the "recorded" reading and 1.3% [0.8, 2.1] under the "strict" reading. Two procedures over the 11,369 participant messages outside that sample agree, giving 3.2% [2.9, 3.5] from the census and 1.73% [1.45, 2.06] from a re-reading under the Stage B instruction. When memory is required, the furthest required message sits a median of 2,157 messages back, with an upper quartile of 5,114 and a maximum of 12,542; even the nearest required message sits a median of 450 messages back.

• Pooled measures mislead. A recency window finds the required message for 95.9% of the 1,477 chat probes with a reference set, of which 72.6% are answerable from the recent thread, but for only 2.2% of those that need memory. At the natural rate, 96% of the gain from supplying recorded evidence comes from messages that need none. Three of the ten participants contribute no memory-bearing item under the recorded reading, and five under the strict one.

• Locating distant evidence defeats standard scorers. On the distant items, hit@5 is 0.005 for Random, 0.022 for Recency, 0.248 for BM25 and 1.000 for Oracle, with the broken oracle at 0.000 as a control must be. On dependent items BM25 reaches 0.347.

• No detector tried can tell when memory is needed on real messages. A detector reading the probe and its three preceding messages reaches AUROC 0.530 [0.488, 0.572] against probes whose referent is absent and 0.548 [0.488, 0.609] against cold opens. Scored against its own target it reaches 0.601 [0.527, 0.676], and its threshold catches 6 of the 40 probes that need memory while raising 40 false alarms.

• Authored questions leak the cue. A detector shown three retrieved messages reaches 0.802 [0.742, 0.857] on the question track against 0.546 [0.490, 0.605] on chat probes, and a keyword scorer's confidence reaches 0.696 [0.629, 0.758]. An authored question shares more words with its evidence than a message sent in conversation does.

• Most of the benefit from context lands where it is not needed. Replacing the last three messages (C1) with the recorded messages (C2) raises content match by +0.140 [0.126, 0.167] over both strata and by +0.108 [0.077, 0.159] on the proportional stratum. That proportional gain splits into +0.004 for probes that need memory and +0.104 for those that do not. On the 167 items whose reference reply depends on the recorded messages, the same replacement is worth +0.455 [0.319, 0.588].

• Which messages are supplied matters as much as how many. With budget, heading and order held fixed, ten messages chosen by BM25 score 0.189 [0.156, 0.247] below the ten preceding ones, and match no context at all (1.426 against 1.431).

• Framing changes the decision to look. Labeling the same ten messages "retrieved memories" instead of "earlier turns from this chat" raises memory use by ten to fourteen points on every population tested, including probes that need no memory. Without context, the misfire rate is zero across the 434 cold opens and the 120 probes whose verified locus is none; with context it rises to 29.2% when messages were selected for the probe and 60.8% when ten were supplied without regard to it.

• Three agent systems reconstruct the persona at equal quality and very different cost. Claude Opus 5.5, Codex GPT-5.6-sol and Antigravity running Gemini 3.8 Flash all reach F1 of 0.686, 0.688 and 0.701 across three runs on each of the ten participants, with run agreement of 0.935, 0.932 and 0.946, while processing 546M, 18M and 36M tokens respectively — a 31-fold difference in cost. Precision of 0.56 to 0.59 against recall of 0.86 to 0.89 means each system adds roughly seventy to eighty further fields for every hundred it recovers, and the three agree with one another at 0.86 to 0.88, above their agreement with the released file.

• Context coverage scales, but only with very large budgets. Coverage of the required message rises from 0.109 at 8k tokens to 0.453 at 128k and 1.000 at one million tokens, where every history fits.

Methodology in Plain English

The researchers started from conversations people actually had with a companion application, taken from the application's operational store so that each message stands where the participant sent it. The release is anonymized: every message was rewritten with direct identifiers replaced by consistent surrogates, keeping what was said, why, and the context it answers.

From each conversation they derived four things. A profile records what the person stated, claim by claim, with a confidence, a window over which the claim held, and the messages that establish it. A persona records how the person thinks, feels and decides, including named working hypotheses with their grounds. A chat ground truth pairs each probe — a message the participant sent verbatim — with the context a correct reply may draw on, a reference reply written from that context alone, and a reasoning trace. A question set poses the same decisions as written questions.

The chat ground truth was built in five stages: resolve what the probe points at, verify that evidence exists in the source conversation and drop what fails, classify the item by one of nineteen fixed rules, write the reference reply from the recorded context alone, and validate that the reply is supported. Verification can only remove a reference and never add one, so verification errors can only lower the measured demand rate. The derivation discarded a great deal of what it first proposed: Stage A proposed a distant referent for 806 unused messages and Stage B verified 422; a final pass matched 1,544 profile claims to referents and kept 752. The tier differs from a pre-verification surface signal on 685 of the 1,533 items.

Probes were sampled in three strata. A proportional stratum (1,169 scoreable items of 1,227) is a random sample and alone answers how often the past is needed. An enriched stratum (364 of 373) over-samples deep-history items so conditional statements can be made. An abstention control of 434 session openers needs nothing prior. Retrieval was scored with five methods (Random, Recency, Recency-user, BM25, Oracle) plus three substitution controls (offset permuted, position randomized, broken oracle), and responses were compared across five context conditions that differ only in what precedes the probe in the model's input. Comparison used one judge scoring all five conditions in a single call, with human adjudication agreement reported in the appendix.

Why This Matters

Impact on research. Earlier memory benchmarks build dialogs around the memory they test, declare personas in advance, and never revise a scripted history. RealCompanion shows what changes when those substitutions are removed: memory demand falls to a few percent of messages and concentrates in the longest relationships, so pooled retrieval scores largely report corpus composition rather than capability. The paper's Proposition 2 formalizes this, and the audit trail makes every label contestable.

Real-world applications:

• Companion and consumer chat products whose memory layers were tuned against generated dialogs and a question-shaped retrieval task.
• Long-horizon personal assistants that must decide, message by message, whether to search months of history at all.
• Support and care-oriented agents, where the corpus shows people disclose family, health and worries but almost never state a census category.
• Agent evaluation pipelines that need to price reconstruction quality against compute, given the 31-fold token difference at equal F1.

Industry relevance. The paper reports that memory layers now shipping inside products were tuned against generated-conversation benchmarks, and that three frontier and flash-tier agent systems reconstruct a persona equally well at very different cost. It also reports a privacy measurement: given a sample of a participant's writing and the ten released histories, a stylometric classifier using no content words matches them at 83.0% and a language model at 98.0% against a chance rate of 10%, so removing identifiers does not remove authorship.

Future Directions

• Building detectors that decide when memory is needed on real messages, given that the best detector tested sits near chance on the proportional stratum.
• Determining whether a memory-need signal learned on authored questions transfers to conversational probes, since authored questions leak the cue at 0.802 AUROC against 0.546 on chat.
• Separating framing effects from real capability, because relabeling identical messages as memories shifts use by ten to fourteen points and the C1/C2 contrast also carries an unseparated heading difference.
• Extending the corpus beyond ten relationships and one companion application, and testing whether the finding that memory demand is a property of relationship depth holds where the two heaviest relationships hold 73% of all messages.
• Resolving the low-demand measurement problem generally, since 96% of the pooled gain from recorded evidence lands on probes that need no memory.

Target Audience

Researchers and engineers working on long-term conversational memory, personalization and agent evaluation; builders of companion, assistant and support products who need to judge whether their memory stack is solving the problem it claims to solve; and privacy and ethics reviewers interested in what anonymization of conversational corpora does and does not guarantee.

Authors’ abstract

A companion that talks with a person for months should come to understand them. It should remember what they said, infer who they are, and know when the past bears on the message in front of it. Testing this requires a real person's record, and such records are private, so benchmarks generate the person and the questions and settle in advance what matters. We release \bench, ten real relationships with an AI companion: 27,218 messages over up to 120 days, released as the conversation and four files derived from it, a profile, a persona, a chat ground truth and a question set, each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation. Three findings follow. First, the past is rarely needed and far away. Pooled measures mislead: a recency window finds the required message for 95.9\% of probes and 2.2\% of those that need memory, and at the natural rate 96\% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector we tried can tell when memory is needed on real messages, authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 at a 31-fold difference in cost.