When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model

AI summary

Overview

Research area: Conversational/agent memory for large language models — specifically the debate over whether memory systems should store LLM-extracted facts or raw conversation turns, and how the retrieval (context) budget changes that answer.

Technical level: Advanced. The paper is written as a pre-registered empirical study with non-inferiority testing, exact McNemar tests, Holm correction, token-matched system comparisons and a blind human audit; readers need comfort with statistical inference and retrieval-system terminology.

Scope: A pre-registered, held-out evaluation of whether raw turns selected by a single call to a typed decision model (Jev) can substitute for an LLM-extraction memory (engram v2) at a tight, matched token budget — and how the value of reranking changes as the budget grows.

What This Paper Is About

Published results disagree about conversational memory: extraction-based systems report gains from distilling conversations into facts, while recent studies find that raw history with good ranking does as well or better — and those recent studies also disagree about whether ranking matters at all. The authors propose that the context budget (how many retrieved items the answer model actually reads) explains part of the disagreement, and they test the first half of that claim under a pre-registered plan. Their goal is to determine whether raw turns, selected by a single reranking call, are non-inferior to a strong LLM-extraction memory at a tight and matched budget, and to measure how the rerank's advantage changes as the budget grows.

Key Contributions

1. A within-study budget result. Reranking's gain over similarity search shrinks as the budget grows: from +17.4 to +1.5 points on LoCoMo and from +9.1 to +1.1 on LongMemEval, as the answer model reads three and then twenty of 30 candidates. The authors offer this as an interpretation for why SmartSearch finds ranking decisive and Fidelity Before Structure finds it marginal.

2. A pre-registered non-inferiority test in conversational-memory evaluation, run on held-out conversations, with a second answer model and blind human grading. It bounds what extraction adds at a tight budget: under the worst grading applied, extraction adds at most 4.7 points; human grading puts the difference between −1.7 and −2.6 points.

3. An answer-level, matched-context evaluation of a typed decision model as the selector. At matched context, Jev is non-inferior to a gpt-4o-mini listwise reranker (S4, lower bound −2.0) at about a third of the latency, and more accurate than Jev-Mem's multi-call Jev graph traversal at matched context (S2).

4. Diagnostics. A per-category decomposition of the selection ceiling — shortlist misses (11.5% of questions) and rerank drops (9.2%), with temporal evidence dropped most at the rerank — plus a finding that the LLM judge's leniency interacts with answer length, so judge–human agreement differs by system.

Main Findings

• Primary test (H1) passes. On 778 scored questions from five held-out LoCoMo conversations, Turns + Jev at k=6 (265 tokens per question) scored 77.0% against engram v2 at k=3 (251 tokens) at 77.5%, a difference of −0.5 points with a one-sided 95% lower bound of −3.0, above the registered −5-point margin. The one-sided p-value is 0.0017.

• Human grading narrows but does not overturn the result. Under strict human grading the difference is −1.7 points (bound −3.9); under lenient grading −2.6 points (bound −4.7, two-sided interval [−5.1, −0.03]). Under lenient grading engram v2 is more accurate, still inside the margin.

• Extraction costs far more to write. Raw turns cost 3,061× less to write: engram v2 costs $1.865 per 1,000 turns, Turns + Jev $0.00061 (embeddings only).

• The rerank closes most of the gap. At the H1 budget, one rerank call closes G = 94% of the accuracy gap between Turns + cosine (68.6% at its matched k of 6) and engram v2 (77.5%).

• Secondary tests S1–S7. After Holm correction, S1 (Turns + Jev 77.2% vs Turns + cosine 59.9%, p = 2.7e−28), S2 (77.0% vs Jev-Mem 70.6%, p = 4.3e−5), S3 (77.2% vs mem0 68.5%, p = 1.7e−6), S4 (non-inferiority vs Turns + LLM, difference −0.4, lower bound −2.0) and S7 (66.8% vs 57.7% on 470 LongMemEval questions, p = 1.3e−6) are rejected. S5 (30 knowledge-update questions, 70.0% vs 70.0%, p = 1.00) and S6 (70 questions, 68.6% vs 65.7%, p = 0.77) are not.

• Extraction wins at generous budgets. engram v2 at k=20 was the most accurate system measured: 82.4% with 1,238 tokens. Jev-Mem at k=40 followed at 80.3% with 1,987 tokens; mem0 at k=20 scored 78.7% with 839 tokens; full context scored 78.3% at 23,631 tokens. Turns + Jev stays near 77.6% at any k.

• Reranking lowers correct abstention. On LongMemEval, Turns + Jev at k=3 abstained on 43.3% of the 30 abstention questions, the same as Turns + cosine at k=3; at k=20, Turns + cosine abstained on 70.0% against Turns + Jev's 63.3%.

• Model robustness. With Llama 3.3 70B Instruct answering from the same contexts, H1 still passes: Turns + Jev 75.3%, engram v2 75.6%, difference −0.3, one-sided bound −2.9. S1 also passes (74.9% vs 57.6%, p = 5.8e−26). Of 3,112 rebuilt contexts, all but 4 matched their recorded token counts exactly.

• Latency. At k=3, Turns + Jev's read p50 is 273 ms and p90 359 ms (one Jev call), against 816 ms and 1,107 ms for the LLM reranker, and 276/319 ms for engram v2.

• Cost of full context. On LongMemEval, full context read 209× the tokens at 29× the cost per question ($0.0169 against $0.00058), scoring 63.0% against 66.8% for Turns + Jev at k=3. That comparison is descriptive (p = 0.13).

• Where the ceiling comes from. Across all nine held-out conversations (1,388 questions), all evidence turns were in the 30-turn shortlist for 76.9% of questions and at least one for 88.5%. About 11.5% of questions are lost to the shortlist and another 9.2% to the rerank. Open-domain evidence reaches the shortlist least often (63.0%) and is dropped most often (31.4%); temporal evidence is dropped by the rerank for 22.7% of questions.

• A post-hoc wider read path. Turns + Jev (wide), tested once after the registered results, takes a 150-turn shortlist with no cut-off and scored 81.5% at k=47 (2,000 tokens, matched to Jev-Mem at 2,000 vs 1,987 tokens). All-evidence recall rose to 91.9% and rerank losses fell to 0.9%. The authors call this a hypothesis for new data, not a finding.

• Judge leniency interacts with answer length. Human agreement with the judge was 81% under the strict mapping and 79% under the lenient one on the 142 judge-discordant questions. Agreement was 77% on engram v2's list-style answers under the lenient mapping, against 82% on Turns + Jev's short answers, which is why human grading widens the gap.

Methodology in Plain English

The authors wrote down their plan, primary test, margin and interpretation before running anything (deposited at 10.5281/zenodo.22970745), then deposited an amendment adding a full LongMemEval run and a second answer model after one batch of results but before any H1 result was seen (10.5281/zenodo.22977848).

They compared memory systems on conversations never used for development: five LoCoMo conversations (3,122 turns, 778 scored questions, plus 209 adversarial) and LongMemEval, first a registered sample of 70 questions and then all 500 questions (470 scored, 30 abstention). Development used a single LoCoMo conversation.

All systems were compared at matched context — the same amount of text the answer model reads. The extraction comparator (engram v2) ran at its natural setting of k=3, and Turns + Jev ran at the k whose rendered token count was closest, ties going to the larger k. Token counts came from a retrieval-only sweep saved before Turns + Jev answered.

The selector is Jev, a typed decision model that answers a fixed-option question with a probability in one short request and does not generate text. Reading starts from a 30-item cosine shortlist; Jev scores every shortlisted turn for relevance, turns above 0.5 are kept, a 10-turn cosine floor follows them, and the answer model reads the first k.

The primary test asks not whether the two systems differ, but whether Turns + Jev is at most 5 points worse — a non-inferiority test on paired, per-question correctness with a one-sided 95% bound. Seven secondary tests use Holm correction at family-wise 0.05. Robustness came from re-answering with Llama 3.3 70B Instruct and from the first author grading, blind to system and judge label, both answers to each of H1's 142 judge-discordant questions.

Why This Matters

This paper shows that a contested question in agent memory — do you need LLM-extracted facts? — may depend on a variable that has often gone unstated: how much context the answer model is allowed to read. It also demonstrates that pre-registration and non-inferiority testing, standard in clinical trials, can be applied to conversational-memory evaluation, which makes reported memory gains harder to overstate. The authors are explicit that the contribution is the test, the within-study budget result and the typed selector, not a new architecture.

Real-world applications suggested by the findings:

• Long-running chat assistants, where storing raw turns with a cheap selector avoids per-message extraction costs of $1.865 per 1,000 turns versus $0.00061.
• Latency-sensitive agents, where one short typed decision (read p50 273 ms) replaces an LLM listwise reranker (816 ms) at comparable accuracy at matched context.
• Cost-constrained or high-volume deployments, where reading 139 tokens per question instead of 23,631 (full context) changes per-query economics by orders of magnitude.
• Systems that must abstain, given the diagnostic that reranking lowers correct abstention (43.3% versus 70.0% at k=20 on LongMemEval) — relevant where a wrong answer is worse than no answer.

Industry relevance: the results give engineering teams a budget-dependent rule of thumb — at tight budgets, spend on selection rather than write-time extraction; at generous budgets, compact extracted facts are more accurate — and quantify the cost and latency trade-offs of each choice.

Future Directions

• A wider read path. Turns + Jev's registered read path tops out near 77.6% because its cutoff and 10-turn cosine floor often cannot fill k=20 (it reads 496 tokens at k=20 against Turns + cosine's 826). The post-hoc wide variant reached 81.5% at k=47 with 91.9% all-evidence recall, but the authors call this a hypothesis that needs new data.

• Testing the budget explanation against more extraction systems. The extraction advantage observed at generous budgets is confounded with the selector's own read-path ceiling (k=20 differences mix the budget with that ceiling), and engram v2 was not run on LongMemEval at all, so these results cannot support a claim that Turns + Jev matches LLM-extracted memory on long histories.

• Disentangling judge leniency from answer length. Judge–human agreement differed by system (77% on engram v2's list-style answers under lenient mapping against 82% on Turns + Jev's short answers); the interaction between leniency and answer length is what this paper adds and what a follow-up would need to isolate.

• Establishing equivalence, not just absence of detection. S5 (30 knowledge-update questions) and S6 (70 questions) detected no difference, which is too few questions to establish equivalence. S3 also carries a caveat: mem0's extraction was served through OpenRouter, about half of it by Azure.

Target Audience

Researchers and engineers working on LLM agent memory, retrieval and reranking, and long-context question answering; evaluation specialists interested in pre-registration, non-inferiority margins and human-audited LLM judging; and practitioners who must choose between write-time extraction and read-time selection under real cost, latency and token-budget constraints. Readers should be comfortable with paired statistical tests and retrieval terminology.

Authors’ abstract

Does conversational memory need LLM-extracted facts, or is selecting the right raw turns enough? Published results disagree. Extraction-based systems report gains from distilled facts. Recent studies find raw history with good ranking does as well, but disagree about whether ranking matters. We ran a pre-registered study on held-out LoCoMo conversations and LongMemEval. At a tight budget on LoCoMo, raw turns selected by a single call to Jev, a typed decision model, are non-inferior to an LLM-extraction memory (one-sided 95% bound -3.0 points against a -5-point margin). Blind human grading narrows the margin but does not change the result. Raw turns cost 3,061 times less to write, and the result holds with a second answer model. Within this study, reranking's gain shrinks as the budget grows. It adds 17.4 points on LoCoMo and 9.1 on LongMemEval when three of 30 candidates are kept. At generous budgets it adds 1.5 and 1.1, and extraction systems are more accurate. This suggests why published results disagree. At matched context, Jev selects as accurately as an LLM reranker (non-inferiority bound -2.0) at a third of the latency, and more accurately than a multi-call graph traversal. Reranking lowers correct abstention. Plans, code and graded answers are released.