Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

AI summary

Overview

Research area: Evaluation methodology for LLM-agent memory systems, at the intersection of long-term memory benchmarking, temporal knowledge representation, and memory security.

Technical level: Advanced. The design ideas are explained plainly, but the evidence rests on paired cluster-aware significance testing, McNemar tests, sign-flip randomization, and risk/odds ratios that assume familiarity with benchmark methodology.

Scope: The paper introduces a synthetic, ground-truth-first corpus for evaluating agent memory across long (multi-week) histories and uses it to benchmark five memory architectures against a no-memory control, revealing that architecture rankings invert as history length grows.

What This Paper Is About

Standard agent-memory benchmarks build conversations first and then extract answer keys from that text, a process the paper says produces label errors and lets real-world entities leak into model pretraining. These benchmarks also overwhelmingly test short interaction histories, the regime where simply reading the whole transcript is still competitive. The paper's goal is to invert that pipeline, generating verifiable ground truth before any text exists, and then to run the benchmark long enough to see whether the memory systems that win short-horizon tests are still the systems a long-lived assistant needs.

Key Contributions

1. A ground-truth-first evaluation instrument for longitudinal, multi-channel, adversarial agent memory, combining per-fact validity intervals, volatility classes, a sent/received trust distinction, injection probes inside a benign harness, and as-of-date question sets. The paper reports none of the five features appeared in the benchmarks it surveyed as of July 2026.

2. A controlled comparison of five memory architectures plus a no-memory control across two history horizons and three stochastic replicates, under a fully logged protocol totaling 52,797 LLM calls including validation.

3. The tenure crossover and its mechanism: a measured inversion of memory-architecture rankings with history length, with the cause isolated as eviction of early-epoch content on the budgeted curated-map backend.

4. A write-quality result with a fact-level audit linking write-stage errors to downstream question-answering failures, plus three further reported findings: injection resistance tracking whether provenance boundaries survive representation, a confident-confabulation failure mode motivating read-time abstention, and Veracium, an open-source provenance-aware memory library (pip install veracium) released with the corpus generator and evaluation harness.

Main Findings

• The short horizon shows specialization, not a winner. With 275 questions across 14 users and 3 replicates (825 judged answers per backend), the no-memory control scored 125/825 (15.2%); the curated map 777/825 (94.2%), the vector backend 751/825 (91.0%), the graph 769/825 (93.2%), hybrid v1 738/825 (89.5%), and the layered hybrid v2 799/825 (96.8%). The paper notes vector's single category win (conflict-resolution, 42/42 vs. 41/42 for both map and graph) is one answer inside replicate noise.

• Naive composition of two curated stores fails. Hybrid v1, a deterministic compile of the graph into a map, scored 738/825 (89.5%), below both of its parents, because the graph stored no narrative for the compiler to use and string-joins amplified extraction noise the parents had been curating away.

• The layered architecture leads the memory systems at the short horizon. Hybrid v2's 96.8% was first by 22 answers, with no significant primary-judge difference from the full-rendered-history baseline. The paper reports significant paired margins vs. the map (45/23 discordant, exact McNemar p = 0.010) and vs. the graph (37/7, p < 10⁻⁴), but notes that under complete cross-family re-judging the graph margin holds (p = 0.018) while the map margin loses significance (p = 0.15).

• Rankings invert with history length (the tenure crossover). On 6 users × 3 replicates (n = 324 per cell) at weeks 3, 6, and 9: the curated map went 81.2 → 79.6 → 78.4, the vector backend 75.3 → 68.8 → 74.7, the graph 75.9 → 79.0 → 90.4, hybrid v2 80.2 → 85.8 → 93.2, and the no-memory control 16.0 → 16.4 → 16.7. The map led the graph by 17 answers at week 3 and trailed by 39 at week 9, with all six users individually showing graph ≥ map at week 9.

• The crossover is judge-robust. The architecture × tenure interaction (graph − map at week 9 minus the same at week 3) was +17.3 pp on average under the primary judge, positive for five of six users (per-user +37.0, +16.7, +9.3, 0.0, +20.4, +20.4; exact six-user sign-flip p = 0.063, described as resolution-limited). Under complete cross-family re-judging it strengthened to +24.1 pp, positive for all six users, with exact p = 0.031, the test's minimum.

• The mechanism is eviction of old content. On the early-chapter probe (the three "-early" question types pooled, n = 54 per cell), the map fell 96.3% → 72.2% as budget eviction consumed the oldest material (13/0 discordant items week 3 → 9, p = 2 × 10⁻⁴), while the graph rose 94.4% → 100% and hybrid v2 held 100% at every checkpoint. Vector's unbounded pool also failed the probe (88.9% → 70.4%). A full-rendered-history baseline held the probe at 160/162 across all checkpoints.

• Write-stage quality is a strong correlate of downstream quality. A write-stage audit (66 audits) scored the two-stage curated writer at 84.5% fact recall and 4.0 errors per audit versus the single-call graph extractor's 74.6% and 8.5, bought with roughly 4.6× the write-path tokens. Across 1,092 observations, questions whose underlying fact was written weakly missed 24.2% of the time versus 1.6% on clean writes (risk ratio ~15, odds ratio 19.6, user-cluster bootstrap 95% CI [10.6, 50.3]); 79% of all QA misses had a weak-written underlying fact. The paper states this is a strong association, not a causal isolation.

• Full history is a strong baseline at short tenure. The full-rendered-history baseline tied or exceeded the tested memory systems at the short horizon. At nine weeks no judge-independent accuracy winner emerged between the layered system and reading the entire raw transcript, at roughly half the per-question read cost for the layered system. A recency window matched to the memory system's mean short-horizon read budget collapsed in both regimes.

• Supersession and representation matter. On knowledge-update questions the curated map scored 120/123 and hybrid v2 121/123 (42/42 on current-value questions). Prose-inline history beat explicit bi-temporal versioned edges at read time (79/81 vs. 72/81), because rendering superseded edges returns stale values to the answerer's view. Separately, only 120/161 (75%) of anchor facts landed in their expected volatility class, and the failure mode is structural: a permanent allergy filed into a functional health-state slot was superseded by a passing cold.

• Injection resistance tracked provenance boundaries, not graph topology. The provenance-typed graph produced zero unsupported assertions across all planted, non-adaptive injection probes (14 unique questions, 42 answer trials) and asserted 0 of 6 on a plausibility-graded attack ladder, while a flattened assertional store asserted 3 of 6 (the subscription, invoice, and spear-phishing rungs). A raw transcript that preserves source framing also resisted.

• Accuracy does not confer caution. The most accurate system almost never abstains when wrong: 93.9% of its judged errors classified as confidently asserted wrong specifics under a deterministic answer-text classifier (8 abstentions in 972 answers), versus 62.7% for the least accurate system. The one surviving injection leak re-enters through the episodic channel.

• The answerability audit surfaced a trust/recall trade-off. 215/220 generated questions validated (97.7%); the five exclusions trace to an oracle declining to assert facts whose only evidence is third-party-authored email, which is exactly the behavior that defeats injection.

• Cost of measurement. The original study logged 34,076 LLM calls end to end, with judging consuming more calls than answering (15.0k vs. 12.7k); review-driven validation added 18,721, for 52,797 in the released logs.

Methodology in Plain English

The researchers flip the usual benchmark construction. A deterministic, seeded "life-script sampler" first invents a fictional user's timeline of facts, and each fact carries a validity interval (when it is true), a volatility class (from permanent through ephemeral), and a channel (chat, or email with sent versus received distinguished, where received mail is third-party content and therefore lower-trust and an injection surface). Only then does an LLM renderer write the chat sessions and emails from per-event fact manifests, and a fidelity verifier checks that every planted fact actually appears in the rendered text. Questions are generated mechanically from the script rather than from the text, so gold answers are correct by construction, and a separate open-book oracle audit checks whether the rendered corpus actually supports each answer.

The corpus is fully synthetic and fictionalized, so it contains no real personal data and entity facts cannot leak from pretraining priors. Two corpora share the generator: a short-horizon corpus (14 users, 275 validated questions) and a tenure corpus (6 users with nine-week histories, 108 questions instantiated as as-of-date sets at weeks 3, 6, and 9). Totals are 383 validated questions across 15 question types.

The comparison holds the answering model fixed (claude-haiku-4-5-20251001) and includes a no-memory control, with a versioned LLM judge (claude-sonnet-5) treated as part of the benchmark definition. The renderer was amazon.nova-pro-v1:0 and the cross-family judge deepseek.v3.2 (both via Bedrock), with Titan-v2 embeddings for the vector backend. Every run was repeated as three independent stochastic replicates, since the providers expose no random-seed control. Because the compared systems differ together in write procedure, representation, retrieval, and token spend, the paper frames the experiment as comparing complete memory-system implementations rather than isolated components, and all headline claims are differential rather than absolute.

Why This Matters

Impact on research. The paper argues that short-horizon benchmarks, which it notes are the field's norm, can mis-rank exactly the systems a long-lived assistant needs, and that recent disputes about whether the write path matters at all look different once horizon is treated as a variable. It offers a specific, mechanistic instance of a ranking inversion in a single variable, with its cause isolated, where prior work had shown rankings are fragile to backbone and benchmark choice. It also argues that write-stage quality, previously compressed at short horizons, becomes a strong correlate of downstream quality outside that regime.

Real-world applications:

• Long-lived personal or work assistants that accumulate months of chat and email with one user, where what the system should remember becomes as consequential as what it can reason about.
• Systems that must track facts whose truth changes over time, such as a subscription the user cancelled or a transient condition that ended, using validity intervals and supersession rather than erasure.
• Deployed agents exposed to untrusted incoming email, where representing third-party claims as claims rather than facts provides what the paper describes as structural quarantine at no added cost.
• Cost planning for production memory: the layered system's advantage against full-history reading at tenure is drastically lower input-token growth, described as roughly half the read cost with no judge-independent accuracy winner at week nine.

Industry relevance. The paper cites commercial memory systems and notes that industry systems have converged on adjacent architectures, such as Perplexity's "Brain" compiling a context graph into an overnight-synthesized wiki, which it cites as evidence of the pattern's practical currency rather than as evaluated results. The winning layered design is released as Veracium, an installable open-source library, and the corpus generator, evaluation harness, audit rubric, all 66 audit records, and per-call logs are released for reproduction.

Future Directions

• A crossed writer × store ablation under matched token budgets, which the paper names as the experiment that would settle causality for the write-quality association it currently reports only as an association.
• A two-sided renderer audit measuring both planted-fact recall and unsupported-fact precision, since the fidelity verifier checks planted-fact presence but not renderer additions, and the answerability audit is only a partial backstop.
• Human or cross-family checking of the audit labels and the judge, because the audit evaluator was the same model as the QA judge, so shared evaluator tendencies could induce correlated measurement error, and auditor–human agreement has not been measured.
• Adaptive and relation-level attacks on graph memories, since the reported probes are planted and non-adaptive, and the laundering channels formalized elsewhere remain real, with the paper's own episodic leak as one example. The write-quality result also has not yet been replicated on the tenure corpus or the hand-built users.

Target Audience

Researchers and engineers building or evaluating long-term memory for LLM agents; benchmark designers concerned with label validity, contamination, and horizon effects; teams deploying assistants that must remember across weeks or months; and security practitioners working on memory poisoning, provenance, and prompt injection. Readers without a statistics background will still follow the design argument, but the results section assumes comfort with paired significance testing and cluster-aware confidence intervals.

Authors’ abstract

Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contamination problems -- and they overwhelmingly measure short interaction histories. We invert the pipeline: a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists; an LLM renderer writes chat and email from per-event fact manifests; a fidelity verifier confirms every planted fact; and questions are instantiated mechanically from the script, so gold answers are script-valid by construction and separately validated for answerability. The synthetic, fictionalized corpus (~380 questions, 15 types) embeds features absent from the benchmarks we survey: per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets. Benchmarking five memory architectures against a no-memory control (fixed answerer, versioned LLM judge, three replicates, two horizons), we find backend rankings invert with history length: the budgeted curated-map memory that leads at three weeks loses recall of evicted content by nine weeks (96% to 72%) while a provenance-typed graph rises to 90%; the inversion is positive for all six users under complete cross-family re-judging (exact p=0.031). A full-rendered-history baseline ties or exceeds the best memory system at the short horizon but shows no judge-independent advantage at nine weeks, at about twice the read cost. Write-stage quality strongly correlates with downstream quality (weakly-written facts fail 24% vs 2%), and injection resistance tracked whether provenance boundaries survive representation. A layered architecture performs best among the memory systems in both regimes (96.8% short-horizon) and is released as Veracium, an open-source library, with the corpus generator and harness.