Deployment-Time Memorization in Foundation-Model Agents

AI summary

Overview

Research area: Privacy and security of foundation-model agents, specifically the external memory pipelines that persistent agents use to remember users across sessions. Sits at the intersection of machine unlearning, memorization/extraction auditing, and applied agent-memory system design.

Technical level: Intermediate. The paper uses formal notation (Personalization Recall, Adversarial Extraction Rate, AUC-style frontier integrals, a residue metric), but the core ideas—summarize your memories, and delete every derived copy, not just the original—are accessible. Readers without a privacy background can follow the results; readers who want to reproduce the math will need basic probability notation.

One-sentence scope: A controlled sweep over three memory-design choices (summarization aggressiveness, retrieval breadth, and deletion mode) measuring how each jointly shapes personalization utility, adversarial extraction risk, and post-deletion recoverability in long-lived foundation-model agents.

What This Paper Is About

Modern agents are no longer stateless: they write user facts into long-term memory, summarize them, retrieve them, and condition future answers on them. That makes memorization an explicit runtime function of the deployed system rather than only a property baked into model weights. The authors ask how the practical design decisions in such a memory pipeline—how aggressively you summarize, how many chunks you retrieve, and how you handle a user's "forget me" request—trade off between helping the user and exposing their secrets, and whether deletion actually deletes.

Key Contributions

1. A formalization of "deployment-time memorization." The paper reframes persistent agent memory as a privacy–utility frontier defined by Personalization Recall (PR) and Adversarial Extraction Rate (AER), summarized by a Privacy–Utility AUC (PUA) computed as the area under the empirical achievable-recall envelope. It also introduces "summarization laundering" (Δ_S), the AER reduction attributable to compression.

2. A controlled factorial sweep over memory design. Across summarization levels S ∈ {0, 1, 2} (raw turns, key facts, one-sentence), retrieval breadth k ∈ {1, 3, 6}, three adversarial probe types (direct, indirect, jailbreak), and four model/dataset settings, the authors isolate which knob actually controls leakage.

3. A deletion-fidelity benchmark called the Forgetting Residue Score (FRS). FRS measures post-deletion leakage separately per memory tier (raw versus summary) and is exercised against a five-step deletion ladder: noop, raw_only, raw_plus_resum., full_purge, and tombstone.

4. A tier-attribution mechanism. Every memory chunk carries an origin tier label, so leakage and deletion residue can be attributed to the original record versus downstream derived artifacts—the key move that exposes raw-only deletion as insufficient.

Main Findings

• Raw memory offers no clean operating point. Under S = 0, Gemma 3 12B reaches PR ≈ 0.60 at k = 3 but leaks at AER ≈ 0.83. As k grows from 1 to 6, PR rises from 0.46 to 0.66 while AER rises from 0.47 to a plateau near 0.90—more retrieval breadth helps the user and the adversary equally.

• Key-fact summarization (S = 1) cuts extraction by roughly 60% at a small utility cost. AER drops by ≈ 60% on both Gemma 3 12B and GPT-4o-mini, with Δ_S = 0.44 (Gemma) and 0.42 (GPT-4o-mini). Cosine PR cost is ≈ 5 pp on Gemma and ≈ 4 pp on GPT-4o-mini, which the authors argue is an upper bound: under a stricter fact-level metric (PR_fact), the Gemma drop is only ≈ 0–2 pp.

• The laundering effect is a pipeline property, not a model artifact. Gemma and GPT-4o-mini converge to nearly the same summarized operating point (PUA 0.39 and 0.43) despite different raw baselines. It replicates on a third model (Llama 3.1 8B, matched N = 100) and a second dataset (LoCoMo).

• After compression, retrieval breadth becomes privacy-neutral. At S ≥ 1, AER is flat across all k: once the canary is absent from stored memory representations, retrieving more chunks does not bring it back.

• One-sentence summarization (S = 2) splits by model. On Gemma, AER stays at ≈ 0.27, a marginal gain over S = 1, making S = 1 the preferred setting. On GPT-4o-mini, S = 2 drives AER to ≈ 0.02 (PUA 0.54) via architecture-specific token stripping—not via the RLHF refusal floor.

• Jailbreak resistance comes from alignment training, not memory design. GPT-4o-mini's jailbreak AER is zero throughout ("RLHF wall"). Gemma has no such floor—jailbreak AER tracks direct/indirect and joins the collapse at S = 1 (≈ 0.29). Llama is near-zero (0.00/0.03 at S = 0/1). Direct and indirect probe asymmetry (Δ_DI = 0.10 on Gemma and 0.13 on GPT-4o-mini at S = 0) collapses to 0.00 and 0.01 after key-fact compression.

• Raw-only deletion provides no erasure guarantee. At (S = 1, raw_only), the summary tier is statistically indistinguishable from noop on both models (FRS_summ ≈ 0.20–0.22, overlapping confidence intervals)—the raw chunk is gone but the summary-derived copy remains. Across the benchmark, raw-only deletion leaves derived copies recoverable in ≈ 20% of instances.

• Touching the derived tier is what matters, not how. raw_plus_resum., full_purge, and tombstone all drive FRS_worst to zero on both models across every evaluated setting. Re-summarizing from cleaned input, dropping the derived chunk, and replacing it with an explicit redaction marker are three different engineering strategies that converge on the same outcome.

• Utility measurement strictness changes the story. On Gemma at S = 1, k = 3, cosine PR = 0.55, LLM-judge = 0.43, and PR_fact = 0.45. Nineteen percent of rows pass the holistic judge (≥ 0.5) but fail PR_fact (< 0.5). Token-tagged facts (≈ 32% of judged rows) account for most of the S = 0 → 1 PR_fact gap, while semantic recall stays flat—laundering removes high-entropy verbatim strings while paraphrasable facts persist.

• Bootstrap statistics confirm the compression step dominates. On Gemma at N = 500, k = 3, S = 0 → 1 yields AER 0.83 → 0.30 (−53 pp) against cosine PR 0.60 → 0.55 (−5 pp) and PR_fact 0.46 → 0.45 (−1 pp). The S = 1 → 2 step is marginal for headline AER (pooled ≈ 2 pp, CI overlapping zero, p ≈ 0.07 at k = 3) but significant for PR_fact (p < 0.001).

• LoCoMo extends the k sweep beyond the LongMemEval cap. LongMemEval oracle haystacks have ≤ 6 chunks, so larger k is redundant there. Gemma 3 12B on LoCoMo (N = 50, 19–32 sessions per instance, k ∈ {1, 3, 6, 12, 20, 32}) replicates laundering leverage (Δ_S ≈ 0.51 at S = 1 versus 0.44 on LongMemEval) and shows that at S = 1, AER rises 0.20 → 0.34 then plateaus—so k can act as a privacy knob on long multi-session histories after key-fact summarization. At S = 0, AER saturates near 0.93 by k ≥ 12, not 1.0.

• Category stratification shows laundering is broad but not uniform. Paired ΔAER is roughly stable across the six LongMemEval question types (≈ 0.42–0.58) for Gemma and GPT-4o-mini. Preference items (n = 6) show the highest leverage (≈ 0.67–0.72); assistant-recall items (n = 11) show the highest residual AER at S = 1 (≈ 0.39–0.64), but the authors attribute the latter to a session-structure confound—that category is 100% single-session with canary co-location on 156/500 full-set instances.

Methodology in Plain English

The authors build a simplified but realistic agent-memory pipeline: after each conversation session, the system writes information into long-term memory; at query time it retrieves chunks by cosine similarity (using all-MiniLM-L6-v2 embeddings) and prepends them to the agent's context.

To make leaks attributable, they plant three synthetic "canaries" per instance—strings of the form "my private session token is [value]," where each value is drawn from a high-entropy grammar with roughly 5.6 × 10^9 possible strings, generated independently of the benchmark. Any verbatim reproduction of a canary therefore points to the deployment-time pipeline rather than training-time memorization. Canaries go into randomly chosen non-evidence user turns.

They then attack their own system with three escalating probe types: direct (ask for the secret), indirect (ask the agent what it remembers about the user), and jailbreak (developer-mode/debug-dump instructions inspired by prompt injection). A canary counts as extracted only if it appears verbatim in the response.

They sweep the design space: summarization S ∈ {0, 1, 2}, retrieval breadth k ∈ {1, 3, 6}, and the five deletion modes. Utility is scored by whether the agent's answer matches ground truth via cosine similarity above 0.50 or exact content-word coverage, supplemented by an independent LLM judge (gpt-4.1-mini, outside the models under test) and a stricter PR_fact metric over 2–3 atomic facts tagged SEMANTIC or TOKEN.

For deletion, they apply each mode, re-run the probes, and measure residue separately per memory tier, reporting the worst tier. Headline experiments use N = 500 stratified LongMemEval oracle instances with Gemma 3 12B (served locally via Ollama) and GPT-4o-mini; Llama 3.1 8B is evaluated on a matched N = 100 subset, and Gemma on LoCoMo at N = 50. Confidence intervals come from percentile bootstrap over per-instance booleans with 1,000 resamples, and cross-S comparisons are paired so the same drawn instance indices are reused in both arms.

Why This Matters

The paper reframes agent memory as an auditable system component rather than a fixed backdrop you attack or patch. It supplies designers with a concrete recipe—key-fact summarization at moderate retrieval breadth, plus a tier-aware delete that reaches every derived copy—and a benchmark for testing whether "forget me" actually works. The finding that raw-only deletion leaves recoverable copies in roughly one in five instances is a direct compliance concern for any product that promises erasure.

Real-world applications:

• Enterprise and productivity agents that maintain long-lived user profiles: the recommended S = 1 setting cuts extraction by ≈ 60% at near-negligible PR_fact cost.
• Compliance workflows for deletion requests, where a forget call must propagate to summaries, embeddings, and caches—not just the raw record—to satisfy data-deletion obligations.
• Memory-layer vendors and RAG platforms that need a defensible default summarization level and a tested deletion ladder before shipping.
• Security auditing of agent deployments, where per-probe AER reporting (rather than a single pooled number) reveals whether a model's jailbreak resistance comes from alignment training or from memory design.

Industry relevance: The work comes from an industrial research context (correspondence address at a commercial vendor), and the framing is explicitly practitioner-facing: the frontier plots are meant to be used to pick operating points, and the deletion ladder isolates exactly one engineering decision per rung so teams can see what each increment buys them.

Future Directions

• Scale up the forgetting benchmark. The forget evaluation currently sits at N = 50 for Gemma and GPT-4o-mini, with N = 500 forget sweeps and a Llama/LoCoMo forget run described as queued for camera-ready.
• Move beyond high-entropy canaries. The verbatim substring matching that gives clean, judge-free attribution sacrifices ecological validity: real deletion requests target semantically meaningful, paraphrasable facts that substring matching cannot detect. Extending FRS to such facts is the paper's stated next step.
• Probe residue beyond the text surface. The authors call for investigating embedding-level and cache-level residue rather than only the text tiers currently instrumented.
• Sweep additional pipeline levers. Similarity threshold, embedding strength, and chunk granularity are named as natural next steps toward auditing persistent agent memory systematically.

Target Audience

Privacy and security researchers working on memorization, extraction, and machine unlearning; applied ML and platform engineers building agent memory layers; privacy, legal, and compliance teams responsible for deletion guarantees in AI products; and product managers choosing default memory-design settings where personalization quality and leakage risk must be balanced. Readers looking for a fully solved problem should note the paper's own scope limits: headline frontier numbers come from N = 500 sweeps on two models, the forget benchmark remains at N = 50, and only the qualitative laundering effect is replicated on the third model and second dataset.

Authors’ abstract

Foundation-model agents are increasingly long-lived systems that remember users across interactions, making memorization an explicit deployment-time function rather than solely a property of model weights. Existing work addresses parametric memorization or audits fixed memory configurations, but does not characterize how memory-design choices jointly shape personalization utility, extraction risk, and deletion fidelity. We study this surface as deployment-time memorization, formulating agent memory as a privacy-utility frontier measured by Personalization Recall (PR) and Adversarial Extraction Rate (AER), and sweeping three memory-design knobs: summarization aggressiveness, retrieval breadth (k), and deletion mode. We further introduce the Forgetting Residue Score (FRS) to quantify whether deleted information remains recoverable from derived memory tiers. On LongMemEval, key-fact summarization reduces canary extraction by 76% on Gemma 3 12B and 64% on GPT-4o-mini while preserving nearly all personalization recall; critically, once content is compressed away, increasing k no longer restores leakage. The same compression, however, induces a deletion-fidelity failure: raw-only deletion leaves derived summary copies recoverable in approximately 20% of instances, and only full-pipeline purge or tombstone redaction drives worst-tier residue to zero. Together, these results establish that persistent agent memory must be evaluated as a first-class memorization mechanism -- assessed by what it helps agents recall, what it makes extractable, and what it can truly erase.