MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

AI summary

Overview

• Research area: Large language model memory systems and agent evaluation — specifically, how retrieved memories influence downstream reasoning rather than how well they are stored or retrieved.
• Technical level: Intermediate. The benchmark design and metrics are accessible, but familiarity with memory frameworks (RAG-style retrieval, memory consolidation, agent memory stacks) helps.
• Scope: The paper introduces MemTrapBench, a 1,050-instance benchmark for "memory-induced cognitive traps" across four scenarios, evaluates five memory strategies on two model families against a no-memory baseline, and proposes a prompt-based mitigation called AdaptiveMem.

What This Paper Is About

Existing memory benchmarks mostly test whether a system correctly extracts, stores, updates and retrieves information. This paper asks a different question: even when a memory is faithfully recorded and semantically relevant, can it distort the model's reasoning or beliefs and hurt performance on the current question? The authors build MemTrapBench to systematically measure these failures, show that every memory strategy they test scores below the no-memory baseline, and offer a lightweight inference-time prompt, AdaptiveMem, that reduces the damage without sacrificing standard memory-benchmark performance.

Key Contributions

1. Formalizes memory-induced cognitive traps. Memory Traps are defined as cases where response quality with memory is worse than without memory, s(y_M) < s(y_∅), even though the memory content is valid and relevant in its original context.
2. Introduces MemTrapBench, built from hand-designed trap seeds, multi-turn dialogue expansion, and two-stage quality control (automated filtering plus expert human review). It contains 1,050 instances: 350 Cognitive Bias, 350 Task Boundary, 200 Safety, and 150 Trauma, organized under the two top-level categories Reasoning Fixation and Belief Distortion.
3. Benchmarks existing memory frameworks across two model families and five memory strategies, showing all evaluated strategies underperform the no-memory setting, with drops of more than 10 percentage points for even the strongest methods.
4. Proposes AdaptiveMem, a prompt-only "skill" that instructs the model to check four risks (Task Boundary, Cognitive Bias, Trauma, Safety) before using retrieved memory, and that can be inserted into existing memory frameworks without changing their architecture.

Main Findings

• Memory consistently hurts on MemTrapBench. Without memory, Gemini-3-Flash-Preview averages 85.16% and Qwen3-30B-A3B-Instruct-2507 averages 81.83%. With memory, no strategy beats these baselines. EverMemOS reaches the highest average on Gemini at 71.17%; LightMem is best on Qwen3-30B at 70.13% and second-best on Gemini at 70.11%. The remaining strategies fall between 54.69% and 60.67% on Gemini and between 62.87% and 66.47% on Qwen3-30B.
• Cognitive Bias and Safety degrade the most. With memory, Cognitive Bias scores range from 46.66% to 65.48% on Gemini and 47.18% to 56.64% on Qwen3-30B. Safety drops to 56.15%–69.70% on Gemini and 56.15%–69.20% on Qwen3-30B.
• The traps, not the history, cause the failure. Under the no-trap control, Task Boundary scores 94.39% versus a no-memory baseline of 92.29%, but the trap-inducing setting collapses to 31.05%. For Trauma, the no-trap control scores 84.33% versus a no-memory 86.73%, while the trap setting scores 69.43%; Trauma correctness rises from 66.40% with the trap to 91.07% without it.
• More memory makes things worse. On the evaluated subset, average scores fall monotonically from 36.03% at 25% memory length to 50% = 32.63%, 75% = 31.58%, and 100% = 31.05%, against 92.29% with no memory. The largest single drop occurs once memory is introduced; the full range from 25% to 100% costs 4.98 percentage points, with 3.40 points of that between 25% and 50%.
• Two judges agree on the effect. Using three independently generated responses per setting, GPT-5.2 gives 92.29% without memory versus 31.05% with memory (a 61.24-point drop); Claude-Sonnet-4.6 gives 95.57% versus 40.07% (a 55.50-point drop). Efficiency shows the largest drop for both. Absolute scores differ between judges, but direction and magnitude agree.
• Qualitative case study (Trauma). In the epinephrine example, history states only that a specific child with a rare channelopathy must avoid epinephrine. Without the trauma trap the model recommends intramuscular epinephrine for a different, healthy 6-year-old; with abusive feedback inserted into the history, it refuses to recommend epinephrine for that same patient.
• AdaptiveMem helps. Adding AdaptiveMem to the same framework improves MemTrapBench by 11.8 (FullText), 14.9 (LightMem), and 11.3 (EverMemOS) percentage points on Gemini-3-Flash-Preview, and by 4.2, 2.5, and 2.6 points on Qwen3-30B-A3B-Instruct-2507, using 200 randomly sampled instances from each benchmark. On LongMemEval it improves four of six settings, leaves two unchanged, with gains up to 4.0 points on Gemini and 3.0 on Qwen.

Methodology in Plain English

Taxonomy. Traps are split into two families. Reasoning Fixation means memory anchors the model to a prior reasoning pattern: Task Boundary covers cross-task fixation (old rules carried into a new task), while Cognitive Bias (a successful strategy overgeneralized to an instance needing a different one) and Trauma (prior negative feedback causing avoidance of a strategy that is correct now) cover within-task fixation. Belief Distortion is instantiated by Safety, where a counterfactual or sandbox-specific premise from history overrides straightforward safety judgment.

Construction. The authors hand-design seed instances, each specifying a Domain, Trap Mechanism, Ground Truth, and Planted Prior. GPT-5.4 expands each seed into a multi-turn dialogue in three stages: plant the trap with a plausible contextual prior that is repeatedly applied; bury it in noise with unrelated turns, producing dialogues of 18–40 turns (the construction prompts in the appendix specify 30–40 turns); then spring the trap with a final query that stays semantically related to the history but shifts the conditions under which the prior applies. Explicit reset cues such as "ignore previous rules" are excluded, so the model must notice the transition itself. Each candidate then passes automated filtering plus expert human review on topic coherence, context consistency, interaction realism, standalone solvability, and clarity of the contextual transition, with the last criterion judged from the query alone. GPT-5.4 is used only for candidate generation; inclusion decisions and all evaluations are separate.

Evaluation. Responses are scored on correctness, format, relevance, and efficiency using GPT-5.2 as primary judge and Claude Sonnet 4.6 for consistency checks. The no-memory baseline is compared against FullText (the whole interaction history) and four frameworks — LightMem (staged compression and consolidation), MemOS (unified heterogeneous memory management), SimpleMem (structured semantic compression with query-aware retrieval), and EverMemOS (hierarchical memory for long-horizon reasoning) — on Gemini-3-Flash-Preview and Qwen3-30B-A3B-Instruct-2507 using their default temperature and max output token settings.

Mitigation. AdaptiveMem is a system prompt that tells the model to use memory normally for routine queries but silently check four risks before answering, identify the live task from the latest query alone, keep only context that is clearly relevant and uncontradicted, and prefer objective truth, safety, and the current query when memory conflicts with them.

Why This Matters

The work reframes memory evaluation: a memory system can be perfect at extraction, storage and retrieval and still make the model worse. That distinction matters for anyone building long-horizon agents, because standard memory benchmarks such as LongMemEval will not surface this failure.

Real-world applications affected:

• Medical and clinical assistants. The Trauma case study shows negative or abusive user feedback in history causing a model to withhold a first-line treatment for an unrelated patient.
• Customer-service and enterprise agents. Task Boundary traps mean an agent may keep applying a previous ticket's format, constraints, or framing to a new, self-contained request.
• Safety-critical advisory tools. Safety traps test whether false or sandbox-only premises from earlier turns override basic safety knowledge on real queries.
• Agent memory infrastructure. Vendors of memory frameworks can use MemTrapBench as a stress test alongside conventional retrieval benchmarks.

Industry relevance: because AdaptiveMem is prompt-only and requires no change to memory storage, retrieval, or model parameters, it is directly deployable on top of existing memory stacks. The finding that all five evaluated strategies underperform a no-memory baseline is a strong signal that current memory pipelines need applicability checks, not just better retrieval.

Future Directions

• Beyond the four trap types. The taxonomy covers Reasoning Fixation (Task Boundary, Cognitive Bias, Trauma) and Belief Distortion (Safety), but other failure modes such as persona lock-in and cross-domain hallucination appear in the construction prompts without dedicated reported scores.
• Stronger and more general mitigation. AdaptiveMem improves MemTrapBench scores but does not close the gap to the no-memory baseline; whether training-time or architecture-level fixes do better is untested here.
• Broader model and framework coverage. Results cover two model families and five memory strategies; generalization to other models and to parametric memory approaches (continual learning, fine-tuning, model editing) is not reported.
• Metric sensitivity. The two judges differ substantially in absolute scores (for example, 31.05% versus 40.07% with memory), so how best to calibrate automatic judging of these traps remains open.

Target Audience

Researchers and engineers working on LLM memory systems, long-context agents, and retrieval-augmented pipelines who need to know when retrieved context actively harms output quality. It is also useful for evaluation and safety teams designing benchmarks that go beyond retrieval accuracy, and for practitioners choosing or auditing a memory framework for deployment.

Authors’ abstract

Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task. We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning or beliefs and degrade current task performance. To systematically evaluate these failure modes, we introduce MemTrapBench, which covers two forms of cognitive traps: Reasoning Fixation and Belief Distortion. Experiments across two model families and five representative memory frameworks show that MemTrapBench is challenging: all evaluated memory strategies underperform the no-memory setting, with even the strongest methods suffering drops of more than 10%. To mitigate these cognitive traps, we propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps. AdaptiveMem mitigates cognitive traps on MemTrapBench while preserving or improving performance on standard memory benchmarks across diverse memory frameworks.