Benchmark Almanac

MemTrapBench

Not in model cards

Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance.

It had 30 Hugging Face paper upvotes and 5 GitHub stars.

In the authors’ words

Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing memory benchmarks mainly evaluate whether information is correctly extracted, stored, and retrieved, while largely overlooking how retrieved memories reshape model reasoning and affect performance on the current task. We identify memory-induced cognitive traps: even faithfully recorded and semantically relevant memories can distort model reasoning or beliefs and degrade current task performance. To systematically evaluate these failure modes, we introduce MemTrapBench, which covers two forms of cognitive traps: Reasoning Fixation and Belief Distortion. Experiments across two model families and five representative memory frameworks show that MemTrapBench is challenging: all evaluated memory strategies underperform the no-memory setting, with even the strongest methods suffering drops of more than 10%. To mitigate these cognitive traps, we propose AdaptiveMem, a simple yet effective inference-time method that instructs LLMs to avoid memory traps. AdaptiveMem mitigates cognitive traps on MemTrapBench while preserving or improving performance on standard memory benchmarks across diverse memory frameworks.

Abstract of MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use (arXiv:2608.20202, CC0)
Made by Zhejiang University (ZJUNLP) Released Aug 2026 Learning from the context All tasks public
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.

No scores here yet

No lab, leaderboard or Epoch AI run has reported a score for MemTrapBench yet.

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Long context

All 66 in Long context
  • MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
  • LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Beam 1.0 66.9%
  • GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 100%
  • OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. K-EXAONE 2.0 94.4%
  • CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
  • AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. Step 5 Preview 88.3%