KnowMe-Bench
Models read long autobiographical narratives and must infer a person's stable motivations and decision principles, not just retrieve facts.
42 HF paper upvotes and about 140 GitHub stars. Aimed at lifelong digital companions.
In the authors’ words
Existing long-horizon memory benchmarks mostly use multi-turn dialogues or synthetic user histories, which makes retrieval performance an imperfect proxy for person understanding. We present \BenchName, a publicly releasable benchmark built from long-form autobiographical narratives, where actions, context, and inner thoughts provide dense evidence for inferring stable motivations and decision principles. \BenchName reconstructs each narrative into a flashback-aware, time-anchored stream and evaluates models with evidence-linked questions spanning factual recall, subjective state attribution, and principle-level reasoning. Across diverse narrative sources, retrieval-augmented systems mainly improve factual accuracy, while errors persist on temporally grounded explanations and higher-level inferences, highlighting the need for memory mechanisms beyond retrieval. Our data is in https://github.com/QuantaAlpha/KnowMeBench.
- Level
- Everyday Most adults, without training
- Scored
- Graded by a model A language model scores each response, usually against a rubric.
- Answers
- Short answer A number, word or short string matched to a key.
No scores here yet
No lab, leaderboard or Epoch AI run has reported a score for KnowMe-Bench yet.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens.
Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Beam 1.0 66.9%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 100%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
K-EXAONE 2.0 94.4%
- CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%