Passkey Retrieval
Mixtral 8x7B 100% Dropped Models find a random passkey hidden at a random position in a long filler prompt. Scored by retrieval accuracy.
Synthetic test from the Landmark Attention paper. Strong long-context models usually score near 100%.
- Level
- Everyday Most adults, without training
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Short answer A number, word or short string matched to a key.
2 models share the top score of 100%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Mixtral 8x7B | 100% | Jan 2024 | Model card | |
| 2 |  Qwen2.5-14B-Instruct-1M | 100% | Jan 2025 | Model card |
The best score passed 85% in Jan 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs dropped it from launch posts once the best score passed 85%
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 27 Jan 2025 | Qwen2.5-14B-Instruct-1M | Passkey Retrieval at 1M tokens (chart only) |
2 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 26 Jan 2025 | Qwen2.5-14B-Instruct-1M technical report | 100 | Passkey Retrieval with documents up to 1M tokens, perfect accuracy stated in text (Figure 1, section 6.1) | Capabilities |
| 8 Jan 2024 | Mixtral 8x7B technical report | 100 | Passkey retrieval, 100% at all context lengths and positions (Section 3.2, Figure 4 Left) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens.
Muse Spark 1.3 98.5% - LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Beam 1.0 66.9%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 100%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
K-EXAONE 2.0 94.4% - CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%