Needle In A Haystack
GLM-4 100% leads since Jan 2024 Dropped
Models retrieve a planted fact from long filler text at varied context lengths and insertion depths. Scored by retrieval accuracy.
Most long-context models score near 100%, and labs run their own variants, so scores are not comparable.
Made by Greg Kamradt Released Nov 2023 Finding facts in long inputs All tasks public Last in a launch post Dec 2024 , dropped after it saturated
Last model added Jan 2025
- Level
- Everyday Most adults, without training
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Short answer A number, word or short string matched to a key.
9
Models scored in lab cards
4
Launch posts list it, from 3 labs
11
System cards report it, from 5 labs
2 models share the top score of 100% among models released by Nov 2024
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Nov 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 GLM-4 100%
- 2
Qwen-Turbo 100% - 3 Claude 3.5 Sonnet 99.7%
- 4 Claude 3 Opus 99.4%
- 5
Llama 3.1-8B 98.8% - 6 Llama 3.1-405B 98.1%
- 7 Llama 3.1-70B 97.5%
- 8 Claude 3 Haiku 95.9%
- 9 Claude 3 Sonnet 95.4%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GLM-4 | 100% | Jan 2024 | Model card | |
| 2 | Qwen-Turbo | 100% | Dec 2024 | Model card | |
| 3 | Claude 3.5 Sonnet | 99.7% | Jun 2024 | Model card | |
| 4 | Claude 3 Opus | 99.4% | Mar 2024 | Model card | |
| 5 | Llama 3.1-8B | 98.8% | Nov 2024 | Model card | |
| 6 | Llama 3.1-405B | 98.1% | Jul 2024 | Model card | |
| 7 | Llama 3.1-70B | 97.5% | Nov 2024 | Model card | |
| 8 | Claude 3 Haiku | 95.9% | Mar 2024 | Model card | |
| 9 | Claude 3 Sonnet | 95.4% | Mar 2024 | Model card |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Z.ai
The best score passed 85% in Jan 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Reported by the lab in its model card
Labs dropped it from launch posts once the best score passed 85%
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 26 Dec 2024 |  DeepSeek-V3 | Needle In A Haystack (NIAH) | |
| 23 Jul 2024 | Llama 3.1-405B | 98.1 | NIH/Multi-needle |
| 7 May 2024 | DeepSeek-V2 (MoE-236B, May 2024) | Needle In A Haystack (NIAH) | |
| 16 Jan 2024 | GLM-4 | 100 | 大海捞针 (128K) |
11 system cards report it, from 5 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 14 Jan 2025 |  MiniMax-Text-01 technical report | 4M vanilla Needle-In-A-Haystack pressure test, heatmap without printed values (Figure 14). Text says the model handles up to 4M tokens | Capabilities | |
| 26 Dec 2024 | DeepSeek-V3 technical report | Needle In A Haystack up to 128K after SFT, heatmap only (Figure 8) | Capabilities | |
| 19 Dec 2024 | Qwen-Turbo technical report | 100 | Passkey Retrieval at 1M tokens, 100% accuracy (5.2.4 text, Figure 2) | Capabilities |
| 23 Nov 2024 | Llama 3.1-8B technical report | 98.8 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| 23 Nov 2024 | Llama 3.1-70B technical report | 97.5 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| 23 Nov 2024 | Llama 3.1-405B technical report | 98.1 | NIH/Multi-needle, average recall of 2 of 4 needles up to 128K, post-trained (Table 2, Table 21) | Capabilities |
| 15 Jul 2024 | Qwen2-72B technical report | Needle in a Haystack, 8K to 128K with YARN beyond 32K, heatmap only (Figure 1) | Capabilities | |
| 20 Jun 2024 | Claude 3.5 Sonnet model card | 99.7 | Needle In A Haystack (Anthropic version, modified prompt), average recall over all context lengths up to 200k. 200k context length 99.7 (Table 5) | Capabilities |
| 4 Mar 2024 | Claude 3 Sonnet model card | 95.4 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 91.4 (Table 7) | Capabilities |
| 4 Mar 2024 | Claude 3 Opus model card | 99.4 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 98.3 (Table 7) | Capabilities |
| 4 Mar 2024 | Claude 3 Haiku model card | 95.9 | Needle In A Haystack (Anthropic version, 30 needles, two haystacks), average recall over all context lengths up to 200k. 200k context length 91.9 (Table 7) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Beam 1.0 66.9%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 100%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
K-EXAONE 2.0 94.4% - CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%