QuALITY
Models answer multiple-choice questions about English passages that average about 5,000 tokens. Scoring is accuracy.
About half the questions were too hard for annotators to answer under tight time limits.
In the authors’ words
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
- Level
- School Students, from grade school to high school
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Multiple choice
Claude 3 Opus leads with 90.5% among models released by Jul 2024
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Jul 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 Claude 3 Opus 90.5%
- 2
Llama 3.1-405B 87.6% - 3 Claude 3 Sonnet 85.9%
- 4 Claude 2 83.2%
- 5 Llama 3.1-70B 82.8%
- 6 Claude 3 Haiku 80.2%
- 7 Llama 3.1-8B 56%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude 3 Opus | 90.5% | Mar 2024 | Model card | |
| 2 | Llama 3.1-405B | 87.6% | Nov 2024 | Model card | |
| 3 | Claude 3 Sonnet | 85.9% | Mar 2024 | Model card | |
| 4 | Claude 2 | 83.2% | Jul 2023 | Model card | |
| 5 | Llama 3.1-70B | 82.8% | Nov 2024 | Model card | |
| 6 | Claude 3 Haiku | 80.2% | Mar 2024 | Model card | |
| 7 | Llama 3.1-8B | 56% | Nov 2024 | Model card |
The benchmark's own leaderboard has 17 entries, 4 of them models we track. We plot every source in the charts below.
2 models have held number one since 2023
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Mar 2024
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No launch post has listed it since Jul 2023
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 11 Jul 2023 | Claude 2 | 83.2 | QuALITY (5-shot) |
7 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 23 Nov 2024 | Llama 3.1-8B technical report | 56 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| 23 Nov 2024 | Llama 3.1-70B technical report | 82.8 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| 23 Nov 2024 | Llama 3.1-405B technical report | 87.6 | QuALITY (5-shot), pretrained long-context (Table 14) | Capabilities |
| 4 Mar 2024 | Claude 3 Sonnet model card | 85.9 | QuALITY, 1-shot. 0-shot 84.9 (Table 6) | Capabilities |
| 4 Mar 2024 | Claude 3 Opus model card | 90.5 | QuALITY, 1-shot. 0-shot 89.2 (Table 6) | Capabilities |
| 4 Mar 2024 | Claude 3 Haiku model card | 80.2 | QuALITY, 1-shot. 0-shot 79.4 (Table 6) | Capabilities |
| 11 Jul 2023 | Claude 2 model card | 83.2 | QuALITY (5-shot) (4.3) | Capabilities |
More in Long context
- MRCR v2 In use Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. Muse Spark 1.3 98.5%
- LongBench v2 In use 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. Beam 1.0 66.9%
- GraphWalks In use A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. Claude Fable 5 100%
- OpenAI MRCR v2 In use Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.
K-EXAONE 2.0 94.4% - CL-bench In use Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. GPT-5.4 27.9%
- AA-LCR 1.1 In use Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.
Step 5 Preview 88.3%