SQuAD

Llama 3.1-405B 89.3% leads since Jul 2024 Dropped

Models extract answer spans from Wikipedia paragraphs for over 100,000 crowdsourced questions. Scored by exact match and F1.

Saturated, with models above human performance since 2018. SQuAD 2.0 added unanswerable questions.

In the authors’ words

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com

Abstract of SQuAD: 100,000+ Questions for Machine Comprehension of Text (arXiv:1606.05250, CC0)

Made by Stanford NLP Released Jun 2016 Reading comprehension Public tasks plus a private test set Last in a launch post Apr 2024 , dropped after it saturated Last model added Jul 2024

Level: Everyday Most adults, without training
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Short answer A number, word or short string matched to a key.

Paper Dataset Leaderboard rajpurkar.github.io

Models scored in lab cards

Launch post lists it, from 1 lab

System cards report it, from 1 lab

Llama 3.1-405B leads with 89.3% among models released by Jul 2024

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

The newest model with a score here came out in Jul 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Llama 3.1-405B 89.3%
2 Llama 3-70B 85.6%
3 Llama 3.1-70B 81.8%
4 Llama 3.1-8B 77%
5 Llama 3-8B 76.4%

#	Model	Score	Date	Source
1	Llama 3.1-405B	89.3%	Jul 2024	Model card
2	Llama 3-70B	85.6%	Apr 2024	Model card
3	Llama 3.1-70B	81.8%	Jul 2024	Model card
4	Llama 3.1-8B	77%	Jul 2024	Model card
5	Llama 3-8B	76.4%	Apr 2024	Model card

2 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

The best score passed 85% in Apr 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Date	Launch post	Score	As the post writes it
18 Apr 2024	Llama 3-70B	85.6	SQuAD (1-shot) [pretrained]

5 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
23 Jul 2024	Llama 3.1-8B model card	77	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
23 Jul 2024	Llama 3.1-70B model card	81.8	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
23 Jul 2024	Llama 3.1-405B model card	89.3	SQuAD, 1-shot, em, pretrained (Base pretrained models table)	Capabilities
18 Apr 2024	Llama 3-8B model card	76.4	SQuAD (1-shot), pretrained (Base pretrained models table)	Capabilities
18 Apr 2024	Llama 3-70B model card	85.6	SQuAD (1-shot), pretrained (Base pretrained models table)	Capabilities