SQuAD v2

Llama 3.1-405B 81.8% leads since Nov 2024 Dropped

Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1.

Saturated. The test set is hidden, so cards report the dev set.

In the authors’ words

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.

Abstract of Know What You Don't Know: Unanswerable Questions for SQuAD (arXiv:1806.03822, CC0)

Made by Stanford NLP Released Jun 2018 Reading comprehension Public tasks plus a private test set Last in a launch post Dec 2023 , dropped before it saturated Last model added Nov 2024

Level: Everyday Most adults, without training
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Short answer A number, word or short string matched to a key.

Paper Dataset Leaderboard rajpurkar.github.io

Models scored in lab cards

Launch posts list it, from 2 labs

System cards report it, from 1 lab

2 models share the top score of 81.8% among models released by Jul 2024

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

The newest model with a score here came out in Jul 2024, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 Llama 3.1-405B 81.8%
2 Llama 3.1-70B 81.8%
3 Llama 2-70B 80.7%
4 PaLM 2-L 80.5%
5 Llama 2-34B 77.4%
6 Llama 3.1-8B 77%
7 Llama 2-13B 72.9%
8 Llama 2-7B 67.2%

#	Model	Score	Date	Source
1	Llama 3.1-405B	81.8%	Nov 2024	Model card
2	Llama 3.1-70B	81.8%	Nov 2024	Model card
3	Llama 2-70B	80.7%	Jul 2023	Model card
4	PaLM 2-L	80.5%	May 2023	Model card
5	Llama 2-34B	77.4%	Jul 2023	Model card
6	Llama 3.1-8B	77%	Nov 2024	Model card
7	Llama 2-13B	72.9%	Jul 2023	Model card
8	Llama 2-7B	67.2%	Jul 2023	Model card

3 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

Other labs Meta

Lab claims Not tested since Nov 2024

Llama 3.1-405B 81.8%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far

No launch post has listed it since Dec 2023

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Google DeepMind

Microsoft

Date	Launch post	Score	As the post writes it
12 Dec 2023	Phi-2		SQuADv2 (2-shot) (in Language Understanding average only)
10 May 2023	PaLM 2-L	80.5	SQuAD v2 (EM), 1-shot

7 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
23 Nov 2024	Llama 3.1-8B technical report	77	SQuAD, pretrained (Table 9). Table 8 lists the reading benchmark as SQuAD V2	Capabilities
23 Nov 2024	Llama 3.1-70B technical report	81.8	SQuAD, pretrained (Table 9). Table 8 lists the reading benchmark as SQuAD V2	Capabilities
23 Nov 2024	Llama 3.1-405B technical report	81.8	SQuAD, pretrained (Table 9). Table 8 lists the reading benchmark as SQuAD V2	Capabilities
19 Jul 2023	Llama 2-7B technical report	67.2	SQuAD (EM), 0-shot, cited as Rajpurkar et al. 2018, pretrained (Table 23)	Capabilities
19 Jul 2023	Llama 2-70B technical report	80.7	SQuAD (EM), 0-shot, cited as Rajpurkar et al. 2018, pretrained (Table 23)	Capabilities
19 Jul 2023	Llama 2-34B technical report	77.4	SQuAD (EM), 0-shot, cited as Rajpurkar et al. 2018, pretrained (Table 23)	Capabilities
19 Jul 2023	Llama 2-13B technical report	72.9	SQuAD (EM), 0-shot, cited as Rajpurkar et al. 2018, pretrained (Table 23)	Capabilities

Versions

Replaces SQuAD