Spider

GPT-4 Jun 2023 86.6% leads since Jun 2023 Dropped

Models turn natural language questions into SQL over 200 multi-table databases from 138 domains. Test databases are unseen in training, and answers are scored by exact match or execution.

10,181 questions and 5,693 unique SQL queries annotated by 11 Yale students. The Spider site announced Spider 2.0 as its successor in 2024.

In the authors’ words

We present Spider, a large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and the exact same programs in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 12.4% exact matching accuracy on a database split setting. This shows that Spider presents a strong challenge for future research. Our dataset and task are publicly available at https://yale-lily.github.io/spider

Abstract of Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task (arXiv:1809.08887, CC0)

Made by Yale University (LILY lab, Tao Yu et al.) Released Sep 2018 Data science and SQL Public tasks plus a private test set Last in a launch post Jan 2025 , dropped after it saturated Last model added Feb 2026 Scores refreshed 2 Nov 2026

Level: University Undergraduates and entry-level professionals
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Code A program or patch run against tests.

taoyds/spider 1,108 stars Paper Leaderboard yale-lily.github.io

Entries on its own leaderboard, 5 models we track

Launch post lists it, from 1 lab

System cards report it, from 3 labs

GPT-4 leads with 86.6% among models released by Jun 2023

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

Bars Table

The newest model with a score here came out in Jun 2023, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

1 GPT-4 Jun 2023 86.6%
2 GPT-4 Mar 2023 85.3%
3 GPT-3.5 Turbo (Mar 2023) 82.3%
4 CodeX 78.2%
5 T5-3B 78%
6 T5-Base 71.1%

#	Entry	Score	Setting	Date	Source
1	GPT-4 Jun 2023	86.6%	DAIL-SQL + GPT-4 + Self-Consistency	Jun 2023	Official leaderboard
2	GPT-4 Mar 2023	85.3%	DIN-SQL + GPT-4	Mar 2023	Official leaderboard
3	GPT-3.5 Turbo (Mar 2023)	82.3%	C3 + ChatGPT + Zero-Shot	Mar 2023	Official leaderboard
4	CodeX	78.2%	DIN-SQL + CodeX	Apr 2023	Official leaderboard
5	T5-3B	78%	T5-3B+NatSQL+Token Preprocessing	Oct 2019	Official leaderboard
6	T5-Base	71.1%	T5-Base+NatSQL+Token Preprocessing	Aug 2021	Official leaderboard

Labs report scores for 4 models in their cards. We plot every source in the charts below.

4 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Google DeepMind OpenAI Mistral AI Cohere Qwen

Own leaderboard Passed 85% in Mar 2023, by GPT-4

GPT-4 86.6%

Lab claims Not tested since Feb 2026

qwen3-coder-next 83.7%

The best score passed 85% in Mar 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Mistral AI

Date	Launch post	Score	As the post writes it
13 Jan 2025	Codestral	66.5	Spider (SQL)

3 system cards report it, from 3 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

Date	Document	Score	As the document writes it	Section
28 Feb 2026	qwen3-coder-next technical report	83.7	Spider (Table 7)	Capabilities
1 Apr 2025	Cohere Command A technical report	80.2	Spider Test execution accuracy (Table 14). Dev 79.5	Capabilities
3 Mar 2025	Phi-4 Mini technical report	42.2	Spider (4-Shot) (Table 8)	Capabilities

Versions

Replaced by Spider 2.0