SuperGLUE
T5-11B 88.9% leads since Oct 2019 Not in model cards Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score.
Saturated, with models above the human baseline since 2021.
In the authors’ words
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.
- Level
- Everyday Most adults, without training
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Multiple choice
T5-11B leads with 88.9% among models released by Jan 2022
Best score per model in Epoch AI’s data. Where a model ran with several settings or harnesses, we keep its best.
The newest model with a score here came out in Jan 2022, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 T5-11B 88.9%
- 2 T5-3B 86.4%
- 3 Switch-Large 84.7%
- 4 T5-Large 82.7%
- 5 T5-Base 76.2%
- 6 Switch-Base 73.3%
- 7 InstructGPT 175B 71.8%
- 8 T5-Small 63.3%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | T5-11B | 88.9% | Oct 2019 | Epoch AI | |
| 2 | T5-3B | 86.4% | Oct 2019 | Epoch AI | |
| 3 | Switch-Large | 84.7% | Epoch AI | ||
| 4 | T5-Large | 82.7% | Epoch AI | ||
| 5 | T5-Base | 76.2% | Epoch AI | ||
| 6 | Switch-Base | 73.3% | Epoch AI | ||
| 7 | InstructGPT 175B | 71.8% | Jan 2022 | Epoch AI | |
| 8 | T5-Small | 63.3% | Epoch AI |
The benchmark's own leaderboard has 26 entries, 3 of them models we track. We plot every source in the charts below.
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and Epoch AI’s runs.
The best score passed 85% in Oct 2019
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Knowledge
- MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects.
Intern-S2-397B 89.8% - SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy.
Qwen 3.6 Max (Preview) 73.9% - WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1.
Tencent Hy4 preview 83.9% - KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry.
Solar Pro 4 79.2% - BIG-Bench Extra Hard In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Gemma 4 31B IT 74.4%
- HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed.
Seed2.0 Pro 73.6%