Benchmark Almanac

MMLU-Redux

Qwen3.7-Max 95% leads since May 2026 Fading

Manually re-annotated subset of MMLU that fixes wrong labels. Multiple-choice accuracy.

Saturated. This first release has 3,000 questions in 30 subjects. The authors now recommend MMLU-Redux 2.0, and cards that write only MMLU-Redux do not say which version.

In the authors’ words

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

Abstract of Are We Done with MMLU? (arXiv:2406.04127, CC0)
Made by University of Edinburgh Released Jun 2024 Exam questions All tasks public Last in a launch post Jun 2026 , dropped after it saturated Last model added Jun 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
aryopg/mmlu-redux 33 stars Paper Dataset
38
Models scored in lab cards
26
Launch posts list it, from 6 labs
25
System cards report it, from 6 labs

Qwen3.7-Max leads with 95% among models released by Jun 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Jun 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

38 models. Scroll sideways for the rest.

  1. 95.0 Qwen3.7-Max
  2. 94.9 Qwen3.5 397B-A17B
  3. 94.5 Qwen 3.6 Plus
  4. 94.5 Qwen3.7-Plus
  5. 94.4 Kimi K2 Thinking
  6. 94.2 Qwen3.5-Omni-Plus
  7. 94.0 Qwen3.5-122B-A10B
  8. 93.8 Qwen3-235B-A22B-Thinking
  9. 93.7 DeepSeek-V3.1
  10. 93.7 Qwen3-VL-235B-A22B
  11. 93.5 Qwen3.6 27B
  12. 93.4 DeepSeek-R1
  13. 93.3 Qwen 3.6 35B-A3B
  14. 92.8 Qwen3-Max
  15. 92.7 Qwen3-235B-A22B
  16. 92.7 Kimi K2
  17. 92.5 Qwen3-Next-80B-A3B
  18. 92.3 EXAONE 4.0 32B
  19. 91.2 qwen3-coder-next
  20. 90.9 Qwen3-32B
  21. 90.8 DeepSeek-V4-Pro
  22. 89.5 Qwen3-30B-A3B
  23. 89.3 LongCat-Flash-Thinking
  24. 89.1 DeepSeek-V3
  25. 88.6 Qwen3-14B
  26. 87.5 Qwen3-8B
  27. 86.8 Qwen2.5-72B
  28. 86.6 Qwen3-Omni-30B-A3B
  29. 86.3 Nemotron-Cascade 2 30B-A3B
  30. 85.9 Qwen2.5-VL-72B
  31. 83.9 Qwen2.5-32B
  32. 83.7 Qwen3-4B
  33. 81.7 Qwen-Turbo
  34. 80.7 Qwen2.5-14B-Instruct-1M
  35. 80.0 Qwen2.5-14B
  36. 75.4 Qwen2.5-7B
  37. 73.9 Qwen3-1.7B
  38. 71.0 Qwen2.5-Omni-7B
  1. 1 Qwen3.7-Max 95%
  2. 2 Qwen3.5 397B-A17B 94.9%
  3. 3 Qwen 3.6 Plus 94.5%
  4. 4 Qwen3.7-Plus 94.5%
  5. 5 Kimi K2 Thinking 94.4%
  6. 6 Qwen3.5-Omni-Plus 94.2%
  7. 7 Qwen3.5-122B-A10B 94%
  8. 8 Qwen3-235B-A22B-Thinking Jul 2025 93.8%
  9. 9 DeepSeek-V3.1 93.7%
  10. 10 Qwen3-VL-235B-A22B 93.7%
  11. 11 Qwen3.6 27B 93.5%
  12. 12 DeepSeek-R1 93.4%
  13. 13 Qwen 3.6 35B-A3B 93.3%
  14. 14 Qwen3-Max 92.8%
  15. 15 Qwen3-235B-A22B 92.7%
  16. 16 Kimi K2 Jul 2025 92.7%
  17. 17 Qwen3-Next-80B-A3B 92.5%
  18. 18 EXAONE 4.0 32B 92.3%
  19. 19 qwen3-coder-next 91.2%
  20. 20 Qwen3-32B 90.9%
  21. 21 DeepSeek-V4-Pro 90.8%
  22. 22 Qwen3-30B-A3B 89.5%
  23. 23 LongCat-Flash-Thinking 89.3%
  24. 24 DeepSeek-V3 89.1%
  25. 25 Qwen3-14B 88.6%
  26. 26 Qwen3-8B 87.5%
  27. 27 Qwen2.5-72B 86.8%
  28. 28 Qwen3-Omni-30B-A3B 86.6%
  29. 29 Nemotron-Cascade 2 30B-A3B 86.3%
  30. 30 Qwen2.5-VL-72B 85.9%
  31. 31 Qwen2.5-32B 83.9%
  32. 32 Qwen3-4B 83.7%
  33. 33 Qwen-Turbo 81.7%
  34. 34 Qwen2.5-14B-Instruct-1M 80.7%
  35. 35 Qwen2.5-14B 80%
  36. 36 Qwen2.5-7B 75.4%
  37. 37 Qwen3-1.7B 73.9%
  38. 38 Qwen2.5-Omni-7B 71%
#ModelScoreSettingDateSource
1 Qwen3.7-Max 95% May 2026 Model card
2 Qwen3.5 397B-A17B 94.9% Feb 2026 Model card
3 Qwen 3.6 Plus 94.5% Apr 2026 Model card
4 Qwen3.7-Plus 94.5% Jun 2026 Model card
5 Kimi K2 Thinking 94.4% Nov 2025 Model card
6 Qwen3.5-Omni-Plus 94.2% Mar 2026 Model card
7 Qwen3.5-122B-A10B 94% Feb 2026 Model card
8 Qwen3-235B-A22B-Thinking Jul 2025 93.8% Jul 2025 Model card
9 DeepSeek-V3.1 93.7% Aug 2025 Model card
10 Qwen3-VL-235B-A22B 93.7% Sep 2025 Model card
11 Qwen3.6 27B 93.5% Apr 2026 Model card
12 DeepSeek-R1 93.4% May 2025 Model card
13 Qwen 3.6 35B-A3B 93.3% Apr 2026 Model card
14 Qwen3-Max 92.8% Jan 2026 Model card
15 Qwen3-235B-A22B 92.7% Apr 2025 Model card
16 Kimi K2 Jul 2025 92.7% Jul 2025 Model card
17 Qwen3-Next-80B-A3B 92.5% Sep 2025 Model card
18 EXAONE 4.0 32B 92.3% Jul 2025 Model card
19 qwen3-coder-next 91.2% Feb 2026 Model card
20 Qwen3-32B 90.9% May 2025 Model card
21 DeepSeek-V4-Pro 90.8% Apr 2026 Model card
22 Qwen3-30B-A3B 89.5% May 2025 Model card
23 LongCat-Flash-Thinking 89.3% Sep 2025 Model card
24 DeepSeek-V3 89.1% Dec 2024 Model card
25 Qwen3-14B 88.6% May 2025 Model card
26 Qwen3-8B 87.5% May 2025 Model card
27 Qwen2.5-72B 86.8% Sep 2024 Model card
28 Qwen3-Omni-30B-A3B 86.6% Sep 2025 Model card
29 Nemotron-Cascade 2 30B-A3B 86.3% Mar 2026 Model card
30 Qwen2.5-VL-72B 85.9% Feb 2025 Model card
31 Qwen2.5-32B 83.9% Dec 2024 Model card
32 Qwen3-4B 83.7% May 2025 Model card
33 Qwen-Turbo 81.7% Dec 2024 Model card
34 Qwen2.5-14B-Instruct-1M 80.7% Jan 2025 Model card
35 Qwen2.5-14B 80% Dec 2024 Model card
36 Qwen2.5-7B 75.4% Dec 2024 Model card
37 Qwen3-1.7B 73.9% May 2025 Model card
38 Qwen2.5-Omni-7B 71% Mar 2025 Model card

7 models have held number one since its release in Jun 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen DeepSeek Moonshot AI
Lab claims Passed 85% in Sep 2024, by Qwen2.5-72B
Qwen3.7-Max 95%

The best score passed 85% in Sep 2024

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.7-Max Kimi K2 Thinking DeepSeek-R1 DeepSeek-R1 Qwen2.5-72B 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen3.7-Max DeepSeek-R1

Labs headline it less often now, last in Jun 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
DeepSeek
Moonshot AI
LG AI Research
Meituan
NVIDIA
DateLaunch postScoreAs the post writes it
1 Jun 2026 Qwen3.7-Plus 94.5 MMLU-Redux
20 May 2026 Qwen3.7-Max 95 MMLU-Redux
24 Apr 2026 DeepSeek-V4-Pro 90.8 MMLU-Redux (EM)
22 Apr 2026 Qwen3.6 27B 93.5 MMLU-Redux
15 Apr 2026 Qwen 3.6 35B-A3B 93.3 MMLU-Redux
2 Apr 2026 Qwen 3.6 Plus 94.5 MMLU-Redux
30 Mar 2026 Qwen3.5-Omni-Plus 94.2 MMLU-Redux
19 Mar 2026 Nemotron-Cascade 2 30B-A3B 86.3 MMLU-Redux
24 Feb 2026 Qwen3.5-122B-A10B 94 MMLU-Redux
16 Feb 2026 Qwen3.5 397B-A17B 94.9 MMLU-Redux
23 Jan 2026 Qwen3-Max 92.8 MMLU-Redux
6 Nov 2025 Kimi K2 Thinking 94.4 MMLU-Redux, no tools
23 Sep 2025 Qwen3-VL-235B-A22B 93.7 MMLU-Redux
22 Sep 2025 Qwen3-Omni-30B-A3B 86.6 MMLU-Redux
21 Sep 2025 LongCat-Flash-Thinking 89.3 MMLU-Redux(acc)
11 Sep 2025 Qwen3-Next-80B-A3B 92.5 MMLU-Redux
21 Aug 2025 DeepSeek-V3.1 93.7 MMLU-Redux (EM)
25 Jul 2025 Qwen3-235B-A22B-Thinking 93.8 MMLU-Redux
15 Jul 2025 EXAONE 4.0 32B 92.3 MMLU-Redux (reasoning)
11 Jul 2025 Kimi K2 92.7 MMLU-Redux, EM
28 May 2025 DeepSeek-R1 93.4 MMLU-Redux (EM)
29 Apr 2025 Qwen3-235B-A22B 92.7 MMLU-Redux
27 Mar 2025 Qwen2.5-Omni-7B 71 MMLU-redux
20 Jan 2025 DeepSeek-R1 92.9 MMLU-Redux (EM)
26 Dec 2024 DeepSeek-V3 89.1 MMLU-Redux (EM)
19 Sep 2024 Qwen2.5-72B 86.8 MMLU-redux

25 system cards report it, from 6 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
17 Apr 2026 Qwen3.5-Omni-Plus technical report 94.2 MMLU-Redux (Table 4) Capabilities
19 Mar 2026 Nemotron-Cascade 2 30B-A3B technical report 86.3 MMLU-Redux (Table 1) Capabilities
28 Feb 2026 qwen3-coder-next technical report 91.2 MMLU-Redux (Table 8) Capabilities
26 Nov 2025 Qwen3-VL-235B-A22B technical report 93.7 MMLU-Redux (Table 6) Capabilities
23 Sep 2025 LongCat-Flash-Thinking technical report 89.3 MMLU-Redux(acc) (Table 2) Capabilities
22 Sep 2025 Qwen3-Omni-30B-A3B technical report 86.6 MMLU-Redux, Instruct (Table 4). Thinking 88.8 (Table 5) Capabilities
21 Jul 2025 Kimi K2 technical report 92.7 MMLU-Redux (EM) (Table 3) Capabilities
15 Jul 2025 EXAONE 4.0 32B technical report 92.3 MMLU-Redux (Table 3, Reasoning). Non-reasoning 89.8 (Table 4) Capabilities
14 May 2025 Qwen3-8B technical report 87.5 MMLU-Redux, thinking mode (Table 17). Non-thinking 79.5 (Table 18) Capabilities
14 May 2025 Qwen3-4B technical report 83.7 MMLU-Redux, thinking mode (Table 17). Non-thinking 77.3 (Table 18) Capabilities
14 May 2025 Qwen3-32B technical report 90.9 MMLU-Redux, thinking mode (Table 13). Non-thinking 85.7 (Table 14) Capabilities
14 May 2025 Qwen3-30B-A3B technical report 89.5 MMLU-Redux, thinking mode (Table 15). Non-thinking 84.1 (Table 16) Capabilities
14 May 2025 Qwen3-235B-A22B technical report 92.7 MMLU-Redux, thinking mode (Table 11). Non-thinking 89.2 (Table 12) Capabilities
14 May 2025 Qwen3-14B technical report 88.6 MMLU-Redux, thinking mode (Table 15). Non-thinking 82.0 (Table 16) Capabilities
14 May 2025 Qwen3-1.7B technical report 73.9 MMLU-Redux, thinking mode (Table 19). Non-thinking 64.4 (Table 20) Capabilities
26 Mar 2025 Qwen2.5-Omni-7B technical report 71 MMLU-redux (Table 1) Capabilities
19 Feb 2025 Qwen2.5-VL-72B technical report 85.9 MMLU-redux (Table 4) Capabilities
26 Jan 2025 Qwen2.5-14B-Instruct-1M technical report 80.7 MMLU-redux (Table 6) Capabilities
20 Jan 2025 DeepSeek-R1 technical report 92.9 MMLU-Redux (EM), DeepSeek-R1 (Table 8, same as final column of Table 3) Capabilities
26 Dec 2024 DeepSeek-V3 technical report 89.1 MMLU-Redux (EM), DeepSeek-V3 (Table 6) Capabilities
19 Dec 2024 Qwen2.5-7B technical report 75.4 MMLU-redux, Qwen2.5-7B-Instruct (Table 8) Capabilities
19 Dec 2024 Qwen2.5-72B technical report 86.8 MMLU-redux, Qwen2.5-72B-Instruct (Table 6) Capabilities
19 Dec 2024 Qwen2.5-32B technical report 83.9 MMLU-redux, Qwen2.5-32B-Instruct (Table 7) Capabilities
19 Dec 2024 Qwen2.5-14B technical report 80 MMLU-redux, Qwen2.5-14B-Instruct (Table 7) Capabilities
19 Dec 2024 Qwen-Turbo technical report 81.7 MMLU-redux, Qwen2.5-Turbo (Table 7) Capabilities

Versions

Replaced by MMLU-Redux 2.0

More in Knowledge

All 109 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • BIG-Bench Extra Hard In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Gemma 4 31B IT 74.4%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%