Benchmark Almanac

MMLU

Seed1.8 92.3% leads since Dec 2025 Fading

57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy.

Saturated and contaminated. Reported only as a base-model metric.

In the authors’ words

We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Abstract of Measuring Massive Multitask Language Understanding (arXiv:2009.03300, CC0)
Made by UC Berkeley (Hendrycks et al.) Released Sep 2020 Exam questions All tasks public Last in a launch post Aug 2026 , dropped after it saturated Last model added Aug 2026 Scores refreshed 31 Oct 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
hendrycks/test 1,625 stars Paper Dataset
127
Models scored in lab cards
99
Launch posts list it, from 29 labs
87
System cards report it, from 26 labs

Seed1.8 leads with 92.3% among models released by Aug 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Aug 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

127 models. Scroll sideways for the rest.

  1. 92.3 Seed1.8
  2. 91.8 o1
  3. 91.7 Gemini 1.5 Pro
  4. 91.3 Cogito v2 Preview 671B MoE
  5. 91.0 Cogito v1 Preview 70B
  6. 90.8 o1-preview
  7. 90.8 DeepSeek-R1
  8. 90.6 Qwen3-VL-235B-A22B
  9. 90.6 Sarvam 105B
  10. 90.5 Claude 3.5 Sonnet (October 2024)
  11. 90.4 Claude 3.5 Sonnet
  12. 90.3 LongCat-Flash-Omni
  13. 90.1 DeepSeek-V4-Pro
  14. 90.0 Gemini 1.0 Ultra
  15. 90.0 gpt-oss-120b
  16. 90.0 GLM-4.5
  17. 89.7 LongCat-Flash-Chat
  18. 89.5 Hunyuan-TurboS
  19. 89.5 Kimi K2
  20. 89.1 InternVL3.5-241B-A28B
  21. 88.7 GPT-4o
  22. 88.6 Llama 3.1-405B
  23. 88.5 DeepSeek-V3
  24. 88.5 MiniMax-Text-01
  25. 88.3 GLM-5
  26. 88.2 Claude 3 Opus
  27. 88.2 Solar Open 100B
  28. 88.2 Solar Pro 2
  29. 87.7 qwen3-coder-next
  30. 87.7 Hermes 4.3 36B
  31. 87.5 Grok-2
  32. 87.5 GPT-4.1 mini
  33. 87.4 Kimi k1.5
  34. 87.4 Amazon Nova Premier
  35. 87.4 GLM-4.5-Air
  36. 87.4 seed-oss-36b-instruct
  37. 87.2 Trinity Large Preview
  38. 87.2 Hermes 4 405B
  39. 87.1 sonar
  40. 87.0 Tulu 3 405B
  41. 87.0 Sarvam-M
  42. 86.9 o3-mini
  43. 86.7 GPT-4 Turbo
  44. 86.5 ERNIE-4.5-300B-A47B
  45. 86.4 GPT-4
  46. 86.4 Olmo 3.1 32B Think
  47. 86.1 Qwen2.5-72B
  48. 86.0 Llama 3.1-70B
  49. 86.0 Llama 3.3 70B Instruct
  50. 85.5 Cohere Command A
  51. 85.5 Llama 4 Maverick
  52. 85.4 Olmo 3 32B Think
  53. 85.3 gpt-oss-20b
  54. 85.1 Sarvam 30B
  55. 85.0 Trinity Mini
  56. 84.8 Phi-4
  57. 84.0 Falcon-H1-34B
  58. 84.0 Mistral Large 2
  59. 83.0 EXAONE Deep 32B
  60. 82.3 Qwen2-72B
  61. 82.0 Llama 3-70B
  62. 82.0 GPT-4o mini
  63. 82.0 Kimi-VL-A3B-Thinking-2506
  64. 81.9 Gemini 1.5 Pro
  65. 81.5 GLM-4
  66. 81.5 Claude 3 Sonnet
  67. 81.3 Grok-1.5
  68. 81.2 Mistral Large
  69. 80.9 Claude 3.5 Haiku
  70. 80.7 Mistral Small 3
  71. 80.6 Mistral Small 3.1
  72. 80.5 Mistral Small 3.2
  73. 80.2 Apple Foundation Model Server (2025)
  74. 80.2 Granite 4.1 30B
  75. 80.1 GPT-4.1 nano
  76. 79.6 Llama 4 Scout
  77. 78.9 Phi-3.5-MoE
  78. 78.6 Gemma 3 27B
  79. 78.5 Claude 2
  80. 78.4 Granite 4.0 H Small
  81. 78.2 phi-3-medium 14B
  82. 77.8 DeepSeek-V2 (MoE-236B, May 2024)
  83. 77.7 Code World Model (CWM)
  84. 77.5 Qwen1.5-72B
  85. 77.3 OLMo 2 32B Instruct
  86. 76.7 Claude 3 Haiku
  87. 76.6 InternLM3-8B-Instruct
  88. 76.2 Gemma 2 27B
  89. 75.7 phi-3-small 7.4B
  90. 73.0 Grok-1
  91. 73.0 Llama 3.1-8B
  92. 72.5 Mixtral 8x7B
  93. 72.3 Gemma 2 9B
  94. 71.9 Gemma 3 12B
  95. 71.1 DeepSeek LLM 67B
  96. 70.0 Moonlight-16B-A3B
  97. 69.6 Apertus 70B
  98. 69.0 Phi-3.5-mini
  99. 68.9 Llama 2-70B
  100. 68.8 phi-3-mini 3.8B
  101. 68.4 Llama 3-8B
  102. 67.3 Phi-4 Mini
  103. 66.8 Granite 3.2 8B Instruct
  104. 66.5 Qwen-14B
  105. 65.9 AFM-4.5B
  106. 65.5 Granite 3.3 8B Instruct
  107. 64.9 Gemma 3n E4B
  108. 64.8 LFM2-8B-A1B
  109. 64.3 Gemma 7B
  110. 63.3 Falcon-Arabic-7B
  111. 62.6 Llama 2-34B
  112. 61.4 ChatGLM3-6B
  113. 60.1 Mistral 7B v0.1
  114. 59.4 granite-4.0-1b
  115. 58.1 Gemma 3 4B
  116. 57.0 Qwen-7B
  117. 56.7 Phi-2
  118. 55.2 LFM2-1.2B
  119. 54.8 Llama 2-13B
  120. 53.2 BitNet b1.58 2B4T
  121. 51.0 LFM2-VL-1.6B
  122. 50.4 North Micro Vision Instruct
  123. 45.3 Llama 2-7B
  124. 43.9 Qwen-1_8B
  125. 42.3 Gemma 2B
  126. 40.0 OLMo 2 1B Instruct
  127. 38.8 Gemma 3 1B
  1. 1 Seed1.8 92.3%
  2. 2 o1 91.8%
  3. 3 Gemini 1.5 Pro May 2024 91.7%
  4. 4 Cogito v2 Preview 671B MoE 91.3%
  5. 5 Cogito v1 Preview 70B 91%
  6. 6 o1-preview 90.8%
  7. 7 DeepSeek-R1 90.8%
  8. 8 Qwen3-VL-235B-A22B 90.6%
  9. 9 Sarvam 105B 90.6%
  10. 10 Claude 3.5 Sonnet (October 2024) 90.5%
  11. 11 Claude 3.5 Sonnet 90.4%
  12. 12 LongCat-Flash-Omni 90.3%
  13. 13 DeepSeek-V4-Pro 90.1%
  14. 14 Gemini 1.0 Ultra 90%
  15. 15 gpt-oss-120b 90%
  16. 16 GLM-4.5 90%
  17. 17 LongCat-Flash-Chat 89.7%
  18. 18 Hunyuan-TurboS 89.5%
  19. 19 Kimi K2 Jul 2025 89.5%
  20. 20 InternVL3.5-241B-A28B 89.1%
  21. 21 GPT-4o May 2024 88.7%
  22. 22 Llama 3.1-405B 88.6%
  23. 23 DeepSeek-V3 88.5%
  24. 24 MiniMax-Text-01 88.5%
  25. 25 GLM-5 88.3%
  26. 26 Claude 3 Opus 88.2%
  27. 27 Solar Open 100B 88.2%
  28. 28 Solar Pro 2 88.2%
  29. 29 qwen3-coder-next 87.7%
  30. 30 Hermes 4.3 36B 87.7%
  31. 31 Grok-2 Aug 2024 87.5%
  32. 32 GPT-4.1 mini 87.5%
  33. 33 Kimi k1.5 87.4%
  34. 34 Amazon Nova Premier 87.4%
  35. 35 GLM-4.5-Air 87.4%
  36. 36 seed-oss-36b-instruct 87.4%
  37. 37 Trinity Large Preview 87.2%
  38. 38 Hermes 4 405B 87.2%
  39. 39 sonar 87.1%
  40. 40 Tulu 3 405B 87%
  41. 41 Sarvam-M 87%
  42. 42 o3-mini 86.9%
  43. 43 GPT-4 Turbo Apr 2024 86.7%
  44. 44 ERNIE-4.5-300B-A47B 86.5%
  45. 45 GPT-4 Mar 2023 86.4%
  46. 46 Olmo 3.1 32B Think 86.4%
  47. 47 Qwen2.5-72B 86.1%
  48. 48 Llama 3.1-70B 86%
  49. 49 Llama 3.3 70B Instruct 86%
  50. 50 Cohere Command A 85.5%
  51. 51 Llama 4 Maverick 85.5%
  52. 52 Olmo 3 32B Think 85.4%
  53. 53 gpt-oss-20b 85.3%
  54. 54 Sarvam 30B 85.1%
  55. 55 Trinity Mini 85%
  56. 56 Phi-4 84.8%
  57. 57 Falcon-H1-34B 84%
  58. 58 Mistral Large 2 Jul 2024 84%
  59. 59 EXAONE Deep 32B 83%
  60. 60 Qwen2-72B 82.3%
  61. 61 Llama 3-70B 82%
  62. 62 GPT-4o mini 82%
  63. 63 Kimi-VL-A3B-Thinking-2506 82%
  64. 64 Gemini 1.5 Pro Feb 2024 81.9%
  65. 65 GLM-4 81.5%
  66. 66 Claude 3 Sonnet 81.5%
  67. 67 Grok-1.5 81.3%
  68. 68 Mistral Large 81.2%
  69. 69 Claude 3.5 Haiku 80.9%
  70. 70 Mistral Small 3 80.7%
  71. 71 Mistral Small 3.1 80.6%
  72. 72 Mistral Small 3.2 80.5%
  73. 73 Apple Foundation Model Server (2025) 80.2%
  74. 74 Granite 4.1 30B 80.2%
  75. 75 GPT-4.1 nano 80.1%
  76. 76 Llama 4 Scout 79.6%
  77. 77 Phi-3.5-MoE 78.9%
  78. 78 Gemma 3 27B 78.6%
  79. 79 Claude 2 78.5%
  80. 80 Granite 4.0 H Small 78.4%
  81. 81 phi-3-medium 14B 78.2%
  82. 82 DeepSeek-V2 (MoE-236B, May 2024) 77.8%
  83. 83 Code World Model (CWM) 77.7%
  84. 84 Qwen1.5-72B 77.5%
  85. 85 OLMo 2 32B Instruct 77.3%
  86. 86 Claude 3 Haiku 76.7%
  87. 87 InternLM3-8B-Instruct 76.6%
  88. 88 Gemma 2 27B 76.2%
  89. 89 phi-3-small 7.4B 75.7%
  90. 90 Grok-1 73%
  91. 91 Llama 3.1-8B 73%
  92. 92 Mixtral 8x7B 72.5%
  93. 93 Gemma 2 9B 72.3%
  94. 94 Gemma 3 12B 71.9%
  95. 95 DeepSeek LLM 67B 71.1%
  96. 96 Moonlight-16B-A3B 70%
  97. 97 Apertus 70B 69.6%
  98. 98 Phi-3.5-mini 69%
  99. 99 Llama 2-70B 68.9%
  100. 100 phi-3-mini 3.8B 68.8%
  101. 101 Llama 3-8B 68.4%
  102. 102 Phi-4 Mini 67.3%
  103. 103 Granite 3.2 8B Instruct 66.8%
  104. 104 Qwen-14B 66.5%
  105. 105 AFM-4.5B 65.9%
  106. 106 Granite 3.3 8B Instruct 65.5%
  107. 107 Gemma 3n E4B 64.9%
  108. 108 LFM2-8B-A1B 64.8%
  109. 109 Gemma 7B 64.3%
  110. 110 Falcon-Arabic-7B 63.3%
  111. 111 Llama 2-34B 62.6%
  112. 112 ChatGLM3-6B 61.4%
  113. 113 Mistral 7B v0.1 60.1%
  114. 114 granite-4.0-1b 59.4%
  115. 115 Gemma 3 4B 58.1%
  116. 116 Qwen-7B 57%
  117. 117 Phi-2 56.7%
  118. 118 LFM2-1.2B 55.2%
  119. 119 Llama 2-13B 54.8%
  120. 120 BitNet b1.58 2B4T 53.2%
  121. 121 LFM2-VL-1.6B 51%
  122. 122 North Micro Vision Instruct 50.4%
  123. 123 Llama 2-7B 45.3%
  124. 124 Qwen-1_8B 43.9%
  125. 125 Gemma 2B 42.3%
  126. 126 OLMo 2 1B Instruct 40%
  127. 127 Gemma 3 1B 38.8%
#ModelScoreSettingDateSource
1 Seed1.8 92.3% Dec 2025 Model card
2 o1 91.8% Dec 2024 Model card
3 Gemini 1.5 Pro May 2024 91.7% May 2024 Model card
4 Cogito v2 Preview 671B MoE 91.3% Jul 2025 Model card
5 Cogito v1 Preview 70B 91% Apr 2025 Model card
6 o1-preview 90.8% Sep 2024 Model card
7 DeepSeek-R1 90.8% Jan 2025 Model card
8 Qwen3-VL-235B-A22B 90.6% Sep 2025 Model card
9 Sarvam 105B 90.6% Mar 2026 Model card
10 Claude 3.5 Sonnet (October 2024) 90.5% Oct 2024 Model card
11 Claude 3.5 Sonnet 90.4% Jun 2024 Model card
12 LongCat-Flash-Omni 90.3% Oct 2025 Model card
13 DeepSeek-V4-Pro 90.1% Apr 2026 Model card
14 Gemini 1.0 Ultra 90% Dec 2023 Model card
15 gpt-oss-120b 90% Aug 2025 Model card
16 GLM-4.5 90% Aug 2025 Model card
17 LongCat-Flash-Chat 89.7% Aug 2025 Model card
18 Hunyuan-TurboS 89.5% Feb 2025 Model card
19 Kimi K2 Jul 2025 89.5% Jul 2025 Model card
20 InternVL3.5-241B-A28B 89.1% Aug 2025 Model card
21 GPT-4o May 2024 88.7% May 2024 Model card
22 Llama 3.1-405B 88.6% Jul 2024 Model card
23 DeepSeek-V3 88.5% Dec 2024 Model card
24 MiniMax-Text-01 88.5% Jan 2025 Model card
25 GLM-5 88.3% Feb 2026 Model card
26 Claude 3 Opus 88.2% Mar 2024 Model card
27 Solar Open 100B 88.2% Dec 2025 Model card
28 Solar Pro 2 88.2% Jul 2025 Model card
29 qwen3-coder-next 87.7% Feb 2026 Model card
30 Hermes 4.3 36B 87.7% Dec 2025 Model card
31 Grok-2 Aug 2024 87.5% Aug 2024 Model card
32 GPT-4.1 mini 87.5% Apr 2025 Model card
33 Kimi k1.5 87.4% Jan 2025 Model card
34 Amazon Nova Premier 87.4% Apr 2025 Model card
35 GLM-4.5-Air 87.4% Aug 2025 Model card
36 seed-oss-36b-instruct 87.4% Aug 2025 Model card
37 Trinity Large Preview 87.2% Feb 2026 Model card
38 Hermes 4 405B 87.2% Aug 2025 Model card
39 sonar 87.1% Feb 2025 Model card
40 Tulu 3 405B 87% Jan 2025 Model card
41 Sarvam-M 87% May 2025 Model card
42 o3-mini 86.9% Jan 2025 Model card
43 GPT-4 Turbo Apr 2024 86.7% Apr 2024 Model card
44 ERNIE-4.5-300B-A47B 86.5% Jun 2025 Model card
45 GPT-4 Mar 2023 86.4% Mar 2023 Model card
46 Olmo 3.1 32B Think 86.4% Dec 2025 Model card
47 Qwen2.5-72B 86.1% Sep 2024 Model card
48 Llama 3.1-70B 86% Jul 2024 Model card
49 Llama 3.3 70B Instruct 86% Dec 2024 Model card
50 Cohere Command A 85.5% Apr 2025 Model card
51 Llama 4 Maverick 85.5% Apr 2025 Model card
52 Olmo 3 32B Think 85.4% Nov 2025 Model card
53 gpt-oss-20b 85.3% Aug 2025 Model card
54 Sarvam 30B 85.1% Mar 2026 Model card
55 Trinity Mini 85% Dec 2025 Model card
56 Phi-4 84.8% Dec 2024 Model card
57 Falcon-H1-34B 84% May 2025 Model card
58 Mistral Large 2 Jul 2024 84% Jul 2024 Model card
59 EXAONE Deep 32B 83% Mar 2025 Model card
60 Qwen2-72B 82.3% Jun 2024 Model card
61 Llama 3-70B 82% Apr 2024 Model card
62 GPT-4o mini 82% Jul 2024 Model card
63 Kimi-VL-A3B-Thinking-2506 82% Jun 2025 Model card
64 Gemini 1.5 Pro Feb 2024 81.9% Feb 2024 Model card
65 GLM-4 81.5% Jan 2024 Model card
66 Claude 3 Sonnet 81.5% Mar 2024 Model card
67 Grok-1.5 81.3% Mar 2024 Model card
68 Mistral Large 81.2% Feb 2024 Model card
69 Claude 3.5 Haiku 80.9% Oct 2024 Model card
70 Mistral Small 3 80.7% Jan 2025 Model card
71 Mistral Small 3.1 80.6% Mar 2025 Model card
72 Mistral Small 3.2 80.5% Jun 2025 Model card
73 Apple Foundation Model Server (2025) 80.2% Jul 2025 Model card
74 Granite 4.1 30B 80.2% Apr 2026 Model card
75 GPT-4.1 nano 80.1% Apr 2025 Model card
76 Llama 4 Scout 79.6% Apr 2025 Model card
77 Phi-3.5-MoE 78.9% Aug 2024 Model card
78 Gemma 3 27B 78.6% Mar 2025 Model card
79 Claude 2 78.5% Jul 2023 Model card
80 Granite 4.0 H Small 78.4% Oct 2025 Model card
81 phi-3-medium 14B 78.2% Apr 2024 Model card
82 DeepSeek-V2 (MoE-236B, May 2024) 77.8% May 2024 Model card
83 Code World Model (CWM) 77.7% Sep 2025 Model card
84 Qwen1.5-72B 77.5% Feb 2024 Model card
85 OLMo 2 32B Instruct 77.3% Mar 2025 Model card
86 Claude 3 Haiku 76.7% Mar 2024 Model card
87 InternLM3-8B-Instruct 76.6% Jan 2025 Model card
88 Gemma 2 27B 76.2% Jul 2024 Model card
89 phi-3-small 7.4B 75.7% Aug 2024 Model card
90 Grok-1 73% Nov 2023 Model card
91 Llama 3.1-8B 73% Jul 2024 Model card
92 Mixtral 8x7B 72.5% Dec 2023 Model card
93 Gemma 2 9B 72.3% Jul 2024 Model card
94 Gemma 3 12B 71.9% Mar 2025 Model card
95 DeepSeek LLM 67B 71.1% Nov 2023 Model card
96 Moonlight-16B-A3B 70% Feb 2025 Model card
97 Apertus 70B 69.6% Sep 2025 Model card
98 Phi-3.5-mini 69% Aug 2024 Model card
99 Llama 2-70B 68.9% Jul 2023 Model card
100 phi-3-mini 3.8B 68.8% Aug 2024 Model card
101 Llama 3-8B 68.4% Apr 2024 Model card
102 Phi-4 Mini 67.3% Feb 2025 Model card
103 Granite 3.2 8B Instruct 66.8% Feb 2025 Model card
104 Qwen-14B 66.5% Sep 2023 Model card
105 AFM-4.5B 65.9% Jul 2025 Model card
106 Granite 3.3 8B Instruct 65.5% Apr 2025 Model card
107 Gemma 3n E4B 64.9% Jun 2025 Model card
108 LFM2-8B-A1B 64.8% Oct 2025 Model card
109 Gemma 7B 64.3% Feb 2024 Model card
110 Falcon-Arabic-7B 63.3% Nov 2025 Model card
111 Llama 2-34B 62.6% Jul 2023 Model card
112 ChatGLM3-6B 61.4% Oct 2023 Model card
113 Mistral 7B v0.1 60.1% Sep 2023 Model card
114 granite-4.0-1b 59.4% Oct 2025 Model card
115 Gemma 3 4B 58.1% Mar 2025 Model card
116 Qwen-7B 57% Sep 2023 Model card
117 Phi-2 56.7% Dec 2023 Model card
118 LFM2-1.2B 55.2% Jul 2025 Model card
119 Llama 2-13B 54.8% Jul 2023 Model card
120 BitNet b1.58 2B4T 53.2% Apr 2025 Model card
121 LFM2-VL-1.6B 51% Aug 2025 Model card
122 North Micro Vision Instruct 50.4% Aug 2026 Model card
123 Llama 2-7B 45.3% Jul 2023 Model card
124 Qwen-1_8B 43.9% Sep 2023 Model card
125 Gemma 2B 42.3% Mar 2024 Model card
126 OLMo 2 1B Instruct 40% Apr 2025 Model card
127 Gemma 3 1B 38.8% Mar 2025 Model card

Epoch AI has scores for 127 models. We plot every source in the charts below.

6 models have held number one since 2023

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for Epoch AI’s runs and what labs claim in their cards.

OpenAI Anthropic Google DeepMind ByteDance Seed
Epoch AI runs Passed 85% in Mar 2023, by GPT-4
GPT-4o 88.1%
Lab claims Passed 85% in Mar 2023, by GPT-4
Seed1.8 92.3%

The best score passed 85% in Mar 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Run or collected by Epoch AI Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2022 2023 2024 2025 2026 Near the ceiling (85%) Random guessing Seed1.8 o1 Gemini 1.0 Ultra davinci-002 Gopher (280B) GPT-J-6B 0% 20% 40% 60% 80% 100% 2022 2023 2024 2025 2026 Near the ceiling (85%) Random guessing Seed1.8 GPT-4

Labs headline it less often now, last in Aug 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

OpenAI
Anthropic
Meta
Qwen
Mistral AI
Z.ai
xAI
DeepSeek
Google DeepMind
Microsoft
Shanghai AI Lab
MiniMax
Moonshot AI
Ai2
Perplexity
IBM
Tencent
Cohere
Deep Cogito
Amazon
Technology Innovation Institute
Sarvam AI
Baidu
Upstage
Liquid AI
Arcee AI
ByteDance Seed
Nous Research
Meituan
DateLaunch postScoreAs the post writes it
12 Aug 2026 North Micro Vision Instruct 50.4 MMLU test
29 Apr 2026 Granite 4.1 30B 80.2 MMLU (5-shot)
24 Apr 2026 DeepSeek-V4-Pro 90.1 MMLU (EM)
6 Mar 2026 Sarvam 30B 85.1 MMLU
6 Mar 2026 Sarvam 105B 90.6 MMLU
27 Jan 2026 Trinity Large Preview 87.2 MMLU
31 Dec 2025 Solar Open 100B 88.2 MMLU
18 Dec 2025 Seed1.8 92.3 MMLU
12 Dec 2025 Olmo 3.1 32B Think 86.4 MMLU
3 Dec 2025 Hermes 4.3 36B 87.7 MMLU
1 Dec 2025 Trinity Mini 85 MMLU (Zero Shot)
20 Nov 2025 Olmo 3 32B Think 85.4 MMLU
31 Oct 2025 LongCat-Flash-Omni 90.3 MMLU(acc)
28 Oct 2025 granite-4.0-1b 59.4 MMLU (5-shot)
7 Oct 2025 LFM2-8B-A1B 64.8 MMLU
2 Oct 2025 Granite 4.0 H Small 78.4 MMLU (5-shot)
23 Sep 2025 Qwen3-VL-235B-A22B 90.6 MMLU
29 Aug 2025 LongCat-Flash-Chat 89.7 MMLU(acc)
26 Aug 2025 Hermes 4 405B 87.2 MMLU (reasoning)
20 Aug 2025 seed-oss-36b-instruct 87.4 MMLU
12 Aug 2025 LFM2-VL-1.6B 51 Text MMLU
5 Aug 2025 gpt-oss-20b 85.3 MMLU
5 Aug 2025 gpt-oss-120b 90 MMLU Questions across academic disciplines
31 Jul 2025 Cogito v2 Preview 671B MoE 91.3 MMLU (thinking)
29 Jul 2025 AFM-4.5B 65.9 MMLU
11 Jul 2025 Kimi K2 89.5 MMLU, EM
10 Jul 2025 LFM2-1.2B 55.2 MMLU
10 Jul 2025 Solar Pro 2 88.2 MMLU (0-shot, CoT)
30 Jun 2025 ERNIE-4.5-300B-A47B 86.5 MMLU
26 Jun 2025 Gemma 3n E4B 64.9 MMLU 0-shot (IT)
21 Jun 2025 Kimi-VL-A3B-Thinking-2506 82 MMLU
20 Jun 2025 Mistral Small 3.2 80.5 MMLU
23 May 2025 Sarvam-M 87 MMLU (0.87)
20 May 2025 Falcon-H1-34B 84 MMLU
30 Apr 2025 Amazon Nova Premier 87.4 MMLU
29 Apr 2025 OLMo 2 1B Instruct 40 MMLU
16 Apr 2025 Granite 3.3 8B Instruct 65.5 MMLU
15 Apr 2025 BitNet b1.58 2B4T 53.2 MMLU
14 Apr 2025 GPT-4.1 nano 80.1 MMLU
14 Apr 2025 GPT-4.1 mini 87.5 MMLU
8 Apr 2025 Cogito v1 Preview 70B 91 MMLU (thinking)
5 Apr 2025 Llama 4 Scout 79.6 MMLU (5-shot) [pretrained]
5 Apr 2025 Llama 4 Maverick 85.5 MMLU (5-shot) [pretrained]
17 Mar 2025 Mistral Small 3.1 80.6 MMLU
13 Mar 2025 Cohere Command A MMLU (chart only)
13 Mar 2025 OLMo 2 32B Instruct 77.3 MMLU
12 Mar 2025 Gemma 3 27B 78.6 MMLU 5-shot (PT)
27 Feb 2025 Hunyuan-TurboS 89.5 MMLU
26 Feb 2025 Phi-4 Mini 67.3 MMLU (5-shot)
26 Feb 2025 Granite 3.2 8B Instruct 66.8 MMLU
22 Feb 2025 Moonlight-16B-A3B 70 MMLU
11 Feb 2025 sonar 87.1 MMLU (0-shot)
31 Jan 2025 o3-mini 86.9 MMLU (pass@1)
30 Jan 2025 Tulu 3 405B 87 MMLU (5 shot, CoT)
30 Jan 2025 Mistral Small 3 80.7 Best of MMLU (5-shot) [base] / French MMLU [base] / German MMLU [base] / Spanish MMLU [base]
28 Jan 2025 Qwen2.5-Max MMLU
20 Jan 2025 Kimi k1.5 87.4 MMLU (EM), short-CoT
20 Jan 2025 DeepSeek-R1 90.8 MMLU (Pass@1)
15 Jan 2025 MiniMax-Text-01 88.5 MMLU* (0-shot CoT)
13 Jan 2025 InternLM3-8B-Instruct 76.6 MMLU(0-shot)
26 Dec 2024 DeepSeek-V3 88.5 MMLU (EM)
17 Dec 2024 o1 91.8 MMLU (pass @1)
12 Dec 2024 Phi-4 84.8 MMLU (simple-evals)
6 Dec 2024 Llama 3.3 70B Instruct 86 MMLU (CoT), 0-shot, macro_avg/acc
22 Oct 2024 Claude 3.5 Sonnet (October 2024) 90.5 MMLU (5-shot CoT)
19 Sep 2024 Qwen2.5-72B 86.1 MMLU
12 Sep 2024 o1-preview 90.8 MMLU pass@1
20 Aug 2024 Phi-3.5-MoE 78.9 MMLU (5-shot)
13 Aug 2024 Grok-2 87.5 MMLU (0-shot CoT)
24 Jul 2024 Mistral Large 2 84 Best of MMLU [base pretrained] / Multilingual MMLU French [base] / Multilingual MMLU German [base] / Multilingual MMLU Spanish [base] / Multilingual MMLU Italian [base]
23 Jul 2024 Llama 3.1-405B 88.6 MMLU (0-shot, CoT)
18 Jul 2024 GPT-4o mini 82 MMLU
27 Jun 2024 Gemma 2 27B 75.2 MMLU (5-shot, top-1)
20 Jun 2024 Claude 3.5 Sonnet 90.4 MMLU (5-shot CoT)
7 Jun 2024 Qwen2-72B 82.3 MMLU
14 May 2024 Gemini 1.5 Pro 91.7 MMLU, maj@32 (pretrained) (5-shot = 85.9%)
13 May 2024 GPT-4o 88.7 MMLU (0-shot CoT)
7 May 2024 DeepSeek-V2 (MoE-236B, May 2024) 77.8 MMLU
22 Apr 2024 phi-3-medium 14B 78.2 MMLU (5-Shot)
18 Apr 2024 Llama 3-70B 82 MMLU (5-shot) [instruct]
11 Apr 2024 GPT-4 Turbo 86.7 MMLU%
28 Mar 2024 Grok-1.5 81.3 MMLU (5-shot)
4 Mar 2024 Claude 3 Opus 88.2 MMLU (5-shot CoT)
26 Feb 2024 Mistral Large 81.2 Best of MMLU / French MMLU / German MMLU / Spanish MMLU / Italian MMLU
21 Feb 2024 Gemma 7B 64.3 MMLU (5-shot, top-1)
15 Feb 2024 Gemini 1.5 Pro 81.9 MMLU, 5-shot
5 Feb 2024 Qwen1.5-72B 77.5 MMLU
16 Jan 2024 GLM-4 81.5 MMLU (5-shot)
12 Dec 2023 Phi-2 56.7 MMLU, Table 2
11 Dec 2023 Mixtral 8x7B 72.5 Best of MMLU (MCQ in 57 subjects) / French MMLU / German MMLU / Spanish MMLU / Italian MMLU
6 Dec 2023 Gemini 1.0 Ultra 90 MMLU, CoT@32 (5-shot = 83.7%)
29 Nov 2023 DeepSeek LLM 67B 71.1 MMLU
3 Nov 2023 Grok-1 73 MMLU (5-shot)
27 Oct 2023 ChatGLM3-6B 61.4 MMLU
27 Sep 2023 Mistral 7B v0.1 60.1 MMLU
25 Sep 2023 Qwen-14B 66.5 MMLU (5-shot)
18 Jul 2023 Llama 2-70B 68.9 MMLU
11 Jul 2023 Claude 2 78.5 MMLU (5-shot CoT)
14 Mar 2023 GPT-4 86.4 MMLU (5-shot)

87 system cards report it, from 26 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
21 Mar 2026 Seed1.8 model card 92.3 MMLU (Table 1) Capabilities
28 Feb 2026 qwen3-coder-next technical report 87.7 MMLU (Table 8) Capabilities
19 Feb 2026 Trinity Large Preview technical report 87.2 MMLU, Trinity Large Preview (Table 4) Capabilities
17 Feb 2026 GLM-5 technical report 88.3 GLM-5-Base, MMLU (EM) (Table 11) Capabilities
11 Jan 2026 Solar Open 100B technical report 88.2 MMLU (Table 6) Capabilities
15 Dec 2025 Olmo 3 32B Think technical report 85.4 MMLU, Final Think 3.0 column (Table 14) Capabilities
15 Dec 2025 Olmo 3.1 32B Think technical report 86.4 MMLU (Tables 1 and 14) Capabilities
28 Nov 2025 LFM2-8B-A1B technical report 64.8 MMLU, LFM2-8B-A1B (Table 7) Capabilities
28 Nov 2025 LFM2-1.2B technical report 55.2 MMLU, LFM2-1.2B (Table 6) Capabilities
8 Nov 2025 Falcon-Arabic-7B technical report 63.3 MMLU 5-shot (Table 8) Capabilities
31 Oct 2025 LongCat-Flash-Omni technical report 90.3 MMLU (acc) (Table 13) Capabilities
8 Oct 2025 OLMo 2 1B Instruct technical report 40 MMLU (Table 7, Table 23) Capabilities
30 Sep 2025 Code World Model (CWM) technical report 77.7 MMLU, without reasoning (Table 13) Capabilities
17 Sep 2025 Apertus 70B technical report 69.6 MMLU, 0-shot CoT exact match (Table 17) Capabilities
1 Sep 2025 LongCat-Flash-Chat technical report 89.7 MMLU (acc), model-judged answer matching (Table 3) Capabilities
25 Aug 2025 InternVL3.5-241B-A28B technical report 89.1 MMLU, OpenCompass (Table 14) Capabilities
25 Aug 2025 Hermes 4 405B technical report 87.2 MMLU, reasoning mode (Table 3). Non-reasoning 73.6 Capabilities
8 Aug 2025 GLM-4.5 technical report 90 MMLU (EM) (Table 6) Capabilities
8 Aug 2025 GLM-4.5-Air technical report 87.4 MMLU (EM) (Table 6) Capabilities
5 Aug 2025 gpt-oss-20b model card 85.3 MMLU, high reasoning (Table 3) Capabilities
5 Aug 2025 gpt-oss-120b model card 90 MMLU, high reasoning (Table 3) Capabilities
30 Jul 2025 Falcon-H1-34B technical report 84 MMLU, logprobs 5-shot (Table 24) Capabilities
21 Jul 2025 Kimi K2 technical report 89.5 MMLU (EM) (Table 3) Capabilities
17 Jul 2025 Voxtral Small technical report MMLU 5-shot, text-only, chart without printed numbers (Figure 6). Speech-synthesized MMLU* 74.3 (Table 8) Capabilities
17 Jul 2025 Apple Foundation Model Server (2025) technical report 80.2 MMLU, Simple-Evals, AFM Server (Table 2). Table 3 prints 80.0 at 16 bits and 79.2 for AFM Server Opt at 3.6 bits per weight Capabilities
29 Jun 2025 ERNIE-4.5-300B-A47B technical report 86.5 MMLU (Table 5) Capabilities
30 Apr 2025 Amazon Nova Premier technical report 87.4 MMLU, 0-shot CoT, macro average (Table 1) Capabilities
16 Apr 2025 BitNet b1.58 2B4T technical report 53.2 MMLU (5-shot; Acc) (Table 1, Table 2, Table 3) Capabilities
5 Apr 2025 Llama 4 Scout model card 79.6 MMLU, 5-shot, macro_avg/acc_char, pretrained (Pre-trained models table) Capabilities
5 Apr 2025 Llama 4 Maverick model card 85.5 MMLU, 5-shot, macro_avg/acc_char, pretrained (Pre-trained models table) Capabilities
1 Apr 2025 Cohere Command A technical report 85.5 MMLU, 5-shot CoT, simple-evals (Table 3, Table 1) Capabilities
25 Mar 2025 Gemma 3 4B technical report 58.1 MMLU, IT, 0-shot (Table 18) Capabilities
25 Mar 2025 Gemma 3 27B technical report 76.9 MMLU, IT, 0-shot (Table 18) Capabilities
25 Mar 2025 Gemma 3 1B technical report 38.8 MMLU, IT, 0-shot (Table 18) Capabilities
25 Mar 2025 Gemma 3 12B technical report 71.9 MMLU, IT, 0-shot (Table 18) Capabilities
16 Mar 2025 EXAONE Deep 32B technical report 83 MMLU, accuracy (Table 3) Capabilities
3 Mar 2025 Phi-4 Mini technical report 67.3 MMLU (5-Shot) (Table 7) Capabilities
20 Jan 2025 Kimi k1.5 technical report 87.4 MMLU (EM), short-CoT (Table 3) Capabilities
20 Jan 2025 DeepSeek-R1 technical report 90.8 MMLU (EM), DeepSeek-R1 (Table 8, same as final column of Table 3) Capabilities
14 Jan 2025 MiniMax-Text-01 technical report 88.5 MMLU, 0-shot CoT (Table 8) Capabilities
31 Dec 2024 OLMo 2 32B Instruct technical report 77.3 MMLU, 0-shot CoT (Tables 7 and 16) Capabilities
26 Dec 2024 DeepSeek-V3 technical report 88.5 MMLU (EM), DeepSeek-V3 (Table 6) Capabilities
12 Dec 2024 Phi-4 technical report 84.8 MMLU, simple-evals (Table 1) Capabilities
6 Dec 2024 Llama 3.3 70B Instruct model card 86 MMLU (CoT), 0-shot, macro_avg/acc (Instruction tuned models table) Capabilities
23 Nov 2024 Llama 3.1-8B technical report 73 MMLU (0-shot, CoT), post-trained (Table 2). 5-shot = 69.4 Capabilities
23 Nov 2024 Llama 3.1-70B technical report 86 MMLU (0-shot, CoT), post-trained (Table 2). 5-shot = 83.6 Capabilities
23 Nov 2024 Llama 3.1-405B technical report 88.6 MMLU (0-shot, CoT), post-trained (Table 2). 5-shot = 87.3 Capabilities
22 Nov 2024 Tulu 3 405B technical report 87 MMLU (5 shot, CoT), RLVR (Table 4) Capabilities
22 Oct 2024 Claude 3.5 Sonnet (October 2024) model card 90.5 MMLU, 5-shot CoT. 5-shot 88.7%, 0-shot CoT 89.3% (Table 7) Capabilities
22 Oct 2024 Claude 3.5 Haiku model card 80.9 MMLU, 5-shot CoT. 5-shot 77.6%, 0-shot CoT 80.3% (Table 8) Capabilities
30 Aug 2024 phi-3-small 7.4B technical report 75.7 MMLU (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-mini 3.8B technical report 68.8 MMLU (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 phi-3-medium 14B technical report 78 MMLU (5-Shot) (Section 3 table) Capabilities
30 Aug 2024 Phi-3.5-MoE technical report 78.9 MMLU (5-shot) (Table 3) Capabilities
30 Aug 2024 Phi-3.5-mini technical report 69 MMLU (5-shot) (Table 3) Capabilities
31 Jul 2024 Gemma 2 9B technical report 72.3 MMLU, IT, 5-shot (Table 17). PT 5-shot (Table 13): 71.3 Capabilities
31 Jul 2024 Gemma 2 27B technical report 76.2 MMLU, IT, 5-shot (Table 17). PT 5-shot (Table 13): 75.2 Capabilities
23 Jul 2024 Llama 3.1-8B model card 73 MMLU (CoT), 0-shot, macro_avg/acc, instruction tuned (Instruction tuned models table). 5-shot = 69.4 Capabilities
23 Jul 2024 Llama 3.1-70B model card 86 MMLU (CoT), 0-shot, macro_avg/acc, instruction tuned (Instruction tuned models table). 5-shot = 83.6 Capabilities
23 Jul 2024 Llama 3.1-405B model card 88.6 MMLU (CoT), 0-shot, macro_avg/acc, instruction tuned (Instruction tuned models table). 5-shot = 87.3 Capabilities
15 Jul 2024 Qwen2-72B technical report 82.3 MMLU, Qwen2-72B-Instruct (Table 6). 83.2 on the non-contaminated subset (Table 15) Capabilities
20 Jun 2024 Claude 3.5 Sonnet model card 90.4 MMLU, 5-shot CoT. 5-shot 88.7, 0-shot CoT 88.3 (Table 1) Capabilities
18 Jun 2024 GLM-4 technical report 81.5 GLM-4 (0116), MMLU (Table 2) Capabilities
18 Jun 2024 ChatGLM3-6B technical report 61.4 ChatGLM3-6B-Base, MMLU (Table 1) Capabilities
7 May 2024 DeepSeek-V2 (MoE-236B, May 2024) technical report 77.8 MMLU 5-shot, DeepSeek-V2 Chat (RL) (Table 3) Capabilities
18 Apr 2024 Llama 3-8B model card 68.4 MMLU (5-shot), instruction tuned (Instruction tuned models table). Pretrained = 66.6 Capabilities
18 Apr 2024 Llama 3-70B model card 82 MMLU (5-shot), instruction tuned (Instruction tuned models table). Pretrained = 79.5 Capabilities
13 Mar 2024 Gemma 7B technical report 64.3 MMLU, PT, 5-shot top-1 (Table 6). HuggingFace H6 run (Table 7): 64.6 Capabilities
13 Mar 2024 Gemma 2B technical report 42.3 MMLU, PT, 5-shot top-1 (Table 6) Capabilities
4 Mar 2024 Claude 3 Sonnet model card 81.5 MMLU, 5-shot CoT. 5-shot 79.0 (Table 1) Capabilities
4 Mar 2024 Claude 3 Opus model card 88.2 MMLU, 5-shot CoT. 5-shot 86.8 (Table 1) Capabilities
4 Mar 2024 Claude 3 Haiku model card 76.7 MMLU, 5-shot CoT. 5-shot 75.2 (Table 1) Capabilities
8 Jan 2024 Mixtral 8x7B technical report 70.6 MMLU (MCQ in 57 subjects), 5-shot (Table 3; same in Table 2) Capabilities
5 Jan 2024 DeepSeek LLM 67B technical report 71.1 MMLU 0-shot, DeepSeek 67B Chat (Table 6) Capabilities
10 Oct 2023 Mistral 7B v0.1 technical report 60.1 MMLU, 5-shot (Table 2) Capabilities
28 Sep 2023 Qwen-7B technical report 57 MMLU 5-shot, higher of the two settings, 0-shot 55.8 (Table 5) Capabilities
28 Sep 2023 Qwen-14B technical report 66.5 MMLU 5-shot, higher of the two settings, 0-shot 64.6 (Table 5) Capabilities
28 Sep 2023 Qwen-1_8B technical report 43.9 MMLU 5-shot, higher of the two settings, 0-shot 42.4 (Table 5) Capabilities
19 Jul 2023 Llama 2-7B technical report 45.3 MMLU (5-shot), pretrained (Table 3, Table 19) Capabilities
19 Jul 2023 Llama 2-70B technical report 68.9 MMLU (5-shot), pretrained (Table 3, Table 19) Capabilities
19 Jul 2023 Llama 2-34B technical report 62.6 MMLU (5-shot), pretrained (Table 3, Table 19) Capabilities
19 Jul 2023 Llama 2-13B technical report 54.8 MMLU (5-shot), pretrained (Table 3, Table 19) Capabilities
18 Jul 2023 Llama 2-7B model card 45.3 MMLU, pretrained (Overall performance on grouped academic benchmarks) Capabilities
18 Jul 2023 Llama 2-70B model card 68.9 MMLU, pretrained (Overall performance on grouped academic benchmarks) Capabilities
18 Jul 2023 Llama 2-13B model card 54.8 MMLU, pretrained (Overall performance on grouped academic benchmarks) Capabilities
11 Jul 2023 Claude 2 model card 78.5 MMLU (5-shot CoT) (4.3) Capabilities
14 Mar 2023 GPT-4 technical report 86.4 MMLU, 5-shot (Table 2, base model) Capabilities

Versions

Replaced by MMLU-Pro

More in Knowledge

All 109 in Knowledge
  • MMLU-Pro In use Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. Intern-S2-397B 89.8%
  • SuperGPQA In use 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. Qwen 3.6 Max (Preview) 73.9%
  • WideSearch In use 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. Tencent Hy4 preview 83.9%
  • KMMLU-Pro In use Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. Solar Pro 4 79.2%
  • BIG-Bench Extra Hard In use Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. Gemma 4 31B IT 74.4%
  • HLE-Verified In use Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. Seed2.0 Pro 73.6%