Benchmark Almanac

MMMLU

Claude Mythos Preview 92.7% leads since Apr 2026 In use

MMLU professionally translated into 14 languages. Multiple-choice accuracy.

Made by OpenAI Released Sep 2024 Knowledge in many languages All tasks public Last in a launch post Jul 2026 Last model added Aug 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
openai/simple-evals 4,690 stars Dataset
62
Models scored in lab cards
48
Launch posts list it, from 13 labs
40
System cards report it, from 9 labs

Claude Mythos Preview leads with 92.7% among models released by Jul 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Jul 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

62 models. Scroll sideways for the rest.

  1. 92.7 Claude Mythos Preview
  2. 92.6 Gemini 3.1 Pro
  3. 91.8 Gemini 3 Pro
  4. 91.8 Gemini 3 Flash
  5. 91.5 Claude Opus 4.7
  6. 91.1 Claude Opus 4.6
  7. 90.8 Claude Opus 4.5
  8. 90.3 DeepSeek-V4-Pro
  9. 90.3 Qwen3.7-Max
  10. 89.5 Claude Opus 4.1
  11. 89.5 Qwen 3.6 Plus
  12. 89.3 Claude Sonnet 4.6
  13. 89.1 Claude Sonnet 4.5
  14. 89.0 Qwen3.7-Plus
  15. 88.9 Gemini 3.1 Flash-Lite
  16. 88.8 o3
  17. 88.8 Claude Opus 4
  18. 88.5 Qwen3.5 397B-A17B
  19. 88.4 Gemma 4 31B IT
  20. 88.1 Seed2.0 Pro
  21. 86.7 Qwen3.5-122B-A10B
  22. 86.6 K-EXAONE 2.0
  23. 86.5 Claude Sonnet 4
  24. 86.3 Gemma 4 26B A4B
  25. 86.2 Cogito v2.1
  26. 86.1 Claude 3.7 Sonnet
  27. 85.8 Llama 4 Behemoth
  28. 85.7 K-EXAONE
  29. 85.6 EXAONE 4.0 32B
  30. 85.5 Mistral Large 3
  31. 85.4 EXAONE 4.5 33B
  32. 85.2 o4-mini
  33. 85.1 GPT-4.5
  34. 84.3 Qwen3-235B-A22B
  35. 83.0 Claude Haiku 4.5
  36. 82.4 Cogito v2 Preview 671B MoE
  37. 81.5 DiffusionGemma 26B A4B
  38. 81.3 gpt-oss-120b
  39. 80.6 Qwen3-32B
  40. 80.5 Cogito v1 Preview 70B
  41. 79.4 DeepSeek-V3
  42. 78.5 GPT-4.1 mini
  43. 78.4 Qwen3-30B-A3B
  44. 78.4 seed-oss-36b-instruct
  45. 78.0 Qwen2-72B
  46. 77.9 Qwen3-14B
  47. 75.7 gpt-oss-20b
  48. 74.6 Apple Foundation Model Server (2025)
  49. 74.4 Qwen3-8B
  50. 73.7 Granite 4.1 30B
  51. 69.9 Phi-3.5-MoE
  52. 69.8 Qwen3-4B
  53. 69.7 Granite 4.0 H Small
  54. 66.9 GPT-4.1 nano
  55. 59.1 Qwen3-1.7B
  56. 55.4 Phi-3.5-mini
  57. 55.3 LFM2-8B-A1B
  58. 49.3 Phi-4 Mini
  59. 47.3 phi-3-mini 3.8B
  60. 46.7 LFM2-1.2B
  61. 45.0 granite-4.0-1b
  62. 35.6 Gemma 3n E4B
  1. 1 Claude Mythos Preview 92.7%
  2. 2 Gemini 3.1 Pro 92.6%
  3. 3 Gemini 3 Pro 91.8%
  4. 4 Gemini 3 Flash 91.8%
  5. 5 Claude Opus 4.7 91.5%
  6. 6 Claude Opus 4.6 91.1%
  7. 7 Claude Opus 4.5 90.8%
  8. 8 DeepSeek-V4-Pro 90.3%
  9. 9 Qwen3.7-Max 90.3%
  10. 10 Claude Opus 4.1 89.5%
  11. 11 Qwen 3.6 Plus 89.5%
  12. 12 Claude Sonnet 4.6 89.3%
  13. 13 Claude Sonnet 4.5 89.1%
  14. 14 Qwen3.7-Plus 89%
  15. 15 Gemini 3.1 Flash-Lite 88.9%
  16. 16 o3 88.8%
  17. 17 Claude Opus 4 88.8%
  18. 18 Qwen3.5 397B-A17B 88.5%
  19. 19 Gemma 4 31B IT 88.4%
  20. 20 Seed2.0 Pro 88.1%
  21. 21 Qwen3.5-122B-A10B 86.7%
  22. 22 K-EXAONE 2.0 86.6%
  23. 23 Claude Sonnet 4 86.5%
  24. 24 Gemma 4 26B A4B 86.3%
  25. 25 Cogito v2.1 86.2%
  26. 26 Claude 3.7 Sonnet 86.1%
  27. 27 Llama 4 Behemoth 85.8%
  28. 28 K-EXAONE 85.7%
  29. 29 EXAONE 4.0 32B 85.6%
  30. 30 Mistral Large 3 85.5%
  31. 31 EXAONE 4.5 33B 85.4%
  32. 32 o4-mini 85.2%
  33. 33 GPT-4.5 85.1%
  34. 34 Qwen3-235B-A22B 84.3%
  35. 35 Claude Haiku 4.5 83%
  36. 36 Cogito v2 Preview 671B MoE 82.4%
  37. 37 DiffusionGemma 26B A4B 81.5%
  38. 38 gpt-oss-120b 81.3%
  39. 39 Qwen3-32B 80.6%
  40. 40 Cogito v1 Preview 70B 80.5%
  41. 41 DeepSeek-V3 79.4%
  42. 42 GPT-4.1 mini 78.5%
  43. 43 Qwen3-30B-A3B 78.4%
  44. 44 seed-oss-36b-instruct 78.4%
  45. 45 Qwen2-72B 78%
  46. 46 Qwen3-14B 77.9%
  47. 47 gpt-oss-20b 75.7%
  48. 48 Apple Foundation Model Server (2025) 74.6%
  49. 49 Qwen3-8B 74.4%
  50. 50 Granite 4.1 30B 73.7%
  51. 51 Phi-3.5-MoE 69.9%
  52. 52 Qwen3-4B 69.8%
  53. 53 Granite 4.0 H Small 69.7%
  54. 54 GPT-4.1 nano 66.9%
  55. 55 Qwen3-1.7B 59.1%
  56. 56 Phi-3.5-mini 55.4%
  57. 57 LFM2-8B-A1B 55.3%
  58. 58 Phi-4 Mini 49.3%
  59. 59 phi-3-mini 3.8B 47.3%
  60. 60 LFM2-1.2B 46.7%
  61. 61 granite-4.0-1b 45%
  62. 62 Gemma 3n E4B 35.6%
#ModelScoreSettingDateSource
1 Claude Mythos Preview 92.7% Apr 2026 Model card
2 Gemini 3.1 Pro 92.6% Feb 2026 Model card
3 Gemini 3 Pro 91.8% Nov 2025 Model card
4 Gemini 3 Flash 91.8% Dec 2025 Model card
5 Claude Opus 4.7 91.5% Apr 2026 Model card
6 Claude Opus 4.6 91.1% Feb 2026 Model card
7 Claude Opus 4.5 90.8% Nov 2025 Model card
8 DeepSeek-V4-Pro 90.3% Apr 2026 Model card
9 Qwen3.7-Max 90.3% May 2026 Model card
10 Claude Opus 4.1 89.5% Aug 2025 Model card
11 Qwen 3.6 Plus 89.5% Apr 2026 Model card
12 Claude Sonnet 4.6 89.3% Feb 2026 Model card
13 Claude Sonnet 4.5 89.1% Sep 2025 Model card
14 Qwen3.7-Plus 89% Jun 2026 Model card
15 Gemini 3.1 Flash-Lite 88.9% Mar 2026 Model card
16 o3 88.8% Apr 2025 Model card
17 Claude Opus 4 88.8% May 2025 Model card
18 Qwen3.5 397B-A17B 88.5% Feb 2026 Model card
19 Gemma 4 31B IT 88.4% Apr 2026 Model card
20 Seed2.0 Pro 88.1% Feb 2026 Model card
21 Qwen3.5-122B-A10B 86.7% Feb 2026 Model card
22 K-EXAONE 2.0 86.6% Jul 2026 Model card
23 Claude Sonnet 4 86.5% May 2025 Model card
24 Gemma 4 26B A4B 86.3% Apr 2026 Model card
25 Cogito v2.1 86.2% Nov 2025 Model card
26 Claude 3.7 Sonnet 86.1% Feb 2025 Model card
27 Llama 4 Behemoth 85.8% Apr 2025 Model card
28 K-EXAONE 85.7% Dec 2025 Model card
29 EXAONE 4.0 32B 85.6% Jul 2025 Model card
30 Mistral Large 3 85.5% Dec 2025 Model card
31 EXAONE 4.5 33B 85.4% Apr 2026 Model card
32 o4-mini 85.2% Apr 2025 Model card
33 GPT-4.5 85.1% Feb 2025 Model card
34 Qwen3-235B-A22B 84.3% Apr 2025 Model card
35 Claude Haiku 4.5 83% Oct 2025 Model card
36 Cogito v2 Preview 671B MoE 82.4% Jul 2025 Model card
37 DiffusionGemma 26B A4B 81.5% Jun 2026 Model card
38 gpt-oss-120b 81.3% Aug 2025 Model card
39 Qwen3-32B 80.6% May 2025 Model card
40 Cogito v1 Preview 70B 80.5% Apr 2025 Model card
41 DeepSeek-V3 79.4% Dec 2024 Model card
42 GPT-4.1 mini 78.5% Apr 2025 Model card
43 Qwen3-30B-A3B 78.4% May 2025 Model card
44 seed-oss-36b-instruct 78.4% Aug 2025 Model card
45 Qwen2-72B 78% Jun 2024 Model card
46 Qwen3-14B 77.9% May 2025 Model card
47 gpt-oss-20b 75.7% Aug 2025 Model card
48 Apple Foundation Model Server (2025) 74.6% Jul 2025 Model card
49 Qwen3-8B 74.4% May 2025 Model card
50 Granite 4.1 30B 73.7% Apr 2026 Model card
51 Phi-3.5-MoE 69.9% Aug 2024 Model card
52 Qwen3-4B 69.8% May 2025 Model card
53 Granite 4.0 H Small 69.7% Oct 2025 Model card
54 GPT-4.1 nano 66.9% Apr 2025 Model card
55 Qwen3-1.7B 59.1% May 2025 Model card
56 Phi-3.5-mini 55.4% Aug 2024 Model card
57 LFM2-8B-A1B 55.3% Oct 2025 Model card
58 Phi-4 Mini 49.3% Feb 2025 Model card
59 phi-3-mini 3.8B 47.3% Aug 2024 Model card
60 LFM2-1.2B 46.7% Jul 2025 Model card
61 granite-4.0-1b 45% Oct 2025 Model card
62 Gemma 3n E4B 35.6% Jun 2025 Model card

8 models have held number one since its release in Sep 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen DeepSeek Anthropic OpenAI Google DeepMind
Lab claims Passed 85% in Feb 2025, by Claude 3.7 Sonnet
Claude Mythos Preview 92.7%

The best score passed 85% in Feb 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Mythos Preview Claude Opus 4.1 Claude 3.7 Sonnet DeepSeek-V3 Qwen2-72B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Claude Mythos Preview Claude 3.7 Sonnet

3 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Microsoft
DeepSeek
Anthropic
OpenAI
Meta
Deep Cogito
Google DeepMind
Liquid AI
LG AI Research
ByteDance Seed
IBM
Mistral AI
DateLaunch postScoreAs the post writes it
31 Jul 2026 K-EXAONE 2.0 86.6 MMMLU (nine non-English supported languages)
10 Jun 2026 DiffusionGemma 26B A4B 81.5 MMMLU
1 Jun 2026 Qwen3.7-Plus 89 MMMLU
20 May 2026 Qwen3.7-Max 90.3 MMMLU
29 Apr 2026 Granite 4.1 30B 73.7 MMMLU (5-shot, 11 languages)
24 Apr 2026 DeepSeek-V4-Pro 90.3 MMMLU (EM)
16 Apr 2026 Claude Opus 4.7 91.5 Multilingual Q&A MMMLU
7 Apr 2026 Claude Mythos Preview 92.7 MMMLU
2 Apr 2026 Gemma 4 26B A4B 86.3 MMMLU
2 Apr 2026 Qwen 3.6 Plus 89.5 MMMLU
2 Apr 2026 Gemma 4 31B IT 88.4 MMMLU
3 Mar 2026 Gemini 3.1 Flash-Lite 88.9 MMMLU
24 Feb 2026 Qwen3.5-122B-A10B 86.7 MMMLU
19 Feb 2026 Gemini 3.1 Pro 92.6 MMMLU
17 Feb 2026 Claude Sonnet 4.6 89.3 Multilingual Q&A MMMLU
16 Feb 2026 Qwen3.5 397B-A17B 88.5 MMMLU
14 Feb 2026 Seed2.0 Pro 88.1 MMMLU
5 Feb 2026 Claude Opus 4.6 91.1 Multilingual Q&A MMMLU
31 Dec 2025 K-EXAONE 85.7 MMMLU
17 Dec 2025 Gemini 3 Flash 91.8 MMMLU
2 Dec 2025 Mistral Large 3 85.5 MMMLU (8-lang average) [base]
24 Nov 2025 Claude Opus 4.5 90.8 Multilingual Q&A MMMLU
19 Nov 2025 Cogito v2.1 86.2 Multilingual MMLU
18 Nov 2025 Gemini 3 Pro 91.8 MMMLU
28 Oct 2025 granite-4.0-1b 45 MMMLU (5-shot, 11 languages)
15 Oct 2025 Claude Haiku 4.5 83 Multilingual Q&A MMMLU
7 Oct 2025 LFM2-8B-A1B 55.3 MMMLU
2 Oct 2025 Granite 4.0 H Small 69.7 MMMLU (5-shot, 11 languages)
29 Sep 2025 Claude Sonnet 4.5 89.1 Multilingual Q&A MMMLU
20 Aug 2025 seed-oss-36b-instruct 78.4 MMMLU
5 Aug 2025 Claude Opus 4.1 89.5 Multilingual Q&A MMMLU
31 Jul 2025 Cogito v2 Preview 671B MoE 82.4 MMMLU (thinking)
15 Jul 2025 EXAONE 4.0 32B 85.6 MMMLU (ES), Spanish only (reasoning)
10 Jul 2025 LFM2-1.2B 46.7 MMMLU
26 Jun 2025 Gemma 3n E4B 35.6 OpenAI MMLU 0-shot (IT)
22 May 2025 Claude Sonnet 4 86.5 Multilingual Q&A MMMLU
22 May 2025 Claude Opus 4 88.8 Multilingual Q&A MMMLU
29 Apr 2025 Qwen3-235B-A22B 84.3 MMMLU 14 languages
14 Apr 2025 GPT-4.1 nano 66.9 Multilingual MMLU
14 Apr 2025 GPT-4.1 mini 78.5 Multilingual MMLU
8 Apr 2025 Cogito v1 Preview 70B 80.5 MMMLU (thinking)
5 Apr 2025 Llama 4 Behemoth 85.8 Multilingual MMLU
27 Feb 2025 GPT-4.5 85.1 MMMLU (multilingual)
26 Feb 2025 Phi-4 Mini 49.3 Multilingual MMLU (5-shot)
24 Feb 2025 Claude 3.7 Sonnet 86.1 Multilingual Q&A MMMLU (64K extended thinking)
26 Dec 2024 DeepSeek-V3 79.4 MMMLU-non-English (Acc.)
20 Aug 2024 Phi-3.5-MoE 69.9 Multilingual MMLU (5-shot)
7 Jun 2024 Qwen2-72B 78 M-MMLU (5-shot)

40 system cards report it, from 9 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
5 Aug 2026 K-EXAONE 2.0 technical report 86.6 MMMLU, nine non-English languages (Table 6, Reasoning mode) Capabilities
31 Jul 2026 DiffusionGemma 26B A4B technical report 81.5 MMMLU, TD, 76.3 no-think (Table 3) Capabilities
24 Jul 2026 Gemma 4 31B IT technical report 88.4 MMMLU, thinking (Table 5) Capabilities
24 Jul 2026 Gemma 4 26B A4B technical report 86.3 MMMLU, thinking (Table 5) Capabilities
16 Apr 2026 Claude Opus 4.7 system card 91.5 MMMLU, non-English language pairings, adaptive thinking at max effort, average of 3 trials (Table 8.1.A, 8.5) Capabilities
9 Apr 2026 EXAONE 4.5 33B technical report 85.4 MMMLU (ko, de, es, ja) (Table 3, Reasoning mode) Capabilities
7 Apr 2026 Claude Mythos Preview system card 92.7 MMMLU, all non-English language pairings, adaptive thinking, max effort, average of 5 trials (Table 6.3.A; 92.67 in 6.7) Capabilities
3 Mar 2026 Gemini 3.1 Flash-Lite model card 88.9 MMMLU (results table, p.4) Capabilities
19 Feb 2026 Gemini 3.1 Pro model card 92.6 MMMLU (results table, p.4) Capabilities
17 Feb 2026 Claude Sonnet 4.6 system card 89.3 MMMLU, non-English language pairings, adaptive thinking, max effort (Table 2.1.A, 2.11) Capabilities
14 Feb 2026 Seed2.0 Pro model card 88.1 MMMLU (Table 3) Capabilities
5 Feb 2026 Claude Opus 4.6 system card 91.1 MMMLU, all non-English language pairings, adaptive thinking, max effort, average of 5 trials (Table 2.3.A; 91.05 in 2.13) Capabilities
5 Jan 2026 K-EXAONE technical report 85.7 MMMLU (ko, de, es, ja) (Table 3, Reasoning mode). Per language ko 85.6, de 85.1, es 86.6, ja 85.5 (Table 5). Non-reasoning 83.8 Capabilities
11 Dec 2025 GPT-5.2 system card addendum MMLU Language (0-shot), gpt-5.2-thinking, 14 languages, no overall score (Table 8) Capabilities
28 Nov 2025 LFM2-8B-A1B technical report 55.3 MMMLU, LFM2-8B-A1B (Table 7) Capabilities
28 Nov 2025 LFM2-1.2B technical report 46.7 MMMLU, LFM2-1.2B (Table 6) Capabilities
24 Nov 2025 Claude Opus 4.5 system card 90.8 MMMLU, 64k thinking, avg of 10 trials over 14 languages (Table 2.3.A; 90.77% in 2.19) Capabilities
7 Aug 2025 GPT-5 system card MMLU Language (0-shot), 14 languages, no average (Table 11) Capabilities
5 Aug 2025 gpt-oss-20b model card 75.7 MMMLU (Average) over 14 languages, high reasoning (Table 2, Table 3) Capabilities
5 Aug 2025 gpt-oss-120b model card 81.3 MMMLU (Average) over 14 languages, high reasoning (Table 2, Table 3) Capabilities
17 Jul 2025 Apple Foundation Model Server (2025) technical report 74.6 MMMLU, AFM-supported languages only, Simple-Evals (Table 2) Capabilities
15 Jul 2025 EXAONE 4.0 32B technical report 85.6 MMMLU (ES), Spanish only (Table 3, Reasoning). Non-reasoning 80.6 (Table 4) Capabilities
14 May 2025 Qwen3-8B technical report 74.4 MMMLU 14 languages, thinking mode (Table 17). Non-thinking 66.9 (Table 18) Capabilities
14 May 2025 Qwen3-4B technical report 69.8 MMMLU 14 languages, thinking mode (Table 17). Non-thinking 61.7 (Table 18) Capabilities
14 May 2025 Qwen3-32B technical report 80.6 MMMLU 14 languages, thinking mode (Table 13). Non-thinking 76.5 (Table 14) Capabilities
14 May 2025 Qwen3-30B-A3B technical report 78.4 MMMLU 14 languages, thinking mode (Table 15). Non-thinking 73.8 (Table 16) Capabilities
14 May 2025 Qwen3-235B-A22B technical report 84.3 MMMLU 14 languages, thinking mode (Table 11). Non-thinking 79.8 (Table 12) Capabilities
14 May 2025 Qwen3-14B technical report 77.9 MMMLU 14 languages, thinking mode (Table 15). Non-thinking 72.6 (Table 16) Capabilities
14 May 2025 Qwen3-1.7B technical report 59.1 MMMLU 14 languages, thinking mode (Table 19). Non-thinking 48.3 (Table 20) Capabilities
16 Apr 2025 o4-mini system card 85.2 MMLU Language (0-shot), average of 14 languages, o4-mini-high (Table 16) Capabilities
16 Apr 2025 o3 system card 88.8 MMLU Language (0-shot), average of 14 languages, o3-high (Table 16) Capabilities
3 Mar 2025 Phi-4 Mini technical report 49.3 Multilingual-MMLU (5-Shot) (Table 7) Capabilities
27 Feb 2025 GPT-4.5 system card MMLU Language (0-shot), 14 human-translated languages plus English, per-language scores with no average (Table 16) Capabilities
31 Jan 2025 o3-mini system card MMLU Language (0-shot), 14 human-translated languages, per-language scores with no average (Table 14) Capabilities
5 Dec 2024 o1 system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 17) Capabilities
12 Sep 2024 o1-preview system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 12) Capabilities
12 Sep 2024 o1-mini system card MMLU Language (0-shot), human translated MMLU in 14 languages plus English, no overall score (Table 12) Capabilities
30 Aug 2024 phi-3-mini 3.8B technical report 47.3 MMLU-multilingual average (Section 4 text, Figure 4) Capabilities
30 Aug 2024 Phi-3.5-MoE technical report 69.9 Ml MMLU (5-shot), multilingual MMLU (Table 3) Capabilities
30 Aug 2024 Phi-3.5-mini technical report 55.4 Ml MMLU (5-shot), multilingual MMLU (Table 3) Capabilities

More in Multilingual

All 115 in Multilingual
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
  • Alyah In use Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. Falcon-Emirati-7B 84.8%