Benchmark Almanac

MMMU-Pro

Kimi K3.1 Pro 84.1% leads since Oct 2026 In use

Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts.

Variant (standard vs vision, 4 vs 10 options) often unstated in cards.

In the authors’ words

This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.

Abstract of MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (arXiv:2409.02813, CC0)
Made by MMMU team Released Sep 2024 General visual questions All tasks public Last in a launch post Oct 2026 Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
MMMU-Benchmark/MMMU 600 stars Paper Dataset Leaderboard
72
Models scored in lab cards
67
Launch posts list it, from 21 labs
40
System cards report it, from 16 labs

Kimi K3.1 Pro leads with 84.1%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

72 models. Scroll sideways for the rest.

  1. 84.1 Kimi K3.1 Pro
  2. 83.6 Gemini 3.5 Flash
  3. 82.7 Seed2.1 Pro
  4. 82.3 Qwen 3.8 Max
  5. 82.0 GPT-5.6 Terra
  6. 81.7 Intern-S2-397B
  7. 81.6 Kimi K3
  8. 81.2 Gemini 3 Flash
  9. 81.2 GPT-5.4
  10. 81.0 Gemini 3 Pro
  11. 80.5 Gemini 3.1 Pro
  12. 80.4 Muse Spark
  13. 79.5 GPT-5.6 Luna
  14. 79.4 Kimi K2.6
  15. 79.0 Qwen3.5 397B-A17B
  16. 79.0 Qwen3.7-Plus
  17. 78.8 Qwen 3.6 Plus
  18. 78.5 Kimi K2.5
  19. 78.4 GPT-5
  20. 78.4 Seed2.0 Lite (0428)
  21. 78.2 Seed2.0 Pro
  22. 78.1 MiniMax-M3
  23. 77.9 MiMo-V2.5
  24. 77.3 Claude Opus 4.6
  25. 76.9 Qwen3.5-122B-A10B
  26. 76.9 Gemma 4 31B IT
  27. 76.8 Gemini 3.1 Flash-Lite
  28. 76.8 MiMo-V2-Omni
  29. 76.0 GPT-5.5 Instant
  30. 76.0 Step 5 Preview
  31. 75.8 Qwen3.6 27B
  32. 75.6 Claude Sonnet 4.6
  33. 75.3 Qwen 3.6 35B-A3B
  34. 74.1 GPT-5 mini
  35. 74.0 Inkling-Small
  36. 74.0 Muse Glimmer
  37. 73.9 Qwen3.5-Omni-Plus
  38. 73.8 Gemma 4 26B A4B
  39. 73.5 Inkling
  40. 73.2 Seed1.8
  41. 72.8 Intern-S1-Pro
  42. 69.5 GPT-5.4 Nano
  43. 69.3 Qwen3-VL-235B-A22B
  44. 68.6 ERNIE 5.0
  45. 68.6 EXAONE 4.5 33B
  46. 67.6 Seed1.5-VL
  47. 66.4 Seed1.6-Thinking
  48. 66.0 GLM-4.6V
  49. 65.2 GLM-4.5V
  50. 64.1 STEP3-VL-10B
  51. 63.5 Nova 2.0 Pro Preview
  52. 63.0 Command A+
  53. 62.6 GPT-5 nano
  54. 61.8 Nova 2 Lite
  55. 61.4 Nova 2 Omni Preview
  56. 60.0 Mistral Small 4
  57. 59.6 Llama 4 Maverick
  58. 57.9 Magistral Medium 1.0
  59. 57.1 GLM-4.1V-9B-Thinking
  60. 57.0 Qwen3-Omni-30B-A3B
  61. 56.5 DeepSeek V4.1 Flash
  62. 54.3 DiffusionGemma 26B A4B
  63. 52.2 Llama 4 Scout
  64. 51.8 Magistral Small 1.0
  65. 51.1 Qwen2.5-VL-72B
  66. 49.3 Mistral Small 3.1
  67. 46.3 Kimi-VL-A3B-Thinking-2506
  68. 42.0 Nemotron 3.5 Bolt E4B
  69. 38.5 Phi-4-multimodal
  70. 36.6 Qwen2.5-Omni-7B
  71. 30.5 LFM2.5-VL-3B
  72. 28.0 Nemotron Nano 12B v2 VL
  1. 1 Kimi K3.1 Pro 84.1%
  2. 2 Gemini 3.5 Flash 83.6%
  3. 3 Seed2.1 Pro 82.7%
  4. 4 Qwen 3.8 Max 82.3%
  5. 5 GPT-5.6 Terra 82%
  6. 6 Intern-S2-397B 81.7%
  7. 7 Kimi K3 81.6%
  8. 8 Gemini 3 Flash 81.2%
  9. 9 GPT-5.4 81.2%
  10. 10 Gemini 3 Pro 81%
  11. 11 Gemini 3.1 Pro 80.5%
  12. 12 Muse Spark 80.4%
  13. 13 GPT-5.6 Luna 79.5%
  14. 14 Kimi K2.6 79.4%
  15. 15 Qwen3.5 397B-A17B 79%
  16. 16 Qwen3.7-Plus 79%
  17. 17 Qwen 3.6 Plus 78.8%
  18. 18 Kimi K2.5 78.5%
  19. 19 GPT-5 78.4%
  20. 20 Seed2.0 Lite (0428) 78.4%
  21. 21 Seed2.0 Pro 78.2%
  22. 22 MiniMax-M3 78.1%
  23. 23 MiMo-V2.5 77.9%
  24. 24 Claude Opus 4.6 77.3%
  25. 25 Qwen3.5-122B-A10B 76.9%
  26. 26 Gemma 4 31B IT 76.9%
  27. 27 Gemini 3.1 Flash-Lite 76.8%
  28. 28 MiMo-V2-Omni 76.8%
  29. 29 GPT-5.5 Instant 76%
  30. 30 Step 5 Preview 76%
  31. 31 Qwen3.6 27B 75.8%
  32. 32 Claude Sonnet 4.6 75.6%
  33. 33 Qwen 3.6 35B-A3B 75.3%
  34. 34 GPT-5 mini 74.1%
  35. 35 Inkling-Small 74%
  36. 36 Muse Glimmer 74%
  37. 37 Qwen3.5-Omni-Plus 73.9%
  38. 38 Gemma 4 26B A4B 73.8%
  39. 39 Inkling 73.5%
  40. 40 Seed1.8 73.2%
  41. 41 Intern-S1-Pro 72.8%
  42. 42 GPT-5.4 Nano 69.5%
  43. 43 Qwen3-VL-235B-A22B 69.3%
  44. 44 ERNIE 5.0 68.6%
  45. 45 EXAONE 4.5 33B 68.6%
  46. 46 Seed1.5-VL 67.6%
  47. 47 Seed1.6-Thinking 66.4%
  48. 48 GLM-4.6V 66%
  49. 49 GLM-4.5V 65.2%
  50. 50 STEP3-VL-10B 64.1%
  51. 51 Nova 2.0 Pro Preview 63.5%
  52. 52 Command A+ 63%
  53. 53 GPT-5 nano 62.6%
  54. 54 Nova 2 Lite 61.8%
  55. 55 Nova 2 Omni Preview 61.4%
  56. 56 Mistral Small 4 60%
  57. 57 Llama 4 Maverick 59.6%
  58. 58 Magistral Medium 1.0 57.9%
  59. 59 GLM-4.1V-9B-Thinking 57.1%
  60. 60 Qwen3-Omni-30B-A3B 57%
  61. 61 DeepSeek V4.1 Flash 56.5%
  62. 62 DiffusionGemma 26B A4B 54.3%
  63. 63 Llama 4 Scout 52.2%
  64. 64 Magistral Small 1.0 51.8%
  65. 65 Qwen2.5-VL-72B 51.1%
  66. 66 Mistral Small 3.1 49.3%
  67. 67 Kimi-VL-A3B-Thinking-2506 46.3%
  68. 68 Nemotron 3.5 Bolt E4B 42%
  69. 69 Phi-4-multimodal 38.5%
  70. 70 Qwen2.5-Omni-7B 36.6%
  71. 71 LFM2.5-VL-3B 30.5%
  72. 72 Nemotron Nano 12B v2 VL 28%
#ModelScoreSettingDateSource
1 Kimi K3.1 Pro 84.1% Oct 2026 Model card
2 Gemini 3.5 Flash 83.6% May 2026 Model card
3 Seed2.1 Pro 82.7% Jun 2026 Model card
4 Qwen 3.8 Max 82.3% Aug 2026 Model card
5 GPT-5.6 Terra 82% Jul 2026 Model card
6 Intern-S2-397B 81.7% Sep 2026 Model card
7 Kimi K3 81.6% Jul 2026 Model card
8 Gemini 3 Flash 81.2% Dec 2025 Model card
9 GPT-5.4 81.2% Mar 2026 Model card
10 Gemini 3 Pro 81% Nov 2025 Model card
11 Gemini 3.1 Pro 80.5% Feb 2026 Model card
12 Muse Spark 80.4% Apr 2026 Model card
13 GPT-5.6 Luna 79.5% Jul 2026 Model card
14 Kimi K2.6 79.4% Apr 2026 Model card
15 Qwen3.5 397B-A17B 79% Feb 2026 Model card
16 Qwen3.7-Plus 79% Jun 2026 Model card
17 Qwen 3.6 Plus 78.8% Apr 2026 Model card
18 Kimi K2.5 78.5% Jan 2026 Model card
19 GPT-5 78.4% Aug 2025 Model card
20 Seed2.0 Lite (0428) 78.4% Apr 2026 Model card
21 Seed2.0 Pro 78.2% Feb 2026 Model card
22 MiniMax-M3 78.1% Jun 2026 Model card
23 MiMo-V2.5 77.9% Apr 2026 Model card
24 Claude Opus 4.6 77.3% Feb 2026 Model card
25 Qwen3.5-122B-A10B 76.9% Feb 2026 Model card
26 Gemma 4 31B IT 76.9% Apr 2026 Model card
27 Gemini 3.1 Flash-Lite 76.8% Mar 2026 Model card
28 MiMo-V2-Omni 76.8% Mar 2026 Model card
29 GPT-5.5 Instant 76% May 2026 Model card
30 Step 5 Preview 76% Sep 2026 Model card
31 Qwen3.6 27B 75.8% Apr 2026 Model card
32 Claude Sonnet 4.6 75.6% Feb 2026 Model card
33 Qwen 3.6 35B-A3B 75.3% Apr 2026 Model card
34 GPT-5 mini 74.1% Aug 2025 Model card
35 Inkling-Small 74% Jul 2026 Model card
36 Muse Glimmer 74% Aug 2026 Model card
37 Qwen3.5-Omni-Plus 73.9% Mar 2026 Model card
38 Gemma 4 26B A4B 73.8% Apr 2026 Model card
39 Inkling 73.5% Jul 2026 Model card
40 Seed1.8 73.2% Dec 2025 Model card
41 Intern-S1-Pro 72.8% Feb 2026 Model card
42 GPT-5.4 Nano 69.5% Mar 2026 Model card
43 Qwen3-VL-235B-A22B 69.3% Sep 2025 Model card
44 ERNIE 5.0 68.6% Feb 2026 Model card
45 EXAONE 4.5 33B 68.6% Apr 2026 Model card
46 Seed1.5-VL 67.6% May 2025 Model card
47 Seed1.6-Thinking 66.4% Jun 2025 Model card
48 GLM-4.6V 66% Dec 2025 Model card
49 GLM-4.5V 65.2% Aug 2025 Model card
50 STEP3-VL-10B 64.1% Jan 2026 Model card
51 Nova 2.0 Pro Preview 63.5% Dec 2025 Model card
52 Command A+ 63% May 2026 Model card
53 GPT-5 nano 62.6% Aug 2025 Model card
54 Nova 2 Lite 61.8% Dec 2025 Model card
55 Nova 2 Omni Preview 61.4% Dec 2025 Model card
56 Mistral Small 4 60% Mar 2026 Model card
57 Llama 4 Maverick 59.6% Apr 2025 Model card
58 Magistral Medium 1.0 57.9% Jun 2025 Model card
59 GLM-4.1V-9B-Thinking 57.1% Jul 2025 Model card
60 Qwen3-Omni-30B-A3B 57% Sep 2025 Model card
61 DeepSeek V4.1 Flash 56.5% Sep 2026 Model card
62 DiffusionGemma 26B A4B 54.3% Jun 2026 Model card
63 Llama 4 Scout 52.2% Apr 2025 Model card
64 Magistral Small 1.0 51.8% Jun 2025 Model card
65 Qwen2.5-VL-72B 51.1% Jan 2025 Model card
66 Mistral Small 3.1 49.3% Mar 2025 Model card
67 Kimi-VL-A3B-Thinking-2506 46.3% Jun 2025 Model card
68 Nemotron 3.5 Bolt E4B 42% Oct 2026 Model card
69 Phi-4-multimodal 38.5% Feb 2025 Model card
70 Qwen2.5-Omni-7B 36.6% Mar 2025 Model card
71 LFM2.5-VL-3B 30.5% Aug 2026 Model card
72 Nemotron Nano 12B v2 VL 28% Nov 2025 Model card

The benchmark's own leaderboard has 61 entries, 57 of them models we track. We plot every source in the charts below.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-5.4
thinking, n…
82.1%
GPT-5.2
thinking, n…
80.4%
GPT-5.1
thinking
79%
GPT-5
non-thinking thinking
78.4%
Claude Opus 4.6
no tools with tools
77.3%
Claude Sonnet 4.6
no tools
75.6%
Claude Sonnet 4.5
no tools
68.9%
Seed1.5-VL
thinking
67.6%
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5.4 thinking, with tools 82.1%
GPT-5.4 thinking, no tools 81.2%
Gemini 3 Pro 81%
Gemini 3.1 Pro thinking, high 80.5%
GPT-5.2 thinking, no python 80.4%
Muse Spark thinking 80.4%
GPT-5.2 thinking, no tools 79.5%
GPT-5.1 thinking 79%
GPT-5 thinking 78.4%
Claude Opus 4.6 with tools 77.3%
Gemma 4 31B IT 76.9%
o3 76.4%
GPT-5.1 76%
Claude Sonnet 4.6 with tools 75.6%
Claude Sonnet 4.6 no tools 74.5%
Claude Opus 4.5 73.9%
Claude Opus 4.6 no tools 73.9%
Gemma 4 26B A4B 73.8%
dots.vlm1 70.1%
Claude Sonnet 4.5 68.9%
Qwen3-VL-235B-A22B 68.1%
Gemini 2.5 Pro May 2025 68%
Seed1.5-VL thinking 67.6%
Seed1.6-Thinking 66.4%
GLM-4.5V thinking 65.2%
Claude Sonnet 4.5 no tools 63.4%
GPT-5 non-thinking 62.7%
Seed1.5-VL 59.9%
GLM-4.1V-9B-Thinking thinking 57.1%
Skywork-R1V3-38B 55.4%
Gemma 4 E4B 52.6%
GPT-4o May 2024 51.9%
Claude 3.5 Sonnet 51.5%
InternVL2_5-78B 48.6%
Qwen2-VL-72B-Instruct 46.2%
Qwen2.5-VL-72B 46.2%
InternVL2.5-38B 46%
Gemma 4 E2B 44.2%
Gemini 1.5 Pro May 2024 43.5%
MiMo-VL-7B-RL 43.3%
MiMo-VL-7B-SFT 42.3%
InternVL2-Llama3-76B 40%
Llama 3.2 90B 39.5%
Qwen2.5-VL-7B 38.3%
GPT-4o mini 37.6%
InternVL2.5-26B 37.1%
InternVL2.5-8B 34.3%
InternVL2-40B 34.2%
Qwen2.5-VL-3B 31.6%
LLaVA-OneVision 72B 31%
InternVL2-8B 29%
Llama 3.2 11B 28.4%
MiniCPM-V-2_6 27.2%
MAmmoTH-VL-8B 25.3%
LLaVA-NeXT-72B 25.1%
LLaVA-OneVision 7B 24.1%
LLaVA-v1.6-34B 23.8%
InternVL2.5-2B 23.7%
Idefics3-8B-Llama3 22.9%
Phi-3.5-vision-instruct 19.7%
MiniCPM-Llama3-V 2.5 19.6%
InternVL2.5-1B 19.4%
llava-v1.6-vicuna-13b 17.2%
llava-v1.6-mistral-7b 17%
llava-v1.6-vicuna-7b 16.1%

6 models have held number one since its release in Sep 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Google DeepMind Qwen Meta ByteDance Seed Moonshot AI
Own leaderboard Not tested since Apr 2026
GPT-5.4 82.1%
Lab claims 8 leaders
Kimi K3.1 Pro 84.1%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Kimi K3.1 Pro GPT-5.4 GPT-5 Llama 4 Maverick GPT-4o LLaVA-NeXT-72B 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Kimi K3.1 Pro o3

12 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Microsoft
Mistral AI
Meta
ByteDance Seed
Moonshot AI
Z.ai
OpenAI
Google DeepMind
Amazon
Shanghai AI Lab
Anthropic
Baidu
Xiaomi Corp
LG AI Research
Cohere
MiniMax
Thinking Machines
Liquid AI
DeepSeek
StepFun
DateLaunch postScoreAs the post writes it
26 Oct 2026 GPT-6.1 Terra MMMU-Pro (chart)
24 Oct 2026 Kimi K3.1 Pro 84.1 MMMU-Pro
20 Sep 2026 Step 5 Preview 76 MMMU-Pro
13 Sep 2026 Intern-S2-397B 81.7 MMMU Pro
10 Sep 2026 DeepSeek V4.1 Flash 56.5 MMMU-Pro (EM)
12 Aug 2026 LFM2.5-VL-3B 30.5 MMMU-Pro
10 Aug 2026 Muse Glimmer 74 MMMU Pro
3 Aug 2026 Qwen 3.8 Max 82.3 MMMU-Pro
30 Jul 2026 Inkling-Small 74 MMMU Pro Standard 10
16 Jul 2026 Kimi K3 81.6 MMMU-Pro (without python)
15 Jul 2026 Inkling 73.5 MMMU Pro Standard 10 (Artificial Analysis)
9 Jul 2026 GPT-5.6 Luna 79.5 MMMU Pro (with tools) 79.5 (no tools 78.4)
9 Jul 2026 GPT-5.6 Terra 82 MMMU Pro (with tools) 82 (no tools 80.7)
23 Jun 2026 Seed2.1 Pro 82.7 MMMU-Pro (w. Tool) 81.6 (82.7)
10 Jun 2026 DiffusionGemma 26B A4B 54.3 MMMU Pro
1 Jun 2026 Qwen3.7-Plus 79 MMMU-Pro
1 Jun 2026 MiniMax-M3 78.1 MMMU-Pro
20 May 2026 Command A+ 63 MMMU Pro
19 May 2026 Gemini 3.5 Flash 83.6 MMMU-Pro (No tools)
5 May 2026 GPT-5.5 Instant 76 MMMU-Pro Standard (Expert Multimodal Reasoning)
28 Apr 2026 Seed2.0 Lite (0428) 78.4 MMMU_Pro
22 Apr 2026 Qwen3.6 27B 75.8 MMMU-Pro
22 Apr 2026 MiMo-V2.5 77.9 MMMU-Pro
20 Apr 2026 Kimi K2.6 79.4 MMMU-Pro
15 Apr 2026 Qwen 3.6 35B-A3B 75.3 MMMU-Pro
9 Apr 2026 EXAONE 4.5 33B 68.6 MMMU-Pro
8 Apr 2026 Muse Spark 80.4 MMMU Pro (Multimodal Understanding)
2 Apr 2026 Gemma 4 26B A4B 73.8 MMMU Pro
2 Apr 2026 Qwen 3.6 Plus 78.8 MMMU-Pro
2 Apr 2026 Gemma 4 31B IT 76.9 MMMU Pro
30 Mar 2026 Qwen3.5-Omni-Plus 73.9 MMMU-Pro
18 Mar 2026 MiMo-V2-Omni 76.8 MMMU-Pro
17 Mar 2026 GPT-5.4 Nano 69.5 MMMUPro w/ Python 69.5 (without 66.1)
16 Mar 2026 Mistral Small 4 60 MMMU-Pro (reasoning; instruct = 46.3)
5 Mar 2026 GPT-5.4 81.2 MMMU Pro (no tools)
3 Mar 2026 Gemini 3.1 Flash-Lite 76.8 MMMU-Pro (No tools)
24 Feb 2026 Qwen3.5-122B-A10B 76.9 MMMU-Pro
19 Feb 2026 Gemini 3.1 Pro 80.5 MMMU-Pro (No tools)
17 Feb 2026 Claude Sonnet 4.6 75.6 Visual reasoning MMMU-Pro (with tools)
16 Feb 2026 Qwen3.5 397B-A17B 79 MMMU-Pro
6 Feb 2026 ERNIE 5.0 68.6 MMMU-Pro
5 Feb 2026 Claude Opus 4.6 77.3 Visual reasoning MMMU Pro (with tools)
2 Feb 2026 Intern-S1-Pro 72.8 MMMU-Pro
27 Jan 2026 Kimi K2.5 78.5 MMMU-Pro
18 Dec 2025 Seed1.8 73.2 MMMU-Pro
17 Dec 2025 Gemini 3 Flash 81.2 MMMU-Pro
8 Dec 2025 GLM-4.6V 66 MMMU_Pro
2 Dec 2025 Nova 2 Omni Preview 61.4 MMMU Pro accuracy
2 Dec 2025 Nova 2 Lite 61.8 MMMU Pro accuracy
2 Dec 2025 Nova 2.0 Pro Preview 63.5 MMMU Pro accuracy
18 Nov 2025 Gemini 3 Pro 81 MMMU-Pro
23 Sep 2025 Qwen3-VL-235B-A22B 69.3 MMMU_Pro
22 Sep 2025 Qwen3-Omni-30B-A3B 57 MMMU_pro
11 Aug 2025 GLM-4.5V 65.2 MMMU Pro
7 Aug 2025 GPT-5 nano 62.6 MMMU-Pro (avg across standard and vision sets)
7 Aug 2025 GPT-5 mini 74.1 MMMU-Pro (avg across standard and vision sets)
7 Aug 2025 GPT-5 78.4 MMMU Pro*** Graduate-level visual problem-solving
1 Jul 2025 GLM-4.1V-9B-Thinking 57.1 MMMU-Pro
25 Jun 2025 Seed1.6-Thinking 66.4 MMMU-pro
21 Jun 2025 Kimi-VL-A3B-Thinking-2506 46.3 MMMU-Pro (Pass@1)
12 May 2025 Seed1.5-VL 67.6 MMMU-Pro
5 Apr 2025 Llama 4 Scout 52.2 MMMU Pro (avg of Standard and Vision)
5 Apr 2025 Llama 4 Maverick 59.6 MMMU Pro (avg of Standard and Vision)
27 Mar 2025 Qwen2.5-Omni-7B 36.6 MMMU-Prooverall
17 Mar 2025 Mistral Small 3.1 49.3 MMMU PRO
26 Feb 2025 Phi-4-multimodal 38.5 MMMU-Pro (std/vision)
28 Jan 2025 Qwen2.5-VL-72B 51.1 MMMU_Pro

40 system cards report it, from 16 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
23 Oct 2026 Nemotron 3.5 Bolt E4B technical report 42 MMMU-Pro Vision, reasoning, 39.08 non-reasoning (Table 8) Capabilities
19 Oct 2026 Muse Glimmer technical report 74 MMMU-Pro, high (Tables 2 and 5) Capabilities
31 Jul 2026 DiffusionGemma 26B A4B technical report 54.3 MMMU Pro, TD, 66.0 no-think (Table 3) Capabilities
30 Jul 2026 Inkling-Small model card 74 MMMU Pro Standard 10 (5. Evaluations table) Capabilities
27 Jul 2026 Kimi K3 technical report 81.6 MMMU-Pro without Python, cell 81.6 / 83.4 (without / with Python) (Table 2) Capabilities
24 Jul 2026 Gemma 4 31B IT technical report 76.9 MMMU Pro, thinking, max resolution 1120 vision tokens (Table 6). At 280 tokens: 75.8 Capabilities
24 Jul 2026 Gemma 4 26B A4B technical report 73.8 MMMU Pro, thinking, max resolution 1120 vision tokens (Table 6). At 280 tokens: 73.2 Capabilities
15 Jul 2026 Inkling model card 73.5 MMMU Pro Standard 10 (5. Evaluations table, effort 0.99) Capabilities
23 Jun 2026 Seed2.1 Pro model card 81.6 MMMU-Pro, no tools, 82.7 with tools (Table 9) Capabilities
19 May 2026 Gemini 3.5 Flash model card 83.6 MMMU-Pro, no tools (results table, p.4) Capabilities
17 Apr 2026 Qwen3.5-Omni-Plus technical report 73.9 MMMU-Pro (Table 6) Capabilities
9 Apr 2026 EXAONE 4.5 33B technical report 68.6 MMMU-Pro (Table 2, Reasoning mode) Capabilities
26 Mar 2026 Intern-S1-Pro technical report 72.8 MMMU-Pro, thinking (Table 2) Capabilities
21 Mar 2026 Seed1.8 model card 73.2 MMMU-Pro (Table 2). 65.4 in NoThink mode (2.4 text) Capabilities
3 Mar 2026 Gemini 3.1 Flash-Lite model card 76.8 MMMU-Pro, no tools (results table, p.4) Capabilities
19 Feb 2026 Gemini 3.1 Pro model card 80.5 MMMU Pro, no tools (results table, p.4) Capabilities
17 Feb 2026 Claude Sonnet 4.6 system card 74.5 MMMU-Pro, no tools, adaptive thinking, max effort; 75.6 with an image cropping tool (Table 2.1.A, 2.17.2) Capabilities
14 Feb 2026 Seed2.0 Pro model card 78.2 MMMU-Pro, Standard (10 options) and Vision combined (Table 8) Capabilities
5 Feb 2026 Claude Opus 4.6 system card 73.9 MMMU-Pro, no tools, adaptive thinking, max effort, average of Standard (10 options) and Vision (Table 2.3.A, 2.19.2); 77.3 with an image cropping tool Capabilities
4 Feb 2026 ERNIE 5.0 technical report 68.6 MMMU-Pro (Table 3) Capabilities
30 Jan 2026 Kimi K2.5 technical report 78.5 MMMU-Pro (Table 4) Capabilities
14 Jan 2026 STEP3-VL-10B technical report 64.1 MMMU-Pro, SeRe (Tables 1 and 3). PaCoRe 67.18 Capabilities
1 Jan 2026 GLM-4.6V technical report 66 MMMU Pro, GLM-4.6V thinking (Table 2) Capabilities
2 Dec 2025 Nova 2 Omni Preview technical report 61.4 MMMU Pro accuracy (Table 6) Capabilities
2 Dec 2025 Nova 2 Lite technical report 61.8 MMMU Pro accuracy (Table 3) Capabilities
2 Dec 2025 Nova 2.0 Pro Preview technical report 63.5 MMMU Pro accuracy (Table 3) Capabilities
26 Nov 2025 Qwen3-VL-235B-A22B technical report 69.3 MMMU-Pro (Table 2) Capabilities
6 Nov 2025 Nemotron Nano 12B v2 VL technical report 28 MMMU Pro, reasoning on (Table 3). Reasoning off 14.5 Capabilities
22 Sep 2025 Qwen3-Omni-30B-A3B technical report 57 MMMU-Pro overall, Instruct (Table 9). Thinking 60.5 (Table 10) Capabilities
13 Aug 2025 GLM-4.5V technical report 65.2 MMMU Pro, GLM-4.5V thinking (Table 2) Capabilities
1 Jul 2025 GLM-4.1V-9B-Thinking technical report 57.1 MMMU Pro, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
23 Jun 2025 Kimi-VL-A3B-Thinking-2506 technical report 46.3 MMMU-Pro (avg) (Pass@1) (Table 4) Capabilities
12 Jun 2025 Magistral Small 1.0 technical report 51.8 MMMU-Pro (Standard), Magistral Small bar, printed label (Figure 10). MMMU-Pro (Vision) 39.2 Capabilities
12 Jun 2025 Magistral Medium 1.0 technical report 57.9 MMMU-Pro (Standard), printed bar label (Figure 10). MMMU-Pro (Vision) 52.1 Capabilities
11 May 2025 Seed1.5-VL technical report 67.6 MMMU-Pro, thinking (Table 6). Non-thinking 59.9 Capabilities
5 Apr 2025 Llama 4 Scout model card 52.2 MMMU Pro, 0-shot, average of Standard and Vision, instruction tuned (Instruction tuned models table) Capabilities
5 Apr 2025 Llama 4 Maverick model card 59.6 MMMU Pro, 0-shot, average of Standard and Vision, instruction tuned (Instruction tuned models table) Capabilities
26 Mar 2025 Qwen2.5-Omni-7B technical report 36.6 MMMU-Pro overall (Table 5) Capabilities
3 Mar 2025 Phi-4-multimodal technical report 38.5 MMMUPro, average of standard 39.7 and vision 37.3 (Table 1) Capabilities
19 Feb 2025 Qwen2.5-VL-72B technical report 51.1 MMMU-Pro overall (Table 3) Capabilities

Versions

Replaces MMMU

More in Vision

All 234 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy. Qwen 3.8 Max 97.7%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%
  • DocVQA In use Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. Seed1.5-VL 96.9%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%