Benchmark Almanac

MATH-Vision (MathVision)

Qwen 3.8 Max 97.7% leads since Aug 2026 In use

3,040 competition math problems with visual context. Answer accuracy.

In the authors’ words

Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io

Abstract of Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset (arXiv:2402.14804, CC0)
Made by CUHK MMLab Released Feb 2024 Visual math and diagrams All tasks public Last in a launch post Oct 2026 Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Competition The strongest students, in olympiad-style contests
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
42
Models scored in lab cards
38
Launch posts list it, from 11 labs
29
System cards report it, from 11 labs

Qwen 3.8 Max leads with 97.7%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

  1. 1 Qwen 3.8 Max 97.7%
  2. 2 Qwen3.8-Omni-Flash 96.2%
  3. 3 Qwen3.8 Flash 95.7%
  4. 4 Qwen 3.8 27B 94.6%
  5. 5 Seed2.1 Pro 94.5%
  6. 6 Kimi K3 94.3%
  7. 7 Qwen3.7-Plus 90.3%
  8. 8 Seed2.0 Lite (0428) 89.8%
  9. 9 Seed2.0 Pro 88.8%
  10. 10 Qwen3.5 397B-A17B 88.6%
  11. 11 Qwen 3.6 Plus 88%
  12. 12 Kimi K2.6 87.4%
  13. 13 Qwen3.5-122B-A10B 86.2%
  14. 14 Gemma 4 31B IT 85.6%
  15. 15 Kimi K2.5 84.2%
  16. 16 Gemma 4 26B A4B 82.4%
  17. 17 Seed1.8 81.3%
  18. 18 Nemotron 3.5 Super VL 78.2%
  19. 19 EXAONE 4.5 33B 75.2%
  20. 20 Qwen3-VL-235B-A22B 74.6%
  21. 21 ERNIE 5.0 74.3%
  22. 22 Qwen3.5-Omni-Plus 73%
  23. 23 STEP3-VL-10B 70.8%
  24. 24 DiffusionGemma 26B A4B 70.5%
  25. 25 Seed1.6-Thinking 70.3%
  26. 26 Seed1.5-VL 68.7%
  27. 27 GLM-4.5V 65.6%
  28. 28 Step 3 64.8%
  29. 29 InternVL3.5-241B-A28B 63.9%
  30. 30 GLM-4.6V 63.5%
  31. 31 Intern-S1 62.5%
  32. 32 ERNIE-4.5-VL-28B-A3B-Thinking 61.9%
  33. 33 Kimi-VL-A3B-Thinking-2506 56.9%
  34. 34 Qwen3-Omni-30B-A3B 56.3%
  35. 35 GLM-4.1V-9B-Thinking 54.4%
  36. 36 Nemotron Nano 12B v2 VL 53.6%
  37. 37 Nemotron 3.5 Bolt E4B 46.3%
  38. 38 Kimi k1.5 38.6%
  39. 39 Qwen2.5-VL-72B 38.1%
  40. 40 Kimi-VL-A3B-Thinking 36.8%
  41. 41 Phi-4-Reasoning-Vision-15B 36.2%
  42. 42 Qwen2.5-Omni-7B 25%
#ModelScoreSettingDateSource
1 Qwen 3.8 Max 97.7% Aug 2026 Model card
2 Qwen3.8-Omni-Flash 96.2% Sep 2026 Model card
3 Qwen3.8 Flash 95.7% Aug 2026 Model card
4 Qwen 3.8 27B 94.6% Aug 2026 Model card
5 Seed2.1 Pro 94.5% Jun 2026 Model card
6 Kimi K3 94.3% Jul 2026 Model card
7 Qwen3.7-Plus 90.3% Jun 2026 Model card
8 Seed2.0 Lite (0428) 89.8% Apr 2026 Model card
9 Seed2.0 Pro 88.8% Feb 2026 Model card
10 Qwen3.5 397B-A17B 88.6% Feb 2026 Model card
11 Qwen 3.6 Plus 88% Apr 2026 Model card
12 Kimi K2.6 87.4% Apr 2026 Model card
13 Qwen3.5-122B-A10B 86.2% Feb 2026 Model card
14 Gemma 4 31B IT 85.6% Apr 2026 Model card
15 Kimi K2.5 84.2% Jan 2026 Model card
16 Gemma 4 26B A4B 82.4% Apr 2026 Model card
17 Seed1.8 81.3% Dec 2025 Model card
18 Nemotron 3.5 Super VL 78.2% Oct 2026 Model card
19 EXAONE 4.5 33B 75.2% Apr 2026 Model card
20 Qwen3-VL-235B-A22B 74.6% Sep 2025 Model card
21 ERNIE 5.0 74.3% Feb 2026 Model card
22 Qwen3.5-Omni-Plus 73% Mar 2026 Model card
23 STEP3-VL-10B 70.8% Jan 2026 Model card
24 DiffusionGemma 26B A4B 70.5% Jun 2026 Model card
25 Seed1.6-Thinking 70.3% Jun 2025 Model card
26 Seed1.5-VL 68.7% May 2025 Model card
27 GLM-4.5V 65.6% Aug 2025 Model card
28 Step 3 64.8% Jul 2025 Model card
29 InternVL3.5-241B-A28B 63.9% Aug 2025 Model card
30 GLM-4.6V 63.5% Jan 2026 Model card
31 Intern-S1 62.5% Jul 2025 Model card
32 ERNIE-4.5-VL-28B-A3B-Thinking 61.9% Nov 2025 Model card
33 Kimi-VL-A3B-Thinking-2506 56.9% Jun 2025 Model card
34 Qwen3-Omni-30B-A3B 56.3% Sep 2025 Model card
35 GLM-4.1V-9B-Thinking 54.4% Jul 2025 Model card
36 Nemotron Nano 12B v2 VL 53.6% Nov 2025 Model card
37 Nemotron 3.5 Bolt E4B 46.3% Oct 2026 Model card
38 Kimi k1.5 38.6% Jan 2025 Model card
39 Qwen2.5-VL-72B 38.1% Jan 2025 Model card
40 Kimi-VL-A3B-Thinking 36.8% Apr 2025 Model card
41 Phi-4-Reasoning-Vision-15B 36.2% Mar 2026 Model card
42 Qwen2.5-Omni-7B 25% Mar 2025 Model card

The benchmark's own leaderboard has 153 entries, 105 of them models we track. We plot every source in the charts below.

More effort adds a median 6 points

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-5.4
xhigh
96.1%
Gemini 3.1 Pro
thinking hi… thinking hi…
95.7%
Kimi K2.6
no tools with python
93.2%
Kimi K2.5
thinking, w…
85%
Claude Opus 4.6
max max, with p…
84.6%
GPT-5
minimal
72%
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5.4 xhigh, with python 96.1%
Gemini 3.1 Pro thinking high, with python 95.7%
Kimi K2.6 with python 93.2%
GPT-5.4 xhigh 92%
Gemini 3 Flash 90.6%
Gemini 3.1 Pro thinking high 89.8%
Seed2.0 Lite (0428) 89.8%
Seed2.0 Pro 88.8%
Qwen3.5 397B-A17B 88.6%
Kimi K2.6 no tools 87.4%
Gemini 3 Pro 86.6%
Seed2.0 Lite (0215) 86.4%
Qwen3.5-122B-A10B 86.2%
Qwen3.5-27B 86%
Gemma 4 31B IT 85.6%
Kimi K2.5 thinking, with python 85%
Claude Opus 4.6 max, with python 84.6%
Kimi K2.5 84.2%
Qwen3.5-35B-A3B 83.9%
GPT-5.2 83%
Gemma 4 26B A4B 82.4%
Seed1.8 81.3%
Seed1.6-Thinking 77.2%
Claude Opus 4.5 77.1%
Qwen3-VL-235B-A22B thinking 74.6%
Gemini 2.5 Pro Mar 2025 73.3%
GPT-5 72%
GPT-5 mini 71.9%
Claude Opus 4.6 max 71.2%
Claude Sonnet 4.5 71.1%
dots.vlm1 69.6%
Seed1.5-VL 68.7%
Claude Opus 4.1 thinking 66%
Qwen3-VL-235B-A22B-Instruct 66%
GLM-4.5V 65.6%
Step 3 64.8%
InternVL3.5-241B-A28B 63.9%
GLM-4.6V 63.5%
MiMo-VL-7B-RL 60.4%
o1 60.3%
MiMo-VL-7B-RL-2508 59.6%
Qwen3-VL-8B-Thinking 59.6%
Gemma 4 E4B 59.5%
Claude 3.7 Sonnet 58.6%
o4-mini 58%
MiMo-VL-7B-SFT 57.9%
Kimi-VL-A3B-Thinking-2506 56.9%
InternVL3.5-30B-A3B 55.7%
GLM-4.1V-9B-Thinking 54.4%
GLM-4.6V-Flash 54%
InternVL3.5-38B 54%
Ovis2.5-9B 53.9%
Skywork-R1V3-38B 52.6%
Gemma 4 E2B 52.4%
InternVL3.5-8B 52%
Qianfan-VL-70B 50.3%
Skywork-R1V2-38B 49.7%
Doubao-1.5-pro 48.6%
Gemini 2.0 Pro 48.1%
GPT-4.5 47.3%
Gemma 3 27B non-thinking 46%
Keye-VL-8B-Preview 46%
GPT-5 minimal 45.8%
InternVL3-78B 43.1%
Gemini 2.0 Flash Feb 2025 41.3%
InternVL3.5-4B 40%
Kimi k1.5 38.6%
Qwen2.5-VL-72B 38.1%
Claude 3.5 Sonnet 38%
Ovis2.5-2B 37.4%
InternVL3-14B 37%
Kimi-VL-A3B-Thinking 36.8%
QVQ-72B-Preview 35.9%
InternVL3-38B 34.5%
Qianfan-VL-8B 32.8%
InternVL2_5-78B 32.2%
Ovis2-34B 31.9%
InternVL2.5-38B 31.8%
GPT-4o May 2024 30.4%
GPT-4 Turbo Apr 2024 30.3%
Ovis2-16B 30.1%
InternVL3-8B 29%
InternVL3-9B 27.6%
Claude 3 Opus 27.1%
Qwen2-VL-72B-Instruct 25.9%
LLaVA-OneVision 72B 25.3%
Qwen2.5-VL-7B 25.1%
MiniCPM-V-2_6 23.4%
InternVL2.5-26B 23.1%
GPT-4V 22.8%
MiniCPM-o-2_6 21.7%
InternVL3-2B 21.7%
Qwen2.5-VL-3B 21.2%
InternVL2.5-8B 19.7%
Gemini 1.5 Pro May 2024 19.2%
InternVL3-1B 18.8%
Ovis1.6-Gemma2-9B 18.8%
Aquila-VL-2B 17.9%
Gemini 1.0 Pro 17.7%
InternVL-Chat-V1-2-Plus 17%
Qwen2-VL-7B-Instruct 16.3%
Qwen-VL-Max 15.6%
InternVL2.5-2B 14.7%
InternLM-XComposer2-VL-7B 14.5%
InternVL2.5-1B 14.4%
Qwen2-VL-2B-Instruct 12.4%
ShareGPT4V-13B 11.9%
LLaVA-v1.5-13B 11.1%
Qwen-VL-Plus 10.7%
ShareGPT4V-7B 10.5%
llava-v1.5-7b 8.52%

12 models have held number one since its release in Feb 2024

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic Google DeepMind ByteDance Seed Moonshot AI Qwen
Own leaderboard Passed 85% in Nov 2025, by Gemini 3 Pro
GPT-5.4 96.1%
Lab claims Passed 85% in Feb 2026, by Seed2.0 Pro
Qwen 3.8 Max 97.7%

The best score passed 85% in Nov 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Qwen 3.8 Max Gemini 3 Flash Seed1.6-Thinking o1 Claude 3.5 Sonnet GPT-4o GPT-4V 0% 20% 40% 60% 80% 100% 2024 2025 2026 2027 Near the ceiling (85%) Qwen 3.8 Max Gemini 3 Flash

5 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Moonshot AI
Qwen
ByteDance Seed
Shanghai AI Lab
StepFun
Z.ai
Baidu
Microsoft
Google DeepMind
LG AI Research
NVIDIA
DateLaunch postScoreAs the post writes it
15 Oct 2026 Nemotron 3.5 Super VL 78.2 MathVision
18 Sep 2026 Qwen3.8-Omni-Flash 96.2 MathVision Without CI 91.8 With CI 96.2
26 Aug 2026 Qwen3.8 Flash 95.7 Visual math problem solving MathVision Without CI 90.6 With CI 95.7
14 Aug 2026 Qwen 3.8 27B 94.6 Visual math problem solving MathVision Without CI 90.0 With CI 94.6
3 Aug 2026 Qwen 3.8 Max 97.7 MathVision (without CI / with CI) 95.2 / 97.7
16 Jul 2026 Kimi K3 94.3 MathVision (without python)
23 Jun 2026 Seed2.1 Pro 94.5 MathVision (w. Tool) 92.6 (94.5)
10 Jun 2026 DiffusionGemma 26B A4B 70.5 MATH-Vision
1 Jun 2026 Qwen3.7-Plus 90.3 MathVision
28 Apr 2026 Seed2.0 Lite (0428) 89.8 MathVision
20 Apr 2026 Kimi K2.6 87.4 MathVision
9 Apr 2026 EXAONE 4.5 33B 75.2 MathVision
2 Apr 2026 Gemma 4 26B A4B 82.4 MATH-Vision
2 Apr 2026 Qwen 3.6 Plus 88 MathVision
2 Apr 2026 Gemma 4 31B IT 85.6 MATH-Vision
30 Mar 2026 Qwen3.5-Omni-Plus 73 MathVision
4 Mar 2026 Phi-4-Reasoning-Vision-15B 36.2 MathVision_MINI (default mode)
24 Feb 2026 Qwen3.5-122B-A10B 86.2 MathVision
16 Feb 2026 Qwen3.5 397B-A17B 88.6 MathVision
14 Feb 2026 Seed2.0 Pro 88.8 MathVision
6 Feb 2026 ERNIE 5.0 74.3 MathVision
27 Jan 2026 Kimi K2.5 84.2 MathVision
13 Jan 2026 STEP3-VL-10B 70.8 MathVision (SeRe)
18 Dec 2025 Seed1.8 81.3 MathVision
11 Nov 2025 ERNIE-4.5-VL-28B-A3B-Thinking 61.9 MathVision
23 Sep 2025 Qwen3-VL-235B-A22B 74.6 MathVision
22 Sep 2025 Qwen3-Omni-30B-A3B 56.3 MathVision_full
25 Aug 2025 InternVL3.5-241B-A28B 63.9 MathVision
11 Aug 2025 GLM-4.5V 65.6 MathVision
31 Jul 2025 Step 3 64.8 MATH-Vision
24 Jul 2025 Intern-S1 62.5 MathVision
25 Jun 2025 Seed1.6-Thinking 70.3 MathVision
21 Jun 2025 Kimi-VL-A3B-Thinking-2506 56.9 MATH-Vision (Pass@1)
12 May 2025 Seed1.5-VL 68.7 MathVision
10 Apr 2025 Kimi-VL-A3B-Thinking 36.8 MathVision (full) (Pass@1)
27 Mar 2025 Qwen2.5-Omni-7B 25 MathVisionfull
28 Jan 2025 Qwen2.5-VL-72B 38.1 MathVision_FULL
20 Jan 2025 Kimi k1.5 38.6 MathVision-Full (Pass@1), long-CoT

29 system cards report it, from 11 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
23 Oct 2026 Nemotron 3.5 Bolt E4B technical report 46.3 MathVision, reasoning, 43.72 non-reasoning (Table 8) Capabilities
15 Oct 2026 Nemotron 3.5 Super VL technical report 78.2 MathVision (Table 10) Capabilities
22 Sep 2026 Qwen3.8-Omni-Flash technical report 91.8 MathVision without CI, some ground truth corrected (Table 3). With CI 96.2 Capabilities
27 Jul 2026 Kimi K3 technical report 94.3 Math-Vision without Python, cell 94.3 / 97.8 (Table 2) Capabilities
24 Jul 2026 Gemma 4 31B IT technical report 85.6 MATH-Vision, thinking, max resolution (Table 6) Capabilities
24 Jul 2026 Gemma 4 26B A4B technical report 82.4 MATH-Vision, thinking, max resolution (Table 6) Capabilities
23 Jun 2026 Seed2.1 Pro model card 92.6 MathVision, no tools, 94.5 with tools (Table 9) Capabilities
17 Apr 2026 Qwen3.5-Omni-Plus technical report 73 MathVision (Table 6) Capabilities
9 Apr 2026 EXAONE 4.5 33B technical report 75.2 MATH-Vision (Table 2, Reasoning mode) Capabilities
21 Mar 2026 Seed1.8 model card 81.3 MathVision (Table 2) Capabilities
4 Mar 2026 Phi-4-Reasoning-Vision-15B technical report 36.2 MathVision MINI, default mode, 34.2 force nothink, 36.2 force thinking (Tables 5 and 6) Capabilities
14 Feb 2026 Seed2.0 Pro model card 88.8 MathVision (Table 8) Capabilities
4 Feb 2026 ERNIE 5.0 technical report 74.3 MathVision (Table 3) Capabilities
30 Jan 2026 Kimi K2.5 technical report 84.2 MathVision (Table 4) Capabilities
14 Jan 2026 STEP3-VL-10B technical report 70.8 MathVision, SeRe (Tables 1 and 3). PaCoRe 75.95 Capabilities
1 Jan 2026 GLM-4.6V technical report 63.5 MathVision, GLM-4.6V thinking (Table 2) Capabilities
26 Nov 2025 Qwen3-VL-235B-A22B technical report 74.6 MathVision (Table 2) Capabilities
6 Nov 2025 Nemotron Nano 12B v2 VL technical report 53.6 MathVision, reasoning on (Table 3). Reasoning off 31.5 Capabilities
22 Sep 2025 Qwen3-Omni-30B-A3B technical report 56.3 MATH-Vision full, Instruct (Table 9). Thinking 62.9 (Table 10) Capabilities
25 Aug 2025 InternVL3.5-241B-A28B technical report 63.9 MathVision (Table 2, Table 3). With parallel thinking 65.9 Capabilities
21 Aug 2025 Intern-S1 technical report 62.5 MathVision (Table 2) Capabilities
13 Aug 2025 GLM-4.5V technical report 65.6 MathVision, GLM-4.5V thinking (Table 2) Capabilities
1 Jul 2025 GLM-4.1V-9B-Thinking technical report 54.4 MathVision, GLM-4.1V-9B-Thinking thinking (Table 2) Capabilities
23 Jun 2025 Kimi-VL-A3B-Thinking-2506 technical report 56.9 MathVision (full) (Pass@1) (Table 4) Capabilities
11 May 2025 Seed1.5-VL technical report 68.7 MathVision, thinking (Table 6). Non-thinking 65.5 Capabilities
10 Apr 2025 Kimi-VL-A3B-Thinking technical report 36.8 MathVision (full) (Pass@1) (Table 4). Figure 13 shows 18.7 at 1k rising to 36.8 at 16k max thinking tokens Capabilities
26 Mar 2025 Qwen2.5-Omni-7B technical report 25 MathVision full (Table 5) Capabilities
19 Feb 2025 Qwen2.5-VL-72B technical report 38.1 MATH-Vision full (Table 3) Capabilities
20 Jan 2025 Kimi k1.5 technical report 38.6 MathVision-Full (Pass@1), long-CoT (Table 2). Short-CoT 31.0 (Table 3) Capabilities

More in Vision

All 234 in Vision
  • MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
  • MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. Kimi K3.1 Pro 84.1%
  • CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
  • ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. Command A Vision 90.9%
  • DocVQA In use Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. Seed1.5-VL 96.9%
  • MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. Qwen 3.8 Max 85.9%