MATH-Vision (MathVision)
3,040 competition math problems with visual context. Answer accuracy.
In the authors’ words
Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io
- Level
- Competition The strongest students, in olympiad-style contests
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Short answer A number, word or short string matched to a key.
Qwen 3.8 Max leads with 97.7%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 97.7
Qwen 3.8 Max
- 96.2
Qwen3.8-Omni-Flash
- 95.7
Qwen3.8 Flash
- 94.6
Qwen 3.8 27B
- 94.5
Seed2.1 Pro
- 94.3 Kimi K3
- 90.3
Qwen3.7-Plus
- 89.8
Seed2.0 Lite (0428)
- 88.8
Seed2.0 Pro
- 88.6
Qwen3.5 397B-A17B
- 88.0
Qwen 3.6 Plus
- 87.4 Kimi K2.6
- 86.2
Qwen3.5-122B-A10B
- 85.6
Gemma 4 31B IT
- 84.2 Kimi K2.5
- 82.4
Gemma 4 26B A4B
- 81.3
Seed1.8
- 78.2
Nemotron 3.5 Super VL
- 75.2
EXAONE 4.5 33B
- 74.6
Qwen3-VL-235B-A22B
- 74.3
ERNIE 5.0
- 73.0
Qwen3.5-Omni-Plus
- 70.8
STEP3-VL-10B
- 70.5
DiffusionGemma 26B A4B
- 70.3
Seed1.6-Thinking
- 68.7
Seed1.5-VL
- 65.6 GLM-4.5V
- 64.8
Step 3
- 63.9
InternVL3.5-241B-A28B
- 63.5 GLM-4.6V
- 62.5
Intern-S1
- 61.9
ERNIE-4.5-VL-28B-A3B-Thinking
- 56.9 Kimi-VL-A3B-Thinking-2506
- 56.3
Qwen3-Omni-30B-A3B
- 54.4 GLM-4.1V-9B-Thinking
- 53.6
Nemotron Nano 12B v2 VL
- 46.3
Nemotron 3.5 Bolt E4B
- 38.6 Kimi k1.5
- 38.1
Qwen2.5-VL-72B
- 36.8 Kimi-VL-A3B-Thinking
- 36.2
Phi-4-Reasoning-Vision-15B
- 25.0
Qwen2.5-Omni-7B
- 1
Qwen 3.8 Max 97.7%
- 2
Qwen3.8-Omni-Flash 96.2%
- 3
Qwen3.8 Flash 95.7%
- 4
Qwen 3.8 27B 94.6%
- 5
Seed2.1 Pro 94.5%
- 6 Kimi K3 94.3%
- 7
Qwen3.7-Plus 90.3%
- 8
Seed2.0 Lite (0428) 89.8%
- 9
Seed2.0 Pro 88.8%
- 10
Qwen3.5 397B-A17B 88.6%
- 11
Qwen 3.6 Plus 88%
- 12 Kimi K2.6 87.4%
- 13
Qwen3.5-122B-A10B 86.2%
- 14
Gemma 4 31B IT 85.6%
- 15 Kimi K2.5 84.2%
- 16
Gemma 4 26B A4B 82.4%
- 17
Seed1.8 81.3%
- 18
Nemotron 3.5 Super VL 78.2%
- 19
EXAONE 4.5 33B 75.2%
- 20
Qwen3-VL-235B-A22B 74.6%
- 21
ERNIE 5.0 74.3%
- 22
Qwen3.5-Omni-Plus 73%
- 23
STEP3-VL-10B 70.8%
- 24
DiffusionGemma 26B A4B 70.5%
- 25
Seed1.6-Thinking 70.3%
- 26
Seed1.5-VL 68.7%
- 27 GLM-4.5V 65.6%
- 28
Step 3 64.8%
- 29
InternVL3.5-241B-A28B 63.9%
- 30 GLM-4.6V 63.5%
- 31
Intern-S1 62.5%
- 32
ERNIE-4.5-VL-28B-A3B-Thinking 61.9%
- 33 Kimi-VL-A3B-Thinking-2506 56.9%
- 34
Qwen3-Omni-30B-A3B 56.3%
- 35 GLM-4.1V-9B-Thinking 54.4%
- 36
Nemotron Nano 12B v2 VL 53.6%
- 37
Nemotron 3.5 Bolt E4B 46.3%
- 38 Kimi k1.5 38.6%
- 39
Qwen2.5-VL-72B 38.1%
- 40 Kimi-VL-A3B-Thinking 36.8%
- 41
Phi-4-Reasoning-Vision-15B 36.2%
- 42
Qwen2.5-Omni-7B 25%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | | 97.7% | Aug 2026 | Model card | |
| 2 | | 96.2% | Sep 2026 | Model card | |
| 3 | | 95.7% | Aug 2026 | Model card | |
| 4 | | 94.6% | Aug 2026 | Model card | |
| 5 | | 94.5% | Jun 2026 | Model card | |
| 6 | Kimi K3 | 94.3% | Jul 2026 | Model card | |
| 7 | | 90.3% | Jun 2026 | Model card | |
| 8 | | 89.8% | Apr 2026 | Model card | |
| 9 | | 88.8% | Feb 2026 | Model card | |
| 10 | | 88.6% | Feb 2026 | Model card | |
| 11 | | 88% | Apr 2026 | Model card | |
| 12 | Kimi K2.6 | 87.4% | Apr 2026 | Model card | |
| 13 | | 86.2% | Feb 2026 | Model card | |
| 14 | | 85.6% | Apr 2026 | Model card | |
| 15 | Kimi K2.5 | 84.2% | Jan 2026 | Model card | |
| 16 | | 82.4% | Apr 2026 | Model card | |
| 17 | | 81.3% | Dec 2025 | Model card | |
| 18 | | 78.2% | Oct 2026 | Model card | |
| 19 | | 75.2% | Apr 2026 | Model card | |
| 20 | | 74.6% | Sep 2025 | Model card | |
| 21 | | 74.3% | Feb 2026 | Model card | |
| 22 | | 73% | Mar 2026 | Model card | |
| 23 | | 70.8% | Jan 2026 | Model card | |
| 24 | | 70.5% | Jun 2026 | Model card | |
| 25 | | 70.3% | Jun 2025 | Model card | |
| 26 | | 68.7% | May 2025 | Model card | |
| 27 | GLM-4.5V | 65.6% | Aug 2025 | Model card | |
| 28 | | 64.8% | Jul 2025 | Model card | |
| 29 | | 63.9% | Aug 2025 | Model card | |
| 30 | GLM-4.6V | 63.5% | Jan 2026 | Model card | |
| 31 | | 62.5% | Jul 2025 | Model card | |
| 32 | | 61.9% | Nov 2025 | Model card | |
| 33 | Kimi-VL-A3B-Thinking-2506 | 56.9% | Jun 2025 | Model card | |
| 34 | | 56.3% | Sep 2025 | Model card | |
| 35 | GLM-4.1V-9B-Thinking | 54.4% | Jul 2025 | Model card | |
| 36 | | 53.6% | Nov 2025 | Model card | |
| 37 | | 46.3% | Oct 2026 | Model card | |
| 38 | Kimi k1.5 | 38.6% | Jan 2025 | Model card | |
| 39 | | 38.1% | Jan 2025 | Model card | |
| 40 | Kimi-VL-A3B-Thinking | 36.8% | Apr 2025 | Model card | |
| 41 | | 36.2% | Mar 2026 | Model card | |
| 42 | | 25% | Mar 2025 | Model card |
The benchmark's own leaderboard has 153 entries, 105 of them models we track. We plot every source in the charts below.
More effort adds a median 6 points
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
12 models have held number one since its release in Feb 2024
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Nov 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
5 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 15 Oct 2026 | 78.2 | MathVision | |
| 18 Sep 2026 | 96.2 | MathVision Without CI 91.8 With CI 96.2 | |
| 26 Aug 2026 | 95.7 | Visual math problem solving MathVision Without CI 90.6 With CI 95.7 | |
| 14 Aug 2026 | 94.6 | Visual math problem solving MathVision Without CI 90.0 With CI 94.6 | |
| 3 Aug 2026 | 97.7 | MathVision (without CI / with CI) 95.2 / 97.7 | |
| 16 Jul 2026 | Kimi K3 | 94.3 | MathVision (without python) |
| 23 Jun 2026 | 94.5 | MathVision (w. Tool) 92.6 (94.5) | |
| 10 Jun 2026 | 70.5 | MATH-Vision | |
| 1 Jun 2026 | 90.3 | MathVision | |
| 28 Apr 2026 | 89.8 | MathVision | |
| 20 Apr 2026 | Kimi K2.6 | 87.4 | MathVision |
| 9 Apr 2026 | 75.2 | MathVision | |
| 2 Apr 2026 | 82.4 | MATH-Vision | |
| 2 Apr 2026 | 88 | MathVision | |
| 2 Apr 2026 | 85.6 | MATH-Vision | |
| 30 Mar 2026 | 73 | MathVision | |
| 4 Mar 2026 | 36.2 | MathVision_MINI (default mode) | |
| 24 Feb 2026 | 86.2 | MathVision | |
| 16 Feb 2026 | 88.6 | MathVision | |
| 14 Feb 2026 | 88.8 | MathVision | |
| 6 Feb 2026 | 74.3 | MathVision | |
| 27 Jan 2026 | Kimi K2.5 | 84.2 | MathVision |
| 13 Jan 2026 | 70.8 | MathVision (SeRe) | |
| 18 Dec 2025 | 81.3 | MathVision | |
| 11 Nov 2025 | 61.9 | MathVision | |
| 23 Sep 2025 | 74.6 | MathVision | |
| 22 Sep 2025 | 56.3 | MathVision_full | |
| 25 Aug 2025 | 63.9 | MathVision | |
| 11 Aug 2025 | GLM-4.5V | 65.6 | MathVision |
| 31 Jul 2025 | 64.8 | MATH-Vision | |
| 24 Jul 2025 | 62.5 | MathVision | |
| 25 Jun 2025 | 70.3 | MathVision | |
| 21 Jun 2025 | Kimi-VL-A3B-Thinking-2506 | 56.9 | MATH-Vision (Pass@1) |
| 12 May 2025 | 68.7 | MathVision | |
| 10 Apr 2025 | Kimi-VL-A3B-Thinking | 36.8 | MathVision (full) (Pass@1) |
| 27 Mar 2025 | 25 | MathVisionfull | |
| 28 Jan 2025 | 38.1 | MathVision_FULL | |
| 20 Jan 2025 | Kimi k1.5 | 38.6 | MathVision-Full (Pass@1), long-CoT |
29 system cards report it, from 11 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 23 Oct 2026 | | 46.3 | MathVision, reasoning, 43.72 non-reasoning (Table 8) | Capabilities |
| 15 Oct 2026 | | 78.2 | MathVision (Table 10) | Capabilities |
| 22 Sep 2026 | | 91.8 | MathVision without CI, some ground truth corrected (Table 3). With CI 96.2 | Capabilities |
| 27 Jul 2026 | Kimi K3 technical report | 94.3 | Math-Vision without Python, cell 94.3 / 97.8 (Table 2) | Capabilities |
| 24 Jul 2026 | | 85.6 | MATH-Vision, thinking, max resolution (Table 6) | Capabilities |
| 24 Jul 2026 | | 82.4 | MATH-Vision, thinking, max resolution (Table 6) | Capabilities |
| 23 Jun 2026 | | 92.6 | MathVision, no tools, 94.5 with tools (Table 9) | Capabilities |
| 17 Apr 2026 | | 73 | MathVision (Table 6) | Capabilities |
| 9 Apr 2026 | | 75.2 | MATH-Vision (Table 2, Reasoning mode) | Capabilities |
| 21 Mar 2026 | | 81.3 | MathVision (Table 2) | Capabilities |
| 4 Mar 2026 | | 36.2 | MathVision MINI, default mode, 34.2 force nothink, 36.2 force thinking (Tables 5 and 6) | Capabilities |
| 14 Feb 2026 | | 88.8 | MathVision (Table 8) | Capabilities |
| 4 Feb 2026 | | 74.3 | MathVision (Table 3) | Capabilities |
| 30 Jan 2026 | Kimi K2.5 technical report | 84.2 | MathVision (Table 4) | Capabilities |
| 14 Jan 2026 | | 70.8 | MathVision, SeRe (Tables 1 and 3). PaCoRe 75.95 | Capabilities |
| 1 Jan 2026 | GLM-4.6V technical report | 63.5 | MathVision, GLM-4.6V thinking (Table 2) | Capabilities |
| 26 Nov 2025 | | 74.6 | MathVision (Table 2) | Capabilities |
| 6 Nov 2025 | | 53.6 | MathVision, reasoning on (Table 3). Reasoning off 31.5 | Capabilities |
| 22 Sep 2025 | | 56.3 | MATH-Vision full, Instruct (Table 9). Thinking 62.9 (Table 10) | Capabilities |
| 25 Aug 2025 | | 63.9 | MathVision (Table 2, Table 3). With parallel thinking 65.9 | Capabilities |
| 21 Aug 2025 | | 62.5 | MathVision (Table 2) | Capabilities |
| 13 Aug 2025 | GLM-4.5V technical report | 65.6 | MathVision, GLM-4.5V thinking (Table 2) | Capabilities |
| 1 Jul 2025 | GLM-4.1V-9B-Thinking technical report | 54.4 | MathVision, GLM-4.1V-9B-Thinking thinking (Table 2) | Capabilities |
| 23 Jun 2025 | Kimi-VL-A3B-Thinking-2506 technical report | 56.9 | MathVision (full) (Pass@1) (Table 4) | Capabilities |
| 11 May 2025 | | 68.7 | MathVision, thinking (Table 6). Non-thinking 65.5 | Capabilities |
| 10 Apr 2025 | Kimi-VL-A3B-Thinking technical report | 36.8 | MathVision (full) (Pass@1) (Table 4). Figure 13 shows 18.7 at 1k rising to 36.8 at 16k max thinking tokens | Capabilities |
| 26 Mar 2025 | | 25 | MathVision full (Table 5) | Capabilities |
| 19 Feb 2025 | | 38.1 | MATH-Vision full (Table 3) | Capabilities |
| 20 Jan 2025 | Kimi k1.5 technical report | 38.6 | MathVision-Full (Pass@1), long-CoT (Table 2). Short-CoT 31.0 (Table 3) | Capabilities |
More in Vision
- MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
- MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. Kimi K3.1 Pro 84.1%
- CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
- ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance.
Command A Vision 90.9%
- DocVQA In use Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers.
Seed1.5-VL 96.9%
- MMStar In use Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy.
Qwen 3.8 Max 85.9%