Benchmark Almanac

MathBench

GPT-4o May 2024 70.9% leads since May 2024 Not in model cards

Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency.

Accepted to ACL 2024 Findings. The repository has about 118 stars.

In the authors’ words

Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, falling short in providing a holistic assessment of the LLMs' math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model's mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs' mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context. The project is released at https://github.com/open-compass/MathBench .

Abstract of MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (arXiv:2405.12209, CC0)
Made by OpenCompass (Shanghai AI Laboratory) Released May 2024 Word problems All tasks public Last model added Jun 2024 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
open-compass/MathBench 118 stars Paper Leaderboard
33
Entries on its own leaderboard, 21 models we track

GPT-4o leads with 70.9% among models released by Jun 2024

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Jun 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

33 entries. Scroll sideways for the rest.

  1. 70.9 GPT-4o
  2. 67.0 Qwen2-72B
  3. 63.0 Claude 3 Opus
  4. 59.9 DeepSeek-V2 (MoE-236B, May 2024)
  5. 59.2 Qwen-Max-0428
  6. 58.8 GPT-4 Turbo (Jan 2024)
  7. 58.4 Qwen1.5-110B
  8. 56.4 Llama 3-70B
  9. 53.4 Qwen2-7B-Instruct
  10. 52.2 Yi-1.5-34B
  11. 51.3 GLM-4
  12. 51.1 Yi-1.5-9B-Chat
  13. 50.3 DeepSeek-Math-7B-RL
  14. 49.7 Qwen-72B-Chat
  15. 47.7 GLM-4-9B-Chat
  16. 42.1 InternLM2-Chat-20B
  17. 41.9 InternLM2-Chat-Math-20B
  18. 40.9 DeepSeekMath-7B-Instruct
  19. 40.3 DeepSeek LLM 67B
  20. 39.5 Qwen-14B
  21. 37.7 InternLM2-Chat-Math-7B
  22. 36.7 Llama 3-8B
  23. 34.1 InternLM2-Chat-7B
  24. 31.6 Yi-34B
  25. 27.7 Qwen-7B
  26. 24.9 Metamath-Llemma-7B
  27. 24.7 MammoTH-70B
  28. 22.3 DeepSeek-7B-Chat
  29. 20.9 Baichuan2-13B-Chat
  30. 19.9 ChatGLM3-6B
  31. 18.5 MammoTH-13B
  32. 17.2 Yi-6B-Chat
  33. 11.3 MammoTH-7B
  1. 1 GPT-4o May 2024 70.9%
  2. 2 Qwen2-72B 67%
  3. 3 Claude 3 Opus 63%
  4. 4 DeepSeek-V2 (MoE-236B, May 2024) 59.9%
  5. 5 Qwen-Max-0428 59.2%
  6. 6 GPT-4 Turbo (Jan 2024) 58.8%
  7. 7 Qwen1.5-110B 58.4%
  8. 8 Llama 3-70B 56.4%
  9. 9 Qwen2-7B-Instruct 53.4%
  10. 10 Yi-1.5-34B 52.2%
  11. 11 GLM-4 51.3%
  12. 12 Yi-1.5-9B-Chat 51.1%
  13. 13 DeepSeek-Math-7B-RL 50.3%
  14. 14 Qwen-72B-Chat 49.7%
  15. 15 GLM-4-9B-Chat 47.7%
  16. 16 InternLM2-Chat-20B 42.1%
  17. 17 InternLM2-Chat-Math-20B 41.9%
  18. 18 DeepSeekMath-7B-Instruct 40.9%
  19. 19 DeepSeek LLM 67B 40.3%
  20. 20 Qwen-14B 39.5%
  21. 21 InternLM2-Chat-Math-7B 37.7%
  22. 22 Llama 3-8B 36.7%
  23. 23 InternLM2-Chat-7B 34.1%
  24. 24 Yi-34B 31.6%
  25. 25 Qwen-7B 27.7%
  26. 26 Metamath-Llemma-7B 24.9%
  27. 27 MammoTH-70B 24.7%
  28. 28 DeepSeek-7B-Chat 22.3%
  29. 29 Baichuan2-13B-Chat 20.9%
  30. 30 ChatGLM3-6B 19.9%
  31. 31 MammoTH-13B 18.5%
  32. 32 Yi-6B-Chat 17.2%
  33. 33 MammoTH-7B 11.3%
#EntryScoreSettingDateSource
1 GPT-4o May 2024 70.9% May 2024 Official leaderboard
2 Qwen2-72B 67% Jun 2024 Official leaderboard
3 Claude 3 Opus 63% Feb 2024 Official leaderboard
4 DeepSeek-V2 (MoE-236B, May 2024) 59.9% May 2024 Official leaderboard
5 Qwen-Max-0428 59.2% Official leaderboard
6 GPT-4 Turbo (Jan 2024) 58.8% Jan 2024 Official leaderboard
7 Qwen1.5-110B 58.4% Apr 2024 Official leaderboard
8 Llama 3-70B 56.4% Apr 2024 Official leaderboard
9 Qwen2-7B-Instruct 53.4% Jun 2024 Official leaderboard
10 Yi-1.5-34B 52.2% May 2024 Official leaderboard
11 GLM-4 51.3% Jan 2024 Official leaderboard
12 Yi-1.5-9B-Chat 51.1% May 2024 Official leaderboard
13 DeepSeek-Math-7B-RL 50.3% Official leaderboard
14 Qwen-72B-Chat 49.7% Official leaderboard
15 GLM-4-9B-Chat 47.7% Jun 2024 Official leaderboard
16 InternLM2-Chat-20B 42.1% Official leaderboard
17 InternLM2-Chat-Math-20B 41.9% Official leaderboard
18 DeepSeekMath-7B-Instruct 40.9% Feb 2024 Official leaderboard
19 DeepSeek LLM 67B 40.3% Nov 2023 Official leaderboard
20 Qwen-14B 39.5% Sep 2023 Official leaderboard
21 InternLM2-Chat-Math-7B 37.7% Official leaderboard
22 Llama 3-8B 36.7% Apr 2024 Official leaderboard
23 InternLM2-Chat-7B 34.1% Official leaderboard
24 Yi-34B 31.6% Nov 2023 Official leaderboard
25 Qwen-7B 27.7% Sep 2023 Official leaderboard
26 Metamath-Llemma-7B 24.9% Official leaderboard
27 MammoTH-70B 24.7% Official leaderboard
28 DeepSeek-7B-Chat 22.3% Official leaderboard
29 Baichuan2-13B-Chat 20.9% Sep 2023 Official leaderboard
30 ChatGLM3-6B 19.9% Oct 2023 Official leaderboard
31 MammoTH-13B 18.5% Official leaderboard
32 Yi-6B-Chat 17.2% Nov 2023 Official leaderboard
33 MammoTH-7B 11.3% Official leaderboard

2 models have held number one since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard Not tested since Jun 2024
GPT-4o 70.9%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) GPT-4o Claude 3 Opus GPT-4 Turbo (Jan 2024) GLM-4 DeepSeek LLM 67B Baichuan2-13B-Chat 0% 20% 40% 60% 80% 100% 2024 Near the ceiling (85%) GPT-4o Claude 3 Opus

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Math

All 105 in Math
  • IMO-AnswerBench In use 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. GLM-5.2 91%
  • AIME 2026 In use Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. GPT-5.5 100%
  • HMMT 2026 In use Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. GPT-5.5 98.5%
  • MathArena (Apex, IMO, USAMO) In use Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. Claude Opus 4.8 81.3%
  • PolyMath In use Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. Qwen3.7-Max 86.5%
  • USAMO 2026 In use Models write full proofs for the six problems of the 2026 USA Mathematical Olympiad. Judges grade each proof from 0 to 7 against rubrics. GPT-5.5 98.2%