MGSM
Cogito v2 Preview 671B MoE 93% leads since Jul 2025 Fading 250 GSM8K problems translated into 10 languages. Final-answer accuracy.
Saturated.
In the authors’ words
We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at https://github.com/google-research/url-nlp.
Made by Google Research Released Oct 2022 Reasoning in many languages All tasks public Last in a launch post Sep 2026 , dropped after it saturated
Last model added Sep 2026
- Level
- School Students, from grade school to high school
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Short answer A number, word or short string matched to a key.
52
Models scored in lab cards
33
Launch posts list it, from 10 labs
36
System cards report it, from 11 labs
Cogito v2 Preview 671B MoE leads with 93% among models released by Sep 2026
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Sep 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- Claude Sonnet 5.5
- and 16 more
- 93.0 Cogito v2 Preview 671B MoE
- 92.7 Cogito v1 Preview 70B
- 92.5 Claude 3.5 Sonnet (October 2024)
- 92.3
Llama 4 Maverick - 92.0 o3-mini
- 91.6 Claude 3.5 Sonnet
- 91.6 Llama 3.1-405B
- 91.1 Llama 3.3 70B Instruct
- 90.7 Claude 3 Opus
- 90.6 Llama 4 Scout
- 90.5 GPT-4o
- 90.1
Cohere Command A - 89.6 GPT-4 Turbo
- 89.3 o1
- 88.7
Gemini 1.5 Pro - 88.2
Qwen2.5-72B - 87.5 Gemini 1.5 Pro
- 87.2 Qwen2.5-32B
- 87.1 Apple Foundation Model Server (2025)
- 87.0 PaLM 2-L
- 87.0 GPT-4o mini
- 86.9 Llama 3.1-70B
- 86.6 Qwen2-72B
- 85.6 Claude 3.5 Haiku
- 84.8 DiffusionGemma 26B A4B
- 84.4
DeepSeek-V4-Pro - 83.5 Claude 3 Sonnet
- 82.3 Qwen2.5-14B
- 80.6
Phi-4 - 80.2 DeepSeek V4.1 Flash
- 79.8 DeepSeek-V3
- 79.5 Aya Vision 32B
- 76.3
Falcon-H1-34B - 75.1 Claude 3 Haiku
- 74.3 Gemma 3 27B
- 74.0 DeepSeek LLM 67B
- 72.4 LFM2-8B-A1B
- 71.1 Granite 4.1 30B
- 68.9 Llama 3.1-8B
- 66.1 Qwen2.5-7B
- 64.3 Gemma 3 12B
- 64.3
Apertus 70B - 63.9 Phi-4 Mini
- 61.7 Qwen1.5-72B
- 60.7 Gemma 3n E4B
- 58.7 Phi-3.5-MoE
- 55.0 LFM2-1.2B
- 47.9 Phi-3.5-mini
- 38.7 Granite 4.0 H Small
- 37.8 granite-4.0-1b
- 34.7 Gemma 3 4B
- 2.04 Gemma 3 1B
- 1 Cogito v2 Preview 671B MoE 93%
- 2 Cogito v1 Preview 70B 92.7%
- 3 Claude 3.5 Sonnet (October 2024) 92.5%
- 4 Llama 4 Maverick 92.3%
- 5 o3-mini 92%
- 6 Claude 3.5 Sonnet 91.6%
- 7 Llama 3.1-405B 91.6%
- 8 Llama 3.3 70B Instruct 91.1%
- 9 Claude 3 Opus 90.7%
- 10 Llama 4 Scout 90.6%
- 11 GPT-4o May 2024 90.5%
- 12 Cohere Command A 90.1%
- 13 GPT-4 Turbo Apr 2024 89.6%
- 14 o1 89.3%
- 15 Gemini 1.5 Pro Feb 2024 88.7%
- 16 Qwen2.5-72B 88.2%
- 17 Gemini 1.5 Pro May 2024 87.5%
- 18 Qwen2.5-32B 87.2%
- 19 Apple Foundation Model Server (2025) 87.1%
- 20 PaLM 2-L 87%
- 21 GPT-4o mini 87%
- 22 Llama 3.1-70B 86.9%
- 23 Qwen2-72B 86.6%
- 24 Claude 3.5 Haiku 85.6%
- 25 DiffusionGemma 26B A4B 84.8%
- 26 DeepSeek-V4-Pro 84.4%
- 27 Claude 3 Sonnet 83.5%
- 28 Qwen2.5-14B 82.3%
- 29 Phi-4 80.6%
- 30 DeepSeek V4.1 Flash 80.2%
- 31 DeepSeek-V3 79.8%
- 32 Aya Vision 32B 79.5%
- 33 Falcon-H1-34B 76.3%
- 34 Claude 3 Haiku 75.1%
- 35 Gemma 3 27B 74.3%
- 36 DeepSeek LLM 67B 74%
- 37 LFM2-8B-A1B 72.4%
- 38 Granite 4.1 30B 71.1%
- 39 Llama 3.1-8B 68.9%
- 40 Qwen2.5-7B 66.1%
- 41 Gemma 3 12B 64.3%
- 42 Apertus 70B 64.3%
- 43 Phi-4 Mini 63.9%
- 44 Qwen1.5-72B 61.7%
- 45 Gemma 3n E4B 60.7%
- 46 Phi-3.5-MoE 58.7%
- 47 LFM2-1.2B 55%
- 48 Phi-3.5-mini 47.9%
- 49 Granite 4.0 H Small 38.7%
- 50 granite-4.0-1b 37.8%
- 51 Gemma 3 4B 34.7%
- 52 Gemma 3 1B 2.04%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Cogito v2 Preview 671B MoE | 93% | Jul 2025 | Model card | |
| 2 | Cogito v1 Preview 70B | 92.7% | Apr 2025 | Model card | |
| 3 | Claude 3.5 Sonnet (October 2024) | 92.5% | Oct 2024 | Model card | |
| 4 | Llama 4 Maverick | 92.3% | Apr 2025 | Model card | |
| 5 | o3-mini | 92% | Jan 2025 | Model card | |
| 6 | Claude 3.5 Sonnet | 91.6% | Jun 2024 | Model card | |
| 7 | Llama 3.1-405B | 91.6% | Jul 2024 | Model card | |
| 8 | Llama 3.3 70B Instruct | 91.1% | Dec 2024 | Model card | |
| 9 | Claude 3 Opus | 90.7% | Mar 2024 | Model card | |
| 10 | Llama 4 Scout | 90.6% | Apr 2025 | Model card | |
| 11 | GPT-4o May 2024 | 90.5% | May 2024 | Model card | |
| 12 | Cohere Command A | 90.1% | Apr 2025 | Model card | |
| 13 | GPT-4 Turbo Apr 2024 | 89.6% | Apr 2024 | Model card | |
| 14 | o1 | 89.3% | Dec 2024 | Model card | |
| 15 | Gemini 1.5 Pro Feb 2024 | 88.7% | Feb 2024 | Model card | |
| 16 | Qwen2.5-72B | 88.2% | Sep 2024 | Model card | |
| 17 | Gemini 1.5 Pro May 2024 | 87.5% | May 2024 | Model card | |
| 18 | Qwen2.5-32B | 87.2% | Dec 2024 | Model card | |
| 19 | Apple Foundation Model Server (2025) | 87.1% | Jul 2025 | Model card | |
| 20 | PaLM 2-L | 87% | May 2023 | Model card | |
| 21 | GPT-4o mini | 87% | Jul 2024 | Model card | |
| 22 | Llama 3.1-70B | 86.9% | Jul 2024 | Model card | |
| 23 | Qwen2-72B | 86.6% | Jun 2024 | Model card | |
| 24 | Claude 3.5 Haiku | 85.6% | Oct 2024 | Model card | |
| 25 | DiffusionGemma 26B A4B | 84.8% | Jul 2026 | Model card | |
| 26 | DeepSeek-V4-Pro | 84.4% | Apr 2026 | Model card | |
| 27 | Claude 3 Sonnet | 83.5% | Mar 2024 | Model card | |
| 28 | Qwen2.5-14B | 82.3% | Dec 2024 | Model card | |
| 29 | Phi-4 | 80.6% | Dec 2024 | Model card | |
| 30 | DeepSeek V4.1 Flash | 80.2% | Sep 2026 | Model card | |
| 31 | DeepSeek-V3 | 79.8% | Dec 2024 | Model card | |
| 32 | Aya Vision 32B | 79.5% | May 2025 | Model card | |
| 33 | Falcon-H1-34B | 76.3% | Jul 2025 | Model card | |
| 34 | Claude 3 Haiku | 75.1% | Mar 2024 | Model card | |
| 35 | Gemma 3 27B | 74.3% | Mar 2025 | Model card | |
| 36 | DeepSeek LLM 67B | 74% | Jan 2024 | Model card | |
| 37 | LFM2-8B-A1B | 72.4% | Oct 2025 | Model card | |
| 38 | Granite 4.1 30B | 71.1% | Apr 2026 | Model card | |
| 39 | Llama 3.1-8B | 68.9% | Jul 2024 | Model card | |
| 40 | Qwen2.5-7B | 66.1% | Dec 2024 | Model card | |
| 41 | Gemma 3 12B | 64.3% | Mar 2025 | Model card | |
| 42 | Apertus 70B | 64.3% | Sep 2025 | Model card | |
| 43 | Phi-4 Mini | 63.9% | Feb 2025 | Model card | |
| 44 | Qwen1.5-72B | 61.7% | Feb 2024 | Model card | |
| 45 | Gemma 3n E4B | 60.7% | Jun 2025 | Model card | |
| 46 | Phi-3.5-MoE | 58.7% | Aug 2024 | Model card | |
| 47 | LFM2-1.2B | 55% | Jul 2025 | Model card | |
| 48 | Phi-3.5-mini | 47.9% | Aug 2024 | Model card | |
| 49 | Granite 4.0 H Small | 38.7% | Oct 2025 | Model card | |
| 50 | granite-4.0-1b | 37.8% | Oct 2025 | Model card | |
| 51 | Gemma 3 4B | 34.7% | Mar 2025 | Model card | |
| 52 | Gemma 3 1B | 2.04% | Mar 2025 | Model card |
7 models have held number one since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Other labs Google DeepMind Anthropic Deep Cogito
The best score passed 85% in May 2023
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Reported by the lab in its model card
Best so far
Labs headline it less often now, last in Sep 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 10 Sep 2026 | DeepSeek V4.1 Flash | 80.2 | MGSM (EM) |
| 29 Apr 2026 | Granite 4.1 30B | 71.1 | MGSM (8-shot, 5 languages) |
| 24 Apr 2026 | DeepSeek-V4-Pro | 84.4 | MGSM (EM) |
| 28 Oct 2025 | granite-4.0-1b | 37.8 | MGSM (8-shot, 5 languages) |
| 7 Oct 2025 | LFM2-8B-A1B | 72.4 | MGSM |
| 2 Oct 2025 | Granite 4.0 H Small | 38.7 | MGSM (8-shot, 5 languages) |
| 31 Jul 2025 | Cogito v2 Preview 671B MoE | 93 | MGSM (thinking) |
| 10 Jul 2025 | LFM2-1.2B | 55 | MGSM |
| 26 Jun 2025 | Gemma 3n E4B | 60.7 | MGSM 0-shot (IT) |
| 8 Apr 2025 | Cogito v1 Preview 70B | 92.7 | MGSM (thinking) |
| 5 Apr 2025 | Llama 4 Scout | 90.6 | MGSM, average/em |
| 5 Apr 2025 | Llama 4 Maverick | 92.3 | MGSM, average/em |
| 12 Mar 2025 | Gemma 3 27B | 74.3 | MGSM (PT) |
| 26 Feb 2025 | Phi-4 Mini | 63.9 | MGSM (0-shot, CoT) |
| 31 Jan 2025 | o3-mini | 92 | MGSM (pass@1) |
| 26 Dec 2024 | DeepSeek-V3 | 79.8 | MGSM (EM) |
| 17 Dec 2024 | o1 | 89.3 | MGSM (pass @1) |
| 12 Dec 2024 | Phi-4 | 80.6 | MGSM (simple-evals) |
| 6 Dec 2024 | Llama 3.3 70B Instruct | 91.1 | MGSM, 0-shot, em |
| 22 Oct 2024 | Claude 3.5 Sonnet (October 2024) | 92.5 | MGSM (0-shot CoT) |
| 19 Sep 2024 | Qwen2.5-72B | 88.2 | MGSM8K (extended) |
| 20 Aug 2024 | Phi-3.5-MoE | 58.7 | MGSM (0-shot CoT) |
| 23 Jul 2024 | Llama 3.1-405B | 91.6 | MGSM (0-shot, CoT) |
| 18 Jul 2024 | GPT-4o mini | 87 | MGSM |
| 20 Jun 2024 | Claude 3.5 Sonnet | 91.6 | MGSM (0-shot CoT) |
| 7 Jun 2024 | Qwen2-72B | 86.6 | MGSM (0-shot, CoT) |
| 14 May 2024 | Gemini 1.5 Pro | 87.5 | MGSM, 8-shot |
| 13 May 2024 | GPT-4o | 90.5 | MGSM (0-shot CoT) |
| 11 Apr 2024 | GPT-4 Turbo | 89.6 | MGSM% |
| 4 Mar 2024 | Claude 3 Opus | 90.7 | MGSM (0-shot) |
| 15 Feb 2024 | Gemini 1.5 Pro | 88.7 | MGSM, 8-shot |
| 5 Feb 2024 | Qwen1.5-72B | 61.7 | Multilingual Math (MGSM, 8-shot) |
| 10 May 2023 | PaLM 2-L | 87 | MGSM, CoT + self-consistency (greedy CoT = 72.2) |
36 system cards report it, from 11 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 31 Jul 2026 | DiffusionGemma 26B A4B technical report | 84.8 | MGSM, TD, 80.7 no-think (Table 3) | Capabilities |
| 28 Nov 2025 | LFM2-8B-A1B technical report | 72.4 | MGSM, LFM2-8B-A1B (Table 7) | Capabilities |
| 28 Nov 2025 | LFM2-1.2B technical report | 55 | MGSM, LFM2-1.2B (Table 6) | Capabilities |
| 17 Sep 2025 | Apertus 70B technical report | 64.3 | MGSM, 0-shot CoT (Table 18) | Capabilities |
| 30 Jul 2025 | Falcon-H1-34B technical report | 76.3 | MGSM, flexible extract 8-shot native CoT, 6 languages (Tables 24 and 35) | Capabilities |
| 17 Jul 2025 | Apple Foundation Model Server (2025) technical report | 87.1 | MGSM, AFM-supported languages only, Simple-Evals (Table 2) | Capabilities |
| 13 May 2025 | Aya Vision 32B technical report | 79.5 | MGSM, Aya Vision languages (Table 5) | Capabilities |
| 5 Apr 2025 | Llama 4 Scout model card | 90.6 | MGSM, 0-shot, average/em, instruction tuned (Instruction tuned models table) | Capabilities |
| 5 Apr 2025 | Llama 4 Maverick model card | 92.3 | MGSM, 0-shot, average/em, instruction tuned (Instruction tuned models table) | Capabilities |
| 1 Apr 2025 | Cohere Command A technical report | 90.1 | MGSM average over 6 languages, simple-evals (Table 24) | Capabilities |
| 25 Mar 2025 | Gemma 3 4B technical report | 34.7 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 27B technical report | 74.3 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 1B technical report | 2.04 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 12B technical report | 64.3 | MGSM, PT, 8-shot (Table 13) | Capabilities |
| 3 Mar 2025 | Phi-4 Mini technical report | 63.9 | MGSM (0-Shot, CoT) (Table 7) | Capabilities |
| 19 Dec 2024 | Qwen2.5-7B technical report | 66.1 | MGSM8K (extended), Qwen2.5-7B-Instruct (Table 14) | Capabilities |
| 19 Dec 2024 | Qwen2.5-72B technical report | 88.2 | MGSM8K (extended), Qwen2.5-72B-Instruct (Table 13) | Capabilities |
| 19 Dec 2024 | Qwen2.5-32B technical report | 87.2 | MGSM8K (extended), Qwen2.5-32B-Instruct (Table 13) | Capabilities |
| 19 Dec 2024 | Qwen2.5-14B technical report | 82.3 | MGSM8K (extended), Qwen2.5-14B-Instruct (Table 14) | Capabilities |
| 12 Dec 2024 | Phi-4 technical report | 80.6 | MGSM, simple-evals (Table 1) | Capabilities |
| 6 Dec 2024 | Llama 3.3 70B Instruct model card | 91.1 | MGSM, 0-shot, em (Instruction tuned models table) | Capabilities |
| 23 Nov 2024 | Llama 3.1-8B technical report | 68.9 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| 23 Nov 2024 | Llama 3.1-70B technical report | 86.9 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| 23 Nov 2024 | Llama 3.1-405B technical report | 91.6 | MGSM (0-shot, CoT), post-trained (Table 2, Table 20) | Capabilities |
| 22 Oct 2024 | Claude 3.5 Sonnet (October 2024) model card | 92.5 | MGSM, 0-shot CoT (Table 7) | Capabilities |
| 22 Oct 2024 | Claude 3.5 Haiku model card | 85.6 | MGSM, 0-shot CoT (Table 8) | Capabilities |
| 30 Aug 2024 | Phi-3.5-MoE technical report | 58.7 | MGSM (0-shot CoT) (Table 3) | Capabilities |
| 30 Aug 2024 | Phi-3.5-mini technical report | 47.9 | MGSM (0-shot CoT) (Table 3) | Capabilities |
| 23 Jul 2024 | Llama 3.1-8B model card | 68.9 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| 23 Jul 2024 | Llama 3.1-70B model card | 86.9 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| 23 Jul 2024 | Llama 3.1-405B model card | 91.6 | Multilingual MGSM (CoT), 0-shot, em, instruction tuned (Instruction tuned models table) | Capabilities |
| 20 Jun 2024 | Claude 3.5 Sonnet model card | 91.6 | MGSM, 0-shot CoT (Table 1) | Capabilities |
| 4 Mar 2024 | Claude 3 Sonnet model card | 83.5 | MGSM, 0-shot. 8-shot 83.7 (Table 1, Table 4) | Capabilities |
| 4 Mar 2024 | Claude 3 Opus model card | 90.7 | MGSM, 0-shot. 8-shot 90.5 (Table 1, Table 4) | Capabilities |
| 4 Mar 2024 | Claude 3 Haiku model card | 75.1 | MGSM, 0-shot. 8-shot 76.5 (Table 1, Table 4) | Capabilities |
| 5 Jan 2024 | DeepSeek LLM 67B technical report | 74 | MGSM-zh (Chinese subset only), chain of thought, DeepSeek LLM 67B Chat. 76.4 with tool-integrated reasoning (Table 16) | Capabilities |
Versions
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5% - MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.
Inkling 88.7% - MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%