- MMLU-Pro 71.1
- MMLU-redux 86.8
- GPQA 49
- MATH 83.1
- GSM8K 95.8
- Best of HumanEval / HumanEval+ 86.6
- Best of MBPP / MBPP+ 88.2
- MultiPL-E 75.1
- LiveCodeBench 2305-2409 55.5
- LiveBench 0831 52.3
- IFeval strict-prompt 84.1
- Arena-Hard 81.2
- AlignBench v1.1 8.16
- MTbench 9.35
- IFEval (multilingual) 87
- AMMLU (Arabic) 72.4
- JMMLU (Japanese) 80.6
- KMMLU (Korean) 62
- IndoMMLU (Indonesian) 69.3
- TurkishMMLU (Turkish) 76.1
- okapi MMLU (translated) 80
- MGSM8K (extended) 88.2
- BLEnD 32.5
- MMLU 86.1
- BBH 86.3
- ARC-C 72.4
- TruthfulQA 60.4
- WindoGrande 83.9
- HellaSwag 87.6
- Theoremqa 42.4
- MMLU-stem 82.7
- Multi-Exam 78.7
- Multi-Understanding 89.6
- Multi-Translation 39
Qwen2.5-72B
Qwen2.5-72B ranks first on 5 of the 92 benchmarks it has scores on.
101
Benchmarks with a score
5
Ranked first, out of five or more models
12
In the top three
Qwen Sep 2024 Qwen2.5-72B
Open weights Flagship
128.9
#166 of 271
Epoch Capabilities Index
- GPQA Diamond 49.1% #147 /219
- OSWorld 5.00% #9 /9
- KOR-Bench 52.2% #9 /33
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ArabicMMLU | 71.4% | #1 of 39 | Official leaderboard | |
| ArbMMLU-HT | 72.2% | #1 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 72.4% | #1 of 39 | Official leaderboard | |
| MT-Bench | 9.35 | #1 of 23 | Model card | |
| AlGhafa | 78.2% | #2 of 39 | Official leaderboard | |
| MadinahQA | 65% | #2 of 39 | Official leaderboard | |
| MMLU-CF | 71.6% | #2 of 37 | Official leaderboard | |
| Okapi MMLU | 80% | #1 of 8 | Model card | |
| RULER | 95.1% | #5 of 38 | Model card | |
| WinoGrande | 83.9% | #9 of 42 | Model card | |
| AMMLU | 72.4% | #1 of 4 | Model card | |
| IndoMMLU | 69.3% | #1 of 4 | Model card | |
| JMMLU | 80.6% | #1 of 4 | Model card | |
| Multilingual IFEval | 87% | #1 of 4 | Model card | |
| TurkishMMLU | 76.1% | #1 of 4 | Model card | |
| Chinese SimpleQA | 50.2% | #7 of 27 | Official leaderboard | |
| BIG-Bench Hard | 86.3% | #19 of 71 | Model card | |
| ALRAGE | 78.8% | #11 of 39 | Official leaderboard | |
| HellaSwag | 87.6% | #14 of 47 | Model card | |
| MGSM | 88.2% | #16 of 52 | Model card | |
| PIQA | 82.6% | #18 of 58 | Epoch AI | |
| EQ-Bench v2 | 79.0 | #27 of 85 | Official leaderboard | |
| LongBench-Chat | 8.72 | #2 of 6 | Model card | |
| Multilingual Exams | 78.7% | #1 of 3 | Model card | |
| Multilingual Understanding | 89.6% | #1 of 3 | Model card | |
| MMLU | 86.1% | #47 of 127 | Model card | |
| BLEnD | 32.5% | #2 of 5 | Model card | |
| Arena-Hard-Auto | 81.2% | #18 of 39 | Model card | |
| TruthfulQA | 60.4% | #14 of 28 | Model card | |
| Flores | 39.0 | #4 of 8 | Model card | |
| KMMLU | 62% | #3 of 6 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.30% | #76 of 148 | Official leaderboard | |
| ARC-Challenge | 72.4% | #25 of 45 | Model card | |
| SuperGPQA | 40.8% | #34 of 58 | Official leaderboard | |
| AlignBench | 8.16 | #9 of 15 | Model card | |
| LLM Confabulation Leaderboard | 19.1% | #36 of 58 | Official leaderboard | |
| FaithJudge | 20.7% | #23 of 37 | Official leaderboard | |
| LongBench v2 | 43.5% | #21 of 33 | Official leaderboard | |
| HELM Arabic | 76.9 | #18 of 28 | Official leaderboard | |
| IFEval | 84.1% | #82 of 126 | Model card | |
| BuzzBench | 45.9 | #15 of 23 | Official leaderboard | |
| LLM-AggreFact | 75.6% | #4 of 6 | Official leaderboard | |
| LMArena (Elo) | 1,303 | #228 of 333 | Official leaderboard | |
| MMLU-Redux | 86.8% | #27 of 38 | Model card | |
| TriviaQA | 71.9% | #27 of 38 | Epoch AI | |
| MMLU-Pro | 71.1% | #110 of 151 | Model card | |
| WritingBench | 57.8 | #33 of 45 | Official leaderboard | |
| EQ-Bench 3 | 896 | #68 of 78 | Official leaderboard | |
| Arena-Hard v2.0 | 10.1% | #23 of 23 | Official leaderboard | |
| MultiLoKo | 16.4% | #7 of 7 | Official leaderboard | |
| MMLU-stem | 82.7% | #2 of 2 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 95.8% | #9 of 87 | Model card | |
| MATH | 83.1% | #15 of 93 | Model card | |
| KOR-Bench | 52.2% | #9 of 33 | Official leaderboard | |
| ForecastBench | 61.5 | #29 of 86 | Official leaderboard | |
| ZebraLogic | 26.6% | #16 of 46 | Official leaderboard | |
| MATH Level 5 | 63.2% | #41 of 94 | Epoch AI | |
| U-MATH | 51.2% | #15 of 30 | Official leaderboard | |
| GPQA Diamond | 49.1% | #147 of 219 | Epoch AI | |
| DTBench | 38.2% | #128 of 178 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 8.06% | #155 of 202 | Epoch AI | |
| PolyMath | 16.9% | #36 of 38 | Official leaderboard | |
| TheoremQA | 42.4% | #2 of 2 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 88.2% | #4 of 66 | Model card | |
| BigCodeBench | 50.8% | #18 of 69 | Official leaderboard | |
| HumanEval | 86.6% | #33 of 95 | Model card | |
| BigCodeBench-Hard | 25.4% | #45 of 96 | Official leaderboard | |
| LiveCodeBench | 55.5% | #37 of 73 | Model card | |
| MultiPL-E | 75.1% | #10 of 18 | Model card | |
| AutoCodeBench | 34.3% | #26 of 34 | Official leaderboard | |
| METR Time Horizon | 5.8 min | #27 of 31 | Official leaderboard | |
| Multi-SWE-bench | 2.57% | #10 of 11 | Official leaderboard | |
| WeirdML | 16% | #105 of 112 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Galileo Agent Leaderboard v2 | 51% | #7 of 21 | Official leaderboard | |
| AceBench | 74.7% | #6 of 16 | Official leaderboard | |
| AgentBench FC | 40.8% | #7 of 12 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.82 | #22 of 31 | Official leaderboard | |
| Step Race | 1 | #45 of 59 | Official leaderboard | |
| BALROG | 16.2% | #28 of 35 | Official leaderboard | |
| SnakeBench | 18 | #150 of 183 | Official leaderboard | |
| LLM Chess | -672 | #118 of 124 | Official leaderboard | |
| OSWorld | 5.00% | #9 of 9 | Epoch AI |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 79.4% | #91 of 138 | Official leaderboard | |
| TheAgentCompany | 5.70% | #11 of 14 | Epoch AI |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Video-MME | 73.5% | #4 of 50 | Epoch AI | |
| GeoBench | 62% | #18 of 27 | Epoch AI | |
| SpatialViz-Bench | 33.3% | #6 of 8 | Epoch AI |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 89.2% | #2 of 39 | Official leaderboard | |
| SimpleSafetyTests | 100% | #6 of 75 | Official leaderboard | |
| Anthropic Red Team | 99.6% | #16 of 75 | Official leaderboard | |
| HELM Safety | 93.1 | #31 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 59% | #53 of 82 | Official leaderboard | |
| BaxBench | 10.4% | #27 of 34 | Official leaderboard | |
| FORTRESS | 56.4% | #59 of 66 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 74.5 | #18 of 71 | Official leaderboard | |
| Political Science LLM Benchmark | 64.8% | #19 of 29 | Official leaderboard | |
| LiveBench | 52.3% | #18 of 27 | Model card | |
| HELM Capabilities | 59.9 | #42 of 60 | Official leaderboard | |
| ACCoRD | 42.5 | #126 of 150 | Official leaderboard | |
| Conceptual Reasoning Index | 24.2 | #127 of 149 | Official leaderboard | |
| LMCA | 13.4% | #121 of 139 | Epoch AI |
Qwen's models over time
Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
Qwen2.5 Technical Report (arXiv 2412.15115) 19 Dec 2024
Capabilities
- MMLU-Pro 71.1
- MMLU-Redux 86.8
- LiveBench 52.3
- GPQA Diamond 49
- MATH 83.1
- GSM8K 95.8
- HumanEval 86.6
- MBPP 88.2
- MultiPL-E 75.1
- LiveCodeBench 55.5
- IFEval 84.1
- Arena-Hard-Auto 81.2
- MT-Bench 9.35
- Multilingual IFEval 87
- AMMLU 72.4
- JMMLU 80.6
- KMMLU 62
- IndoMMLU 69.3
- TurkishMMLU 76.1
- Okapi MMLU 80
- MGSM 88.2
- BLEnD 32.5
- Qwen in-house English automatic evaluation
- Qwen in-house Chinese automatic evaluation
- RULER 95.1
- LV-Eval
- LongBench-Chat 8.72