Qwen2-72B ranks first on 2 of the 57 benchmarks it has scores on.
Qwen Jun 2024 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AlGhafa | 78.6% | #1 of 39 | Official leaderboard | |
| ArbMMLU-HT | 68.1% | #2 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 70.1% | #3 of 39 | Official leaderboard | |
| MadinahQA | 61.4% | #4 of 39 | Official leaderboard | |
| ArabicMMLU | 66.5% | #6 of 39 | Official leaderboard | |
| WinoGrande | 85.1% | #7 of 42 | Model card | |
| AlpacaEval 2.0 | 38.1% | #8 of 40 | Official leaderboard | |
| ThaiExam | 63.8% | #5 of 24 | Official leaderboard | |
| EQ-Bench v2 | 81.3 | #18 of 85 | Official leaderboard | |
| MT-Bench | 9.12 | #5 of 23 | Model card | |
| HellaSwag | 87.6% | #13 of 47 | Model card | |
| CMMLU | 90.1% | #5 of 17 | Model card | |
| MMLU-CF | 63.7% | #14 of 37 | Official leaderboard | |
| WildBench | 46.4 | #18 of 47 | Official leaderboard | |
| BIG-Bench Hard | 82.4% | #29 of 71 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.70% | #63 of 148 | Official leaderboard | |
| ToRR | 45.4 | #6 of 14 | Official leaderboard | |
| MGSM | 86.6% | #23 of 52 | Model card | |
| ALRAGE | 73.4% | #18 of 39 | Official leaderboard | |
| MMLU | 82.3% | #60 of 127 | Model card | |
| MixEval | 86.7% | #1 of 2 | Model card | |
| AlignBench | 8.27 | #8 of 15 | Model card | |
| TruthfulQA | 54.8% | #16 of 28 | Model card | |
| ARC-Challenge | 68.9% | #28 of 45 | Model card | |
| Flores | 37.8 | #5 of 8 | Model card | |
| Multilingual Exams | 76.6% | #2 of 3 | Model card | |
| Multilingual Understanding | 80.7% | #2 of 3 | Model card | |
| MMMLU | 78% | #45 of 62 | Model card | |
| C-Eval | 83.8% | #28 of 38 | Model card | |
| LMArena (Elo) | 1,262 | #255 of 333 | Official leaderboard | |
| IFEval | 77.6% | #104 of 126 | Model card | |
| MMLU-Pro | 64.4% | #125 of 151 | Model card | |
| Arena-Hard-Auto | 48.1% | #34 of 39 | Model card | |
| Multilingual human evaluation | 3.93 | #1 of 1 | Model card | |
| Qwen in-house Chinese automatic evaluation | 69.6% | #1 of 1 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MathBench | 67% | #2 of 21 | Official leaderboard | |
| OlympicArena | 34.1% | #5 of 25 | Official leaderboard | |
| GSM8K | 93.2% | #23 of 87 | Model card | |
| ZebraLogic | 21.4% | #19 of 46 | Official leaderboard | |
| KOR-Bench | 47% | #15 of 33 | Official leaderboard | |
| MATH | 69% | #45 of 93 | Model card | |
| TheoremQA | 44.4% | #1 of 2 | Model card | |
| MATH Level 5 | 39.1% | #64 of 94 | Epoch AI | |
| GPQA Diamond | 40.8% | #173 of 219 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 80.2% | #17 of 66 | Model card | |
| HumanEval | 86% | #36 of 95 | Model card | |
| BigCodeBench | 46.3% | #31 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 20.6% | #56 of 96 | Official leaderboard | |
| EvalPlus | 79% | #4 of 6 | Model card | |
| Aider-Edit | 55.6% | #26 of 37 | Official leaderboard | |
| SciCode | 1.50% | #13 of 18 | Official leaderboard | |
| LiveCodeBench | 35.7% | #54 of 73 | Model card | |
| MultiPL-E | 69.2% | #14 of 18 | Model card | |
| METR Time Horizon | 2.5 min | #28 of 31 | Official leaderboard | |
| WeirdML | 11.3% | #123 of 130 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SuperCLUE-Agent | 74.0 | #2 of 7 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TheAgentCompany | 1.10% | #14 of 14 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 89.2% | #1 of 39 | Official leaderboard | |
| HELM Safety | 93.3 | #29 of 75 | Official leaderboard | |
| Anthropic Red Team | 99.1% | #35 of 75 | Official leaderboard | |
| SimpleSafetyTests | 98.5% | #43 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 62.1% | #50 of 82 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 77.0 | #17 of 71 | Official leaderboard | |
| Qwen in-house English automatic evaluation | 72.9% | #1 of 1 | Model card |
Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.