Qwen-14B.
Qwen Sep 2023 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ARC-Challenge | 84.4% | #14 of 75 | Epoch AI | |
| BoolQ | 86.2% | #18 of 75 | Epoch AI | |
| Qwen Chinese human evaluation | 30.7% | #1 of 2 | Model card | |
| CriticEval | 5.35 | #5 of 8 | Official leaderboard | |
| PIQA | 79.9% | #37 of 58 | Epoch AI | |
| EQ-Bench v2 | 63.5 | #61 of 85 | Official leaderboard | |
| LAMBADA | 71.1% | #21 of 26 | Epoch AI | |
| MMLU | 66.5% | #104 of 127 | Model card | |
| BIG-Bench Hard | 58.7% | #59 of 71 | Model card | |
| C-Eval | 71.7% | #33 of 38 | Model card | |
| CMMLU | 71% | #15 of 17 | Model card | |
| LMArena (Elo) | 1,139 | #310 of 333 | Official leaderboard | |
| VCSUM | 17.3 | #1 of 1 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEvalPack | 33.7% | #2 of 3 | Model card | |
| HumanEval | 43.9% | #82 of 95 | Model card | |
| MBPP | 46.4% | #58 of 66 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese Tool-Use Benchmark | 98% | #1 of 4 | Model card | |
| Code Interpreter Benchmark | 81.7% | #1 of 4 | Model card | |
| HuggingFace Agent Benchmark | 93.5% | #1 of 3 | Model card |
Qwen's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.