Qwen Technical Report (arXiv 2309.16609) 28 Sep 2023
Capabilities
- MMLU 43.9
- C-Eval 50.7
- GSM8K 27.8
- BIG-Bench Hard 27.1
- HumanEval 14.6
- Chinese Tool-Use Benchmark 92
- Code Interpreter Benchmark 26.8
- HuggingFace Agent Benchmark 85.2
Qwen-1_8B
Qwen-1_8B.
Released 30 Nov 2023
13
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Unknown lab Nov 2023
Qwen-1_8B
92.8
#260 of 271
Epoch Capabilities Index
- MMLU 43.9% #124 /127
- C-Eval 50.7% #38 /38
- BIG-Bench Hard 27.1% #70 /71
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ARC-Challenge | 53.2% | #42 of 75 | Epoch AI | |
| BoolQ | 68% | #58 of 75 | Epoch AI | |
| PIQA | 73.3% | #54 of 58 | Epoch AI | |
| EQ-Bench v2 | 30.0 | #81 of 85 | Official leaderboard | |
| LAMBADA | 58.4% | #25 of 26 | Epoch AI | |
| MMLU | 43.9% | #124 of 127 | Model card | |
| BIG-Bench Hard | 27.1% | #70 of 71 | Model card | |
| C-Eval | 50.7% | #38 of 38 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 27.8% | #85 of 87 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 14.6% | #94 of 95 | Model card |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese Tool-Use Benchmark | 92% | #4 of 4 | Model card | |
| Code Interpreter Benchmark | 26.8% | #4 of 4 | Model card | |
| HuggingFace Agent Benchmark | 85.2% | #3 of 3 | Model card |
The lab's models over time
's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its system card
Every result in the longer documents the lab published with it. 8 of them aren't in the launch post.