Qwen-Turbo places in the top three on Needle In A Haystack.
Qwen Released 1 Nov 2024 Qwen Nov 2024 Closed weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Needle In A Haystack | 100% | #2 of 9 | Model card | |
| RULER | 93.1% | #12 of 38 | Model card | |
| MT-Bench | 8.81 | #11 of 23 | Model card | |
| LongBench-Chat | 8.34 | #4 of 6 | Model card | |
| Arena-Hard-Auto | 67.1% | #29 of 39 | Model card | |
| MMLU-Pro | 64.5% | #124 of 151 | Model card | |
| IFEval | 76.3% | #107 of 126 | Model card | |
| MMLU-Redux | 81.7% | #33 of 38 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH | 81.1% | #19 of 93 | Model card | |
| GSM8K | 93.8% | #21 of 87 | Model card | |
| MATH Level 5 | 56.2% | #45 of 94 | Epoch AI | |
| GPQA Diamond | 41.8% | #171 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 6.11% | #169 of 202 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 82.8% | #13 of 66 | Model card | |
| HumanEval | 86.6% | #34 of 95 | Model card | |
| MultiPL-E | 73.7% | #11 of 18 | Model card | |
| LiveCodeBench | 37.8% | #53 of 73 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SnakeBench | 26 | #57 of 183 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LiveBench | 42.3% | #25 of 27 | Model card |
Every result in the longer documents the lab published with it. 19 of them aren't in the launch post.