Benchmark Almanac

Qwen2.5-14B-Instruct-1M

Qwen2.5-14B-Instruct-1M ranks first on LongBench-Chat.

Qwen Released 27 Jan 2025 Family Qwen2.5-1M The lab's model card
17
Benchmarks with a score
1
Ranked first, out of five or more models
1
In the top three
Qwen Jan 2025

Qwen2.5-14B-Instruct-1M

Open weights Small

No Epoch Capabilities Index score yet
  • MMLU-Pro 63.3% #128 /151
  • IFEval 84.3% #80 /126
  • LiveCodeBench 38.6% #51 /73

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
RULER 95.7% #4 of 38 Model card
LongBench-Chat 8.76 #1 of 6 Model card
MT-Bench 8.89 #9 of 23 Model card
IFEval 84.3% #80 of 126 Model card
Arena-Hard-Auto 70.2% #27 of 39 Model card
MMLU-Pro 63.3% #128 of 151 Model card
MMLU-Redux 80.7% #34 of 38 Model card
Passkey Retrieval 100% #2 of 2 Model card
LV-Eval 49.8% #1 of 1 Model card

Math and science

BenchmarkScoreRankSource
GSM8K 94.8% #16 of 87 Model card
MATH 79.5% #24 of 93 Model card
GPQA Diamond 39.9% #259 of 284 Model card

Coding

BenchmarkScoreRankSource
HumanEval 88.4% #23 of 95 Model card
MBPP 80.2% #19 of 66 Model card
MultiPL-E 77.1% #6 of 18 Model card
LiveCodeBench 38.6% #51 of 73 Model card

Other

BenchmarkScoreRankSource
LiveBench 44.6% #23 of 27 Model card

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Its system card

Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.