DeepSeek-V2 model card 7 May 2024
- MMLU 77.8
- BBH 79.7
- C-Eval 78
- CMMLU 81.6
- HumanEval 81.1
- MBPP 72
- LiveCodeBench (0901-0401) 32.5
- GSM8K 92.2
- Math 53.9
- Needle In A Haystack (NIAH)
- AlpacaEval 2.0
- MTBench
- Alignbench (总分) 7.91
DeepSeek-V2 (MoE-236B, May 2024)
DeepSeek-V2 (MoE-236B, May 2024) ranks first on Natural Questions.
30
Benchmarks with a score
1
Ranked first, out of five or more models
2
In the top three
DeepSeek May 2024 DeepSeek-V2 (MoE-236B, May 2024)
Open weights Flagship
124.7
#184 of 271
Epoch Capabilities Index
- MathBench 59.9% #4 /21
- Natural Questions 53.4% #1 /20
- IFEval 63.8% #120 /126
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Natural Questions | 53.4% | #1 of 20 | Model card | |
| WinoGrande | 86.3% | #5 of 76 | Epoch AI | |
| HellaSwag | 87.1% | #5 of 74 | Epoch AI | |
| ARC-Easy | 98.1% | #2 of 23 | Model card | |
| PIQA | 83.9% | #8 of 58 | Epoch AI | |
| TriviaQA | 86.7% | #7 of 39 | Model card | |
| WildBench | 50.0 | #9 of 47 | Official leaderboard | |
| ARC-Challenge | 92.3% | #11 of 45 | Model card | |
| RepoQA | 83.4% | #6 of 22 | Official leaderboard | |
| MT-Bench | 8.97 | #7 of 23 | Model card | |
| MixEval | 83.7% | #12 of 34 | Official leaderboard | |
| AGIEval | 61.4% | #15 of 35 | Model card | |
| BIG-Bench Hard | 79.7% | #35 of 71 | Model card | |
| CLUEWSC | 89.9% | #4 of 8 | Model card | |
| MMLU-Pro | 54.8% | #127 of 212 | Official leaderboard | |
| MMLU | 77.8% | #82 of 127 | Model card | |
| AlignBench | 7.91 | #11 of 15 | Model card | |
| CMMLU | 81.6% | #13 of 17 | Model card | |
| C-Eval | 78% | #30 of 38 | Model card | |
| AlpacaEval 2.0 | 38.9% | #14 of 16 | Model card | |
| IFEval | 63.8% | #120 of 126 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MathBench | 59.9% | #4 of 21 | Official leaderboard | |
| GSM8K | 92.2% | #29 of 87 | Model card | |
| MATH | 53.9% | #57 of 93 | Model card | |
| CMATH | 81.9% | #4 of 6 | Model card | |
| SuperCLUE-Math6 | 83.3 | #1 of 1 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 81.1% | #50 of 95 | Model card | |
| CRUXEval | 61.5% | #5 of 9 | Model card | |
| MBPP | 72% | #37 of 66 | Model card | |
| LiveCodeBench | 32.5% | #60 of 73 | Model card |
DeepSeek's models over time
DeepSeek's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 10 of them aren't in the launch post.
Capabilities
- TriviaQA 86.7
- Natural Questions 53.4
- MMLU 77.8
- ARC-Easy 98.1
- ARC-Challenge 92.3
- BIG-Bench Hard 79.7
- AGIEval 61.4
- IFEval 63.8
- HumanEval 81.1
- MBPP 72
- CRUXEval 61.5
- LiveCodeBench 32.5
- GSM8K 92.2
- MATH 53.9
- CMATH 81.9
- CLUEWSC 89.9
- C-Eval 78
- CMMLU 81.6
- MT-Bench 8.97
- AlpacaEval 2.0 38.9
- AlignBench 7.91
- SuperCLUE-Math6 83.3