DeepSeek-V2.5 places in the top three on 2 benchmarks.
DeepSeek Sep 2024 Sep 2024 snapshot Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese SimpleQA | 55.7% | #4 of 27 | Official leaderboard | |
| EQ-Bench v2 | 82.0 | #17 of 85 | Official leaderboard | |
| MT-Bench | 9.02 | #6 of 23 | Model card | |
| MMLU-Pro | 65.8% | #98 of 212 | Official leaderboard | |
| Arena-Hard-Auto | 76.2% | #21 of 39 | Model card | |
| AlpacaEval 2.0 | 50.5% | #10 of 16 | Model card | |
| LMArena (Elo) | 1,307 | #220 of 333 | Official leaderboard | |
| AlignBench | 8.04 | #10 of 15 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 2.40% | #117 of 148 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 47.8% | #14 of 33 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 83.5% | #8 of 46 | Official leaderboard | |
| HumanEval | 89% | #17 of 95 | Model card | |
| MBPP+ | 74.1% | #7 of 35 | Official leaderboard | |
| MHPP | 40% | #9 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 26.1% | #39 of 96 | Official leaderboard | |
| Aider-Edit | 72.2% | #3 of 5 | Model card | |
| HumanEval Multi | 73.8% | #3 of 5 | Model card | |
| LiveCodeBench | 41.8% | #49 of 73 | Model card | |
| Aider Polyglot | 17.8% | #45 of 54 | Epoch AI | |
| SWE-bench Verified | 16.8% | #150 of 150 | Model card | |
| DS-Arena-Code | 63.1% | #1 of 1 | Model card | |
| DS-FIM-Eval | 78.3% | #1 of 1 | Model card |
The benchmarks the lab chose to headline, with the scores it gave.