DeepSeek LLM 67B places in the top three on Natural Questions.
DeepSeek Nov 2023 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Natural Questions | 47% | #3 of 20 | Model card | |
| Chinese SimpleQA | 47% | #10 of 27 | Official leaderboard | |
| TriviaQA | 81.5% | #15 of 39 | Model card | |
| OpenBookQA | 63.2% | #9 of 20 | Model card | |
| PIQA | 82.6% | #16 of 33 | Model card | |
| HellaSwag | 84% | #25 of 47 | Model card | |
| WinoGrande | 76% | #23 of 42 | Model card | |
| ARC-Easy | 81.6% | #13 of 23 | Model card | |
| BIG-Bench Hard | 71.7% | #42 of 71 | Model card | |
| RACE-M | 70.9% | #2 of 3 | Model card | |
| AGIEval | 46.4% | #24 of 35 | Model card | |
| ARC-Challenge | 64.1% | #31 of 45 | Model card | |
| MGSM | 74% | #36 of 52 | Model card | |
| MMLU | 71.1% | #95 of 127 | Model card | |
| MT-Bench | 8.35 | #18 of 23 | Model card | |
| C3 | 77% | #5 of 6 | Model card | |
| DROP | 71.9% | #34 of 40 | Model card | |
| RACE-H | 56% | #6 of 7 | Model card | |
| LMArena (Elo) | 1,185 | #288 of 333 | Official leaderboard | |
| CMMLU | 67.8% | #16 of 17 | Model card | |
| C-Eval | 65.2% | #36 of 38 | Model card | |
| IFEval | 55.5% | #124 of 126 | Model card | |
| AlignBench | 6.43 | #15 of 15 | Model card | |
| CLUEWSC | 60% | #8 of 8 | Model card | |
| CCPM | 84.9% | #2 of 2 | Model card | |
| CHID | 72.6% | #1 of 1 | Model card | |
| ChineseQA | 85.1% | #1 of 1 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.1% | #54 of 87 | Model card | |
| MathBench | 40.3% | #14 of 21 | Official leaderboard | |
| MATH | 32.6% | #79 of 93 | Model card | |
| MATH Level 5 | 6.39% | #90 of 94 | Epoch AI | |
| GPQA Diamond | 24.6% | #211 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 0.83% | #195 of 202 | Epoch AI | |
| CMATH | 80.3% | #6 of 6 | Model card | |
| Hungarian National High-School Exam | 58% | #2 of 2 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 73.8% | #61 of 95 | Model card | |
| MBPP | 61.4% | #50 of 66 | Model card | |
| HumanEval Multi | 53.3% | #5 of 5 | Model card | |
| LeetCode Weekly Contest | 17.5% | #1 of 1 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 9.51% | #10 of 15 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.4% | #22 of 75 | Official leaderboard | |
| Do-Not-Answer | 97.8% | #1 of 2 | Model card | |
| HELM Safety | 87.3 | #59 of 75 | Official leaderboard | |
| SimpleSafetyTests | 96.8% | #59 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 50.5% | #66 of 82 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 48.8 | #40 of 71 | Official leaderboard |
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 23 of them aren't in the launch post.