Mistral 7B v0.1.
Mistral AI Sep 2023 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenBookQA | 79.8% | #7 of 41 | Epoch AI | |
| BoolQ | 87.4% | #15 of 75 | Epoch AI | |
| BIG-Bench Hard | 56.1% | #19 of 45 | Epoch AI | |
| PIQA | 83% | #14 of 33 | Model card | |
| WinoGrande | 75.3% | #25 of 42 | Model card | |
| HellaSwag | 81.3% | #31 of 47 | Model card | |
| Natural Questions | 28.8% | #14 of 20 | Model card | |
| ANLI | 47.1% | #8 of 11 | Epoch AI | |
| ARC-Easy | 80% | #17 of 23 | Model card | |
| TriviaQA | 69.9% | #30 of 39 | Model card | |
| ARC-Challenge | 55.5% | #36 of 45 | Model card | |
| MMLU-Pro | 30.9% | #180 of 212 | Official leaderboard | |
| MMLU | 60.1% | #113 of 127 | Model card | |
| MT-Bench | 6.84 | #22 of 23 | Model card | |
| LMArena (Elo) | 1,031 | #13 of 13 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CRUXEval | 34.3% | #8 of 11 | Official leaderboard | |
| MBPP | 47.5% | #56 of 66 | Model card | |
| MBPP+ | 42.1% | #31 of 35 | Official leaderboard | |
| HumanEval+ | 23.8% | #42 of 46 | Official leaderboard | |
| HumanEval | 30.5% | #89 of 95 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 53.8% | #133 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Mistral 175 unsafe prompts | 100% | #1 of 1 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 29.2 | #60 of 71 | Official leaderboard |
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 6 of them aren't in the launch post.