- Best of MMLU [base pretrained] / Multilingual MMLU French [base] / Multilingual MMLU German [base] / Multilingual MMLU Spanish [base] / Multilingual MMLU Italian [base] 84
- MT Bench 8.63
- Wild Bench 56.3
- Arena Hard 73.2
- Best of Human Eval / Human Eval Plus 92
- Best of MBPP Base / MBPP Plus 80
- MultiPL-E Average (Python, C++, Bash, Java, TypeScript, PHP, C#) 76.9
- GSM8K (8-shot) 93
- Math Instruct (0-shot, CoT) [0-shot no CoT = 70%] 71.5
Mistral Large 2 Jul 2024
Mistral Large 2.
34
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jul 2024 Mistral Large 2
Jul 2024 snapshot Open weights Flagship
127.6
#171 of 271
Epoch Capabilities Index
- GPQA Diamond 49% #149 /219
- KOR-Bench 53.1% #8 /33
- SWT-bench 16.3% #5 /9
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| EQ-Bench v2 | 85.0 | #5 of 85 | Official leaderboard | |
| WildBench | 55.8 | #5 of 47 | Official leaderboard | |
| MixEval | 86.1% | #5 of 34 | Official leaderboard | |
| MMLU | 84% | #58 of 127 | Model card | |
| MMLU-Pro | 65.9% | #97 of 212 | Official leaderboard | |
| SEAL Precise Instruction Following | 82.8 | #9 of 18 | Official leaderboard | |
| SEAL Multilingual | 1,032 | #10 of 19 | Official leaderboard | |
| MT-Bench | 8.63 | #14 of 23 | Model card | |
| LMArena (Elo) | 1,314 | #213 of 333 | Official leaderboard | |
| Arena-Hard-Auto | 73.2% | #25 of 39 | Model card | |
| Lech Mazur Creative Writing | 6.90 | #37 of 44 | Epoch AI | |
| LongBench v2 | 33.6% | #29 of 33 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ZebraLogic | 29% | #11 of 46 | Official leaderboard | |
| KOR-Bench | 53.1% | #8 of 33 | Official leaderboard | |
| GSM8K | 93% | #25 of 87 | Model card | |
| ForecastBench | 61.7 | #27 of 83 | Epoch AI | |
| MATH | 71.5% | #37 of 93 | Model card | |
| GSM1k | 93.9% | #7 of 15 | Official leaderboard | |
| MATH Level 5 | 44.8% | #57 of 94 | Epoch AI | |
| GPQA Diamond | 49% | #149 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 8.47% | #153 of 202 | Epoch AI | |
| DTBench | 35.4% | #139 of 178 | Official leaderboard | |
| SimpleBench | 22.5% | #75 of 82 | Epoch AI |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 92% | #10 of 95 | Model card | |
| MBPP | 80% | #21 of 66 | Model card | |
| MHPP | 41.4% | #8 of 23 | Official leaderboard | |
| MultiPL-E | 76.9% | #7 of 18 | Model card | |
| BigCodeBench-Hard | 26% | #41 of 96 | Official leaderboard | |
| SEAL Coding | 1,029 | #13 of 26 | Official leaderboard | |
| SWT-bench | 16.3% | #5 of 9 | Official leaderboard | |
| Aider-Edit | 60.2% | #22 of 37 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ToolComp | 50.4% | #25 of 32 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AIR-Bench 2024 | 35.3% | #77 of 82 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 74.4 | #19 of 71 | Official leaderboard |
Mistral AI's models over time
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.