Mistral Large places in the top three on OR-Bench.
Mistral AI Feb 2024 Closed weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| EQ-Bench v2 | 85.2 | #4 of 85 | Official leaderboard | |
| WinoGrande | 86.7% | #5 of 42 | Model card | |
| HellaSwag | 89.2% | #7 of 47 | Model card | |
| ARC-Challenge | 94% | #8 of 45 | Model card | |
| MixEval | 84.2% | #8 of 34 | Official leaderboard | |
| AlpacaEval 2.0 | 32.6% | #14 of 40 | Official leaderboard | |
| TriviaQA | 82.7% | #14 of 39 | Model card | |
| WildBench | 42.3 | #23 of 47 | Official leaderboard | |
| MMLU | 81.2% | #68 of 127 | Model card | |
| TruthfulQA | 50.5% | #18 of 28 | Model card | |
| LMArena (Elo) | 1,243 | #262 of 333 | Official leaderboard | |
| SEAL Precise Instruction Following | 79.9 | #16 of 18 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 53.6 | #9 of 86 | Official leaderboard | |
| GSM8K | 91.2% | #34 of 87 | Model card | |
| MATH Level 5 | 24.5% | #71 of 94 | Epoch AI | |
| MATH | 45% | #71 of 93 | Model card | |
| DTBench | 55.9% | #143 of 177 | Epoch AI | |
| GPQA Diamond | 38.8% | #178 of 219 | Epoch AI | |
| GSM1k | 87.5% | #13 of 15 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.94% | #187 of 202 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 73.1% | #34 of 66 | Model card | |
| HumanEval+ | 62.2% | #25 of 46 | Official leaderboard | |
| MBPP+ | 59.5% | #22 of 35 | Official leaderboard | |
| BigCodeBench | 34.1% | #58 of 69 | Official leaderboard | |
| HumanEval | 45.1% | #81 of 95 | Model card | |
| BigCodeBench-Hard | 8.40% | #86 of 96 | Official leaderboard | |
| SEAL Coding | 811 | #24 of 26 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 17.1% | #5 of 15 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OR-Bench | 9.80% | #3 of 28 | Official leaderboard | |
| SEAL Adversarial Robustness | 37.0 | #7 of 8 | Official leaderboard | |
| DistressBench | 59.1% | #39 of 42 | Official leaderboard |
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.