Mistral NeMo
Mistral NeMo.
Mistral AI Released 18 Jul 2024 23
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jul 2024 Mistral NeMo
Open weights
118.6
#202 of 271
Epoch Capabilities Index
- GPQA Diamond 29.9% #199 /219
- Vectara Hallucination Leaderboard (HHEM) 11.2% #17 /148
- BigCodeBench-Hard 16.6% #69 /96
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 11.2% | #17 of 148 | Official leaderboard | |
| PIQA | 83.5% | #11 of 58 | Epoch AI | |
| EQ-Bench v2 | 77.1 | #30 of 85 | Official leaderboard | |
| WildBench | 46.9 | #17 of 47 | Official leaderboard | |
| BoolQ | 82.5% | #34 of 75 | Epoch AI | |
| Slop Score | 54.4 | #10 of 17 | Official leaderboard | |
| MMLU-Pro | 44.8% | #146 of 212 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 224.0 | #59 of 76 | Official leaderboard | |
| Creative Writing v3 | 48.0 | #110 of 120 | Official leaderboard | |
| Longform Creative Writing (EQ-Bench) | 28.7 | #114 of 119 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.2% | #15 of 82 | Epoch AI | |
| ZebraLogic | 11.8% | #33 of 46 | Official leaderboard | |
| GPQA Diamond | 29.9% | #199 of 219 | Epoch AI | |
| MATH Level 5 | 10.8% | #88 of 94 | Epoch AI | |
| DTBench | 14.3% | #168 of 178 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench-Hard | 16.6% | #69 of 96 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BFCL V4 | 27.6% | #45 of 65 | Official leaderboard | |
| BALROG | 17.6% | #25 of 35 | Official leaderboard | |
| Berkeley Function Calling Leaderboard (BFCL) | 51.5% | #48 of 67 | Official leaderboard | |
| SnakeBench | 15 | #168 of 183 | Official leaderboard | |
| LisanBench | 48.7 | #125 of 133 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.66 | #30 of 31 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BrowseSafe++ | 59.7% | #14 of 15 | Official leaderboard |
Mistral AI's models over time
Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.