Codestral 25.01 (Mistral news) 13 Jan 2025
- HumanEval (Python) 86.6
- MBPP 80.2
- CruxEval 55.5
- LiveCodeBench (version not stated) 37.9
- RepoBench 38
- Spider (SQL) 66.5
- CanItEdit 50.5
- HumanEval (average on several languages) 71.4
Codestral
Codestral.
Mistral AI Released 13 Jan 2025 14
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
Mistral AI Jan 2025 Codestral
Open weights
No Epoch Capabilities Index score yet
- SimpleQA Verified 5.56% #54 /56
- BaxBench 15.8% #24 /34
- HumanEval Multi 71.4% #4 /5
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| FACTS Grounding | 59.7% | #32 of 34 | Official leaderboard | |
| ECLeKTic | 3.20% | #33 of 35 | Official leaderboard | |
| SimpleQA Verified | 5.56% | #54 of 56 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 80.2% | #18 of 66 | Model card | |
| HumanEval | 86.6% | #35 of 95 | Model card | |
| CanItEdit | 50.5% | #2 of 3 | Model card | |
| LiveCodeBench | 37.9% | #52 of 73 | Model card | |
| Spider | 66.5% | #3 of 4 | Model card | |
| CRUXEval | 55.5% | #7 of 9 | Model card | |
| HumanEval Multi | 71.4% | #4 of 5 | Model card | |
| Aider Polyglot | 11.1% | #50 of 54 | Epoch AI | |
| ALE-Bench | 138 | #121 of 121 | Epoch AI | |
| RepoBench | 38% | #1 of 1 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BaxBench | 15.8% | #24 of 34 | Official leaderboard |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.