Benchmark Almanac

Mixtral 8x7B

Mixtral 8x7B places in the top three on CriticEval.

Mistral AI Released 11 Dec 2023 Family Mixtral The lab's model card
49
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Mistral AI Dec 2023

Mixtral 8x7B

Open weights Open model

118.4 #205 of 271
Epoch Capabilities Index
  • GPQA Diamond 30.6% #198 /219
  • LiveSQLBench 2.33% #36 /36
  • Vectara Hallucination Leaderboard (HHEM) 20.1% #4 /148

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

BenchmarkScoreRankSource
Vectara Hallucination Leaderboard (HHEM) 20.1% #4 of 148 Official leaderboard
OpenBookQA 85.8% #5 of 41 Epoch AI
AlpacaEval 94.8% #4 of 19 Official leaderboard
PIQA 83.6% #11 of 33 Model card
CriticEval 5.75 #3 of 8 Official leaderboard
HellaSwag 86.7% #18 of 47 Model card
ARC-Challenge 85.8% #18 of 45 Model card
WinoGrande 81.2% #17 of 42 Model card
RepoQA 68% #10 of 22 Official leaderboard
ANLI 55.2% #5 of 11 Epoch AI
ARC-Easy 83.1% #11 of 23 Model card
Passkey Retrieval 100% #1 of 2 Model card
MMLU-CF 58.3% #19 of 37 Official leaderboard
EQ-Bench v2 72.4 #47 of 85 Official leaderboard
AlpacaEval 2.0 23.7% #24 of 40 Official leaderboard
Natural Questions 30.6% #12 of 20 Model card
WildBench 35.0 #31 of 47 Official leaderboard
TriviaQA 71.5% #28 of 39 Model card
MMLU-Pro 43.3% #153 of 212 Official leaderboard
MMLU 72.5% #92 of 127 Model card
RewardBench 74.5% #35 of 48 Official leaderboard
LMArena (Elo) 1,197 #284 of 333 Official leaderboard
ToRR 34.7 #12 of 14 Official leaderboard
SuperGPQA 22.1% #50 of 58 Official leaderboard
MT-Bench 8.30 #20 of 23 Model card
ThaiExam 30% #22 of 24 Official leaderboard

Math and science

BenchmarkScoreRankSource
ForecastBench 58.1 #17 of 86 Official leaderboard
ZebraLogic 8.70% #40 of 46 Official leaderboard
GSM8K 58.4% #76 of 87 Model card
MATH 28.4% #82 of 93 Model card
GPQA Diamond 30.6% #198 of 219 Epoch AI
DTBench 49.6% #163 of 177 Epoch AI
MATH Level 5 9.95% #89 of 94 Epoch AI

Coding

BenchmarkScoreRankSource
CRUXEval 40.5% #6 of 11 Official leaderboard
HumanEval+ 39.6% #34 of 46 Official leaderboard
MBPP 60.7% #51 of 66 Model card
MBPP+ 49.7% #28 of 35 Official leaderboard
HumanEval 40.2% #85 of 95 Model card
MHPP 11% #21 of 23 Official leaderboard
LiveSQLBench 2.33% #36 of 36 Official leaderboard

Agents and tools

BenchmarkScoreRankSource
GTA 9.77% #9 of 15 Official leaderboard

Professional work

BenchmarkScoreRankSource
LegalBench 55.8% #131 of 138 Official leaderboard

Safety and security

BenchmarkScoreRankSource
AIR-Bench 2024 39.1% #76 of 82 Official leaderboard
HELM Safety 81.4 #70 of 75 Official leaderboard
BBQ (Bias Benchmark for QA) 56% #23 of 24 Model card
Anthropic Red Team 92.8% #72 of 75 Official leaderboard
SimpleSafetyTests 90.5% #72 of 75 Official leaderboard

Other

BenchmarkScoreRankSource
HELM Lite 51.0 #37 of 71 Official leaderboard
HELM Capabilities 39.7 #59 of 60 Official leaderboard

Mistral AI's models over time

Mistral AI's models in color, against every other model on the Epoch Capabilities Index, by release date.

50 75 100 125 150 175 2023 2024 2025 2026 2027 Mistral Large 4 Mistral Medium 3.5 Magistral Small 1.0 Ministral 3B Mistral Large 2 Mixtral 8x7B Mistral 7B v0.1 50 75 100 125 150 175 2023 2024 2025 2026 2027 Mistral Large 4 Mistral Medium 3.5

Its launch post

The benchmarks the lab chose to headline, with the scores it gave.

Mixtral of experts (launch post) 11 Dec 2023

Its system card

Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.