Grok-2 Beta Release 13 Aug 2024
Grok-2 Aug 2024
Grok-2.
11
Benchmarks with a score
0
Ranked first, out of five or more models
0
In the top three
xAI Aug 2024
Grok-2
Aug 2024 snapshot Closed weights Flagship
No Epoch Capabilities Index score yet
- MMLU-Pro 75.5% #100 /151
- MathVista 69% #43 /64
- MMLU 87.5% #31 /127
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU | 87.5% | #31 of 127 | Model card | |
| LMArena (Elo) | 1,336 | #190 of 333 | Official leaderboard | |
| MMLU-Pro | 75.5% | #100 of 151 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MATH | 76.1% | #30 of 93 | Model card | |
| GPQA Diamond | 56% | #219 of 284 | Model card | |
| SimpleBench | 22.7% | #90 of 100 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 88.4% | #20 of 95 | Model card | |
| Aider-Edit | 58.6% | #24 of 37 | Official leaderboard |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.