- AlpacaEval 2 LC 42.8
- BBH 70.6
- DROP 78
- GSM8k 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- Safety (Ai2 safety suite average) 85.9
- PopQA 37.5
- TruthQA 73.2
OLMo 2 32B Instruct
OLMo 2 32B Instruct places in the top three on 2 benchmarks.
22
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Ai2 Mar 2025 OLMo 2 32B Instruct
Open weights Flagship
No Epoch Capabilities Index score yet
- PopQA 37.5% #2 /7
- Misguided Attention 19.2% #45 /45
- MMLU-Pro 52.7% #138 /151
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TruthfulQA | 73.2% | #4 of 28 | Model card | |
| AGIEval | 68.3% | #10 of 35 | Model card | |
| PopQA | 37.5% | #2 of 7 | Model card | |
| IFEval | 85.6% | #70 of 126 | Model card | |
| BIG-Bench Hard | 70.6% | #45 of 71 | Model card | |
| MMLU | 77.3% | #85 of 127 | Model card | |
| DROP | 78% | #29 of 40 | Model card | |
| AlpacaEval 2.0 | 42.8% | #12 of 16 | Model card | |
| LMArena (Elo) | 1,252 | #260 of 333 | Official leaderboard | |
| MMLU-Pro | 52.7% | #138 of 151 | Model card | |
| IFEval-OOD | 33.1% | #1 of 1 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 87.6% | #48 of 87 | Model card | |
| MATH | 49.7% | #64 of 93 | Model card | |
| DeepMind Mathematics | 34.7% | #3 of 4 | Model card | |
| GPQA Diamond | 35.9% | #266 of 284 | Model card | |
| Misguided Attention | 19.2% | #45 of 45 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SnakeBench | 18 | #145 of 183 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.3% | #26 of 75 | Official leaderboard | |
| Tulu 3 Safety Eval | 85.9% | #3 of 6 | Model card | |
| SimpleSafetyTests | 98% | #47 of 75 | Official leaderboard | |
| HELM Safety | 89.7 | #51 of 75 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Capabilities | 47.5 | #56 of 60 | Official leaderboard |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
2 OLMo 2 Furious 31 Dec 2024
Capabilities
- AlpacaEval 2.0 42.8
- BIG-Bench Hard 70.6
- DROP 78
- GSM8K 87.6
- IFEval 85.6
- MATH 49.7
- MMLU 77.3
- PopQA 37.5
- TruthfulQA 73.2
- AGIEval 68.3
- DeepMind Mathematics 34.7
- GPQA Diamond 35.9
- IFEval-OOD 33.1
- MMLU-Pro 52.7
Safety
- Tulu 3 Safety Eval 85.9