Tulu 3 405B ranks first on PopQA.
Ai2 Jan 2025 Open weights Flagship
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BIG-Bench Hard | 88.6% | #8 of 71 | Model card | |
| PopQA | 55.5% | #1 of 7 | Model card | |
| MMLU | 87% | #40 of 127 | Model card | |
| AlpacaEval 2.0 | 51.4% | #8 of 16 | Model card | |
| IFEval | 86% | #66 of 126 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 95.9% | #4 of 95 | Model card | |
| HumanEval+ | 92.9% | #2 of 27 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Tulu 3 Safety Eval | 86.7% | #2 of 6 | Model card |
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it.