Capabilities
- MMLU 68.4
- GPQA Diamond 34.2
- HumanEval 62.2
- GSM8K 79.6
- MATH 30
- AGIEval 45.9
- CommonSenseQA 72.6
- WinoGrande 76.1
- BIG-Bench Hard 61.1
- ARC-Challenge 78.6
- TriviaQA 78.5
- SQuAD 76.4
- QuAC 44.4
- BoolQ 75.7
- DROP 58.4
Llama 3-8B
Llama 3-8B places in the top three on ANLI.
Meta Released 18 Apr 2024 44
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Meta Apr 2024 Llama 3-8B
Open weights
116.4
#211 of 271
Epoch Capabilities Index
- GPQA Diamond 26.1% #209 /219
- KOR-Bench 32.8% #25 /33
- StructEval 51.6% #27 /28
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenBookQA | 82.6% | #6 of 41 | Epoch AI | |
| Vectara Hallucination Leaderboard (HHEM) | 7.40% | #36 of 148 | Official leaderboard | |
| ANLI | 57.3% | #3 of 11 | Epoch AI | |
| WinoGrande | 76.1% | #22 of 42 | Model card | |
| ARC-Challenge | 78.6% | #24 of 45 | Model card | |
| TriviaQA | 78.5% | #21 of 39 | Model card | |
| AlpacaEval 2.0 | 22.9% | #25 of 40 | Official leaderboard | |
| MMLU-CF | 57.3% | #24 of 37 | Official leaderboard | |
| EQ-Bench v2 | 68.9 | #56 of 85 | Official leaderboard | |
| CommonSenseQA | 72.6% | #12 of 17 | Model card | |
| AGIEval | 45.9% | #25 of 35 | Model card | |
| RepoQA | 53.6% | #16 of 22 | Official leaderboard | |
| MixEval | 75% | #25 of 34 | Official leaderboard | |
| WildBench | 33.2 | #35 of 47 | Official leaderboard | |
| MMLU-Pro | 41% | #160 of 212 | Official leaderboard | |
| BIG-Bench Hard | 61.1% | #56 of 71 | Model card | |
| MMLU | 68.4% | #101 of 127 | Model card | |
| LMArena (Elo) | 1,224 | #272 of 333 | Official leaderboard | |
| BoolQ | 75.7% | #24 of 28 | Model card | |
| ThaiExam | 19.8% | #24 of 24 | Official leaderboard | |
| SQuAD | 76.4% | #5 of 5 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 55.6 | #13 of 86 | Official leaderboard | |
| GSM8K | 79.6% | #60 of 87 | Model card | |
| ZebraLogic | 11.9% | #32 of 46 | Official leaderboard | |
| KOR-Bench | 32.8% | #25 of 33 | Official leaderboard | |
| MathBench | 36.7% | #16 of 21 | Official leaderboard | |
| MATH | 30% | #81 of 93 | Model card | |
| OTIS Mock AIME 2024-2025 | 1.94% | #188 of 202 | Epoch AI | |
| GPQA Diamond | 26.1% | #209 of 219 | Epoch AI | |
| MATH Level 5 | 6.13% | #91 of 94 | Epoch AI | |
| DTBench | 6.56% | #176 of 178 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 56.7% | #29 of 46 | Official leaderboard | |
| MBPP+ | 54.8% | #25 of 35 | Official leaderboard | |
| HumanEval | 62.2% | #73 of 95 | Model card | |
| BigCodeBench | 34.4% | #57 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 7.40% | #88 of 96 | Official leaderboard | |
| StructEval | 51.6% | #27 of 28 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GTA | 3.10% | #14 of 15 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SimpleSafetyTests | 99.3% | #28 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 70.9% | #34 of 82 | Official leaderboard | |
| Anthropic Red Team | 98.8% | #42 of 75 | Official leaderboard | |
| OR-Bench | 69.4% | #18 of 28 | Official leaderboard | |
| HELM Safety | 88.6 | #55 of 75 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 38.7 | #51 of 71 | Official leaderboard |
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its system card
Every result in the longer documents the lab published with it. 15 of them aren't in the launch post.