- MMLU (5-shot) [instruct] 82
- GPQA (0-shot) [instruct] 39.5
- HumanEval (0-shot) [instruct] 81.7
- GSM-8K (8-shot, CoT) [instruct] 93
- MATH (4-shot, CoT) [instruct] 50.4
- AGIEval English (3-5 shot) [pretrained] 63
- CommonSenseQA (7-shot) [pretrained] 83.8
- Winogrande (5-shot) [pretrained] 83.1
- BIG-Bench Hard (3-shot, CoT) [pretrained] 81.3
- ARC-Challenge (25-shot) [pretrained] 93
- TriviaQA-Wiki (5-shot) [pretrained] 89.7
- SQuAD (1-shot) [pretrained] 85.6
- QuAC (1-shot, F1) [pretrained] 51.1
- BoolQ (0-shot) [pretrained] 79
- DROP (3-shot, F1) [pretrained] 79.7
Llama 3-70B
Llama 3-70B places in the top three on SQuAD.
53
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Meta Apr 2024 Llama 3-70B
Open weights Open model
123.0
#187 of 271
Epoch Capabilities Index
- GPQA Diamond 40.6% #175 /219
- KOR-Bench 49.2% #13 /33
- Vectara Hallucination Leaderboard (HHEM) 4.10% #82 /148
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 89.7% | #4 of 39 | Model card | |
| MMLU-CF | 68.9% | #5 of 37 | Official leaderboard | |
| EQ-Bench v2 | 82.1 | #15 of 85 | Official leaderboard | |
| WildBench | 49.6 | #10 of 47 | Official leaderboard | |
| ARC-Challenge | 93% | #10 of 45 | Model card | |
| WinoGrande | 83.1% | #12 of 42 | Model card | |
| CommonSenseQA | 83.8% | #5 of 17 | Model card | |
| RepoQA | 82.2% | #7 of 22 | Official leaderboard | |
| MixEval | 84% | #11 of 34 | Official leaderboard | |
| AlpacaEval 2.0 | 34.4% | #13 of 40 | Official leaderboard | |
| AGIEval | 63% | #14 of 35 | Model card | |
| SQuAD | 85.6% | #2 of 5 | Model card | |
| ThaiExam | 56.8% | #11 of 24 | Official leaderboard | |
| BIG-Bench Hard | 81.3% | #34 of 71 | Model card | |
| MMLU | 82% | #61 of 127 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.10% | #82 of 148 | Official leaderboard | |
| MMLU-Pro | 56.2% | #123 of 212 | Official leaderboard | |
| DROP | 79.7% | #24 of 40 | Model card | |
| RewardBench | 76.3% | #31 of 48 | Official leaderboard | |
| BoolQ | 79% | #19 of 28 | Model card | |
| SEAL Precise Instruction Following | 81.2 | #13 of 18 | Official leaderboard | |
| LMArena (Elo) | 1,277 | #246 of 333 | Official leaderboard | |
| QuAC | 51.1% | #1 of 1 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 57.7 | #16 of 86 | Official leaderboard | |
| GSM8K | 93% | #24 of 87 | Model card | |
| OlympicArena | 26.7% | #8 of 25 | Official leaderboard | |
| MathBench | 56.4% | #7 of 21 | Official leaderboard | |
| KOR-Bench | 49.2% | #13 of 33 | Official leaderboard | |
| ZebraLogic | 16.8% | #24 of 46 | Official leaderboard | |
| MATH | 50.4% | #62 of 93 | Model card | |
| GSM1k | 90.1% | #11 of 15 | Official leaderboard | |
| GPQA Diamond | 40.6% | #175 of 219 | Epoch AI | |
| MATH Level 5 | 22.6% | #76 of 94 | Epoch AI | |
| DTBench | 23.6% | #152 of 178 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 4.31% | #175 of 202 | Epoch AI |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 72% | #17 of 46 | Official leaderboard | |
| BigCodeBench | 49% | #26 of 69 | Official leaderboard | |
| MBPP+ | 69% | #15 of 35 | Official leaderboard | |
| BigCodeBench-Hard | 24.6% | #48 of 96 | Official leaderboard | |
| HumanEval | 81.7% | #49 of 95 | Model card | |
| SEAL Coding | 871 | #23 of 26 | Official leaderboard | |
| SciCode | 0% | #17 of 18 | Official leaderboard |
Agents and tools
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OR-Bench | 37.7% | #10 of 28 | Official leaderboard | |
| SimpleSafetyTests | 99% | #33 of 75 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 49.3% | #8 of 17 | Official leaderboard | |
| AIR-Bench 2024 | 64.6% | #44 of 82 | Official leaderboard | |
| HELM Safety | 89.5 | #52 of 75 | Official leaderboard | |
| AgentDojo | 25.6% | #14 of 20 | Official leaderboard | |
| Anthropic Red Team | 96.7% | #63 of 75 | Official leaderboard | |
| Cybench | 5.00% | #24 of 24 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 79.3 | #16 of 71 | Official leaderboard |
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it.
Capabilities
- MMLU 82
- GPQA Diamond 39.5
- HumanEval 81.7
- GSM8K 93
- MATH 50.4
- AGIEval 63
- CommonSenseQA 83.8
- WinoGrande 83.1
- BIG-Bench Hard 81.3
- ARC-Challenge 93
- TriviaQA 89.7
- SQuAD 85.6
- QuAC 51.1
- BoolQ 79
- DROP 79.7