Capabilities
- MMLU 45.3
- BIG-Bench Hard 32.6
- AGIEval 29.3
Safety
- TruthfulQA 33.3
- ToxiGen 21.3
Llama 2-7B
Llama 2-7B places in the top three on DecodingTrust.
Meta Released 18 Jul 2023 36
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Meta Jul 2023 Llama 2-7B
Open weights
98.8
#248 of 271
Epoch Capabilities Index
- Vectara Hallucination Leaderboard (HHEM) 11.3% #16 /148
- MMLU-CF 39.4% #36 /37
- SIQA 48.3% #22 /22
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 11.3% | #16 of 148 | Official leaderboard | |
| OpenBookQA | 58.6% | #11 of 20 | Model card | |
| LAMBADA | 73.3% | #17 of 26 | Epoch AI | |
| TriviaQA | 72.1% | #27 of 39 | Model card | |
| Natural Questions | 25.7% | #15 of 20 | Model card | |
| BoolQ | 77.4% | #22 of 28 | Model card | |
| HellaSwag | 77.2% | #37 of 47 | Model card | |
| WinoGrande | 69.2% | #35 of 42 | Model card | |
| PIQA | 78.8% | #28 of 33 | Model card | |
| ARC-Easy | 75.2% | #20 of 23 | Model card | |
| TruthfulQA | 33.3% | #26 of 28 | Model card | |
| MMLU-Pro | 20.3% | #197 of 212 | Official leaderboard | |
| ARC-Challenge | 45.9% | #42 of 45 | Model card | |
| EQ-Bench v2 | 36.3 | #80 of 85 | Official leaderboard | |
| WildBench | 15.2 | #45 of 47 | Official leaderboard | |
| LMArena (Elo) | 1,108 | #322 of 333 | Official leaderboard | |
| MMLU | 45.3% | #123 of 127 | Model card | |
| AGIEval | 29.3% | #34 of 35 | Model card | |
| BIG-Bench Hard | 32.6% | #69 of 71 | Model card | |
| MMLU-CF | 39.4% | #36 of 37 | Official leaderboard | |
| AlpacaEval 2.0 | 5.35% | #39 of 40 | Official leaderboard | |
| SIQA | 48.3% | #22 of 22 | Model card | |
| CommonSenseQA | 57.8% | #17 of 17 | Model card | |
| SQuAD v2 | 67.2% | #8 of 8 | Model card | |
| L-Eval | 36.8% | #5 of 5 | Official leaderboard |
Math and science
Coding
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 51.9% | #135 of 138 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ScienceQA | 43.1% | #22 of 23 | Epoch AI |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| DecodingTrust | 74.7 | #2 of 5 | Official leaderboard | |
| OR-Bench | 87.5% | #21 of 28 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 32.9% | #16 of 17 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Classic | 60.7 | #9 of 17 | Official leaderboard | |
| HELM Lite | 15.2 | #70 of 71 | Official leaderboard |
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its system cards
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
Capabilities
- MMLU 45.3
- BIG-Bench Hard 32.6
- AGIEval 29.3
- HumanEval 12.8
- MBPP 20.8
- BoolQ 77.4
- PIQA 78.8
- SIQA 48.3
- HellaSwag 77.2
- WinoGrande 69.2
- ARC-Easy 75.2
- ARC-Challenge 45.9
- OpenBookQA 58.6
- CommonSenseQA 57.8
- Natural Questions 25.7
- TriviaQA 72.1
- SQuAD v2 67.2
- QuAC 39.4
- GSM8K 14.6
- MATH 2.5
- Llama 2-Chat helpfulness human evaluation (Meta)