Capabilities
- MMLU 54.8
- BIG-Bench Hard 39.4
- AGIEval 39.1
Safety
- TruthfulQA 41.9
- ToxiGen 26.1
Llama 2-13B
Llama 2-13B places in the top three on 2 benchmarks.
Meta Released 18 Jul 2023 36
Benchmarks with a score
0
Ranked first, out of five or more models
2
In the top three
Meta Jul 2023 Llama 2-13B
Open weights
105.7
#237 of 271
Epoch Capabilities Index
- Vectara Hallucination Leaderboard (HHEM) 10.5% #18 /148
- MMLU-CF 42.8% #34 /37
- SIQA 50.3% #17 /22
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 10.5% | #18 of 148 | Official leaderboard | |
| LAMBADA | 76.5% | #11 of 26 | Epoch AI | |
| BoolQ | 81.7% | #13 of 28 | Model card | |
| TriviaQA | 79.6% | #19 of 39 | Model card | |
| Natural Questions | 31.2% | #11 of 20 | Model card | |
| L-Eval | 45.7% | #3 of 5 | Official leaderboard | |
| OpenBookQA | 57% | #14 of 20 | Model card | |
| HellaSwag | 80.7% | #33 of 47 | Model card | |
| WinoGrande | 72.8% | #30 of 42 | Model card | |
| PIQA | 80.5% | #25 of 33 | Model card | |
| SIQA | 50.3% | #17 of 22 | Model card | |
| ARC-Easy | 77.3% | #19 of 23 | Model card | |
| SQuAD v2 | 72.9% | #7 of 8 | Model card | |
| CommonSenseQA | 67.3% | #15 of 17 | Model card | |
| AGIEval | 39.1% | #31 of 35 | Model card | |
| MMLU-Pro | 25.3% | #189 of 212 | Official leaderboard | |
| TruthfulQA | 41.9% | #25 of 28 | Model card | |
| AlpacaEval 2.0 | 8.44% | #36 of 40 | Official leaderboard | |
| ARC-Challenge | 49.4% | #41 of 45 | Model card | |
| BIG-Bench Hard | 39.4% | #65 of 71 | Model card | |
| MMLU-CF | 42.8% | #34 of 37 | Official leaderboard | |
| LMArena (Elo) | 1,141 | #309 of 333 | Official leaderboard | |
| EQ-Bench v2 | 49.1 | #79 of 85 | Official leaderboard | |
| MMLU | 54.8% | #119 of 127 | Model card |
Math and science
Coding
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 53.7% | #134 of 138 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ScienceQA | 55.8% | #20 of 23 | Epoch AI |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OR-Bench | 91% | #23 of 28 | Official leaderboard | |
| Situational Awareness Dataset (SAD) | 33.5% | #15 of 17 | Official leaderboard | |
| SuperCLUE-Safety | 77.5 | #7 of 7 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Classic | 82.3 | #3 of 17 | Official leaderboard | |
| HELM Lite | 23.3 | #66 of 71 | Official leaderboard |
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its system cards
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
Capabilities
- MMLU 54.8
- BIG-Bench Hard 39.4
- AGIEval 39.1
- HumanEval 18.3
- MBPP 30.6
- BoolQ 81.7
- PIQA 80.5
- SIQA 50.3
- HellaSwag 80.7
- WinoGrande 72.8
- ARC-Easy 77.3
- ARC-Challenge 49.4
- OpenBookQA 57
- CommonSenseQA 67.3
- Natural Questions 31.2
- TriviaQA 79.6
- SQuAD v2 72.9
- QuAC 42.7
- GSM8K 28.7
- MATH 3.9
- Llama 2-Chat helpfulness human evaluation (Meta)