- MMLU (0-shot, CoT) 88.6
- MMLU-Pro (5-shot, CoT) 73.3
- IFEval 88.6
- HumanEval (0-shot) 89
- MBPP EvalPlus (0-shot) 88.6
- GSM8K (8-shot, CoT) 96.8
- MATH (0-shot, CoT) 73.8
- ARC Challenge (0-shot) 96.9
- GPQA (0-shot, CoT) 51.1
- BFCL 88.5
- Nexus 58.7
- ZeroSCROLLS/QuALITY 95.2
- InfiniteBench/En.MC 83.4
- NIH/Multi-needle 98.1
- MGSM (0-shot, CoT) 91.6
Llama 3.1-405B
Llama 3.1-405B ranks first on 7 of the 77 benchmarks it has scores on.
95
Benchmarks with a score
7
Ranked first, out of five or more models
17
In the top three
Meta Jul 2024 Llama 3.1-405B
Open weights Open model
128.8
#168 of 271
Epoch Capabilities Index
- SWE-Bench Pro 11.2% #22 /24
- CritPt 0% #114 /160
- GPQA Diamond 50.9% #144 /219
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ARC-Challenge | 96.9% | #1 of 45 | Model card | |
| TriviaQA | 91.8% | #1 of 39 | Model card | |
| HellaSwag | 89.2% | #2 of 74 | Epoch AI | |
| Phare | 76.4% | #6 of 62 | Official leaderboard | |
| PIQA | 85.6% | #4 of 33 | Model card | |
| ThaiExam | 68.2% | #3 of 24 | Official leaderboard | |
| SQuAD v2 | 81.8% | #1 of 8 | Model card | |
| EQ-Bench v2 | 83.0 | #11 of 85 | Official leaderboard | |
| MGSM | 91.6% | #7 of 52 | Model card | |
| WinoGrande | 86.7% | #6 of 42 | Model card | |
| MultiLoKo | 31% | #1 of 7 | Official leaderboard | |
| MMLU | 88.6% | #22 of 127 | Model card | |
| AlpacaEval 2.0 | 39.3% | #7 of 40 | Official leaderboard | |
| CommonSenseQA | 85.8% | #3 of 17 | Model card | |
| SQuAD | 89.3% | #1 of 5 | Model card | |
| AGIEval | 71.6% | #8 of 35 | Model card | |
| QuALITY | 87.6% | #2 of 7 | Model card | |
| BIG-Bench Hard | 85.9% | #23 of 71 | Model card | |
| IFEval | 88.6% | #42 of 126 | Model card | |
| AP Calculus | 88.6% | #1 of 3 | Model card | |
| AP Physics | 92.9% | #1 of 3 | Model card | |
| GMAT Verbal | 86.6% | #1 of 3 | Model card | |
| InfiniteBench/En.MC | 83.4% | #1 of 3 | Model card | |
| Multilingual MMLU (Meta) | 83.2% | #1 of 3 | Model card | |
| SAT Reading | 74.8% | #1 of 3 | Model card | |
| ZeroSCROLLS/QuALITY | 95.2% | #1 of 3 | Model card | |
| ToRR | 46.1 | #5 of 14 | Official leaderboard | |
| SEAL Precise Instruction Following | 84.8 | #7 of 18 | Official leaderboard | |
| SIQA | 53.7% | #9 of 22 | Model card | |
| GRE Verbal | 166.0 | #3 of 6 | Model card | |
| WorldSense | 63.7% | #2 of 4 | Model card | |
| LLM Confabulation Leaderboard | 17.6% | #32 of 58 | Official leaderboard | |
| SuperGPQA | 43.1% | #32 of 58 | Official leaderboard | |
| LMArena (Elo) | 1,335 | #193 of 333 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 3.90% | #88 of 148 | Official leaderboard | |
| BoolQ | 80% | #17 of 28 | Model card | |
| BuzzBench | 45.9 | #14 of 23 | Official leaderboard | |
| Needle In A Haystack | 98.1% | #6 of 9 | Model card | |
| InfiniteBench/En.QA | 30.5 | #2 of 3 | Model card | |
| MMLU-Pro | 73.3% | #106 of 151 | Model card | |
| RACE | 58.1% | #3 of 4 | Model card | |
| OpenBookQA | 49.2% | #16 of 20 | Model card | |
| Creative Writing v3 | 51.5 | #104 of 120 | Official leaderboard | |
| EQ-Bench 3 | 853 | #72 of 78 | Official leaderboard | |
| SEAL Multilingual | 915 | #19 of 19 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 96.8% | #1 of 87 | Model card | |
| KOR-Bench | 55.4% | #5 of 33 | Official leaderboard | |
| ZebraLogic | 32.6% | #9 of 46 | Official leaderboard | |
| GSM1k | 95.6% | #3 of 15 | Official leaderboard | |
| GRE Quantitative | 162.0 | #2 of 6 | Model card | |
| GMAT Quantitative | 96% | #1 of 3 | Model card | |
| GRE Physics | 80% | #1 of 3 | Model card | |
| MATH | 73.8% | #34 of 93 | Model card | |
| ForecastBench | 61.7 | #33 of 86 | Official leaderboard | |
| MATH Level 5 | 49.8% | #52 of 94 | Epoch AI | |
| GPQA Diamond | 50.9% | #144 of 219 | Epoch AI | |
| CritPt | 0% | #114 of 160 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 9.72% | #152 of 202 | Epoch AI | |
| DTBench | 35.6% | #136 of 178 | Official leaderboard | |
| SimpleBench | 23% | #88 of 100 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 88.6% | #3 of 66 | Model card | |
| HumanEval | 89% | #16 of 95 | Model card | |
| BigO(Bench) | 38.3% | #5 of 18 | Official leaderboard | |
| BigCodeBench-Hard | 26.4% | #36 of 96 | Official leaderboard | |
| MHPP | 39.5% | #10 of 23 | Official leaderboard | |
| SEAL Coding | 1,022 | #14 of 26 | Official leaderboard | |
| HumanEval+ | 82.3% | #18 of 27 | Model card | |
| SciCode | 1.50% | #14 of 18 | Official leaderboard | |
| WeirdML | 21.4% | #115 of 130 | Epoch AI | |
| SWE-Bench Pro | 11.2% | #22 of 24 | Official leaderboard | |
| MLE-bench | 3.33% | #12 of 12 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 88.5% | #1 of 47 | Model card | |
| CRMArena | 51.3% | #3 of 9 | Official leaderboard | |
| API-Bank | 92.3% | #1 of 3 | Model card | |
| Gorilla APIBench | 35.3% | #1 of 3 | Model card | |
| Nexus | 58.7% | #1 of 3 | Model card | |
| LisanBench | 200.0 | #88 of 133 | Official leaderboard | |
| ToolComp | 50.4% | #24 of 32 | Official leaderboard | |
| Elimination Game | 3 | #46 of 56 | Official leaderboard | |
| Step Race | 1 | #55 of 59 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LEXam | 43.1% | #19 of 31 | Official leaderboard | |
| TheAgentCompany | 7.40% | #9 of 14 | Epoch AI | |
| TaxEval v2 | 60.9% | #111 of 129 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SEAL Adversarial Robustness | 10.0 | #2 of 8 | Official leaderboard | |
| FORTRESS | 20.6% | #28 of 66 | Official leaderboard | |
| CyberSecEval Spear Phishing | 14% | #1 of 2 | Model card | |
| SimpleSafetyTests | 98.8% | #39 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 58.6% | #54 of 82 | Official leaderboard | |
| HELM Safety | 89.7 | #50 of 75 | Official leaderboard | |
| CyberSecEval Prompt Injection | 21.7% | #2 of 3 | Model card | |
| Anthropic Red Team | 96.5% | #65 of 75 | Official leaderboard | |
| Cybench | 7.50% | #23 of 24 | Official leaderboard | |
| CyberSecEval Code Interpreter Abuse | 10.4% | #2 of 2 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 85.4 | #8 of 71 | Official leaderboard | |
| HELM Capabilities | 61.8 | #38 of 60 | Official leaderboard |
Meta's models over time
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system cards
Every result in the longer documents the lab published with it. 13 of them aren't in the launch post.
Capabilities
- MMLU 88.6
- MMLU-Pro 73.3
- IFEval 88.6
- ARC-Challenge 96.9
- GPQA Diamond 50.7
- HumanEval 89
- MBPP 88.6
- MultiPL-E
- GSM8K 96.8
- MATH 73.8
- API-Bank 92
- Berkeley Function Calling Leaderboard (BFCL) 88.5
- Gorilla APIBench 35.3
- Nexus 58.7
- MGSM 91.6
- Multilingual MMLU (Meta)
- AGIEval 71.6
- CommonSenseQA 85.8
- WinoGrande 86.7
- BIG-Bench Hard 85.9
- TriviaQA 91.8
- SQuAD 89.3
- QuAC 53.6
- BoolQ 80
- DROP 84.8
The Llama 3 Herd of Models 23 Nov 2024
Capabilities
- MMLU 88.6
- MMLU-Pro 73.3
- IFEval 88.6
- HumanEval 89
- MBPP 88.6
- HumanEval+ 82.3
- MultiPL-E
- GSM8K 96.8
- MATH 73.8
- ARC-Challenge 96.9
- GPQA Diamond 51.1
- Berkeley Function Calling Leaderboard (BFCL) 88.5
- Nexus 58.7
- API-Bank 92.3
- Gorilla APIBench 35.3
- ZeroSCROLLS/QuALITY 95.2
- Qasper 49.8
- SQuALITY 15.4
- InfiniteBench/En.MC 83.4
- InfiniteBench/En.QA 30.5
- Needle In A Haystack 98.1
- MGSM 91.6
- Multilingual MMLU (Meta) 83.2
- LSAT 81.1
- SAT Reading 74.8
- SAT Math 94.9
- GMAT Quantitative 96
- GMAT Verbal 86.6
- GRE Physics 80
- AP Art History 86.7
- AP Biology 100
- AP Calculus 88.6
- AP Chemistry 90.6
- AP English Language and Composition 94.3
- AP English Literature and Composition 83.3
- AP Environmental Science 93.5
- AP Macroeconomics 98.3
- AP Microeconomics 93.8
- AP Physics 92.9
- AP Psychology 100
- AP Statistics 85.2
- AP US Government 97.6
- AP US History 97.6
- AP World History 100
- GRE Quantitative 162
- GRE Verbal 166
- SQuAD v2 81.8
- QuAC 53.6
- RACE 58.1
- CommonSenseQA 85.8
- PIQA 85.6
- SIQA 53.7
- OpenBookQA 49.2
- WinoGrande 82.2
- DROP 84.8
- WorldSense 63.7
- AGIEval 71.6
- BIG-Bench Hard 85.9
- QuALITY 87.6
- Adversarial SQuAD
- Dynabench SQuAD
- GSM-Plus
- PAWS
- Llama 3 pairwise human evaluation (Meta)