Llama 3.1-70B places in the top three on 4 benchmarks.
Meta Released 23 Jul 2024 Meta Jul 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TriviaQA | 89.8% | #3 of 39 | Model card | |
| ARC-Challenge | 94.8% | #7 of 45 | Model card | |
| MMLU-CF | 68.7% | #7 of 37 | Official leaderboard | |
| AlpacaEval 2.0 | 38.1% | #9 of 40 | Official leaderboard | |
| CommonSenseQA | 84.1% | #4 of 17 | Model card | |
| SQuAD v2 | 81.8% | #2 of 8 | Model card | |
| WinoGrande | 83.5% | #11 of 42 | Model card | |
| ThaiExam | 62.3% | #7 of 24 | Official leaderboard | |
| PIQA | 83.8% | #10 of 33 | Model card | |
| InfiniteBench/En.QA | 36.7 | #1 of 3 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 5.00% | #54 of 148 | Official leaderboard | |
| AGIEval | 64.6% | #13 of 35 | Model card | |
| MMLU | 86% | #48 of 127 | Model card | |
| IFEval | 87.5% | #52 of 126 | Model card | |
| RewardBench | 84% | #20 of 48 | Official leaderboard | |
| MGSM | 86.9% | #22 of 52 | Model card | |
| BIG-Bench Hard | 81.6% | #32 of 71 | Model card | |
| Chinese SimpleQA | 40.2% | #13 of 27 | Official leaderboard | |
| RACE | 59% | #2 of 4 | Model card | |
| SIQA | 52.2% | #12 of 22 | Model card | |
| ToRR | 44.4 | #8 of 14 | Official leaderboard | |
| MultiLoKo | 24.3% | #4 of 7 | Official leaderboard | |
| SQuAD | 81.8% | #3 of 5 | Model card | |
| BoolQ | 79.4% | #18 of 28 | Model card | |
| AgentIF | 56.3% | #8 of 12 | Official leaderboard | |
| GRE Verbal | 166.0 | #4 of 6 | Model card | |
| AP Physics | 78.6% | #2 of 3 | Model card | |
| GMAT Verbal | 85.1% | #2 of 3 | Model card | |
| InfiniteBench/En.MC | 78.2% | #2 of 3 | Model card | |
| Multilingual MMLU (Meta) | 78.2% | #2 of 3 | Model card | |
| SAT Reading | 71.4% | #2 of 3 | Model card | |
| ZeroSCROLLS/QuALITY | 90.5% | #2 of 3 | Model card | |
| LMArena (Elo) | 1,293 | #231 of 333 | Official leaderboard | |
| SuperGPQA | 34.9% | #41 of 58 | Official leaderboard | |
| QuALITY | 82.8% | #5 of 7 | Model card | |
| WorldSense | 61.1% | #3 of 4 | Model card | |
| LongBench v2 | 36.2% | #25 of 33 | Official leaderboard | |
| Needle In A Haystack | 97.5% | #7 of 9 | Model card | |
| MMLU-Pro | 66.4% | #120 of 151 | Model card | |
| Creative Writing v3 | 50.8 | #107 of 120 | Official leaderboard | |
| OpenBookQA | 47.6% | #18 of 20 | Model card | |
| AP Calculus | 54.3% | #3 of 3 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 95.1% | #12 of 87 | Model card | |
| KOR-Bench | 50% | #11 of 33 | Official leaderboard | |
| ZebraLogic | 24.9% | #18 of 46 | Official leaderboard | |
| MATH | 68% | #46 of 93 | Model card | |
| GRE Quantitative | 158.0 | #4 of 6 | Model card | |
| GMAT Quantitative | 84% | #2 of 3 | Model card | |
| GRE Physics | 74.7% | #2 of 3 | Model card | |
| MATH Level 5 | 36.7% | #66 of 94 | Epoch AI | |
| GPQA Diamond | 44.2% | #168 of 219 | Epoch AI | |
| Omni-MATH | 24.2% | #7 of 9 | Official leaderboard | |
| DTBench | 33.3% | #142 of 178 | Official leaderboard | |
| U-MATH | 34.3% | #26 of 30 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 3.61% | #178 of 202 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 86% | #7 of 66 | Model card | |
| BigCodeBench | 50.5% | #21 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 25.4% | #44 of 96 | Official leaderboard | |
| MHPP | 33.8% | #11 of 23 | Official leaderboard | |
| HumanEval | 80.5% | #51 of 95 | Model card | |
| BigO(Bench) | 33.8% | #11 of 18 | Official leaderboard | |
| HumanEval+ | 74.4% | #22 of 27 | Model card | |
| WeirdML | 8.97% | #126 of 130 | Epoch AI | |
| SciCode | 0% | #18 of 18 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 84.8% | #2 of 47 | Model card | |
| BALROG | 27.9% | #19 of 35 | Official leaderboard | |
| CRMArena | 41.1% | #5 of 9 | Official leaderboard | |
| AceBench | 60.4% | #10 of 16 | Official leaderboard | |
| API-Bank | 90% | #2 of 3 | Model card | |
| Gorilla APIBench | 29.7% | #2 of 3 | Model card | |
| Nexus | 56.7% | #2 of 3 | Model card | |
| LisanBench | 151.3 | #99 of 133 | Official leaderboard | |
| ToolComp | 37.2% | #28 of 32 | Official leaderboard | |
| PinchBench 2.0 | 10.7% | #56 of 57 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TheAgentCompany | 6.90% | #10 of 14 | Epoch AI | |
| TaxEval v2 | 56.2% | #122 of 129 | Official leaderboard | |
| CorpFin v2 | 38.9% | #113 of 119 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CyberSecEval Code Interpreter Abuse | 3.80% | #1 of 2 | Model card | |
| FORTRESS | 44.2% | #47 of 66 | Official leaderboard | |
| HELM Safety | 84.5 | #67 of 75 | Official leaderboard | |
| SimpleSafetyTests | 92.5% | #67 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 42.5% | #74 of 82 | Official leaderboard | |
| Anthropic Red Team | 93.2% | #71 of 75 | Official leaderboard | |
| CyberSecEval Prompt Injection | 26% | #3 of 3 | Model card | |
| CyberSecEval Spear Phishing | 24% | #2 of 2 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 80.8 | #14 of 71 | Official leaderboard | |
| Political Science LLM Benchmark | 67.5% | #8 of 29 | Official leaderboard | |
| HELM Capabilities | 57.4 | #47 of 60 | Official leaderboard | |
| ACCoRD | 43.2 | #125 of 150 | Official leaderboard | |
| Conceptual Reasoning Index | 24.2 | #126 of 149 | Official leaderboard | |
| LMCA | 14.8% | #119 of 139 | Epoch AI |
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.