Llama 3.1-8B.
Meta Released 23 Jul 2024 Meta Jul 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AlGhafa | 70.9% | #11 of 39 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 5.40% | #51 of 148 | Official leaderboard | |
| ArabicMMLU | 53.6% | #15 of 39 | Official leaderboard | |
| ArbMMLU-HT | 47.9% | #15 of 39 | Official leaderboard | |
| BullshitBench | 34.5% | #43 of 107 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 55.4% | #17 of 39 | Official leaderboard | |
| ARC-Challenge | 83.4% | #23 of 45 | Model card | |
| MadinahQA | 43.1% | #21 of 39 | Official leaderboard | |
| Needle In A Haystack | 98.8% | #5 of 9 | Model card | |
| WinoGrande | 75.7% | #24 of 42 | Model card | |
| CommonSenseQA | 75% | #10 of 17 | Model card | |
| TriviaQA | 77.6% | #23 of 39 | Model card | |
| AGIEval | 47.8% | #23 of 35 | Model card | |
| AP Calculus | 57.1% | #2 of 3 | Model card | |
| AlpacaEval 2.0 | 20.9% | #27 of 40 | Official leaderboard | |
| MMLU-CF | 57.1% | #25 of 37 | Official leaderboard | |
| Phare | 62.5% | #42 of 62 | Official leaderboard | |
| PIQA | 81% | #23 of 33 | Model card | |
| MMLU | 73% | #91 of 127 | Model card | |
| FaithJudge | 28.4% | #27 of 37 | Official leaderboard | |
| MGSM | 68.9% | #39 of 52 | Model card | |
| SQuAD v2 | 77% | #6 of 8 | Model card | |
| BIG-Bench Hard | 64.2% | #54 of 71 | Model card | |
| IFEval | 80.4% | #97 of 126 | Model card | |
| BullshitBench V2 | 14% | #97 of 122 | Official leaderboard | |
| SQuAD | 77% | #4 of 5 | Model card | |
| Chinese SimpleQA | 17.7% | #22 of 27 | Official leaderboard | |
| BuzzBench | 32.9 | #19 of 23 | Official leaderboard | |
| LMArena (Elo) | 1,211 | #278 of 333 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 118.0 | #66 of 76 | Official leaderboard | |
| ALRAGE | 49.9% | #34 of 39 | Official leaderboard | |
| BoolQ | 75% | #25 of 28 | Model card | |
| SuperGPQA | 20.5% | #52 of 58 | Official leaderboard | |
| SIQA | 49.5% | #20 of 22 | Model card | |
| AgentIF | 53.6% | #11 of 12 | Official leaderboard | |
| Creative Writing v3 | 47.2 | #112 of 120 | Official leaderboard | |
| MMLU-Pro | 48.3% | #141 of 151 | Model card | |
| OpenBookQA | 45% | #19 of 20 | Model card | |
| WritingBench | 40.2 | #43 of 45 | Official leaderboard | |
| ThaiExam | 24% | #23 of 24 | Official leaderboard | |
| SEA-HELM | 37.2 | #60 of 62 | Official leaderboard | |
| Judgemark v2.1 | 19.2 | #42 of 42 | Official leaderboard | |
| LongBench v2 | 30.4% | #33 of 33 | Official leaderboard | |
| ToRR | 28.9 | #14 of 14 | Official leaderboard | |
| BIG-Bench Extra Hard | 10.6% | #11 of 11 | Official leaderboard | |
| QuALITY | 56% | #7 of 7 | Model card | |
| GRE Verbal | 149.0 | #6 of 6 | Model card | |
| RACE | 54.3% | #4 of 4 | Model card | |
| WorldSense | 45.5% | #4 of 4 | Model card | |
| AP Physics | 57.1% | #3 of 3 | Model card | |
| GMAT Verbal | 65.7% | #3 of 3 | Model card | |
| InfiniteBench/En.MC | 65.1% | #3 of 3 | Model card | |
| InfiniteBench/En.QA | 27.1 | #3 of 3 | Model card | |
| Multilingual MMLU (Meta) | 58.6% | #3 of 3 | Model card | |
| SAT Reading | 57.4% | #3 of 3 | Model card | |
| ZeroSCROLLS/QuALITY | 81% | #3 of 3 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 84.5% | #52 of 87 | Model card | |
| OlympicArena | 19.4% | #16 of 25 | Official leaderboard | |
| MATH | 51.9% | #60 of 93 | Model card | |
| ZebraLogic | 12.8% | #30 of 46 | Official leaderboard | |
| KOR-Bench | 37.2% | #23 of 33 | Official leaderboard | |
| CritPt | 0% | #115 of 160 | Epoch AI | |
| MATH Level 5 | 22.9% | #74 of 94 | Epoch AI | |
| DTBench | 18.2% | #160 of 178 | Official leaderboard | |
| U-MATH | 29.6% | #27 of 30 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 1.67% | #190 of 202 | Epoch AI | |
| GPQA Diamond | 27% | #207 of 219 | Epoch AI | |
| GRE Quantitative | 152.0 | #6 of 6 | Model card | |
| GMAT Quantitative | 56% | #3 of 3 | Model card | |
| GRE Physics | 48% | #3 of 3 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 72.8% | #36 of 66 | Model card | |
| HumanEval | 72.6% | #64 of 95 | Model card | |
| MBPP+ | 55.6% | #24 of 35 | Official leaderboard | |
| BigCodeBench | 36.6% | #53 of 69 | Official leaderboard | |
| BigCodeBench-Hard | 13.5% | #75 of 96 | Official leaderboard | |
| MHPP | 17.1% | #20 of 23 | Official leaderboard | |
| StructEval | 61.8% | #25 of 28 | Official leaderboard | |
| HumanEval+ | 67.1% | #26 of 27 | Model card | |
| WeirdML | 1.73% | #130 of 130 | Epoch AI | |
| BigO(Bench) | 21.8% | #18 of 18 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 76.1% | #9 of 47 | Model card | |
| BFCL V4 | 25.8% | #50 of 65 | Official leaderboard | |
| BALROG | 15.1% | #29 of 35 | Official leaderboard | |
| AceBench | 33.4% | #14 of 16 | Official leaderboard | |
| LLM Chess | -796 | #121 of 124 | Official leaderboard | |
| LisanBench | 12.7 | #131 of 133 | Official leaderboard | |
| ToolComp | 17.4% | #32 of 32 | Official leaderboard | |
| CRMArena | 3.10% | #9 of 9 | Official leaderboard | |
| API-Bank | 82.6% | #3 of 3 | Model card | |
| Gorilla APIBench | 8.20% | #3 of 3 | Model card | |
| Nexus | 38.5% | #3 of 3 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CorpFin v2 | 37.8% | #116 of 119 | Official leaderboard | |
| TaxEval v2 | 32.3% | #128 of 129 | Official leaderboard | |
| LEXam | 10% | #31 of 31 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CyberSecEval Prompt Injection | 19% | #1 of 3 | Model card | |
| AraTrust | 75.6% | #19 of 39 | Official leaderboard | |
| SimpleSafetyTests | 98.8% | #40 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 62.3% | #49 of 82 | Official leaderboard | |
| Anthropic Red Team | 96.8% | #61 of 75 | Official leaderboard | |
| HELM Safety | 86.2 | #61 of 75 | Official leaderboard | |
| BrowseSafe++ | 23.1% | #15 of 15 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 30.3 | #58 of 71 | Official leaderboard | |
| ACCoRD | 33.7 | #136 of 150 | Official leaderboard | |
| Conceptual Reasoning Index | 13.6 | #138 of 149 | Official leaderboard | |
| HELM Capabilities | 44.4 | #58 of 60 | Official leaderboard | |
| LMCA | 5.37% | #136 of 139 | Epoch AI |
Meta's models in color, against every other model on the Epoch Capabilities Index, by release date.
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.