Gemma 2 9B places in the top three on ALRAGE.
Google DeepMind Released 24 Jun 2024 Google DeepMind Jun 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ALRAGE | 80.5% | #3 of 39 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 10.1% | #20 of 148 | Official leaderboard | |
| EQ-Bench v2 | 80.5 | #22 of 85 | Official leaderboard | |
| BoolQ | 84.2% | #8 of 28 | Model card | |
| WildBench | 45.4 | #19 of 47 | Official leaderboard | |
| WinoGrande | 80.6% | #18 of 42 | Model card | |
| ARC-Easy | 88% | #10 of 23 | Model card | |
| SIQA | 53.4% | #10 of 22 | Model card | |
| AGIEval | 52.8% | #20 of 35 | Model card | |
| PIQA | 81.7% | #20 of 33 | Model card | |
| TriviaQA | 76.6% | #24 of 39 | Model card | |
| MMLU-Pro | 52.1% | #131 of 212 | Official leaderboard | |
| HellaSwag | 81.9% | #30 of 47 | Model card | |
| Natural Questions | 29.2% | #13 of 20 | Model card | |
| ARC-Challenge | 68.4% | #30 of 45 | Model card | |
| TruthfulQA | 50.3% | #19 of 28 | Model card | |
| BIG-Bench Hard | 68.2% | #49 of 71 | Model card | |
| AlGhafa | 47.9% | #27 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 46.6% | #27 of 39 | Official leaderboard | |
| ArabicMMLU | 36.3% | #28 of 39 | Official leaderboard | |
| MMLU-CF | 53.7% | #27 of 37 | Official leaderboard | |
| MMLU | 72.3% | #93 of 127 | Model card | |
| LMArena (Elo) | 1,267 | #253 of 333 | Official leaderboard | |
| MadinahQA | 31.7% | #30 of 39 | Official leaderboard | |
| SuperGPQA | 24% | #47 of 58 | Official leaderboard | |
| ArbMMLU-HT | 25% | #33 of 39 | Official leaderboard | |
| Creative Writing v3 | 49.5 | #108 of 120 | Official leaderboard | |
| EQ-Bench 3 | 636 | #76 of 78 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 41.6% | #19 of 33 | Official leaderboard | |
| ZebraLogic | 12.8% | #29 of 46 | Official leaderboard | |
| GSM8K | 68.6% | #69 of 87 | Model card | |
| MATH | 36.6% | #76 of 93 | Model card | |
| MATH Level 5 | 21% | #77 of 94 | Epoch AI | |
| GPQA Diamond | 27.5% | #204 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 0.56% | #198 of 202 | Epoch AI |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 37.7% | #50 of 69 | Official leaderboard | |
| MBPP | 59.2% | #52 of 66 | Model card | |
| MHPP | 18.1% | #19 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 10.1% | #83 of 96 | Official leaderboard | |
| HumanEval | 40.2% | #86 of 95 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SnakeBench | 19 | #138 of 183 | Official leaderboard | |
| LLM Chess | -546 | #111 of 124 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LEXam | 27.4% | #27 of 31 | Official leaderboard | |
| LegalBench | 68.1% | #122 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Winogender | 79.2% | #1 of 4 | Model card | |
| AraTrust | 77.2% | #18 of 39 | Official leaderboard | |
| OR-Bench | 79.9% | #19 of 28 | Official leaderboard | |
| CrowS-Pairs | 37.5% | #3 of 4 | Model card | |
| ToxiGen | 39.3 | #3 of 4 | Model card |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Every result in the longer documents the lab published with it. 25 of them aren't in the launch post.