Gemma 2 27B.
Google DeepMind Jun 2024 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ALRAGE | 80.5% | #4 of 39 | Official leaderboard | |
| BoolQ | 84.8% | #5 of 28 | Model card | |
| AlGhafa | 72.2% | #9 of 39 | Official leaderboard | |
| ArbMMLU-HT | 61.2% | #9 of 39 | Official leaderboard | |
| WildBench | 49.4 | #11 of 47 | Official leaderboard | |
| EQ-Bench v2 | 80.5 | #20 of 85 | Official leaderboard | |
| WinoGrande | 83.7% | #10 of 42 | Model card | |
| TriviaQA | 83.7% | #12 of 39 | Model card | |
| ARC-Easy | 88.6% | #8 of 23 | Model card | |
| MMLU-CF | 63.9% | #13 of 37 | Official leaderboard | |
| SIQA | 53.7% | #8 of 22 | Model card | |
| PIQA | 83.2% | #13 of 33 | Model card | |
| Natural Questions | 34.5% | #8 of 20 | Model card | |
| HellaSwag | 86.4% | #19 of 47 | Model card | |
| Open Arabic LLM Leaderboard v2 | 55.5% | #16 of 39 | Official leaderboard | |
| AGIEval | 55.1% | #17 of 35 | Model card | |
| BIG-Bench Hard | 74.9% | #39 of 71 | Model card | |
| RewardBench | 80.9% | #27 of 48 | Official leaderboard | |
| MMLU-Pro | 56.5% | #121 of 212 | Official leaderboard | |
| ARC-Challenge | 71.4% | #26 of 45 | Model card | |
| TruthfulQA | 51.6% | #17 of 28 | Model card | |
| ArabicMMLU | 37.1% | #26 of 39 | Official leaderboard | |
| MMLU | 76.2% | #88 of 127 | Model card | |
| LMArena (Elo) | 1,289 | #235 of 333 | Official leaderboard | |
| MadinahQA | 33.3% | #28 of 39 | Official leaderboard | |
| SuperGPQA | 27.4% | #44 of 58 | Official leaderboard | |
| BIG-Bench Extra Hard | 14.8% | #9 of 11 | Official leaderboard | |
| DROP | 72.2% | #33 of 40 | Model card | |
| SEAL Multilingual | 951 | #16 of 19 | Official leaderboard | |
| LLM Confabulation Leaderboard | 27.1% | #51 of 58 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OlympicArena | 26.3% | #9 of 25 | Official leaderboard | |
| KOR-Bench | 44.5% | #16 of 33 | Official leaderboard | |
| ZebraLogic | 16.3% | #26 of 46 | Official leaderboard | |
| MATH Level 5 | 27.9% | #67 of 94 | Epoch AI | |
| GSM8K | 74% | #66 of 87 | Model card | |
| MATH | 42.3% | #72 of 93 | Model card | |
| GPQA Diamond | 36.5% | #183 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 1.39% | #191 of 202 | Epoch AI | |
| DTBench | 13.3% | #170 of 178 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 47.6% | #30 of 69 | Official leaderboard | |
| MHPP | 28.6% | #12 of 23 | Official leaderboard | |
| BigCodeBench-Hard | 20% | #60 of 96 | Official leaderboard | |
| MBPP | 67.4% | #44 of 66 | Model card | |
| HumanEval | 51.8% | #79 of 95 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 68% | #123 of 138 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 80.5% | #14 of 39 | Official leaderboard | |
| ToxiGen | 38.4 | #2 of 4 | Model card | |
| Winogender | 77.2% | #2 of 4 | Model card | |
| PrimeVul | 63% | #1 of 2 | Model card | |
| OR-Bench | 62% | #17 of 28 | Official leaderboard | |
| BaxBench | 3.95% | #30 of 34 | Official leaderboard | |
| CrowS-Pairs | 36.7% | #4 of 4 | Model card | |
| Hack the Box challenge set (Google DeepMind) | 0% | #4 of 4 | Model card | |
| In-house CTF suite (Phuong et al. 2024) | 7.70% | #4 of 4 | Model card | |
| InterCode-CTF | 44.7% | #4 of 4 | Model card | |
| DiverseVul | 57% | #1 of 1 | Model card | |
| Money talks | 3.72 | #1 of 1 | Model card | |
| SecretPatch | 72% | #1 of 1 | Model card | |
| Self-proliferation challenges (Phuong et al. 2024) | 0% | #1 of 1 | Model card | |
| SPI (security patch identification) | 53% | #1 of 1 | Model card |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 20 of them aren't in the launch post.