Gemma 2B.
Google DeepMind Released 21 Feb 2024 Google DeepMind Feb 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| TruthfulQA | 44.8% | #23 of 28 | Model card | |
| OpenBookQA | 47.8% | #17 of 20 | Model card | |
| SIQA | 49.7% | #19 of 22 | Model card | |
| HellaSwag | 71.4% | #41 of 47 | Model card | |
| PIQA | 77.3% | #30 of 33 | Model card | |
| TriviaQA | 53.2% | #36 of 39 | Model card | |
| WinoGrande | 65.4% | #39 of 42 | Model card | |
| CommonSenseQA | 65.3% | #16 of 17 | Model card | |
| MMLU-Pro | 15.8% | #200 of 212 | Official leaderboard | |
| BIG-Bench Hard | 35.2% | #67 of 71 | Model card | |
| AlpacaEval 2.0 | 5.44% | #38 of 40 | Official leaderboard | |
| Natural Questions | 12.5% | #19 of 20 | Model card | |
| ARC-Challenge | 42.1% | #43 of 45 | Model card | |
| ARC-Easy | 73.2% | #22 of 23 | Model card | |
| BoolQ | 69.4% | #27 of 28 | Model card | |
| LMArena (Elo) | 1,094 | #323 of 333 | Official leaderboard | |
| MMLU | 42.3% | #125 of 127 | Model card | |
| EQ-Bench v2 | 23.3 | #84 of 85 | Official leaderboard | |
| WildBench | -5.25 | #47 of 47 | Official leaderboard | |
| AGIEval | 24.2% | #35 of 35 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 20.7% | #43 of 46 | Official leaderboard | |
| HumanEval | 22% | #92 of 95 | Model card | |
| MBPP+ | 34.1% | #34 of 35 | Official leaderboard | |
| MBPP | 29.2% | #65 of 66 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ToxiGen | 29.8 | #1 of 4 | Model card | |
| CrowS-Pairs | 45.8% | #2 of 4 | Model card | |
| Winogender | 51.3% | #4 of 4 | Model card | |
| BOLD | 45.6 | #2 of 2 | Model card |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Every result in the longer documents the lab published with it. 26 of them aren't in the launch post.