Gemma 7B.
Google DeepMind Feb 2024 Open weights Open model
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard (HHEM) | 14.8% | #11 of 148 | Official leaderboard | |
| BoolQ | 83.2% | #11 of 28 | Model card | |
| PIQA | 81.2% | #21 of 33 | Model card | |
| SIQA | 51.8% | #14 of 22 | Model card | |
| ANLI | 48.7% | #7 of 11 | Epoch AI | |
| ARC-Easy | 81.5% | #15 of 23 | Model card | |
| HellaSwag | 81.2% | #32 of 47 | Model card | |
| WinoGrande | 72.3% | #31 of 42 | Model card | |
| OpenBookQA | 52.8% | #15 of 20 | Model card | |
| EQ-Bench v2 | 61.7 | #64 of 85 | Official leaderboard | |
| Natural Questions | 23% | #16 of 20 | Model card | |
| MMLU-Pro | 33.7% | #174 of 212 | Official leaderboard | |
| CommonSenseQA | 71.3% | #14 of 17 | Model card | |
| AGIEval | 41.7% | #29 of 35 | Model card | |
| BIG-Bench Hard | 55.1% | #60 of 71 | Model card | |
| MMLU | 64.3% | #109 of 127 | Model card | |
| ARC-Challenge | 53.2% | #39 of 45 | Model card | |
| TriviaQA | 63.4% | #34 of 39 | Model card | |
| AlpacaEval 2.0 | 10.4% | #35 of 40 | Official leaderboard | |
| LMArena (Elo) | 1,138 | #311 of 333 | Official leaderboard | |
| TruthfulQA | 31.8% | #27 of 28 | Model card | |
| WildBench | 10.2 | #46 of 47 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP+ | 43.4% | #30 of 35 | Official leaderboard | |
| HumanEval+ | 28.7% | #40 of 46 | Official leaderboard | |
| MBPP | 44.4% | #60 of 66 | Model card | |
| HumanEval | 32.3% | #88 of 95 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SuperCLUE-Agent | 51.3 | #7 of 7 | Official leaderboard |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CrowS-Pairs | 51.3% | #1 of 4 | Model card | |
| OR-Bench | 26.3% | #9 of 28 | Official leaderboard | |
| BOLD | 49.1 | #1 of 2 | Model card | |
| Winogender | 54.2% | #3 of 4 | Model card | |
| ToxiGen | 39.6 | #4 of 4 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 33.6 | #56 of 71 | Official leaderboard |
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
The benchmarks the lab chose to headline, with the scores it gave.
Every result in the longer documents the lab published with it. 8 of them aren't in the launch post.