- GPQA Diamond 0-shot (IT); PT GPQA 5-shot = 24.3 42.4
- SimpleQA 0-shot (IT) 10
- FACTS Grounding (IT) 74.9
- BIG-Bench Hard 0-shot (IT); PT few-shot = 77.7 87.6
- BIG-Bench Extra Hard 0-shot (IT) 19.3
- IFEval 0-shot (IT) 90.4
- HellaSwag 10-shot (PT) 85.6
- BoolQ 0-shot (PT) 82.4
- PIQA 0-shot (PT) 83.3
- SocialIQA 0-shot (PT) 54.9
- TriviaQA 5-shot (PT) 85.5
- Natural Questions 5-shot (PT) 36.1
- ARC-c 25-shot (PT) 70.6
- ARC-e 0-shot (PT) 89
- WinoGrande 5-shot (PT) 78.8
- DROP 1-shot (PT) 77.2
- MMLU (Pro) 0-shot (IT); PT MMLU (Pro COT) 5-shot = 52.2 67.5
- LiveCodeBench 0-shot (IT) 29.7
- Bird-SQL (dev) (IT) 54.4
- Math 0-shot (IT); PT MATH 4-shot = 50.0 89
- HiddenMath 0-shot (IT) 60.3
- MBPP 3-shot (IT); PT = 65.6 74.4
- HumanEval 0-shot (IT); PT = 48.8 87.8
- Natural2Code 0-shot (IT) 84.5
- GSM8K 0-shot (IT); PT 8-shot = 82.6 95.9
- MMLU 5-shot (PT) 78.6
- AGIEval 3-5-shot (PT) 66.2
- Global-MMLU-Lite (PT); IT 0-shot = 75.1 75.7
- ECLeKTic (PT); IT 0-shot = 16.7 24.4
- WMT24++ (ChrF) (PT); IT 0-shot = 53.4 55.7
- MGSM (PT) 74.3
- FloRes (PT) 48.8
- XQuAD (all) (PT) 76.8
- IndicGenBench (PT) 63.4
- MMMU (val) (IT); PT MMMU = 56.1 64.9
- DocVQA (IT); PT DocVQA (val) = 85.6 86.6
- InfoVQA (IT); PT InfoVQA (val) = 59.4 70.6
- TextVQA (val) (PT); IT = 65.1 68.6
- AI2D (IT); PT = 79.0 84.5
- ChartQA (IT); PT = 76.3 78
- VQAv2 (PT); IT VQAv2 (val) = 71.0 72.9
- MathVista (testmini) (IT) 67.6
- COCOcap (PT) 116
- RealWorldQA (PT) 53.9
- ReMI (PT) 44.8
- BLINK (PT) 39.6
- OKVQA (PT) 60.2
- TallyQA (PT) 54.3
- SpatialSense VQA (PT) 59.4
- CountBenchQA (PT) 68
Gemma 3 27B
Gemma 3 27B ranks first on 2 of the 105 benchmarks it has scores on.
112
Benchmarks with a score
2
Ranked first, out of five or more models
6
In the top three
Google DeepMind Mar 2025 Gemma 3 27B
Open weights Open model
129.9
#161 of 271
Epoch Capabilities Index
- SWE-Bench Pro 11.4% #21 /24
- CritPt 0% #126 /160
- SimpleQA Verified 10.7% #46 /56
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MadinahQA | 69.4% | #1 of 39 | Official leaderboard | |
| ArabicMMLU | 70.8% | #2 of 39 | Official leaderboard | |
| Open Arabic LLM Leaderboard v2 | 71.4% | #2 of 39 | Official leaderboard | |
| AlGhafa | 77.9% | #4 of 39 | Official leaderboard | |
| ArbMMLU-HT | 66.9% | #4 of 39 | Official leaderboard | |
| Flores | 48.8 | #1 of 8 | Model card | |
| BIG-Bench Hard | 87.6% | #11 of 71 | Model card | |
| IFEval | 90.4% | #21 of 126 | Model card | |
| IndicGenBench | 63.4 | #1 of 4 | Model card | |
| XQuAD | 76.8 | #1 of 4 | Model card | |
| TriviaQA | 85.5% | #10 of 39 | Model card | |
| ARC-Easy | 89% | #7 of 23 | Model card | |
| AGIEval | 66.2% | #11 of 35 | Model card | |
| SIQA | 54.9% | #7 of 22 | Model card | |
| MMBench-CN v1.1 | 78.2% | #60 of 178 | Official leaderboard | |
| Vectara Hallucination Leaderboard (2025 dataset) | 7.40% | #36 of 104 | Official leaderboard | |
| Natural Questions | 36.1% | #7 of 20 | Model card | |
| PIQA | 83.3% | #12 of 33 | Model card | |
| BoolQ | 82.4% | #12 of 28 | Model card | |
| HellaSwag | 85.6% | #21 of 47 | Model card | |
| Lech Mazur Creative Writing | 7.99 | #20 of 44 | Epoch AI | |
| WinoGrande | 78.8% | #20 of 42 | Model card | |
| LMArena (Elo) | 1,365 | #161 of 333 | Official leaderboard | |
| ALRAGE | 73.3% | #19 of 39 | Official leaderboard | |
| ARC-Challenge | 70.6% | #27 of 45 | Model card | |
| ECLeKTic | 24.4% | #6 of 10 | Model card | |
| MMLU | 78.6% | #78 of 127 | Model card | |
| EQ-Bench 3 | 1,105 | #51 of 78 | Official leaderboard | |
| MRCR (Michelangelo, original) | 59.3% | #6 of 9 | Model card | |
| MGSM | 74.3% | #35 of 52 | Model card | |
| SEA-HELM | 64.5 | #42 of 62 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 3.00% | #105 of 148 | Official leaderboard | |
| Creative Writing v3 | 58.7 | #86 of 120 | Official leaderboard | |
| BIG-Bench Extra Hard | 19.3% | #8 of 11 | Model card | |
| Wer wird Millionär? LLM Benchmark | 289.0 | #56 of 76 | Official leaderboard | |
| DROP | 77.2% | #30 of 40 | Model card | |
| MMLU-Pro | 67.5% | #118 of 151 | Model card | |
| Longform Creative Writing (EQ-Bench) | 41.9 | #95 of 119 | Official leaderboard | |
| SimpleQA Verified | 10.7% | #46 of 56 | Official leaderboard | |
| Slop Score | 69.5 | #14 of 17 | Official leaderboard | |
| Arena-Hard v2.0 | 15% | #19 of 23 | Official leaderboard | |
| SimpleQA | 10% | #49 of 59 | Model card | |
| Global MMLU Lite | 76.3% | #36 of 43 | Official leaderboard | |
| FACTS Grounding | 74.9% | #13 of 15 | Model card | |
| FACTS Multimodal | 24.6% | #34 of 39 | Official leaderboard | |
| FACTS Parametric | 6.27% | #40 of 45 | Official leaderboard | |
| RULER | 66% | #35 of 38 | Model card | |
| Phare | 57.4% | #59 of 62 | Official leaderboard | |
| Fiction.LiveBench | 33.3% | #55 of 57 | Epoch AI | |
| FACTS Search | 41.4% | #30 of 31 | Official leaderboard | |
| FACTS Benchmark Suite | 24.6% | #39 of 40 | Official leaderboard | |
| BullshitBench | 3.64% | #105 of 107 | Official leaderboard | |
| BullshitBench V2 | 3.00% | #120 of 122 | Official leaderboard | |
| LLM Confabulation Leaderboard | 40.3% | #58 of 58 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 95.9% | #8 of 87 | Model card | |
| MATH | 89% | #11 of 93 | Model card | |
| MATH Level 5 | 74% | #31 of 94 | Epoch AI | |
| HiddenMath | 60.3% | #5 of 11 | Model card | |
| OTIS Mock AIME 2024-2025 | 22.5% | #142 of 202 | Epoch AI | |
| GPQA Diamond | 47.7% | #154 of 219 | Epoch AI | |
| CritPt | 0% | #126 of 160 | Epoch AI | |
| PolyMath | 20.3% | #33 of 38 | Official leaderboard | |
| DTBench | 20.9% | #155 of 178 | Official leaderboard | |
| Misguided Attention | 23.4% | #43 of 45 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval | 87.8% | #29 of 95 | Model card | |
| BigO(Bench) | 37.6% | #6 of 18 | Official leaderboard | |
| Natural2Code | 84.5% | #3 of 9 | Model card | |
| BigCodeBench-Hard | 26% | #42 of 96 | Official leaderboard | |
| MBPP | 74.4% | #31 of 66 | Model card | |
| Bird-SQL | 54.4% | #8 of 13 | Model card | |
| LiveCodeBench | 29.7% | #62 of 73 | Model card | |
| SWE-Bench Pro | 11.4% | #21 of 24 | Official leaderboard | |
| Aider Polyglot | 4.90% | #53 of 54 | Epoch AI |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 59.1% | #25 of 67 | Official leaderboard | |
| SnakeBench | 25 | #84 of 183 | Official leaderboard | |
| BFCL V4 | 29.5% | #40 of 65 | Official leaderboard | |
| LisanBench | 175.3 | #95 of 133 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| A-Bench | 76.3% | #22 of 116 | Official leaderboard | |
| MMBench | 82.3% | #37 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 79.6% | #56 of 178 | Official leaderboard | |
| MMMU | 64.9% | #45 of 136 | Official leaderboard | |
| COCO Captions | 116.0 | #1 of 3 | Model card | |
| OK-VQA | 60.2% | #1 of 3 | Model card | |
| MMStar | 59.6% | #61 of 173 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 53.3% | #57 of 152 | Official leaderboard | |
| ReMI | 44.8% | #2 of 4 | Model card | |
| TallyQA | 54.3% | #2 of 4 | Model card | |
| MATH-Vision (MathVision) | 46% | #56 of 105 | Official leaderboard | |
| Perception Test MCVQA | 58.1% | #4 of 6 | Model card | |
| SpatialSense VQA | 59.4% | #2 of 3 | Model card | |
| Vision Arena | 1,160 | #96 of 136 | Official leaderboard | |
| Math-VR | 16.1% | #19 of 26 | Official leaderboard | |
| MathVista | 67.6% | #48 of 64 | Model card | |
| ActivityNet-QA | 52.8% | #6 of 8 | Model card | |
| VQAv2 | 72.9% | #6 of 8 | Model card | |
| InfographicVQA | 70.6% | #17 of 22 | Model card | |
| AI2D | 84.5% | #35 of 44 | Model card | |
| GeoBench | 52% | #41 of 48 | Official leaderboard | |
| TextVQA | 68.6% | #15 of 17 | Model card | |
| CountBench | 68% | #18 of 20 | Model card | |
| BLINK | 39.6% | #26 of 28 | Model card | |
| RealWorldQA | 53.9% | #28 of 30 | Model card | |
| DocVQA | 86.6% | #31 of 33 | Model card | |
| ChartQA | 78% | #35 of 37 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AraTrust | 89.1% | #3 of 39 | Official leaderboard | |
| Spiral-Bench v1.2 | 25.2 | #18 of 21 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 67.5 | #47 of 173 | Official leaderboard | |
| Political Science LLM Benchmark | 66% | #15 of 29 | Official leaderboard | |
| LiveBench | 50% | #26 of 48 | Epoch AI | |
| ACCoRD | 40.5 | #130 of 150 | Official leaderboard | |
| LMCA | 12.3% | #122 of 139 | Epoch AI | |
| Conceptual Reasoning Index | 19.6 | #134 of 149 | Official leaderboard |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 5 of them aren't in the launch post.
Gemma 3 Technical Report (arXiv 2503.19786v1) 25 Mar 2025
Capabilities
- LMArena (Elo) 1338
- MMLU-Pro 67.5
- LiveCodeBench 29.7
- Bird-SQL 54.4
- GPQA Diamond 42.4
- SimpleQA 10
- FACTS Grounding 74.9
- Global MMLU Lite 75.1
- MATH 89
- HiddenMath 60.3
- MMMU 64.9
- MMLU 76.9
- MBPP 74.4
- HumanEval 87.8
- Natural2Code 84.5
- GSM8K 95.9
- BIG-Bench Hard 87.6
- BIG-Bench Extra Hard 19.3
- IFEval 90.4
- ECLeKTic 16.7
- WMT24++ 53.4
- RULER 66
- MRCR (Michelangelo, original) 59.3
- DocVQA 86.6
- InfographicVQA 70.6
- TextVQA 65.1
- AI2D 84.5
- ChartQA 78
- VQAv2 71
- MathVista 67.6
- Perception Test MCVQA 58.1
- ActivityNet-QA 52.8
- HellaSwag 85.6
- BoolQ 82.4
- PIQA 83.3
- SIQA 54.9
- TriviaQA 85.5
- Natural Questions 36.1
- ARC-Challenge 70.6
- ARC-Easy 89
- WinoGrande 78.8
- DROP 77.2
- AGIEval 66.2
- COCO Captions 116
- RealWorldQA 53.9
- ReMI 44.8
- BLINK 39.6
- OK-VQA 60.2
- TallyQA 54.3
- SpatialSense VQA 59.4
- CountBench 68
- MGSM 74.3
- Flores 48.8
- XQuAD 76.8
- IndicGenBench 63.4