Gemma 3 4B

Gemma 3 4B places in the top three on Flores.

Google DeepMind Released 12 Mar 2025

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind Mar 2025

Gemma 3 4B

Open weights

115.8 #214 of 271

Epoch Capabilities Index

SimpleQA Verified 4.60% #56 /56
BLINK 38% #27 /28
GPQA Diamond 23.2% #214 /219

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
IFEval	90.2%	#24 of 126	Model card
Vectara Hallucination Leaderboard (2025 dataset)	6.40%	#28 of 104	Official leaderboard
ArabicMMLU	55.3%	#14 of 39	Official leaderboard
Flores	39.2	#3 of 8	Model card
ArbMMLU-HT	45.2%	#18 of 39	Official leaderboard
MadinahQA	44.1%	#20 of 39	Official leaderboard
ARC-Easy	82.4%	#12 of 23	Model card
AlGhafa	54.2%	#22 of 39	Official leaderboard
Open Arabic LLM Leaderboard v2	50.5%	#22 of 39	Official leaderboard
BIG-Bench Hard	72.2%	#41 of 71	Model card
SIQA	51.9%	#13 of 22	Model card
Vectara Hallucination Leaderboard (HHEM)	3.70%	#92 of 148	Official leaderboard
MMBench-CN v1.1	66.8%	#114 of 178	Official leaderboard
LMArena (Elo)	1,303	#227 of 333	Official leaderboard
IndicGenBench	57.2	#3 of 4	Model card
XQuAD	68.0	#3 of 4	Model card
ARC-Challenge	56.2%	#35 of 45	Model card
EQ-Bench 3	1,001	#62 of 78	Official leaderboard
AGIEval	42.1%	#28 of 35	Model card
HellaSwag	77.2%	#38 of 47	Model card
Creative Writing v3	54.1	#98 of 120	Official leaderboard
PIQA	79.6%	#27 of 33	Model card
TriviaQA	65.8%	#32 of 39	Model card
Natural Questions	20%	#18 of 20	Model card
ECLeKTic	4.60%	#9 of 10	Model card
SEA-HELM	47.8	#56 of 62	Official leaderboard
MMLU	58.1%	#115 of 127	Model card
Longform Creative Writing (EQ-Bench)	34.4	#108 of 119	Official leaderboard
BIG-Bench Extra Hard	11%	#10 of 11	Model card
BoolQ	72.3%	#26 of 28	Model card
FACTS Grounding	70.1%	#14 of 15	Model card
WinoGrande	64.7%	#40 of 42	Model card
Global MMLU Lite	65.1%	#41 of 43	Official leaderboard
MMLU-Pro	43.6%	#145 of 151	Model card
Wer wird Millionär? LLM Benchmark	64.0	#73 of 76	Official leaderboard
SimpleQA	4.00%	#57 of 59	Model card
ALRAGE	33.4%	#38 of 39	Official leaderboard
MGSM	34.7%	#51 of 52	Model card
SimpleQA Verified	4.60%	#56 of 56	Official leaderboard
FACTS Parametric	1.57%	#45 of 45	Official leaderboard
FACTS Multimodal	13.6%	#39 of 39	Official leaderboard
RULER	46.8%	#38 of 38	Model card
Slop Score	85.2	#17 of 17	Official leaderboard
MRCR (Michelangelo, original)	44.6%	#9 of 9	Model card

Math and science

Benchmark	Score	Rank	Source
MATH	75.6%	#31 of 93	Model card
GSM8K	89.2%	#41 of 87	Model card
HiddenMath	43%	#8 of 11	Model card
OTIS Mock AIME 2024-2025	7.50%	#159 of 202	Epoch AI
DTBench	18.2%	#161 of 178	Official leaderboard
GPQA Diamond	23.2%	#214 of 219	Epoch AI

Coding

Benchmark	Score	Rank	Source
HumanEval	71.3%	#67 of 95	Model card
MBPP	63.2%	#49 of 66	Model card
Natural2Code	70.3%	#8 of 9	Model card
Bird-SQL	36.3%	#12 of 13	Model card
LiveCodeBench	12.6%	#72 of 73	Model card

Agents and tools

Benchmark	Score	Rank	Source
BFCL V4	19.6%	#58 of 65	Official leaderboard
Berkeley Function Calling Leaderboard (BFCL)	39.6%	#60 of 67	Official leaderboard
LisanBench	50.0	#124 of 133	Official leaderboard
SnakeBench	8	#181 of 183	Official leaderboard

Vision, audio and robotics

Benchmark	Score	Rank	Source
MMStar	47.9%	#111 of 173	Official leaderboard
CCBench (MMBench Chinese Culture)	36.9%	#99 of 152	Official leaderboard
MMMU	48.8%	#90 of 136	Official leaderboard
MMBench	69.6%	#115 of 155	Official leaderboard
A-Bench	67%	#90 of 116	Official leaderboard
MMBench-EN v1.1	66.1%	#141 of 178	Official leaderboard
CountBench	26.1%	#19 of 20	Model card
MathVista	50%	#61 of 64	Model card
BLINK	38%	#27 of 28	Model card
AI2D	74.8%	#44 of 44	Model card
ChartQA	68.8%	#37 of 37	Model card
DocVQA	75.8%	#33 of 33	Model card
RealWorldQA	45.5%	#30 of 30	Model card
InfographicVQA	50%	#22 of 22	Model card
TextVQA	57.8%	#17 of 17	Model card
ActivityNet-QA	46.3%	#8 of 8	Model card
VQAv2	62.4%	#8 of 8	Model card
Perception Test MCVQA	50.6%	#6 of 6	Model card
ReMI	27.3%	#4 of 4	Model card
TallyQA	42.5%	#4 of 4	Model card
COCO Captions	102.0	#3 of 3	Model card
OK-VQA	51%	#3 of 3	Model card
SpatialSense VQA	50.9%	#3 of 3	Model card

Safety and security

Benchmark	Score		Rank	Source
AraTrust	78.3%		#15 of 39	Official leaderboard

Other

Benchmark	Score	Rank	Source
OpenVLM Leaderboard	55.4	#106 of 173	Official leaderboard
ACCoRD	23.8	#144 of 150	Official leaderboard
Conceptual Reasoning Index	10.1	#148 of 149	Official leaderboard
LMCA	2.78%	#139 of 139	Epoch AI

Google DeepMind's models over time

Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.

Its system card

Every result in the longer documents the lab published with it. 54 of them aren't in the launch post.

Gemma 3 Technical Report (arXiv 2503.19786v1) 25 Mar 2025