Gemma 3 12B

Gemma 3 12B ranks first on MindCube.

Google DeepMind Released 12 Mar 2025

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind Mar 2025

Gemma 3 12B

Open weights

123.5 #185 of 271

Epoch Capabilities Index

CritPt 0% #125 /160
SimpleQA Verified 6.70% #52 /56
BLINK 35.9% #28 /28

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Knowledge and language

Benchmark	Score	Rank	Source
Vectara Hallucination Leaderboard (2025 dataset)	4.40%	#7 of 104	Official leaderboard
ArabicMMLU	65.8%	#7 of 39	Official leaderboard
ArbMMLU-HT	61.2%	#8 of 39	Official leaderboard
Flores	46.0	#2 of 8	Model card
MadinahQA	53%	#12 of 39	Official leaderboard
IFEval	88.9%	#39 of 126	Model card
Open Arabic LLM Leaderboard v2	58.2%	#13 of 39	Official leaderboard
BIG-Bench Hard	85.7%	#24 of 71	Model card
ARC-Easy	88.3%	#9 of 23	Model card
MMBench-CN v1.1	74.3%	#81 of 178	Official leaderboard
AGIEval	57.4%	#16 of 35	Model card
AlGhafa	60.7%	#19 of 39	Official leaderboard
SIQA	53.4%	#11 of 22	Model card
Natural Questions	31.4%	#10 of 20	Model card
IndicGenBench	61.7	#2 of 4	Model card
XQuAD	74.5	#2 of 4	Model card
HellaSwag	84.2%	#24 of 47	Model card
LMArena (Elo)	1,342	#186 of 333	Official leaderboard
TriviaQA	78.2%	#22 of 39	Model card
PIQA	81.8%	#19 of 33	Model card
WinoGrande	74.3%	#27 of 42	Model card
ARC-Challenge	68.9%	#29 of 45	Model card
BoolQ	78.8%	#20 of 28	Model card
MMLU	71.9%	#94 of 127	Model card
SEA-HELM	62.4	#46 of 62	Official leaderboard
Wer wird Millionär? LLM Benchmark	244.0	#57 of 76	Official leaderboard
Global MMLU Lite	69.5%	#12 of 16	Model card
Vectara Hallucination Leaderboard (HHEM)	2.80%	#112 of 148	Official leaderboard
Creative Writing v3	57.0	#93 of 120	Official leaderboard
MGSM	64.3%	#41 of 52	Model card
FACTS Grounding	75.8%	#12 of 15	Model card
ECLeKTic	10.3%	#8 of 10	Model card
Longform Creative Writing (EQ-Bench)	41.8	#96 of 119	Official leaderboard
BIG-Bench Extra Hard	16.3%	#9 of 11	Model card
MMLU-Pro	60.6%	#129 of 151	Model card
Slop Score	70.6	#15 of 17	Official leaderboard
MRCR (Michelangelo, original)	49.8%	#8 of 9	Model card
SimpleQA Verified	6.70%	#52 of 56	Official leaderboard
SimpleQA	6.30%	#55 of 59	Model card
FACTS Multimodal	20.8%	#37 of 39	Official leaderboard
FACTS Parametric	3.56%	#43 of 45	Official leaderboard
RULER	57.1%	#37 of 38	Model card
ALRAGE	25.3%	#39 of 39	Official leaderboard

Math and science

Benchmark	Score	Rank	Source
MATH	83.8%	#13 of 93	Model card
GSM8K	94.4%	#18 of 87	Model card
HiddenMath	54.5%	#7 of 11	Model card
OTIS Mock AIME 2024-2025	16.7%	#146 of 202	Epoch AI
CritPt	0%	#125 of 160	Epoch AI
GPQA Diamond	39.5%	#177 of 219	Epoch AI
DTBench	14.7%	#167 of 178	Official leaderboard

Coding

Benchmark	Score	Rank	Source
HumanEval	85.4%	#37 of 95	Model card
MBPP	73%	#35 of 66	Model card
Natural2Code	80.7%	#5 of 9	Model card
Bird-SQL	47.9%	#9 of 13	Model card
LiveCodeBench	24.6%	#66 of 73	Model card

Agents and tools

Benchmark	Score	Rank	Source
BFCL V4	30.4%	#39 of 65	Official leaderboard
Berkeley Function Calling Leaderboard (BFCL)	50.6%	#49 of 67	Official leaderboard
SnakeBench	19	#139 of 183	Official leaderboard
LisanBench	114.0	#112 of 133	Official leaderboard

Professional work

Benchmark	Score		Rank	Source
LEXam	41.3%		#21 of 31	Official leaderboard

Vision, audio and robotics

Benchmark	Score	Rank	Source
MindCube	46.7%	#1 of 5	Epoch AI
SpatialSense VQA	60%	#1 of 3	Model card
MMMU	59.6%	#58 of 136	Official leaderboard
MMStar	56.1%	#75 of 173	Official leaderboard
CCBench (MMBench Chinese Culture)	49%	#68 of 152	Official leaderboard
MMBench	77.5%	#75 of 155	Official leaderboard
MMBench-EN v1.1	74.9%	#91 of 178	Official leaderboard
A-Bench	70.3%	#71 of 116	Official leaderboard
COCO Captions	111.0	#2 of 3	Model card
OK-VQA	58.7%	#2 of 3	Model card
ReMI	38.5%	#3 of 4	Model card
TallyQA	51.8%	#3 of 4	Model card
AI2D	84.2%	#36 of 44	Model card
MathVista	62.9%	#53 of 64	Model card
Perception Test MCVQA	54.9%	#5 of 6	Model card
ActivityNet-QA	50.4%	#7 of 8	Model card
VQAv2	71.6%	#7 of 8	Model card
DocVQA	87.1%	#30 of 33	Model card
InfographicVQA	64.9%	#20 of 22	Model card
TextVQA	67.7%	#16 of 17	Model card
RealWorldQA	52.2%	#29 of 30	Model card
ChartQA	75.7%	#36 of 37	Model card
BLINK	35.9%	#28 of 28	Model card
CountBench	17.8%	#20 of 20	Model card

Safety and security

Benchmark	Score		Rank	Source
AraTrust	87%		#6 of 39	Official leaderboard

Other

Benchmark	Score	Rank	Source
OpenVLM Leaderboard	62.8	#72 of 173	Official leaderboard
ACCoRD	38.3	#133 of 150	Official leaderboard
Conceptual Reasoning Index	13.3	#140 of 149	Official leaderboard
LMCA	4.46%	#138 of 139	Epoch AI

Google DeepMind's models over time

Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.

Its system card

Every result in the longer documents the lab published with it. 54 of them aren't in the launch post.

Gemma 3 Technical Report (arXiv 2503.19786v1) 25 Mar 2025