Gemini 1.0 Ultra

Gemini 1.0 Ultra places in the top three on 2 benchmarks.

Google DeepMind Released 6 Dec 2023 Family Gemini 1.0 The lab's model card

Benchmarks with a score

Ranked first, out of five or more models

In the top three

Google DeepMind Dec 2023

Flagship

No Epoch Capabilities Index score yet

Where it ranks

Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.

Benchmark	Score	Rank	Source
MMLU	90%	#14 of 127	Model card
HellaSwag	87.8%	#12 of 47	Model card
BIG-Bench Hard	83.6%	#27 of 71	Model card
DROP	82.4%	#22 of 40	Model card
WMT23	74.4	#3 of 3	Model card

Benchmark	Score		Rank	Source
GSM8K	94.4%		#17 of 87	Model card
MATH	53.2%		#58 of 93	Model card

Benchmark	Score	Rank	Source
CodeEditorBench	85.5%	#2 of 7	Official leaderboard
HumanEval	74.4%	#57 of 95	Model card
Natural2Code	74.9%	#7 of 9	Model card

Benchmark	Score	Rank	Source
TextVQA	82.3%	#6 of 17	Model card
VQAv2	77.8%	#3 of 8	Model card
MTVQA-Test	23.2%	#7 of 18	Official leaderboard
MMMU	59.4%	#59 of 136	Official leaderboard
InfographicVQA	80.3%	#13 of 22	Model card
DocVQA	90.9%	#26 of 33	Model card
ChartQA	80.8%	#31 of 37	Model card
MathVista	53%	#57 of 64	Model card
AI2D	79.5%	#40 of 44	Model card

The benchmarks the lab chose to headline, with the scores it gave.