- GPQA, 0-shot 46.2
- MATH, maj@64 (pretrained) (4-shot Minerva prompt = 67.7%) 77.9
- PhysicsFinals, 0-shot 63.9
- HiddenMath, 0-shot 20.1
- Functional MATH (December snapshot), 0-shot 64.6
- AMC 2022-23, 4-shot 46.4
- GSM8K, 11-shot 90.8
- BigBench - Hard, 3-shot 89.2
- DROP, Variable shots (F1) 74.9
- MMLU, maj@32 (pretrained) (5-shot = 85.9%) 91.7
- Hellaswag, 10-shot 93.3
- HumanEval chat preamble, 0-shot 84.1
- Natural2Code chat preamble, 0-shot 82.6
- WMT23, 1-shot (BLEURT) 75.3
- MGSM, 8-shot 87.5
- MMMU (val), 0-shot 62.2
- MathViSTA (testmini), 0-shot 63.9
- Ai2D (test), 0-shot 94.4
- ChemicalDiagramQA (internal), 0-shot 69.7
- ChartQA (test), 0-shot 87.2
- BetterChartQA (internal), 0-shot 65.8
- DocVQA (test), 0-shot (ANLS, with Cloud OCR annotations) 93.1
- DUDE (test), 0-shot (ANLS) 46
- TAT-DQA (test), 0-shot 37.8
- InfographicVQA (test), 0-shot (ANLS) 81
- TextVQA (val), 0-shot 78.7
- VQAv2 (test-dev), 0-shot 80.2
- V* bench (test), 0-shot 71.7
- Blink (val), 0-shot 61.4
- Realworld QA (test), 0-shot 70.4
- VATEX (test), 4-shot (CIDEr) 64.6
- VATEX ZH (val), 4-shot (CIDEr) 55.3
- YouCook2 (val), 4-shot (CIDEr) 106.5
- ActivityNet-QA (test), 0-shot 57.5
- EgoSchema (test), 0-shot 72.2
- OpenEQA (val), 0-shot (LLM-judge score) 57.9
Gemini 1.5 Pro May 2024
Gemini 1.5 Pro ranks first on SEAL Adversarial Robustness.
86
Benchmarks with a score
1
Ranked first, out of five or more models
9
In the top three
Google DeepMind May 2024 Gemini 1.5 Pro
May 2024 snapshot Closed weights Flagship
126.9
#176 of 271
Epoch Capabilities Index
- BLINK 61.4% #18 /28
- GPQA Diamond 45.9% #164 /219
- KOR-Bench 43.4% #18 /33
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU | 91.7% | #3 of 127 | Model card | |
| BIG-Bench Hard | 89.2% | #6 of 71 | Model card | |
| HellaSwag | 93.3% | #4 of 47 | Model card | |
| RepoQA | 90.6% | #3 of 22 | Official leaderboard | |
| ThaiExam | 64.7% | #4 of 24 | Official leaderboard | |
| MixEval | 84.8% | #6 of 34 | Official leaderboard | |
| RewardBench | 88.2% | #11 of 48 | Official leaderboard | |
| WildBench | 47.4 | #15 of 47 | Official leaderboard | |
| MGSM | 87.5% | #17 of 52 | Model card | |
| WMT23 | 75.3 | #1 of 3 | Model card | |
| SEAL Multilingual | 1,069 | #7 of 19 | Official leaderboard | |
| MMBench-CN v1.1 | 73.2% | #85 of 178 | Official leaderboard | |
| LMArena (Elo) | 1,324 | #202 of 333 | Official leaderboard | |
| DROP | 74.9% | #32 of 40 | Model card | |
| SEAL Precise Instruction Following | 79.4 | #18 of 18 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OlympicArena | 35.1% | #3 of 25 | Official leaderboard | |
| ForecastBench | 58.8 | #10 of 83 | Epoch AI | |
| MATH | 77.9% | #26 of 93 | Model card | |
| GSM8K | 90.8% | #37 of 87 | Model card | |
| PhysicsFinals | 63.9% | #1 of 2 | Model card | |
| KOR-Bench | 43.4% | #18 of 33 | Official leaderboard | |
| GSM1k | 92.3% | #9 of 15 | Official leaderboard | |
| MATH Level 5 | 40.8% | #61 of 94 | Epoch AI | |
| GPQA Diamond | 45.9% | #164 of 219 | Epoch AI | |
| AMC | 46.4% | #3 of 4 | Model card | |
| OTIS Mock AIME 2024-2025 | 6.81% | #162 of 202 | Epoch AI | |
| DTBench | 56.5% | #142 of 177 | Epoch AI | |
| HiddenMath | 20.1% | #10 of 11 | Model card | |
| Functional MATH | 64.6% | #1 of 1 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| BigCodeBench | 50.6% | #19 of 69 | Official leaderboard | |
| Aider-Edit | 66.9% | #16 of 37 | Official leaderboard | |
| Natural2Code | 82.6% | #4 of 9 | Model card | |
| BigCodeBench-Hard | 25.4% | #43 of 96 | Official leaderboard | |
| HumanEval | 84.1% | #44 of 95 | Model card | |
| SEAL Coding | 994 | #15 of 26 | Official leaderboard | |
| SciCode | 1.50% | #12 of 18 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ToolComp | 40.4% | #27 of 32 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedXpertQA-MM | 34.1% | #4 of 9 | Official leaderboard | |
| MedHELM | 22.1 | #13 of 13 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Video-MME | 81.3% | #2 of 26 | Official leaderboard | |
| AI2D | 94.4% | #5 of 44 | Model card | |
| TOMATO | 36.1% | #3 of 22 | Official leaderboard | |
| LongVideoBench | 64.4% | #2 of 13 | Official leaderboard | |
| CharXiv | 43.3% | #15 of 77 | Official leaderboard | |
| VCR (Visual Caption Restoration) | 38.1% | #9 of 42 | Official leaderboard | |
| A-Bench | 75.4% | #28 of 116 | Official leaderboard | |
| VQAv2 | 80.2% | #2 of 8 | Model card | |
| MMStar | 59.1% | #62 of 173 | Official leaderboard | |
| MMMU | 62.2% | #51 of 136 | Official leaderboard | |
| ChartQA | 87.2% | #16 of 37 | Model card | |
| InfographicVQA | 81% | #11 of 22 | Model card | |
| EgoSchema | 72.2% | #6 of 12 | Model card | |
| ActivityNet-QA | 57.5% | #4 of 8 | Model card | |
| VATEX ZH | 55.3 | #2 of 4 | Model card | |
| OpenEQA | 57.9 | #1 of 2 | Model card | |
| MMBench-EN v1.1 | 74.6% | #92 of 178 | Official leaderboard | |
| TextVQA | 78.7% | #9 of 17 | Model card | |
| MMMU-Pro | 43.5% | #31 of 57 | Official leaderboard | |
| VHELM | 50.4 | #23 of 41 | Official leaderboard | |
| MMBench | 73.9% | #87 of 155 | Official leaderboard | |
| DocVQA | 93.1% | #20 of 33 | Model card | |
| BLINK | 61.4% | #18 of 28 | Model card | |
| VATEX | 64.6 | #3 of 4 | Model card | |
| CCBench (MMBench Chinese Culture) | 28.4% | #116 of 152 | Official leaderboard | |
| RealWorldQA | 70.4% | #23 of 30 | Model card | |
| MathVista | 63.9% | #51 of 64 | Model card | |
| AHELM | 36.2 | #8 of 10 | Official leaderboard | |
| Vision Arena | 1,120 | #111 of 136 | Official leaderboard | |
| MATH-Vision (MathVision) | 19.2% | #89 of 105 | Official leaderboard | |
| LVBench | 33.1% | #8 of 9 | Official leaderboard | |
| V* | 71.7% | #12 of 12 | Model card | |
| DUDE | 46% | #4 of 4 | Model card | |
| YouCook2 | 106.5 | #4 of 4 | Model card | |
| BetterChartQA | 65.8% | #3 of 3 | Model card | |
| ChemicalDiagramQA | 69.7% | #1 of 1 | Model card | |
| TAT-DQA | 37.8% | #1 of 1 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Anthropic Red Team | 99.9% | #3 of 75 | Official leaderboard | |
| SEAL Adversarial Robustness | 8.00 | #1 of 8 | Official leaderboard | |
| AIR-Bench 2024 | 82.8% | #17 of 82 | Official leaderboard | |
| HELM Safety | 92.5 | #36 of 75 | Official leaderboard | |
| SimpleSafetyTests | 97.5% | #52 of 75 | Official leaderboard | |
| OR-Bench | 88% | #22 of 28 | Official leaderboard | |
| AgentDojo | 28.6% | #18 of 20 | Official leaderboard | |
| Cybench | 7.50% | #22 of 24 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 73.9 | #20 of 71 | Official leaderboard | |
| OpenVLM Leaderboard | 64.5 | #59 of 173 | Official leaderboard |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.