Gemini 1.5 Pro Sep 2024
Gemini 1.5 Pro places in the top three on 4 benchmarks.
Google DeepMind Released 24 Sep 2024 57
Benchmarks with a score
0
Ranked first, out of five or more models
4
In the top three
Google DeepMind Sep 2024 Gemini 1.5 Pro
Sep 2024 snapshot Closed weights
131.7
#157 of 271
Epoch Capabilities Index
- Humanity's Last Exam 4.60% #42 /47
- ZeroBench 2.00% #56 /66
- OCRBench v2 51.6% #16 /65
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMLU | 86.9% | #4 of 127 | Epoch AI | |
| EQ-Bench v2 | 83.5 | #9 of 85 | Official leaderboard | |
| Chinese SimpleQA | 56.7% | #3 of 27 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 9.10% | #22 of 148 | Official leaderboard | |
| Creation-MMBench | -5.49 | #3 of 19 | Official leaderboard | |
| MMBench-CN v1.1 | 81.1% | #37 of 178 | Official leaderboard | |
| LLM Confabulation Leaderboard | 13.5% | #15 of 58 | Official leaderboard | |
| FACTS Grounding | 81.2% | #9 of 34 | Official leaderboard | |
| RewardBench | 86.8% | #13 of 48 | Official leaderboard | |
| ToRR | 48.0 | #4 of 14 | Official leaderboard | |
| ECLeKTic | 20.2% | #14 of 35 | Official leaderboard | |
| MMLU-Pro | 70.3% | #86 of 212 | Official leaderboard | |
| LMArena (Elo) | 1,351 | #176 of 333 | Official leaderboard | |
| WritingBench | 58.8 | #32 of 45 | Official leaderboard | |
| Humanity's Last Exam | 4.60% | #42 of 47 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 52.9 | #6 of 86 | Official leaderboard | |
| U-MATH | 60.1% | #9 of 30 | Official leaderboard | |
| MATH Level 5 | 70.4% | #33 of 94 | Epoch AI | |
| GPQA Diamond | 57.2% | #129 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 23.1% | #141 of 202 | Epoch AI | |
| DTBench | 59% | #138 of 177 | Epoch AI | |
| SimpleBench | 27.1% | #82 of 100 | Official leaderboard | |
| ARC-AGI-2 | 0.80% | #83 of 95 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP+ | 74.6% | #5 of 35 | Official leaderboard | |
| HumanEval-V | 22.9% | #4 of 24 | Official leaderboard | |
| HumanEval+ | 79.3% | #13 of 46 | Official leaderboard | |
| BigCodeBench-Hard | 26.6% | #34 of 96 | Official leaderboard | |
| WeirdML | 22.2% | #114 of 130 | Epoch AI |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Galileo Agent Leaderboard | 0.89 | #14 of 31 | Official leaderboard | |
| AceBench | 70.7% | #8 of 16 | Official leaderboard | |
| BALROG | 21% | #22 of 35 | Official leaderboard | |
| Step Race | 1 | #52 of 59 | Official leaderboard | |
| LLM Chess | -651 | #117 of 124 | Official leaderboard | |
| AndroidLab | 18.8% | #2 of 2 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| CadEval | 34% | #9 of 15 | Epoch AI | |
| MortgageTax | 56.8% | #70 of 86 | Official leaderboard | |
| TheAgentCompany | 3.40% | #12 of 14 | Epoch AI | |
| LegalBench | 69.1% | #119 of 138 | Official leaderboard | |
| TaxEval v2 | 59.5% | #116 of 129 | Official leaderboard | |
| CorpFin v2 | 40.5% | #111 of 119 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMBench-EN v1.1 | 84.4% | #21 of 178 | Official leaderboard | |
| Image2Struct | 79.0 | #3 of 18 | Official leaderboard | |
| MMVU | 65.4% | #5 of 28 | Official leaderboard | |
| MEGA-Bench | 49.6% | #4 of 22 | Official leaderboard | |
| OCRBench v2 | 51.6% | #16 of 65 | Official leaderboard | |
| VisualPuzzles | 45% | #8 of 29 | Official leaderboard | |
| AHELM | 59.2 | #3 of 10 | Official leaderboard | |
| Video-MMMU | 53.9% | #8 of 23 | Official leaderboard | |
| VHELM | 59.6 | #17 of 41 | Official leaderboard | |
| Vision Arena | 1,179 | #90 of 136 | Official leaderboard | |
| VISTA | 37.1% | #39 of 54 | Official leaderboard | |
| ZeroBench | 2.00% | #56 of 66 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AIR-Bench 2024 | 67.3% | #39 of 82 | Official leaderboard | |
| AgentDojo | 17% | #11 of 20 | Official leaderboard | |
| FORTRESS | 53.9% | #55 of 66 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 84.2 | #10 of 71 | Official leaderboard | |
| HELM Capabilities | 65.7 | #30 of 60 | Official leaderboard |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.