Gemini 1.5 Flash Sep 2024
Gemini 1.5 Flash ranks first on OVBench.
Google DeepMind Released 27 Aug 2024 51
Benchmarks with a score
1
Ranked first, out of five or more models
3
In the top three
Google DeepMind Aug 2024 Gemini 1.5 Flash
Sep 2024 snapshot Closed weights
129.3
#164 of 271
Epoch Capabilities Index
- ZeroBench 2.00% #55 /66
- GPQA Diamond 47.3% #158 /219
- FrontierMath 0% #69 /70
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 87.5% | #3 of 58 | Epoch AI | |
| MMBench-CN v1.1 | 81% | #38 of 178 | Official leaderboard | |
| FACTS Grounding | 82.3% | #8 of 34 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 6.60% | #42 of 148 | Official leaderboard | |
| MMLU-CF | 64.8% | #11 of 37 | Official leaderboard | |
| MMLU | 73.9% | #44 of 127 | Epoch AI | |
| MMLU-Pro | 64.1% | #102 of 212 | Official leaderboard | |
| ToRR | 45.4 | #7 of 14 | Official leaderboard | |
| LMArena (Elo) | 1,309 | #218 of 333 | Official leaderboard | |
| ECLeKTic | 11.6% | #24 of 35 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ForecastBench | 50.4 | #4 of 86 | Official leaderboard | |
| U-MATH | 57.8% | #11 of 30 | Official leaderboard | |
| MATH Level 5 | 61.9% | #43 of 94 | Epoch AI | |
| GPQA Diamond | 47.3% | #158 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 16.3% | #147 of 202 | Epoch AI | |
| DTBench | 53.8% | #148 of 177 | Epoch AI | |
| FrontierMath | 0% | #69 of 70 | Epoch AI |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval-V | 17.4% | #6 of 24 | Official leaderboard | |
| HumanEval+ | 75.6% | #15 of 46 | Official leaderboard | |
| MBPP+ | 67.5% | #17 of 35 | Official leaderboard | |
| WeirdML | 24.9% | #111 of 130 | Epoch AI | |
| Aider-Edit | 52.6% | #32 of 37 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Galileo Agent Leaderboard | 0.90 | #11 of 31 | Official leaderboard | |
| BALROG | 14.6% | #30 of 35 | Official leaderboard | |
| Step Race | 1 | #54 of 59 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LegalBench | 63.2% | #126 of 138 | Official leaderboard | |
| MortgageTax | 42.8% | #79 of 86 | Official leaderboard | |
| CorpFin v2 | 38.2% | #114 of 119 | Official leaderboard | |
| TaxEval v2 | 48.2% | #125 of 129 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OVBench | 50.7% | #1 of 7 | Official leaderboard | |
| MMBench-EN v1.1 | 81.6% | #42 of 178 | Official leaderboard | |
| MEGA-Bench | 43.8% | #6 of 22 | Official leaderboard | |
| Image2Struct | 71.0 | #5 of 18 | Official leaderboard | |
| MMVU | 58.8% | #9 of 28 | Official leaderboard | |
| GeoBench | 76% | #18 of 48 | Official leaderboard | |
| MMBench | 79.4% | #60 of 155 | Official leaderboard | |
| A-Bench | 73.6% | #46 of 116 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 52.4% | #62 of 152 | Official leaderboard | |
| Video-MMMU | 49.8% | #10 of 23 | Official leaderboard | |
| MMStar | 55.8% | #77 of 173 | Official leaderboard | |
| VHELM | 51.9 | #21 of 41 | Official leaderboard | |
| AHELM | 40.1 | #7 of 10 | Official leaderboard | |
| Vision Arena | 1,142 | #104 of 136 | Official leaderboard | |
| VISTA | 34% | #43 of 54 | Official leaderboard | |
| ZeroBench | 2.00% | #55 of 66 | Official leaderboard |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| AgentDojo | 3.50% | #3 of 20 | Official leaderboard | |
| AIR-Bench 2024 | 67.1% | #40 of 82 | Official leaderboard | |
| FORTRESS | 50.6% | #51 of 66 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 63.5 | #69 of 173 | Official leaderboard | |
| HELM Lite | 57.4 | #29 of 71 | Official leaderboard | |
| HELM Capabilities | 60.9 | #41 of 60 | Official leaderboard |
Google DeepMind's models over time
Google DeepMind's models in color, against every other model on the Epoch Capabilities Index, by release date.