- Hellaswag, 10-shot 92.5
- MMLU, 5-shot 81.9
- GSM8K, 11-shot 91.7
- GPQA, 0-shot (4-shot = 37.9%) 41.5
- MATH, 7-shot (4-shot Minerva prompt = 58.5%) 59.4
- PhysicsFinals, 0-shot 60.7
- AMC 2022-23, 4-shot 37.2
- BigBench - Hard, 3-shot 84
- DROP, Variable shots (F1) 78.9
- HumanEval chat preamble, 0-shot 71.9
- Natural2Code chat preamble, 0-shot 77.7
- WMT23, 1-shot (BLEURT) 75.2
- MGSM, 8-shot 88.7
- MMMU (val), 0-shot 58.5
- Ai2D (test), 0-shot 80.3
- MathViSTA (testmini), 0-shot 52.1
- ChartQA (test), 0-shot 81.3
- VQAv2 (test-dev), 0-shot 73.2
- TextVQA (val), 0-shot 73.5
- DocVQA (test), 0-shot (ANLS) 86.5
- InfographicVQA (test), 0-shot (ANLS) 72.7
- VATEX (test), 4-shot (CIDEr) 63
- VATEX ZH (val), 4-shot (CIDEr) 54.9
- YouCook2 (val), 4-shot (CIDEr) 134.2
- ActivityNet-QA (test), 0-shot 56.7
- EgoSchema (test), 0-shot 63.2
Gemini 1.5 Pro Feb 2024
Gemini 1.5 Pro places in the top three on Video-MME.
33
Benchmarks with a score
0
Ranked first, out of five or more models
1
In the top three
Google DeepMind Feb 2024 Gemini 1.5 Pro
Feb 2024 snapshot Closed weights Flagship
No Epoch Capabilities Index score yet
- StructEval 71.8% #19 /28
- TextVQA 73.5% #13 /17
- EgoSchema 63.2% #12 /12
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HellaSwag | 92.5% | #5 of 47 | Model card | |
| MGSM | 88.7% | #15 of 52 | Model card | |
| BIG-Bench Hard | 84% | #25 of 71 | Model card | |
| MMLU | 81.9% | #64 of 127 | Model card | |
| SEAL Multilingual | 1,005 | #12 of 19 | Official leaderboard | |
| WMT23 | 75.2 | #2 of 3 | Model card | |
| DROP | 78.9% | #27 of 40 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91.7% | #31 of 87 | Model card | |
| MATH | 59.4% | #55 of 93 | Model card | |
| GSM1k | 90.5% | #10 of 15 | Official leaderboard | |
| GPQA Diamond | 41.5% | #252 of 284 | Model card | |
| AMC | 37.2% | #4 of 4 | Model card | |
| PhysicsFinals | 60.7% | #2 of 2 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Natural2Code | 77.7% | #6 of 9 | Model card | |
| StructEval | 71.8% | #19 of 28 | Official leaderboard | |
| HumanEval | 71.9% | #66 of 95 | Model card | |
| SEAL Coding | 891 | #21 of 26 | Official leaderboard |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LisanBench | 492.0 | #68 of 133 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Video-MME | 75% | #2 of 50 | Epoch AI | |
| ActivityNet-QA | 56.7% | #5 of 8 | Model card | |
| VQAv2 | 73.2% | #5 of 8 | Model card | |
| InfographicVQA | 72.7% | #15 of 22 | Model card | |
| VATEX ZH | 54.9 | #3 of 4 | Model card | |
| YouCook2 | 134.2 | #3 of 4 | Model card | |
| ChartQA | 81.3% | #28 of 37 | Model card | |
| TextVQA | 73.5% | #13 of 17 | Model card | |
| MMMU | 58.5% | #84 of 97 | Model card | |
| AI2D | 80.3% | #39 of 44 | Model card | |
| MathVista | 52.1% | #58 of 64 | Model card | |
| DocVQA | 86.5% | #32 of 33 | Model card | |
| EgoSchema | 63.2% | #12 of 12 | Model card | |
| VATEX | 63.0 | #4 of 4 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Cybench | 7.50% | #18 of 21 | Epoch AI |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.