Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) 30 Aug 2024
Capabilities
Hallucination
- POPE 86.1
Safety
- Microsoft internal multimodal RAI benchmark 8.16
- RTVLM 5.44
- VLGuard 9.1
Phi-3.5-vision-instruct
Phi-3.5-vision-instruct ranks first on ScienceQA.
Released 16 Aug 2024
29
Benchmarks with a score
1
Ranked first, out of five or more models
1
In the top three
Unknown lab Aug 2024
Phi-3.5-vision-instruct
No Epoch Capabilities Index score yet
- BLINK 57% #22 /28
- HumanEval-V 0% #24 /24
- TextVQA 72% #14 /17
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MMBench-CN v1.1 | 62.8% | #137 of 178 | Official leaderboard | |
| POPE | 86.1% | #6 of 7 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval-V | 0% | #24 of 24 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| ScienceQA | 91.3% | #1 of 23 | Epoch AI | |
| CharXiv | 32.7% | #38 of 77 | Official leaderboard | |
| MMBench | 76% | #77 of 155 | Official leaderboard | |
| MMBench-EN v1.1 | 72.1% | #104 of 178 | Official leaderboard | |
| MMStar | 47.5% | #113 of 173 | Official leaderboard | |
| ChartQA | 81.8% | #25 of 37 | Model card | |
| VCR (Visual Caption Restoration) | 18.1% | #30 of 42 | Official leaderboard | |
| MEGA-Bench | 25.4% | #16 of 22 | Official leaderboard | |
| MMVU | 38.1% | #21 of 28 | Official leaderboard | |
| BLINK | 57% | #22 of 28 | Model card | |
| MMMU | 43% | #108 of 136 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 27.6% | #121 of 152 | Official leaderboard | |
| TextVQA | 72% | #14 of 17 | Model card | |
| TOMATO | 20.7% | #19 of 22 | Official leaderboard | |
| KRETA | 42.6% | #15 of 17 | Official leaderboard | |
| MMMU-Pro | 19.7% | #52 of 57 | Official leaderboard | |
| AI2D | 78.1% | #42 of 44 | Model card | |
| Vision Arena | 919 | #135 of 136 | Official leaderboard | |
| MathVista | 43.9% | #64 of 64 | Model card | |
| VISTA | 15.2% | #54 of 54 | Official leaderboard | |
| Video-MME | 50.8% | #42 of 42 | Model card | |
| Inter-GPS (Geometry3K) | 36.3% | #2 of 2 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft internal multimodal RAI benchmark | 8.16 | #1 of 2 | Model card | |
| VLGuard | 9.10 | #1 of 2 | Model card | |
| RTVLM | 5.44 | #2 of 2 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| OpenVLM Leaderboard | 53.0 | #117 of 173 | Official leaderboard |
Its system card
Every result in the longer documents the lab published with it. 14 of them aren't in the launch post.