- Arena Hard 37.9
- BigBench Hard CoT (0-shot) 79.1
- MMLU (5-shot) 78.9
- MMLU-Pro (0-shot, CoT) 54.3
- ARC Challenge (10-shot) 91
- BoolQ (2-shot) 84.6
- GPQA (0-shot, CoT) 36.8
- HellaSwag (5-shot) 83.8
- OpenBookQA (10-shot) 89.6
- PIQA (5-shot) 88.6
- Social IQA (5-shot) 78
- TruthfulQA (MC2) (10-shot) 77.5
- WinoGrande (5-shot) 81.3
- Multilingual MMLU (5-shot) 69.9
- MGSM (0-shot CoT) 58.7
- GSM8K (8-shot, CoT) 88.7
- MATH (0-shot, CoT) 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval (0-shot) 70.7
- MBPP (3-shot) 80.8
Phi-3.5-MoE
Phi-3.5-MoE ranks first on 2 of the 28 benchmarks it has scores on.
32
Benchmarks with a score
2
Ranked first, out of five or more models
7
In the top three
Microsoft Aug 2024 Phi-3.5-MoE
Open weights Small
No Epoch Capabilities Index score yet
- KOR-Bench 43.9% #17 /33
- Vectara Hallucination Leaderboard (HHEM) 2.50% #116 /148
- MMLU-CF 64.6% #12 /37
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 88.6% | #1 of 33 | Model card | |
| TruthfulQA | 77.5% | #2 of 28 | Model card | |
| SIQA | 78% | #3 of 22 | Model card | |
| OpenBookQA | 89.6% | #3 of 20 | Model card | |
| BoolQ | 84.6% | #7 of 28 | Model card | |
| MMLU-CF | 64.6% | #12 of 37 | Official leaderboard | |
| ARC-Challenge | 91% | #15 of 45 | Model card | |
| EQ-Bench v2 | 77.0 | #31 of 85 | Official leaderboard | |
| WinoGrande | 81.3% | #16 of 42 | Model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.23 | #2 of 4 | Model card | |
| BIG-Bench Hard | 79.1% | #36 of 71 | Model card | |
| HellaSwag | 83.8% | #26 of 47 | Model card | |
| MMLU | 78.9% | #77 of 127 | Model card | |
| RULER | 87.1% | #24 of 38 | Model card | |
| SQuALITY | 24.1% | #2 of 3 | Model card | |
| RepoQA | 85% | #3 of 4 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 2.50% | #116 of 148 | Official leaderboard | |
| MMMLU | 69.9% | #51 of 62 | Model card | |
| MGSM | 58.7% | #46 of 52 | Model card | |
| MMLU-Pro | 54.3% | #136 of 151 | Model card | |
| Arena-Hard-Auto | 37.9% | #37 of 39 | Model card | |
| Qasper | 40% | #3 of 3 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 88.7% | #43 of 87 | Model card | |
| KOR-Bench | 43.9% | #17 of 33 | Official leaderboard | |
| MATH | 59.5% | #54 of 93 | Model card | |
| GPQA Diamond | 36.8% | #264 of 284 | Model card |
Coding
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 10.5% | #1 of 5 | Model card | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 0.50% | #2 of 5 | Model card | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 10.6% | #2 of 5 | Model card | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 12% | #5 of 5 | Model card |
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 7 of them aren't in the launch post.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) 30 Aug 2024
Capabilities
- Arena-Hard-Auto 37.9
- BIG-Bench Hard 79.1
- MMLU 78.9
- MMLU-Pro 54.3
- ARC-Challenge 91
- BoolQ 84.6
- GPQA Diamond 36.8
- HellaSwag 83.8
- OpenBookQA 89.6
- PIQA 88.6
- SIQA 78
- WinoGrande 81.3
- MMMLU 69.9
- MGSM 58.7
- GSM8K 88.7
- MATH 59.5
- Qasper 40
- SQuALITY 24.1
- HumanEval 70.7
- MBPP 80.8
- RepoQA 85
- RULER 87.1