- MMLU (5-Shot) 78.2
- HellaSwag (5-Shot) 83
- ANLI (7-Shot) 58.7
- GSM-8K (0-Shot; CoT) 90.3
- MedQA (2-Shot) 69.4
- AGIEval (0-Shot) 48.4
- TriviaQA (5-Shot) 75.6
- Arc-C (10-Shot) 91
- Arc-E (10-Shot) 97.8
- PIQA (5-Shot) 87.7
- SociQA (5-Shot) 80.2
- BigBench-Hard (0-Shot) 81.3
- WinoGrande (5-Shot) 81.4
- OpenBookQA (10-Shot) 87.2
- BoolQ (0-Shot) 86.6
- CommonSenseQA (10-Shot) 82.6
- TruthfulQA (10-Shot) 75.7
- HumanEval (0-Shot) 55.5
- MBPP (3-Shot) 74.5
- GPQA (2-Shot; CoT)
- MT Bench (2 round ave.) 8.91
phi-3-medium 14B
phi-3-medium 14B ranks first on SIQA.
33
Benchmarks with a score
1
Ranked first, out of five or more models
5
In the top three
Microsoft Apr 2024 phi-3-medium 14B
Open weights Small
121.2
#193 of 271
Epoch Capabilities Index
- GPQA Diamond 27.6% #203 /219
- MHPP 21% #16 /23
- SIQA 80.2% #1 /22
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SIQA | 80.2% | #1 of 22 | Model card | |
| PIQA | 87.9% | #2 of 33 | Model card | |
| BoolQ | 86.6% | #3 of 28 | Model card | |
| TruthfulQA | 75.7% | #3 of 28 | Model card | |
| ARC-Easy | 97.8% | #3 of 23 | Model card | |
| OpenBookQA | 87.4% | #5 of 20 | Model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.21 | #1 of 4 | Model card | |
| ARC-Challenge | 91.6% | #12 of 45 | Model card | |
| WinoGrande | 81.5% | #13 of 42 | Model card | |
| MT-Bench | 8.91 | #8 of 23 | Model card | |
| CommonSenseQA | 82.8% | #6 of 17 | Model card | |
| ANLI | 55.8% | #4 of 11 | Epoch AI | |
| EQ-Bench v2 | 76.3 | #35 of 85 | Official leaderboard | |
| BIG-Bench Hard | 81.4% | #33 of 71 | Model card | |
| MMLU-Pro | 55.7% | #124 of 212 | Official leaderboard | |
| HellaSwag | 83% | #28 of 47 | Model card | |
| AGIEval | 50.2% | #21 of 35 | Model card | |
| MMLU | 78.2% | #81 of 127 | Model card | |
| TriviaQA | 75.6% | #25 of 39 | Model card | |
| LMArena (Elo) | 1,198 | #283 of 333 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91% | #36 of 87 | Model card | |
| MATH | 53.1% | #59 of 93 | Model card | |
| MATH Level 5 | 17.6% | #79 of 94 | Epoch AI | |
| GPQA Diamond | 27.6% | #203 of 219 | Epoch AI |
Coding
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 69.9% | #2 of 4 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 1.00% | #4 of 5 | Model card | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 11.2% | #4 of 5 | Model card | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 11.1% | #4 of 5 | Model card | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 25.1% | #4 of 5 | Model card |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 50.9 | #38 of 71 | Official leaderboard |
Microsoft's models over time
Microsoft's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 6 of them aren't in the launch post.
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (arXiv v4) 30 Aug 2024
Capabilities
- MMLU 78
- HellaSwag 82.4
- ANLI 55.8
- GSM8K 91
- MATH 53.1
- MedQA 69.9
- AGIEval 50.2
- TriviaQA 73.9
- ARC-Challenge 91.6
- ARC-Easy 97.7
- PIQA 87.9
- SIQA 80.2
- BIG-Bench Hard 81.4
- WinoGrande 81.5
- OpenBookQA 87.4
- BoolQ 86.5
- CommonSenseQA 82.8
- HumanEval 62.2
- MBPP 75.2
- MT-Bench 8.91