phi-3-mini 3.8B ranks first on Microsoft RAI benchmark: Harmful Content Summarization (DR-3).
Microsoft Released 23 Apr 2024 Microsoft Apr 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SIQA | 76.6% | #4 of 22 | Model card | |
| ARC-Easy | 94.6% | #5 of 23 | Model card | |
| PIQA | 84.2% | #9 of 33 | Model card | |
| OpenBookQA | 83.2% | #6 of 20 | Model card | |
| TruthfulQA | 65% | #11 of 28 | Model card | |
| CommonSenseQA | 80.2% | #7 of 17 | Model card | |
| ARC-Challenge | 84.9% | #20 of 45 | Model card | |
| ANLI | 52.8% | #6 of 11 | Epoch AI | |
| Vectara Hallucination Leaderboard (HHEM) | 4.00% | #85 of 148 | Official leaderboard | |
| BIG-Bench Hard | 71.7% | #43 of 71 | Model card | |
| MMLU-Pro | 45.7% | #143 of 212 | Official leaderboard | |
| MT-Bench | 8.38 | #17 of 23 | Model card | |
| MMLU | 68.8% | #100 of 127 | Model card | |
| WinoGrande | 70.8% | #34 of 42 | Model card | |
| BoolQ | 77.2% | #23 of 28 | Model card | |
| EQ-Bench v2 | 58.1 | #70 of 85 | Official leaderboard | |
| TriviaQA | 64% | #33 of 39 | Model card | |
| HellaSwag | 76.7% | #40 of 47 | Model card | |
| AGIEval | 37.5% | #32 of 35 | Model card | |
| LMArena (Elo) | 1,128 | #315 of 333 | Official leaderboard | |
| MMMLU | 47.3% | #59 of 62 | Model card | |
| Microsoft RAI benchmark: Ungroundedness | 0.60 | #4 of 4 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 82.5% | #56 of 87 | Model card | |
| ZebraLogic | 11.6% | #34 of 46 | Official leaderboard | |
| MATH | 41.3% | #73 of 93 | Model card | |
| GPQA Diamond | 32.8% | #272 of 284 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 70% | #40 of 66 | Model card | |
| HumanEval+ | 59.1% | #28 of 46 | Official leaderboard | |
| MHPP | 19.5% | #17 of 23 | Official leaderboard | |
| MBPP+ | 54.2% | #27 of 35 | Official leaderboard | |
| HumanEval | 58.5% | #78 of 95 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MedQA | 53.8% | #4 of 4 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) | 10% | #1 of 5 | Model card | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) | 0.70% | #3 of 5 | Model card | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) | 24% | #3 of 5 | Model card | |
| Microsoft RAI benchmark: Jailbreak (DR-1) | 12.3% | #5 of 5 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LiveBench | 22.4% | #47 of 48 | Epoch AI |
Microsoft's models in color, against every other model on the Epoch Capabilities Index, by release date.
Every result in the longer documents the lab published with it. 28 of them aren't in the launch post.