Phi-3.5-mini.
Microsoft Released 16 Aug 2024 Microsoft Aug 2024 Open weights
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| SIQA | 74.7% | #5 of 22 | Model card | |
| Qasper | 41.9% | #1 of 3 | Model card | |
| SQuALITY | 24.3% | #1 of 3 | Model card | |
| OpenBookQA | 79.2% | #7 of 20 | Model card | |
| TruthfulQA | 64% | #12 of 28 | Model card | |
| ARC-Challenge | 84.6% | #21 of 45 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 4.10% | #83 of 148 | Official leaderboard | |
| MMLU-CF | 57.9% | #22 of 37 | Official leaderboard | |
| PIQA | 81% | #22 of 33 | Model card | |
| BIG-Bench Hard | 69% | #48 of 71 | Model card | |
| BoolQ | 78% | #21 of 28 | Model card | |
| RULER | 84.1% | #29 of 38 | Model card | |
| MMLU | 69% | #98 of 127 | Model card | |
| WinoGrande | 68.5% | #36 of 42 | Model card | |
| EQ-Bench v2 | 54.7 | #74 of 85 | Official leaderboard | |
| MMMLU | 55.4% | #56 of 62 | Model card | |
| HellaSwag | 69.4% | #43 of 47 | Model card | |
| MGSM | 47.9% | #48 of 52 | Model card | |
| MMLU-Pro | 47.5% | #142 of 151 | Model card | |
| Arena-Hard-Auto | 37% | #38 of 39 | Model card | |
| RepoQA | 77% | #4 of 4 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 86.2% | #50 of 87 | Model card | |
| KOR-Bench | 39% | #21 of 33 | Official leaderboard | |
| MATH | 48.5% | #66 of 93 | Model card | |
| ZebraLogic | 6.40% | #43 of 46 | Official leaderboard | |
| GPQA Diamond | 27.2% | #281 of 284 | Model card |
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP | 68.6% | #42 of 66 | Model card | |
| BigCodeBench-Hard | 14.8% | #70 of 96 | Official leaderboard | |
| HumanEval | 61.5% | #75 of 95 | Model card | |
| BigCodeBench | 35.6% | #55 of 69 | Official leaderboard |
Every result in the longer documents the lab published with it. 23 of them aren't in the launch post.