GPT-4o mini
GPT-4o mini ranks first on PIQA.
121
Benchmarks with a score
1
Ranked first, out of five or more models
2
In the top three
OpenAI Jul 2024
GPT-4o mini
Closed weights Small
126.5
#178 of 271
Epoch Capabilities Index
- ZeroBench 2.00% #54 /66
- OCRBench v2 44.1% #31 /65
- SimpleQA Verified 9.06% #48 /56
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| PIQA | 88.7% | #1 of 58 | Epoch AI | |
| WildBench | 57.3 | #3 of 47 | Official leaderboard | |
| AlpacaEval 2.0 | 50.7% | #4 of 40 | Official leaderboard | |
| BoolQ | 88.7% | #8 of 75 | Epoch AI | |
| MixEval | 84.2% | #9 of 34 | Official leaderboard | |
| MMLU-CF | 65.5% | #10 of 37 | Official leaderboard | |
| Creation-MMBench | -13.56 | #7 of 19 | Official leaderboard | |
| EQ-Bench v2 | 76.9 | #32 of 85 | Official leaderboard | |
| MGSM | 87% | #21 of 52 | Model card | |
| MMBench-CN v1.1 | 75% | #79 of 178 | Official leaderboard | |
| MMLU | 82% | #62 of 127 | Model card | |
| MMLU-Pro | 63.1% | #106 of 212 | Official leaderboard | |
| ThaiExam | 55.7% | #12 of 24 | Official leaderboard | |
| Chinese SimpleQA | 37.8% | #14 of 27 | Official leaderboard | |
| FaithJudge | 18.6% | #20 of 37 | Official leaderboard | |
| Wer wird Millionär? LLM Benchmark | 1267.0 | #47 of 76 | Official leaderboard | |
| RewardBench | 80.1% | #30 of 48 | Official leaderboard | |
| DROP | 79.7% | #25 of 40 | Model card | |
| LMArena (Elo) | 1,318 | #210 of 333 | Official leaderboard | |
| ToRR | 43.1 | #9 of 14 | Official leaderboard | |
| ECLeKTic | 12.4% | #23 of 35 | Official leaderboard | |
| WritingBench | 56.4 | #34 of 45 | Official leaderboard | |
| RewardBench 2 | 58% | #33 of 42 | Official leaderboard | |
| FACTS Grounding | 68.4% | #27 of 34 | Official leaderboard | |
| Creative Writing v3 | 51.9 | #100 of 120 | Official leaderboard | |
| SimpleQA Verified | 9.06% | #48 of 56 | Official leaderboard | |
| Lech Mazur Creative Writing | 6.72 | #38 of 44 | Epoch AI | |
| Phare | 58.3% | #56 of 62 | Official leaderboard | |
| LongBench v2 | 32.4% | #30 of 33 | Official leaderboard | |
| Vectara Hallucination Leaderboard (HHEM) | 1.70% | #135 of 148 | Official leaderboard | |
| BullshitBench | 7.27% | #101 of 107 | Official leaderboard | |
| LLM Confabulation Leaderboard | 37.2% | #57 of 58 | Official leaderboard | |
| BullshitBench V2 | 2.00% | #121 of 122 | Official leaderboard | |
| QuALITY | 71.7% | #4 of 4 | Official leaderboard |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| GSM8K | 91.3% | #5 of 82 | Epoch AI | |
| OlympicArena | 34.1% | #6 of 25 | Official leaderboard | |
| ZebraLogic | 20.1% | #20 of 46 | Official leaderboard | |
| MATH | 70.2% | #41 of 93 | Model card | |
| MATH Level 5 | 52.6% | #48 of 94 | Epoch AI | |
| U-MATH | 43.4% | #19 of 30 | Official leaderboard | |
| PutnamBench | 0% | #10 of 14 | Official leaderboard | |
| OTIS Mock AIME 2024-2025 | 6.94% | #161 of 202 | Epoch AI | |
| GPQA Diamond | 37.7% | #181 of 219 | Epoch AI | |
| DTBench | 24% | #151 of 178 | Official leaderboard | |
| ARC-AGI-2 | 0% | #87 of 95 | Official leaderboard | |
| FrontierMath Tiers 1-3 v2 | 0.70% | #87 of 89 | Epoch AI | |
| SimpleBench | 10.7% | #99 of 100 | Official leaderboard |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HumanEval+ | 83.5% | #7 of 46 | Official leaderboard | |
| BigCodeBench | 51.8% | #14 of 69 | Official leaderboard | |
| MHPP | 42.4% | #7 of 23 | Official leaderboard | |
| MBPP+ | 72.2% | #11 of 35 | Official leaderboard | |
| HumanEval | 87.2% | #30 of 95 | Model card | |
| HumanEval-V | 15.8% | #8 of 24 | Official leaderboard | |
| BigCodeBench-Hard | 25.3% | #47 of 96 | Official leaderboard | |
| StructEval | 73.2% | #16 of 28 | Official leaderboard | |
| Aider-Edit | 55.6% | #27 of 37 | Official leaderboard | |
| SWT-bench | 9.80% | #7 of 9 | Official leaderboard | |
| LiveCodeBench v6 | 27.5% | #19 of 21 | Official leaderboard | |
| LiveCodeBench v5 | 27.7% | #20 of 21 | Official leaderboard | |
| WeirdML | 11.8% | #109 of 112 | Official leaderboard | |
| Aider Polyglot | 3.60% | #54 of 54 | Epoch AI |
Agents and tools
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) | 64% | #18 of 67 | Official leaderboard | |
| AceBench | 72.5% | #7 of 16 | Official leaderboard | |
| Galileo Agent Leaderboard | 0.83 | #20 of 31 | Official leaderboard | |
| ToolComp | 51.7% | #22 of 32 | Official leaderboard | |
| PinchBench | 64.4% | #34 of 48 | Official leaderboard | |
| SnakeBench | 20 | #134 of 183 | Official leaderboard | |
| GAIA | 26.9% | #42 of 57 | Official leaderboard | |
| BALROG | 17.4% | #26 of 35 | Official leaderboard | |
| Elimination Game | 3 | #43 of 56 | Official leaderboard | |
| CRMArena | 28.3% | #7 of 9 | Official leaderboard | |
| Mystery Game Puzzles | 12% | #63 of 80 | Epoch AI | |
| LLM Chess | -252 | #99 of 124 | Official leaderboard | |
| LisanBench | 89.7 | #114 of 133 | Official leaderboard | |
| Chess Puzzles | 3.00% | #80 of 83 | Epoch AI | |
| Step Race | 1 | #57 of 59 | Official leaderboard | |
| VitaBench 2.0 | 6.70% | #20 of 20 | Official leaderboard | |
| Factorio Learning Environment | 26756.0 | #6 of 6 | Official leaderboard |
Professional work
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| LEXam | 42.5% | #20 of 31 | Official leaderboard | |
| MortgageTax | 54.5% | #71 of 86 | Official leaderboard | |
| Vals Finance Agent v1 | 10.8% | #39 of 47 | Official leaderboard | |
| MedHELM | 33.3 | #11 of 13 | Official leaderboard | |
| CorpFin v2 | 45.5% | #104 of 119 | Official leaderboard | |
| TaxEval v2 | 60.5% | #114 of 129 | Official leaderboard | |
| MedXpertQA-MM | 28.1% | #9 of 9 | Official leaderboard |
Vision, audio and robotics
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| VCR (Visual Caption Restoration) | 42% | #7 of 42 | Official leaderboard | |
| KRETA | 73.3% | #4 of 17 | Official leaderboard | |
| CCBench (MMBench Chinese Culture) | 61.4% | #38 of 152 | Official leaderboard | |
| MMVU | 61.6% | #8 of 28 | Official leaderboard | |
| MEGA-Bench | 43.1% | #7 of 22 | Official leaderboard | |
| TOMATO | 28.8% | #7 of 22 | Official leaderboard | |
| Image2Struct | 70.2 | #6 of 18 | Official leaderboard | |
| MTVQA-Test | 25.5% | #6 of 18 | Official leaderboard | |
| MMBench-EN v1.1 | 77.1% | #77 of 178 | Official leaderboard | |
| CharXiv | 34.1% | #34 of 77 | Official leaderboard | |
| MMMU | 59.4% | #61 of 136 | Official leaderboard | |
| Video-MME | 68.9% | #12 of 26 | Official leaderboard | |
| MMBench | 77.6% | #73 of 155 | Official leaderboard | |
| OCRBench v2 | 44.1% | #31 of 65 | Official leaderboard | |
| MMStar | 54.8% | #86 of 173 | Official leaderboard | |
| VHELM | 46.5 | #26 of 41 | Official leaderboard | |
| GeoBench | 64% | #31 of 48 | Official leaderboard | |
| MMMU-Pro | 37.6% | #37 of 57 | Official leaderboard | |
| ZeroBench | 2.00% | #54 of 66 | Official leaderboard | |
| Vision Arena | 1,097 | #116 of 136 | Official leaderboard | |
| VPCT | 34% | #24 of 28 | Epoch AI | |
| MathVista | 56.7% | #55 of 64 | Model card |
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Safety | 93.0 | #32 of 75 | Official leaderboard | |
| Anthropic Red Team | 98.3% | #49 of 75 | Official leaderboard | |
| SimpleSafetyTests | 97.8% | #49 of 75 | Official leaderboard | |
| AIR-Bench 2024 | 56.3% | #57 of 82 | Official leaderboard | |
| FORTRESS | 48.1% | #50 of 66 | Official leaderboard | |
| AgentDojo | 27.2% | #16 of 20 | Official leaderboard | |
| Human Pathogen Capabilities Test v2 (HPCT v2) | 13.1% | #71 of 71 | Official leaderboard |
Other
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| HELM Lite | 70.1 | #23 of 71 | Official leaderboard | |
| OpenVLM Leaderboard | 64.1 | #64 of 173 | Official leaderboard | |
| LiveBench | 41.3% | #38 of 48 | Epoch AI | |
| HELM Capabilities | 56.5 | #48 of 60 | Official leaderboard | |
| LMCA | 10.4% | #124 of 139 | Epoch AI | |
| Conceptual Reasoning Index | 17.5 | #136 of 149 | Official leaderboard | |
| ACCoRD | 32.6 | #141 of 150 | Official leaderboard |
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.