Learning to Reason with LLMs (Appendix A) 12 Sep 2024
o1-preview
o1-preview ranks first on 7 of the 57 benchmarks it has scores on.
64
Benchmarks with a score
7
Ranked first, out of five or more models
15
In the top three
OpenAI Sep 2024
o1-preview
Closed weights Flagship
134.6
#149 of 271
Epoch Capabilities Index
- GPQA Diamond 50.3% #145 /219
- KOR-Bench 72.9% #1 /33
- Vectara Hallucination Leaderboard (HHEM) 3.30% #98 /148
Where it ranks
Its best score on each benchmark, and where that ranks among every model with a score there. We rank each model once, at its best setting.
Knowledge and language
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Chinese SimpleQA | 67.9% | #1 of 27 | Official leaderboard | |
| MMLU | 90.8% | #6 of 127 | Model card | |
| SEAL Multilingual | 1,111 | #3 of 19 | Official leaderboard | |
| SEAL Precise Instruction Following | 86.6 | #3 of 18 | Official leaderboard | |
| LongBench v2 | 57.7% | #6 of 33 | Official leaderboard | |
| LLM Confabulation Leaderboard | 13% | #12 of 58 | Official leaderboard | |
| EQ-Bench v2 | 80.5 | #21 of 85 | Official leaderboard | |
| SimpleQA | 42% | #19 of 59 | Model card | |
| LMArena (Elo) | 1,389 | #142 of 333 | Official leaderboard | |
| SimpleQA Hallucination Rate | 44% | #6 of 12 | Model card | |
| Open Ended Questions (hallucination) | 0.78 | #1 of 2 | Model card | |
| Vectara Hallucination Leaderboard (HHEM) | 3.30% | #98 of 148 | Official leaderboard | |
| WritingBench | 63.4 | #30 of 45 | Official leaderboard | |
| BirthdayFacts | 32% | #2 of 2 | Model card |
Math and science
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| KOR-Bench | 72.9% | #1 of 33 | Official leaderboard | |
| ZebraLogic | 71.4% | #4 of 46 | Official leaderboard | |
| Omni-MATH | 52.5% | #2 of 9 | Official leaderboard | |
| MATH Level 5 | 81.7% | #27 of 94 | Epoch AI | |
| SimpleBench | 41.7% | #63 of 100 | Official leaderboard | |
| GPQA Diamond | 50.3% | #145 of 219 | Epoch AI | |
| OTIS Mock AIME 2024-2025 | 31.1% | #134 of 202 | Epoch AI | |
| AIME 2024 | 56.7% | #71 of 87 | Model card | |
| MATH-500 | 85.5% | #55 of 67 | Model card | |
| ARC-AGI | 18% | #82 of 96 | Epoch AI | |
| QuantBench | 36% | #3 of 3 | Model card |
Coding
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| MBPP+ | 80.2% | #1 of 35 | Official leaderboard | |
| HumanEval+ | 89% | #2 of 46 | Official leaderboard | |
| MHPP | 68.1% | #1 of 23 | Official leaderboard | |
| Aider-Edit | 79.7% | #3 of 37 | Official leaderboard | |
| SciCode | 7.70% | #2 of 18 | Official leaderboard | |
| BIRD-CRITIC | 35.5% | #1 of 9 | Official leaderboard | |
| SEAL Coding | 1,123 | #4 of 26 | Official leaderboard | |
| BigCodeBench-Hard | 28.8% | #24 of 96 | Official leaderboard | |
| SWE-bench Verified | 64.6% | #23 of 60 | Official leaderboard | |
| WeirdML | 47.6% | #53 of 112 | Official leaderboard | |
| METR Time Horizon | 23 min | #18 of 31 | Official leaderboard | |
| METR Time Horizon 1.1 | 20 min | #17 of 26 | Official leaderboard | |
| MLE-bench | 22.4% | #9 of 12 | Official leaderboard | |
| OpenAI Research Engineer Interview: Multiple Choice | 76.8% | #8 of 9 | Model card | |
| Codeforces rating | 1,258 | #39 of 41 | Model card |
Agents and tools
Safety and security
| Benchmark | Score | Rank | Source | |
|---|---|---|---|---|
| Situational Awareness Dataset (SAD) | 61.9% | #1 of 17 | Official leaderboard | |
| Challenging Refusal Evaluation | 93.4% | #1 of 9 | Model card | |
| Political Persuasion Parallel Generation | 53.8% | #1 of 4 | Model card | |
| MakeMeSay | 49% | #3 of 9 | Model card | |
| Jailbreak Augmented Examples | 100% | #2 of 5 | Model card | |
| Standard Refusal Evaluation | 99.5% | #4 of 8 | Model card | |
| External red team jailbreaks | 44% | #1 of 2 | Model card | |
| LAB-Bench ProtocolQA | 81% | #8 of 15 | Model card | |
| StrongREJECT | 84% | #12 of 22 | Model card | |
| MakeMePay | 11.6% | #4 of 6 | Model card | |
| Human Sourced Jailbreaks | 96% | #6 of 8 | Model card | |
| WildChat (toxic conversations) | 97.1% | #3 of 4 | Model card | |
| Capture the Flag (High School) | 43% | #7 of 9 | Model card | |
| Tacit Knowledge and Troubleshooting | 66.7% | #23 of 29 | Model card | |
| ChangeMyView | 81.8% | #4 of 5 | Model card | |
| Production Jailbreaks | 99% | #4 of 5 | Model card | |
| Capture the Flag (Professional) | 7.30% | #17 of 21 | Model card | |
| Cybench | 10% | #20 of 24 | Official leaderboard | |
| XSTest | 97.6% | #16 of 16 | Model card | |
| Capture the Flag (Collegiate) | 0% | #13 of 13 | Model card | |
| LAB-Bench CloningScenarios | 36.4% | #11 of 11 | Model card | |
| CoT Deception Monitoring | 0.79% | #2 of 2 | Model card |
OpenAI's models over time
OpenAI's models in color, against every other model on the Epoch Capabilities Index, by release date.
Its launch post
The benchmarks the lab chose to headline, with the scores it gave.
Its system card
Every result in the longer documents the lab published with it. 39 of them aren't in the launch post.
OpenAI o1 System Card (o1-preview and o1-mini) 12 Sep 2024
Capabilities
- MMMLU
- QuantBench 36
Hallucination
Safety
Robustness
Bio and chem risk
Cyber risk
AI R&D risk
Autonomy
Persuasion
- ChangeMyView 81.8
- Political Persuasion Parallel Generation 53.8
- MakeMePay 11.6
- MakeMeSay 49