MLE-bench
Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards.
OpenAI runs an internal Revised version with 72 problems. The public leaderboard in the repo README stopped taking submissions on 2026-04-24. MLE-bench Lite is the 22-competition Low split.
In the authors’ words
We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup–OpenAI's o1-preview with AIDE scaffolding–achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training. We open-source our benchmark code (github.com/openai/mle-bench/) to facilitate future research in understanding the ML engineering capabilities of AI agents.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
GPT-6 Astra leads with 93.8% among models released by Sep 2026
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Sep 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- Claude Sonnet 5.5
- and 14 more
- 93.8 GPT-6 Astra
- 66.6
MiniMax-M2.7 - 63.9
Gemini 3.6 Flash - 51.5 MiniMax-M2.5
- 40.0 MiniMax-M2
- 39.2 Gemini 3.5 Flash-Lite
- 36.7 GPT-5.5
- 23.3 GPT-5.4
- 17.0 GPT-5.1-Codex-Max
- 16.0 GPT-5.2
- 15.8
Muse Spark - 14.0 o1
- 11.0 o3-mini
- 11.0 OpenAI deep research
- 11.0 GPT-4.5
- 10.0 GPT-5.2 Codex
- 8.00 GPT-5
- 3.00 GPT-5 mini
- 1 GPT-6 Astra 93.8%
- 2 MiniMax-M2.7 66.6%
- 3 Gemini 3.6 Flash 63.9%
- 4 MiniMax-M2.5 51.5%
- 5 MiniMax-M2 40%
- 6 Gemini 3.5 Flash-Lite 39.2%
- 7 GPT-5.5 36.7%
- 8 GPT-5.4 23.3%
- 9 GPT-5.1-Codex-Max 17%
- 10 GPT-5.2 16%
- 11 Muse Spark 15.8%
- 12 o1 14%
- 13 o3-mini 11%
- 14 OpenAI deep research 11%
- 15 GPT-4.5 11%
- 16 GPT-5.2 Codex 10%
- 17 GPT-5 8.00%
- 18 GPT-5 mini 3.00%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra | 93.8% | Sep 2026 | Model card | |
| 2 | MiniMax-M2.7 | 66.6% | Mar 2026 | Model card | |
| 3 | Gemini 3.6 Flash | 63.9% | Jul 2026 | Model card | |
| 4 | MiniMax-M2.5 | 51.5% | May 2026 | Model card | |
| 5 | MiniMax-M2 | 40% | May 2026 | Model card | |
| 6 | Gemini 3.5 Flash-Lite | 39.2% | Jul 2026 | Model card | |
| 7 | GPT-5.5 | 36.7% | Apr 2026 | Model card | |
| 8 | GPT-5.4 | 23.3% | Mar 2026 | Model card | |
| 9 | GPT-5.1-Codex-Max | 17% | Nov 2025 | Model card | |
| 10 | GPT-5.2 | 16% | Dec 2025 | Model card | |
| 11 | Muse Spark | 15.8% | Jun 2026 | Model card | |
| 12 | o1 | 14% | Dec 2024 | Model card | |
| 13 | o3-mini | 11% | Jan 2025 | Model card | |
| 14 | OpenAI deep research | 11% | Feb 2025 | Model card | |
| 15 | GPT-4.5 | 11% | Feb 2025 | Model card | |
| 16 | GPT-5.2 Codex | 10% | Dec 2025 | Model card | |
| 17 | GPT-5 | 8.00% | Aug 2025 | Model card | |
| 18 | GPT-5 mini | 3.00% | Aug 2025 | Model card |
The benchmark's own leaderboard has 12 models. We plot every source in the charts below.
5 models have held number one since its release in Oct 2024
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Sep 2026
All the scores we have, one dot each, so a model tested at several settings appears more than once.
One lab put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 21 Jul 2026 | Gemini 3.5 Flash-Lite | 39.2 | MLE-Bench Machine Learning Engineering |
| 21 Jul 2026 | Gemini 3.6 Flash | 63.9 | MLE-Bench: Machine Learning Engineering |
| 18 Mar 2026 | MiniMax-M2.7 | 66.6 | MLE-Bench lite (medal rate) |
24 system cards report it, from 4 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 3 Sep 2026 | GPT-6 Astra system card | 93.8 | MLE-Bench Revised (72 problems), mean reward, max effort, printed chart label (Figure 56) | AI R&D risk |
| 21 Jul 2026 | Gemini 3.6 Flash model card | 63.9 | MLE-Bench (results table, p.4) | Capabilities |
| 21 Jul 2026 | Gemini 3.5 Flash-Lite model card | 39.2 | MLE-Bench (results table, p.4) | Capabilities |
| 9 Jul 2026 | GPT-5.6 Terra system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 9 Jul 2026 | GPT-5.6 Sol system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 9 Jul 2026 | GPT-5.6 Luna system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 25 Jun 2026 | GPT-5.6 Terra system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 25 Jun 2026 | GPT-5.6 Sol system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 25 Jun 2026 | GPT-5.6 Luna system card | MLE-Bench Revised, 72 problems, mean reward by cost and latency, chart without printed numbers (Figures 46, 47, 9.1.3.5) | AI R&D risk | |
| 12 Jun 2026 | Muse Spark system card | 15.8 | MLE-Bench any medal rate, 30-task subset, 24 hours (Table 1, Table 11) | AI R&D risk |
| 26 May 2026 | MiniMax-M2 technical report | 40 | MLE-Bench Lite, MiniMax-M2 bar, printed label (Figure 9) | AI R&D risk |
| 26 May 2026 | MiniMax-M2.7 technical report | 66.6 | MLE Bench Lite, medal rate, mean of 3 24-hour trials (22 competitions) (Table 4) | AI R&D risk |
| 26 May 2026 | MiniMax-M2.5 technical report | 51.5 | MLE Bench Lite, medal rate, mean of 3 24-hour trials (22 competitions) (Table 4) | AI R&D risk |
| 23 Apr 2026 | GPT-5.5 system card | 36.7 | MLE-Bench-30 (30-competition subset), bronze or better, pass@1, printed chart label (Figure 19, 9.1.3.2) | AI R&D risk |
| 5 Mar 2026 | GPT-5.4 system card | 23.3 | MLE-Bench (30-competition subset), pass@1, printed chart label (Figure 16, 5.1.3.2) | AI R&D risk |
| 18 Dec 2025 | GPT-5.2 Codex system card addendum | 10 | MLE-Bench-30 (30-competition subset), bronze pass@1, printed chart label (Figure 10, 5.1.3.2) | AI R&D risk |
| 11 Dec 2025 | GPT-5.2 system card addendum | 16 | MLE-Bench-30, 30-competition subset, bronze pass@1, no browsing, printed chart label (Figure 14) | AI R&D risk |
| 18 Nov 2025 | GPT-5.1-Codex-Max system card | 17 | MLE-Bench-30 (30-competition subset), bronze pass@1, printed chart label (Figure 12, 5.1.3.3) | AI R&D risk |
| 7 Aug 2025 | GPT-5 system card | 8 | MLE-Bench-30, bronze pass@1, no browsing launch candidate (Figure 23) | AI R&D risk |
| 7 Aug 2025 | GPT-5 mini system card | 3 | MLE-Bench-30, bronze pass@1, gpt-5-thinking-mini launch candidate (Figure 23) | AI R&D risk |
| 27 Feb 2025 | GPT-4.5 system card | 11 | MLE-Bench, 75 Kaggle competitions, AIDE agent, bronze medal rate, post-mitigation (4.6.4) | Autonomy |
| 25 Feb 2025 | OpenAI deep research system card | 11 | MLE-Bench, bronze pass@1, internal tool scaffold, post-mitigation with and without browsing (3.4.7) | Autonomy |
| 31 Jan 2025 | o3-mini system card | 11 | MLE-bench (AIDE), bronze pass@1, o3-mini (Post), pass@10 20%, printed chart label (5.7.4) | Autonomy |
| 5 Dec 2024 | o1 system card | 14 | MLE-bench (AIDE), bronze pass@1, o1 (post-mitigation) (bronze pass@10 24%), printed chart label (5.8.4) | AI R&D risk |
More in AI research
- MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
- PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics.
Qwen 3.8 Max 93% - PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
- KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
- OpenAI PRs Only in system cards Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. GPT-5.2 55%
- OpenAI Research Engineer Interview: Multiple Choice Only in system cards Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. o4-mini 83%