AI research
We track 83 benchmarks in AI research, or 87 if you count each version. They run from expert level to research frontier level.
OpenAI held the top aI research score 38% of the contested time
That's about 18 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
OpenAI DeepSeek Google DeepMind MiniMax Anthropic Moonshot AI Qwen
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Share of contested time at number one
OpenAI 38% Anthropic 27% 
Google DeepMind 14% Moonshot AI 13% 
MiniMax 8%
Google DeepMind 14% Moonshot AI 13% MiniMax 8% Closed out: 3 benchmarks passed 85%
OpenAI 1 
Qwen 1 Anthropic 1
Qwen 1 Anthropic 1 - Sep 2026 MLE-bench GPT-6 Astra
- Aug 2026 PaperBench Qwen 3.8 Max
- Jun 2026 WeirdML Claude Fable 5
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
OpenAI DeepSeek Google DeepMind Anthropic
Expert Specialists with years of training, often over hours of work
Research frontier Researchers, on open problems or tasks that take days
Share of contested time at number one, independent runs only
OpenAI 51% Anthropic 30% Google DeepMind 19% 
DeepSeek 1%
Google DeepMind 19% DeepSeek 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From expert to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Expert
Specialists with years of training, often over hours of work
- WeirdML
- MLE-bench
- WeirdML v3
- OpenAI Research Engineer Interview: Multiple Choice
- SWE-bench Verified (hard subset)
- Time series forecasting task (Anthropic)
- Quadruped reinforcement learning task (Anthropic)
- OpenAI Research Engineer Interview: Coding
- Text-based reinforcement learning task (Anthropic)
- Internal AI research evaluation suite 2
- Internal model use survey (Anthropic)
- Agentic tasks (autonomous replication and adaptation)
- Novel compiler task (Anthropic)
- METR data deduplication
- KernelBench
- CoBench (Anthropic)
- InferenceEval
- KernelBenchInternal
- RE-Bench subset (Anthropic)
- RelBench
- Select machine learning engineering tasks from METR
- SpaceXAI MTS Eval
- AgentKernelArena
- BackendBench
- DataPrep-Bench
- EvoPolicyGym
- FlashInfer-Bench
- HCAST
- Kernels task (Anthropic)
- LLM training task (Anthropic)
- METR general autonomy task suite
- MLAgentBench
- MLGym-Bench
- OpenAI internal AI self-improvement evals
- ParallelKernelBench
- robust-kbench
- S3Gym
- SeedKernelBench
- Simreal ML Benchmark
- SOL-ExecBench
- SpaceXAI internal R&D evals
- TASTE
- TritonBench
Research frontier
Researchers, on open problems or tasks that take days
- METR Time Horizon
- PostTrainBench
- METR Time Horizon 1.1
- Vals RSI Index
- MLS-Bench
- TasteVal
- PostTrainBench 1.2
- PaperBench
- RSI-Exam
- FrontierCS 2
- PostTrainBench 1.1
- OpenAI PRs
- AutoResearchExam
- OpenAI-Proof Q&A
- RSI Bench
- DeepRSI
- Internal Research Debugging Evaluation
- Monorepo-Bench
- KernelGen 1P
- AIRS-Bench
- Google RE-Bench (GRB)
- Key insight rediscovery on an unpublished ML task (external testing)
- OptimBench
- ResearchTasteBench
- Agent4Sci
- AQuA
- Aspire
- Automated LLM Speedrunning Benchmark
- AutoResearch
- Doc2Harness
- EXP-Bench
- FrogsGame post-training task
- FrontierMemory
- HarnessDev
- InnovationEval
- METR pre-deployment AI R&D tasks
- METR Time Horizon suite
- NanoGPT Speedrun
- NanoGPT-Bench
- OpenRSI Index
- RE-Bench
- RSI Arena
- RSI Index (OpenAI)
- RSIBench-Data
By kind
Machine learning engineering
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| WeirdML Models write PyTorch code to solve 17 unusual machine learning tasks, with 5 attempts and feedback each. Scoring is mean best test accuracy. | Expert | Checked automatically | 93.6% | Not in cards |
| PostTrainBench CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Checked automatically | 34.3% | In use |
| MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. | Expert | Checked automatically | 93.8% | In use |
| WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. | Expert | Checked automatically | 42.2% | Not in cards |
| PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Checked automatically | 44.6% | In use |
| PostTrainBench 1.1 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | Research frontier | Checked automatically | 45.6% | In use |
| AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. | Research frontier | Checked automatically | 0.60 | Not in cards |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Expert | Checked automatically | 48.2% | System cards only |
| Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. | Expert | Checked automatically | 4.55 | System cards only |
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Expert | Checked automatically | 31.3 | System cards only |
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Expert | Checked automatically | 0.97 | System cards only |
| Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. | Expert | Checked automatically | 80.9% | System cards only |
| METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. | Expert | Checked automatically | 32.6% | System cards only |
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Research frontier | Checked automatically |  64.2% | System cards only |
| RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. | Expert | Checked automatically | 0.24 | System cards only |
| RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. | Expert | Checked automatically | 38.4% | System cards only |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | Expert | Checked automatically | 0% | System cards only |
| DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. | Expert | Checked automatically | Not in cards | |
| FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. | Research frontier | Checked automatically | Not in cards | |
| LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. | Expert | Checked automatically | System cards only | |
| MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. | Expert | Checked automatically | Not in cards | |
| MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. | Expert | Checked automatically | Not in cards | |
| NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. | Research frontier | Checked automatically | System cards only | |
| RSI Arena Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback. | Research frontier | Ranked by votes | Not in cards | |
| RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. | Research frontier | Checked automatically | Not in cards | |
| Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. | Expert | Settled by events | Not in cards |
AI research tasks
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. | Research frontier | Checked automatically | 0.37 | Not in cards |
| MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. | Research frontier | Checked automatically | 50.3% | In use |
| TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. | Research frontier | Checked automatically | 230.4× | Not in cards |
| PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. | Research frontier | Graded by a model | 93% | In use |
| RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. | Research frontier | Checked automatically | 53.6% | Not in cards |
| FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. | Research frontier | Checked automatically | 58.3% | Not in cards |
| RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. | Research frontier | Checked automatically | 38.9 | Not in cards |
| DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. | Research frontier | Checked automatically | 1,629 | Not in cards |
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Expert | Checked automatically | 0.65 | System cards only |
| AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. | Research frontier | Checked automatically | 0.77 | System cards only |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Research frontier | Graded by people | 80% | System cards only |
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Research frontier | Checked automatically | 64.7% | System cards only |
| Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. | Research frontier | Checked automatically | Not in cards | |
| AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. | Research frontier | Checked automatically | Not in cards | |
| Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. | Research frontier | Checked automatically | Not in cards | |
| Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. | Research frontier | Checked automatically | Not in cards | |
| AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. | Research frontier | Checked automatically | Not in cards | |
| Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. | Research frontier | Checked automatically | Not in cards | |
| EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. | Expert | Checked automatically | Not in cards | |
| EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. | Research frontier | Graded by a model | Not in cards | |
| FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. | Research frontier | Checked automatically | Not in cards | |
| HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. | Research frontier | Checked automatically | Not in cards | |
| InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. | Research frontier | Checked automatically | Not in cards | |
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | Research frontier | Checked automatically | Not in cards | |
| NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. | Research frontier | Checked automatically | Not in cards | |
| OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. | Research frontier | Checked automatically | Not in cards | |
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | Research frontier | Checked automatically | System cards only | |
| S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. | Expert | Checked automatically | Not in cards | |
| TASTE 92 pairwise comparisons of AI safety research proposals. Model must match experienced researchers' preference. Accuracy versus ~77% human agreement. | Expert | Checked automatically | Not in cards |
GPU kernels
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | Research frontier | Graded by a model | 60.4% | System cards only |
| KernelBench Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. | Expert | Checked automatically | 28.8% | In use |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | Expert | Checked automatically | 37.2% | System cards only |
| AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. | Expert | Checked automatically | Not in cards | |
| BackendBench LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed. | Expert | Checked automatically | Not in cards | |
| FlashInfer-Bench Agents generate GPU kernels for LLM inference that are benchmarked on workloads from real serving traces and can be deployed into the FlashInfer stack. | Expert | Checked automatically | Not in cards | |
| Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. | Expert | Checked automatically | System cards only | |
| ParallelKernelBench Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline. | Expert | Checked automatically | Not in cards | |
| robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. | Expert | Checked automatically | Not in cards | |
| SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. | Expert | Checked automatically | System cards only | |
| SOL-ExecBench Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes. | Expert | Checked automatically | Not in cards | |
| TritonBench Models write Triton GPU operators in two channels, one from real GitHub kernels (G) and one matching PyTorch operators (T). Execution checks correctness and speedup. | Expert | Checked automatically | Not in cards |
Task length
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| METR Time Horizon Agents attempt software and research tasks timed against skilled humans. The score is the human task length at which the agent succeeds half the time. | Research frontier | Checked automatically | 4.1 h | System cards only |
| METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. | Research frontier | Checked automatically | 17.4 h | Not in cards |
| HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. | Expert | Checked automatically | Not in cards | |
| METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. | Expert | Checked automatically | System cards only | |
| METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. | Research frontier | Checked automatically | Not in cards |
Labs' internal R&D tests
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | Research frontier | Checked automatically | 55% | System cards only |
| OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. | Expert | Checked automatically | 83% | System cards only |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | Research frontier | Graded by a model | 8.00% | System cards only |
| OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. | Expert | Checked automatically | 100% | System cards only |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | Research frontier | Graded by a model | 78% | System cards only |
| Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. | Expert | Graded by people | 50% | System cards only |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | Research frontier | Checked automatically | 76% | System cards only |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | Expert | Checked automatically | 78% | System cards only |
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Expert | Graded by a model | 55.8% | System cards only |
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Research frontier | Checked automatically | 27% | System cards only |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | Expert | Checked automatically | 46.9% | System cards only |
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | Expert | Checked automatically | 61.1% | System cards only |
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | Expert | Checked automatically | Not in cards | |
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | Research frontier | Checked automatically | Not in cards | |
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | Expert | Checked automatically | Not in cards |