Coding

We track 243 benchmarks in coding, or 272 if you count each version. They run from everyday level to research frontier level.

44 in use 25 past 85% 121 with public code

OpenAI held the top coding score 41% of the contested time

That's about 19 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.

Include lab claims

OpenAI Anthropic DeepSeek Google DeepMind Ai2 Reka Upstage Baidu Meta Mistral AI Z.ai NVIDIA Sarvam AI Qwen xAI MiniMax Moonshot AI Ant Group Inception Labs ByteDance Seed

School Students, from grade school to high school

HumanEval Passed 85% in Apr 2024, by GPT-4 Turbo

Solar Pro 2 98.8%

HumanEval+ Passed 85% in Apr 2024, by GPT-4 Turbo

ERNIE 5.0 94.5%

MBPP Passed 85% in Mar 2024, by Claude 3 Opus

Sarvam 30B 92.7%

University Undergraduates and entry-level professionals

Aider Polyglot Passed 85% in Aug 2025, by GPT-5

GPT-5 88%

BigCodeBench-Hard Not tested since Jun 2025

Claude 3.7 Sonnet 35.8%

Competition The strongest students, in olympiad-style contests

LiveCodeBench v6 Passed 85% in Aug 2025, by Gemini 2.5 Deep Think

DeepSeek-V4-Pro 93.5%

Expert Specialists with years of training, often over hours of work

SciCode 9 leaders

Seed2.1 Pro 59.8%

SWE-bench Verified Passed 85% in Apr 2026, by Claude Mythos Preview

Claude Opus 5 96%

SWE-bench Multilingual Passed 85% in Apr 2026, by Claude Mythos Preview

Claude Opus 5.5 93.9%

SWE-Bench Pro V2 Passed 85% in Sep 2026, by Claude Opus 5

Claude Opus 5 99.4%

Vibe Code Bench Passed 85% in Jun 2026, by Claude Fable 5

Claude Sonnet 5.5 92.4%

DeepSWE 6 leaders

Muse Wonder Full 79%

Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.

OpenAI DeepSeek Anthropic Google DeepMind Meta

School Students, from grade school to high school

HumanEval+ Passed 85% in Apr 2024, by GPT-4 Turbo

o1-mini 89%

MBPP+ Not tested since Dec 2024

o1-preview 80.2%

University Undergraduates and entry-level professionals

Aider Polyglot Passed 85% in Aug 2025, by GPT-5

GPT-5 88%

BigCodeBench Not tested since Apr 2025

GPT-4o 56.1%

BigCodeBench-Hard Not tested since Apr 2025

Claude 3.7 Sonnet 35.8%

Competition The strongest students, in olympiad-style contests

LiveCodeBench v6 Not tested since Jul 2025

o4-mini 80.2%

Expert Specialists with years of training, often over hours of work

SciCode Not tested since Jan 2025

o3-mini 10.8%

SWE-bench Verified Not tested since Jun 2026

Claude Opus 4.7 83.5%

SWE-bench Multilingual Not tested since Feb 2026

Gemini 3 Flash 72.7%

SWE-Bench Pro V2 Passed 85% in Sep 2026, by Claude Opus 5

Claude Opus 5 99.4%

Vibe Code Bench Passed 85% in Jun 2026, by Claude Fable 5

Claude Sonnet 5.5 92.4%

DeepSWE Not tested since Sep 2026

GPT-6 Astra 74.1%

Code Migration v1.1 Passed 85% in Oct 2026, by Claude Sonnet 5.5

Claude Sonnet 5.5 86.9%

Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.

From everyday to research frontier level

Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.

School

Students, from grade school to high school

University

Undergraduates and entry-level professionals

Competition

The strongest students, in olympiad-style contests

Expert

Specialists with years of training, often over hours of work

Research frontier

Researchers, on open problems or tasks that take days

Another 7 have no level. Levels don’t apply to arenas, safety evaluations or composite indices.

By kind

Function-level problems

Benchmark	Level	Scored	Best score	Status
HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests.	School	Checked automatically	98.8%	Fading
MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests.	School	Checked automatically	92.7%	Fading
BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests.	University	Checked automatically	35.8%	Dropped
Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests.	University	Checked automatically	88%	Fading
HumanEval+ Models write Python functions for the 164 HumanEval problems. EvalPlus checks them against about 80 times more tests than the original.	School	Checked automatically	94.5%	Fading
BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests.	University	Checked automatically	60.6%	Fading
MBPP+ Models write short Python functions for MBPP problems, and EvalPlus checks each one against many more generated tests than the original.	School	Checked automatically	86.2%	Fading
Aider-Edit Models edit Python source files to complete 133 Exercism exercises through Aider. Scored by the share of exercises whose unit tests all pass.	School	Checked automatically	79.7%	Dropped
AutoCodeBench Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution.	University	Checked automatically	52.4%	Not in cards
MultiPL-E Models write code for HumanEval and MBPP problems translated into 18 other programming languages. Scored by pass@1 on unit tests.	School	Checked automatically	88.2%	Fading
CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1.	School	Checked automatically	95.9%	Fading
HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning.	University	Checked automatically	40.6%	Fading
SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores.	University	Graded by people	1,237	Not in cards
MHPP Models write Python functions for 210 human-curated problems that need careful reading of specifications and multi-step reasoning. Unit tests check each solution.	University	Checked automatically	68.1%	Dropped
Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness.	School	Checked automatically	94%	Dropped
FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer.	University	Checked automatically	68.3%	Fading
HumanEval Multi Models complete HumanEval problems in Python and seven more languages, including C++, Java, PHP, and Bash. Unit tests score pass@1.	School	Checked automatically	82.6%	Dropped
EvalPlus HumanEval+/MBPP+ with 80x more tests. Showed many 'passing' solutions were wrong.	School	Checked automatically	87%	Dropped
CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains.	Competition	Checked automatically	85.5%	System cards only
CanItEdit Models edit a given block of code to follow an instruction, such as adding a feature, fixing a described bug or changing the approach. Hidden tests check the edited code.	University	Checked automatically	54.3%	Dropped
MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution.	School	Checked automatically	78.4%	Dropped
HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1.	University	Checked automatically	76.2%	System cards only
NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution.	University	Checked automatically	49.6%	Dropped
DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate.	University	Graded by a model	63.1%	Dropped
DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion.	University	Checked automatically	78.3%	Dropped
LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1.	University	Checked automatically	81%	System cards only
CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item.	University	Graded by a model	71.9%	System cards only
Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring.	University	Graded by a model	90.2%	System cards only
HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds.	Expert	Checked automatically	94%	System cards only
HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function.	School	Checked automatically	84.8%	System cards only
HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1.	University	Checked automatically	25.3%	System cards only
McEval Models complete, explain and generate code in 40 programming languages across about 16,000 test samples. Scored by test execution.	School	Checked automatically	59.6%	System cards only
BigCodeArena Users ask for code, both models' programs run in a sandbox, and users try the results before voting for the better answer. Votes give Elo ratings.	Everyday	Ranked by votes		Not in cards
ClassEval Models generate whole Python classes for 100 hand-built tasks. Tests check each method and the class as a whole.	University	Checked automatically		Not in cards
Copilot Arena Developers inside their IDE see paired code completions from two models and pick one. Pairwise votes give an Elo-style ranking of code models.	Everyday	Ranked by votes		Not in cards
TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one.	Expert	Checked automatically		Not in cards

Competitive programming

Benchmark	Level	Scored	Best score	Status
LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.	Competition	Checked automatically	93.5%	In use
LiveCodeBench Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.	Competition	Checked automatically	93.5%	Fading
ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants.	Competition	Checked automatically	2,951	Not in cards
LiveCodeBench v5 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests.	Competition	Checked automatically	81.2%	Fading
Codeforces rating Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance.	Competition	Checked automatically	3,471	In use
IOI (Vals) Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask.	Competition	Checked automatically	91.7%	Not in cards
IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run.	Competition	Checked automatically	100%	Not in cards
OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1.	Competition	Checked automatically	70%	Fading
FrontierCS Models write programs for 156 open-ended computer science problems whose optimum is unknown. An automatic evaluator gives partial scores against an expert reference solution.	Research frontier	Checked automatically	76.4	System cards only
LiveCodeBench Pro Continuously updated Codeforces, ICPC and IOI problems annotated by olympiad medalists. Reported as Elo.	Competition	Checked automatically	2,887	Fading
AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution.	Competition	Checked automatically	65.8%	Fading
ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved.	Competition	Checked automatically	10.0	Fading
APPS Models write Python programs for 10,000 coding problems that range from introductory to competition level. Generated code runs against test cases.	University	Checked automatically	70.2%	Dropped
IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600).	Competition	Checked automatically		Fading
LeetCode Weekly Contest Models solve 126 LeetCode contest problems from July to November 2023. A solution counts only if it passes all of its 20+ test cases.	University	Checked automatically	17.5%	Dropped
OIBench Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions.	Competition	Checked automatically	47.7%	Fading
CF-Div2-Stepfun Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions.	Competition	Checked automatically	86.1%	System cards only
Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved.	University	Checked automatically	75.6%	System cards only
Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved.	University	Checked automatically	6.70%	System cards only
Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved.	University	Checked automatically	26.3%	System cards only
Multi-LCB Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation.	Competition	Checked automatically		Not in cards
USACO Models solve 307 USA Computing Olympiad problems. Hidden unit tests check correctness and efficiency, scored as pass@1.	Competition	Checked automatically		Not in cards

Fixing real repositories

Benchmark	Level	Scored	Best score	Status
SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests.	Expert	Checked automatically	83.5%	In use
SWE-Bench Pro 1,865 long-horizon issue-resolution tasks from 41 repos in four languages. Agent patch must pass hidden tests. Pass@1.	Expert	Checked automatically	61.5%	In use
SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests.	Expert	Checked automatically	93.9%	In use
DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions.	Expert	Checked automatically	74.1%	In use
Code Migration Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.	Expert	Checked automatically	69.8%	Not in cards
FrontierCode Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero.	Expert	Checked automatically	34.3%	In use
SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards.	Expert	Graded by a model	63.2%	In use
AndroidBench Android development tasks from popular open-source apps: agent resolves issues or builds features, verified by test suites in emulator environments.	Expert	Checked automatically	91.8%	In use
OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported.	Expert	Checked automatically	81%	Not in cards
Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests.	Expert	Checked automatically	52.7%	Fading
Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.	Expert	Checked automatically	86.9%	Not in cards
FrontierCode 1.1 Real-repo coding tasks graded for mergeability: correctness, regressions, lint, tests, scope and code quality. Failing any blocker scores zero.	Expert	Checked automatically	53.5%	In use
FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change.	Expert	Checked automatically	45.5%	In use
SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests.	Expert	Checked automatically	61.4%	In use
Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate.	Expert	Checked automatically	39.3%	Not in cards
SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests.	Expert	Graded by a model	67%	Not in cards
CursorBench Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.0 covers edit, refactor and bugfix problems. Agentic graders score correctness, with cost reported.	Expert	Graded by a model	57.8%	System cards only
SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned.	Expert	Checked automatically	81.4%	Fading
Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score.	Expert	Graded by a model	37.9	Not in cards
SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol.	Research frontier	Checked automatically	98%	In use
SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result.	Expert	Graded by a model	59%	Not in cards
SWE-Bench Pro V2 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container.	Expert	Checked automatically	99.4%	Not in cards
SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch.	Expert	Checked automatically	60.3%	Not in cards
OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation.	Expert	Checked automatically	29%	Not in cards
SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch.	Expert	Checked automatically	71.7%	Dropped
SWE-rebench Agents fix fresh GitHub issues that an automated pipeline mines each month. Tests decide whether the issue is resolved.	Expert	Checked automatically	64.5%	System cards only
Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1.	Expert	Checked automatically	76%	Not in cards
SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved.	Research frontier	Checked automatically	9.50%	In use
SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix.	Expert	Checked automatically	56.2%	Fading
ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change.	Expert	Checked automatically	75.2%	Not in cards
Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search.	Expert	Checked automatically	81.6%	Not in cards
SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass.	Expert	Checked automatically	52.6%	Not in cards
Artificial Analysis Coding Agent Index v1.1 Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE.		Checked automatically	80.0	In use
CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages.	Expert	Checked automatically	27.9%	Not in cards
CursorBench 3.1 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness.	Expert	Graded by a model	72.9%	Fading
CursorBench 4.0 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported.	Expert	Graded by a model	57.8%	In use
Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task.	Expert	Checked automatically	64.2%	In use
Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages.	Expert	Checked automatically	77.1%	In use
SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution.	Expert	Checked automatically	14.1%	Not in cards
CursorBench 3.2 Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness.	Expert	Graded by a model	73.4%	In use
Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published.	Expert	Checked automatically	35.2%	In use
KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages.	University	Checked automatically	53.1%	In use
MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE.	Expert	Checked automatically	73.7%	Fading
QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished.	Expert	Checked automatically	80.7%	In use
ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1.	Expert	Graded by a model	73.3%	Not in cards
SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1.	Expert	Checked automatically	51%	System cards only
APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry.	Expert	Checked automatically	56.4%	In use
Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass.	Expert	Checked automatically	78%	System cards only
Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing.	Expert	Checked automatically	25.8%	Dropped
Artificial Analysis Coding Agent Index v1.4 Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero.		Checked automatically	67.0	In use
Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea.	Expert	Checked automatically	51.3%	Dropped
Cursor Bench (2025 internal version) Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions.	Expert	Graded by a model		Fading
Hy-Backend Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published.	Expert	Checked automatically	54.7%	Fading
MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details.	Expert	Checked automatically	63.2%	In use
Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing.	Expert	Checked automatically	27%	Dropped
OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items.	University	Graded by a model	26.1%	Fading
QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit).	Expert	Checked automatically	58.4%	In use
RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline.	Expert	Checked automatically	38%	Dropped
Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses.	Expert	Checked automatically	67.1%	Fading
StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup.	Expert	Checked automatically	49%	In use
SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail.	Expert	Checked automatically	66.6%	Dropped
SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks.	Expert	Checked automatically	8.50%	Fading
SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects.	Expert	Graded by a model	8.90%	Fading
Artificial Analysis Coding Agent Index v2 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores.		Checked automatically		In use
CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass.	Expert	Checked automatically	25.8%	System cards only
CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file.	Expert	Checked automatically	65.6%	System cards only
CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step.	Expert	Checked automatically	52.3%	System cards only
Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability.	Expert	Graded by a model	59.9%	System cards only
Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy.	Expert	Graded by a model		Dropped
DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task.	Expert	Graded by people	67%	System cards only
End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score.	Expert	Checked automatically		Dropped
SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk.	Expert	Checked automatically	42.9%	System cards only
AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues.	Expert	Graded by a model		Not in cards
Artificial Analysis Coding Agent Index v1.0 Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks.		Checked automatically		Not in cards
Artificial Analysis Coding Agent Index v1.5 Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1.		Checked automatically		Not in cards
CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback.	Expert	Checked automatically		Not in cards
Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches.	Expert	Checked automatically		Not in cards
Code Review Bench (Martian) AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments.	Expert	Graded by a model		Not in cards
CoreCodeBench Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels.	Expert	Checked automatically		Not in cards
Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading.	Expert	Checked automatically		Not in cards
FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically.	Expert	Checked automatically		Not in cards
GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it.	Expert	Checked automatically		Not in cards
HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary.	Expert	Checked automatically		Not in cards
Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks.	Expert	Checked automatically		Not in cards
Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass.	Expert	Checked automatically		Not in cards
MacroscopeBench Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs.	Expert	Graded by a model		Not in cards
Qodo AI Code Review Benchmark Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1.	Expert	Checked automatically		Not in cards
Real-SWE Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors.	Expert	Checked automatically		Not in cards
RepoChat Arena Users give a GitHub repository link and a question. A retriever model picks files, two answer models reply, and users vote for the better answer.	Everyday	Ranked by votes		Not in cards
sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change.	Expert	Graded by people		Not in cards
SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores.	Expert	Graded by a model		Not in cards
SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result.	Research frontier	Checked automatically		Not in cards
SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions.	Expert	Checked automatically		Not in cards
SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores.	Expert	Checked automatically		Not in cards
SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes.	Expert	Checked automatically		Not in cards
SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail.	Expert	Checked automatically		Not in cards
SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session.	Expert	Checked automatically		Not in cards
Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup.	Expert	Checked automatically		System cards only
Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished.	Expert	Checked automatically		Not in cards

Building whole projects

Benchmark	Level	Scored	Best score	Status
Vibe Code Bench End-to-end web application development from natural-language specs. Built apps are tested against functional requirements.	Expert	Graded by a model	92.4%	In use
NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate.	Expert	Checked automatically	68.2%	In use
ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original.	Research frontier	Checked automatically	18.5%	In use
SWE-Marathon 20 realistic multi-hour software tasks (library reproductions, product clones, ML work). Verifiers combine hidden tests, parity and integrity checks.	Expert	Checked automatically	29%	In use
MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass.	Research frontier	Checked automatically	97.8%	Not in cards
FrontierSWE 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100.	Research frontier	Checked automatically	65.5%	In use
Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it.	Expert	Checked automatically	34%	Not in cards
GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2.	Research frontier	Checked automatically	79.6%	Not in cards
SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials.	Expert	Checked automatically	53.1%	Not in cards
Vibe Code Bench v1 Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements.	Expert	Graded by a model	41.3%	Not in cards
SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity.	Expert	Checked automatically	28.1%	Not in cards
EvoCode-Bench Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container.	Expert	Checked automatically	54%	Not in cards
EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit.	Expert	Checked automatically	59.1%	Not in cards
Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run.	Expert	Checked automatically	40%	Not in cards
GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result.	University	Checked automatically	70.6%	Not in cards
SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models.	Research frontier	Checked automatically	32.3%	In use
ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans.	Expert	Graded by a model	64.7%	Not in cards
Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests.	Research frontier	Checked automatically	34.2%	Not in cards
CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings.	Expert	Ranked by matches	1,385	Not in cards
DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring.	Expert	Checked automatically	71.1%	In use
VIBE-Pro (MiniMax) Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output.	Expert	Graded by a model	55.6%	Fading
FrontierSWE v1 Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. Models are ranked by average task rank and dominance.	Research frontier	Checked automatically		In use
Hy-CompanyBench Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published.	Expert	Checked automatically	41.7%	In use
Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published.	Expert	Checked automatically	62.4%	In use
Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published.	Expert	Checked automatically	50.5%	Fading
HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials.	Expert	Graded by a model	67.6%	System cards only
Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score.	Expert	Checked automatically	61.2%	In use
RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests.	Expert	Checked automatically	54.3%	In use
VIBE (MiniMax) Agent builds complete web, simulation, Android, iOS or backend apps from scratch. Agent verifiers check execution, interaction and visuals.	Expert	Graded by a model	88.6%	Fading
VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output.	Expert	Graded by a model	50.1%	In use
CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code, and scores separate first-round skill from improvement.	Expert	Ranked by matches		Not in cards
CC-Bench Claude Code runs realistic frontend, backend, tooling and data coding tasks in fresh containers. Humans compare models head-to-head (win rate).	Expert	Ranked by votes		System cards only
cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability.	Expert	Checked automatically		Not in cards
DesignArena Agentic Game Dev Coding agents build multi-file games from a user prompt. Users play the final games and vote, and the logs keep tool calls, retries and failures.	Everyday	Ranked by votes		Not in cards
DesignArena Android (Kotlin) Models build native Android apps in Kotlin from real user prompts. The apps run in an Android emulator, and users compare them and vote.	Everyday	Ranked by votes		Not in cards
DesignArena Builder AI app builders make full-stack web and mobile apps from the same user prompt. Users compare the results and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Full Stack Coding agents build complete React apps with sign-in, databases and backend code from real user prompts. Users try the apps and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Godot Game Dev Coding agents build complete Godot projects with GDScript, scenes and generated assets, exported to run in the browser. Users play the games and vote.	Everyday	Ranked by votes		Not in cards
DesignArena React Native Models build cross-platform mobile apps in React Native from real user prompts. Users compare the running apps and vote for the better one.	Everyday	Ranked by votes		Not in cards
DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design.	Expert	Graded by a model		Not in cards
Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours.	Expert	Checked automatically		Not in cards
FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1.	Expert	Checked automatically		Not in cards
GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together.	Expert	Checked automatically		Not in cards
LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity.	Expert	Checked automatically		Not in cards
LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit.	Expert	Checked automatically		Not in cards
PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria.	Expert	Graded by a model		Not in cards
Snorkel Agentic Coding Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory.	Expert	Checked automatically		Not in cards
Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior.	Research frontier	Checked automatically		Not in cards
Surge AI blind coding evaluation Surge AI raters blindly score model coding outputs on a 1-5 scale. Mistral reports five models.	University	Graded by people		Not in cards
SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build.	Expert	Checked automatically		Not in cards
τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users.	Expert	Checked automatically		Not in cards

Web and app front ends

Benchmark	Level	Scored	Best score	Status
Code Arena Models act as coding agents that build web apps from user prompts, and users vote between two results. Votes yield Elo-style ratings.		Ranked by votes	1,810	In use
WebDev Arena Two anonymous models build web apps from a user's prompt, and the user votes for the better one. Votes yield Elo-style ratings.		Ranked by votes	1,820	Fading
Image-to-WebDev Arena Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals.	Everyday	Ranked by votes	1,753	Not in cards
ArtifactsBench Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist.	University	Graded by a model	66.8%	In use
OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact.	Everyday	Graded by a model	91.0	Not in cards
Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them.	Expert	Graded by a model	82.3%	In use
Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost.	Expert	Checked automatically	97%	Not in cards
QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings.	University	Graded by a model	1,532	Fading
Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity.	Everyday	Checked automatically	93.4	Fading
Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k.	University	Checked automatically	86.3%	Fading
QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages.	University	Graded by a model	1,608	Fading
QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings.	University	Graded by a model	1,568	Fading
MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code.	Expert	Graded by a model	72.3%	In use
QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring.	University	Graded by a model	69.9%	In use
QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo.	Expert	Graded by a model	1,724	In use
QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo.	University	Graded by a model	1,713	In use
QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages.	University	Graded by a model	1,772	In use
SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references.	Everyday	Checked automatically	83.5	Fading
SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs.	University	Graded by a model	63.7%	In use
U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need.	Everyday	Graded by a model	49.2%	Fading
CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app.	Expert	Graded by a model	74.8%	System cards only
ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original.	University	Checked automatically	78.4	System cards only
Kimi Webdev Bench Models build games, 3D scenes and website clones from real prompts. Blind experts compare two outputs and report win, tie and loss rates.	Expert	Ranked by votes	58.6%	System cards only
AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings.	Expert	Ranked by votes		Not in cards
DesignArena 3D Design Models build an interactive 3D scene that runs in the browser as a single HTML file. Users explore the scenes and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote.	Everyday	Ranked by votes		Not in cards
DesignArena Code Users give a prompt, and anonymous models each build a website, UI component, browser game, data visualization or 3D scene as one HTML file. Users vote, and votes become Bradley-Terry ratings.	Everyday	Ranked by votes		Not in cards
DesignArena Frontend Web App Coding agents build multi-file React frontends from real user prompts. Users compare the running apps and vote, and the run logs record agent behavior.	Everyday	Ranked by votes		Not in cards
DesignArena Game Dev Models build a playable browser game as a single HTML file from a real user prompt. Users play the games and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Image to HTML Models recreate a reference image or mockup as a working single-file HTML website. Users compare the rendered pages and vote for the closer, better one.	Everyday	Ranked by votes		Not in cards
DesignArena SVG Models write SVG code for a design request such as an icon, logo, badge or illustration. Users compare the rendered graphics and vote.	Everyday	Ranked by votes		Not in cards
DesignArena UI Component Models build one interface component as a single HTML file from a real user prompt. Users compare the rendered components and vote for the better one.	Everyday	Ranked by votes		Not in cards
DesignArena Video to Website Models watch a reference video and recreate its content as a working single-file HTML website. Users compare the rendered pages and vote.	Everyday	Ranked by votes		Not in cards
DesignArena Website Models build a complete website as one HTML file from a real user prompt. Users compare the rendered pages and vote for the better one.	Everyday	Ranked by votes		Not in cards

Speed and optimization

Benchmark	Level	Scored	Best score	Status
GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup.	Expert	Checked automatically	88.2%	Not in cards
AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code.	Expert	Checked automatically	2.05×	Not in cards
SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch.	Expert	Checked automatically	22.5%	In use
BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels.	Competition	Checked automatically	41.4%	System cards only
Kernel Bench L3 Models write fast GPU kernels for the 50 full model architectures in KernelBench Level 3. Scored on correctness and speedup over PyTorch.	Expert	Checked automatically	2.06	Fading
SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain.	Expert	Checked automatically	3.10%	Fading
ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code.	Expert	Checked automatically		Not in cards

Data science and SQL

Benchmark	Level	Scored	Best score	Status
LiveSQLBench Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases.	University	Checked automatically	40.2%	In use
Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy.	University	Checked automatically	63.6%	Fading
StructEval Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones.	Everyday	Checked automatically	81.2%	In use
Spider Models turn natural language questions into SQL over 200 multi-table databases from 138 domains. Test databases are unseen in training, and answers are scored by exact match or execution.	University	Checked automatically	86.6%	Dropped
BIRD-CRITIC Models fix real user SQL issues in MySQL, PostgreSQL, SQL Server and Oracle applications. Test cases check each fix.	Expert	Checked automatically	35.5%	Not in cards
DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap.	Expert	Checked automatically	89.9%	Dropped
OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason.	Expert	Checked automatically	34.9%	System cards only
Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them.	Expert	Checked automatically	31.1%	Dropped
Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task.	Expert	Graded by a model	39.6%	System cards only
Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases.	University	Checked automatically	55.3%	System cards only
AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios.	Expert	Checked automatically		Not in cards
BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests.	Expert	Checked automatically		Not in cards
DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically.	Expert	Checked automatically		Not in cards
DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights.	Expert	Graded by a model		Not in cards
DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.	Expert	Graded by a model		Not in cards
DesignArena Data Visualization Models turn a user prompt and its data into an interactive visualization as a single HTML file. Users compare the rendered charts and vote.	Everyday	Ranked by votes		Not in cards
DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer.	University	Checked automatically		Not in cards

Scientific code

Benchmark	Level	Scored	Best score	Status
SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases.	Expert	Checked automatically	59.8%	In use
AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change.	Expert	Checked automatically	47.7%	Fading
AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages.	Expert	Checked automatically		Not in cards
ArXivCS Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026.	Research frontier	Checked automatically		Not in cards
SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix.	Expert	Checked automatically		Not in cards