Benchmarks
We track 2,721 benchmarks, or 2,914 if you count each version. Open one to see its leaderboard, score history and the model cards that report it. Arenas, where people vote between answers, also have their own page.
303 in use 321 fading 221 dropped 1,261 with public code Scores refreshed 2 Nov 2026
- All areas
- Knowledge and language 460
- Knowledge 109
- Exam questions 32
- Expert-written questions 8
- Commonsense 11
- Reading comprehension 20
- Trivia and open questions 7
- Questions that need search 10
- Language reasoning 21
- Factuality 70
- Short factual questions 13
- Faithful to sources 16
- Long-form factuality 7
- Hallucination rates 34
- Multilingual 115
- Knowledge in many languages 51
- Translation 15
- Reasoning in many languages 21
- Reading in many languages 12
- Chat and instructions in many languages 16
- Long context 66
- Finding facts in long inputs 17
- Reasoning over long documents 32
- Learning from the context 17
- Instructions and chat 76
- Following precise instructions 34
- Multi-turn conversation 11
- Overall chat quality 31
- Writing and taste 24
- Creative writing 14
- Taste and style 10
- Math and science 205
- Math 105
- Word problems 21
- Competition problems with answers 39
- Written proofs 15
- Formal proofs 15
- Research mathematics 15
- Science and reasoning 88
- Science questions 14
- Science olympiads 8
- Research science 41
- Lab and data work 10
- Abstract puzzles and logic 15
- Forecasting 12
- Real-world events 10
- Social simulation 2
- Coding 326
- Coding 243
- Function-level problems 35
- Competitive programming 18
- Fixing real repositories 85
- Building whole projects 43
- Web and app front ends 34
- Speed and optimization 7
- Data science and SQL 16
- Scientific code 5
- AI research 83
- Machine learning engineering 23
- AI research tasks 29
- GPU kernels 12
- Task length 4
- Labs' internal R&D tests 15
- Agents and tools 267
- Terminal agents 18
- Terminal tasks 15
- Long terminal sessions 3
- Computer and browser use 56
- Desktop apps 15
- Phones 9
- Web browsing 24
- Finding things on screen 8
- Tool use 125
- Function calling 27
- Customer service 14
- Apps and MCP servers 11
- Deep research 37
- General assistants 36
- Games and long tasks 68
- Puzzles and chess 27
- Video games 23
- Multi-agent games 18
- Professional work 249
- Economic work 91
- Tasks from real jobs 25
- Running a business 20
- Documents and spreadsheets 29
- Simulated companies 12
- Composite work indices 5
- Law, finance and medicine 140
- Law 22
- Finance 42
- Medicine 67
- Other professions 9
- Engineering and CAD 18
- CAD models 14
- Electronics 4
- Vision, audio and robotics 522
- Vision 234
- General visual questions 82
- Documents, charts and text in images 51
- Visual math and diagrams 17
- Spatial reasoning and counting 50
- Detection and captioning 18
- Image generation and editing 16
- Audio and video 168
- Video understanding 55
- Long videos 18
- Video captioning 9
- Speech recognition and translation 30
- Audio understanding and voice 31
- Video generation 25
- Robotics 86
- Simulated robots 48
- Real robots 22
- Embodied reasoning 16
- World models 34
- Video world models 10
- Interactive world models 11
- Physical prediction 13
- Safety and security 651
- Safety and alignment 444
- Refusing harmful requests 145
- Jailbreaks and prompt injection 95
- Deception and alignment 136
- Bias, fairness and politics 23
- Wellbeing and sycophancy 22
- Dangerous capabilities 23
- Cybersecurity 111
- Capture-the-flag challenges 34
- Finding and exploiting real bugs 60
- Secure code and review 17
- Biology and chemistry 96
- Lab protocols 12
- Genomics and life sciences 44
- Hazardous knowledge 40
- Other 41
- Other 41
- Composite indices 24
- Other 17
Nothing matches.
- Any level
- Everyday 1016
- School 122
- University 271
- Competition 88
- Expert 1005
- Research frontier 121
- Any status
- In use 303
- Fading 321
- Dropped 221
- Only in system cards 866
- Not in model cards 1010
- Last model: any time
- New model within 3 months 730
- New model within 6 months 1074
- New model within a year 1421
- Quiet for a year or more 633
- No scores yet 667
- With or without code
- Public code 1261
| Benchmark | Made by | Kind | Level | Status | Best score | Launch posts | System cards | Released | Last added |
|---|---|---|---|---|---|---|---|---|---|
| GPQA Diamond 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. | NYU (Rein et al.) | Science questions | Expert | In use |  Gemini 4 Argon 96.4% | 244 | 159 | Nov 2023 | Oct 2026 |
| SWE-bench Verified 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. | Princeton NLP + OpenAI | Fixing real repositories | Expert | In use | Claude Opus 4.7 83.5% | 138 | 80 | Aug 2024 | Oct 2026 |
| MMLU-Pro Harder 10-option version of MMLU with reasoning-focused questions across 14 subjects. | TIGER-Lab (Waterloo) | Exam questions | University | In use |  Intern-S2-397B 89.8% | 127 | 95 | Jun 2024 | Sep 2026 |
| MMLU 57-subject multiple-choice exam questions from elementary to professional level. Scored by accuracy. | UC Berkeley (Hendrycks et al.) | Exam questions | University | Fading |  Seed1.8 92.3% | 99 | 87 | Sep 2020 | Aug 2026 |
| IFEval Models answer about 500 prompts containing verifiable instructions such as word limits or required keywords. Code checks each instruction for strict and loose accuracy. | Following precise instructions | Everyday | In use | Gemma 4 31B IT 98.9% | 94 | 86 | Nov 2023 | Oct 2026 | |
| MMMU 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. | MMMU team (OSU, Waterloo and others) | General visual questions | University | In use | GPT-5.1 85.4% | 89 | 51 | Nov 2023 | Aug 2026 |
| BrowseComp Hard-to-find facts that need persistent web browsing. Short verifiable answers graded for exact correctness. | OpenAI | Web browsing | Expert | In use | Claude Fable 5 93.3% | 73 | 34 | Apr 2025 | Oct 2026 |
| MMMU-Pro Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. | MMMU team | General visual questions | University | In use | Kimi K3.1 Pro 84.1% | 67 | 40 | Sep 2024 | Oct 2026 |
| HumanEval 164 hand-written Python function problems. Generated code is scored pass@1 against unit tests. | OpenAI | Function-level problems | School | Fading |  Solar Pro 2 98.8% | 65 | 64 | Jul 2021 | Sep 2026 |
| MATH 12,500 competition math problems with step-by-step solutions. Final answer scored by exact match. | UC Berkeley (Hendrycks et al.) | Competition problems with answers | Competition | Fading |  Cogito v2 Preview 671B MoE 98.2% | 64 | 66 | Mar 2021 | Sep 2026 |
| SWE-bench Multilingual 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. | SWE-bench team (Princeton/Stanford) | Fixing real repositories | Expert | In use | Claude Opus 5.5 93.9% | 63 | 36 | Apr 2025 | Oct 2026 |
| LiveCodeBench v6 Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. | UC Berkeley / MIT / Cornell | Competitive programming | Competition | In use |  DeepSeek-V4-Pro 93.5% | 61 | 47 | Apr 2025 | Oct 2026 |
| LiveCodeBench v5 Replaced in Apr 2025 | Fading |  Nemotron 3 Super 81.2% | 33 | 29 | Jan 2025 | Apr 2026 | |||
| LiveCodeBench v1 to v4 Replaced in Jan 2025 | Fading | DeepSeek-V4-Pro 93.5% | 60 | 41 | Mar 2024 | Sep 2026 | |||
| IFBench 58 new out-of-domain verifiable output constraints. Scored by programmatic constraint checks (strict or loose). | AllenAI | Following precise instructions | Everyday | In use | Nemotron-Cascade 2 30B-A3B 82.9% | 61 | 39 | Jul 2025 | Oct 2026 |
| MathVista Math reasoning in visual contexts (charts, geometry, puzzles) from 31 datasets. Answer accuracy. | UCLA + Microsoft | Visual math and diagrams | School | Fading | Seed2.1 Pro 90.7% | 58 | 42 | Oct 2023 | Aug 2026 |
| CharXiv Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. | Princeton NLP | Documents, charts and text in images | University | In use | Claude Fable 5 93.5% | 58 | 34 | Jun 2024 | Oct 2026 |
| GSM8K 8.5K grade-school math word problems. Final numeric answer scored by exact match. | OpenAI | Word problems | School | Fading |  Llama 3.1-405B 96.8% | 57 | 67 | Oct 2021 | Sep 2026 |
| MATH-500 Models solve 500 competition math problems drawn from the MATH test set. Final answers are checked against the reference answers. | OpenAI | Competition problems with answers | Competition | Fading |  LongCat-Flash-Thinking 99.2% | 50 | 43 | May 2023 | Oct 2026 |
| BIG-Bench Hard Models solve 23 BIG-Bench tasks where earlier models trailed average human raters, such as logic puzzles and date math. Scoring is exact match. | Suzgun et al. (Stanford University and Google Research) | Language reasoning | School | Fading |  ERNIE-4.5-300B-A47B 94.3% | 49 | 57 | Oct 2022 | Sep 2026 |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | OpenAI | Knowledge in many languages | University | In use | Claude Mythos Preview 92.7% | 48 | 40 | Sep 2024 | Aug 2026 |
| DeepSWE 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. | Datacurve | Fixing real repositories | Expert | In use | GPT-6 Astra 74.1% | 47 | 19 | May 2026 | Oct 2026 |
| MBPP Models write short Python functions from a one-line task description. A solution counts as correct if it passes the task's three assert tests. | Google Research | Function-level problems | School | Fading | Sarvam 30B 92.7% | 41 | 50 | Aug 2021 | Apr 2026 |
| SciCode Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. | SciCode team | Scientific code | Expert | In use | Seed2.1 Pro 59.8% | 41 | 24 | Jul 2024 | Oct 2026 |
| AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. | Zapier | Apps and MCP servers | University | In use | Gemini 4 Carbon 53.9% | 41 | 14 | Apr 2026 | Oct 2026 |
| MATH-Vision (MathVision) 3,040 competition math problems with visual context. Answer accuracy. | CUHK MMLab | Visual math and diagrams | Competition | In use |  Qwen 3.8 Max 97.7% | 38 | 29 | Feb 2024 | Oct 2026 |
| AI2D Models answer multiple choice questions about grade school science diagrams. The set has about 5,000 diagrams and 15,000 questions, scored by accuracy. | Allen Institute for AI (AI2) | Visual math and diagrams | School | Fading | ERNIE 5.0 96.9% | 38 | 28 | Mar 2016 | Aug 2026 |
| DROP Reading comprehension requiring discrete operations (counting, sorting, arithmetic) over paragraphs. Scored by F1. | AllenAI + UC Irvine | Reading comprehension | School | Fading | Kimi K2 93.5% | 36 | 38 | Mar 2019 | Sep 2026 |
| MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. | Scale AI | Apps and MCP servers | University | In use | Muse Spark 1.1 88.1% | 36 | 21 | Jan 2026 | Oct 2026 |
| IMO-AnswerBench 400 olympiad problems with short verifiable answers, robustified against memorization. Scored by answer match. | Google DeepMind | Competition problems with answers | Competition | In use | GLM-5.2 91% | 32 | 20 | Nov 2025 | Oct 2026 |
| AIME 2026 Models solve the 30 integer-answer problems from the 2026 AIME I and II math competitions. Scored by exact answer match. | Mathematical Association of America (exam), MathArena (evaluation) | Competition problems with answers | Competition | In use | GPT-5.5 100% | 31 | 20 | Feb 2026 | Oct 2026 |
| AIME 2025 Replaced in Feb 2026 | Mathematical Association of America (MAA) | In use | Grok 4 Heavy 100% | 129 | 81 | Feb 2025 | Aug 2026 | ||
| AIME 2024 Replaced in Feb 2025 | Mathematical Association of America | Fading | gpt-oss-120b 96.6% | 70 | 57 | Feb 2024 | Feb 2026 | ||
| AIME 2023 Replaced in Feb 2024 | Mathematical Association of America | Not in cards | Feb 2023 | ||||||
| ChartQA Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance. | York University and Nanyang Technological University | Documents, charts and text in images | School | In use |  Command A Vision 90.9% | 30 | 23 | Mar 2022 | Aug 2026 |
| MultiChallenge Multi-turn conversations testing instruction retention, inference memory, versioned editing and self-coherence. Graded by LLM judge with rubrics. | Scale AI | Multi-turn conversation | Everyday | In use | GPT-6.1 Sol 82% | 30 | 21 | Jan 2025 | Oct 2026 |
| DocVQA Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers. | CVC, Universitat Autònoma de Barcelona and IIIT Hyderabad | Documents, charts and text in images | Everyday | In use | Seed1.5-VL 96.9% | 30 | 20 | Jul 2020 | Sep 2026 |
| HellaSwag Commonsense sentence completion with adversarially filtered wrong endings. Multiple-choice accuracy. | University of Washington + AllenAI | Commonsense | Everyday | Fading | InternVL3.5-241B-A28B 97% | 29 | 33 | May 2019 | Sep 2026 |
| C-Eval 13,948 Chinese multiple-choice exam questions across 52 disciplines and four difficulty levels. Accuracy. | SJTU + Tsinghua + Edinburgh | Exam questions | University | Fading | Qwen3-Max 93.7% | 29 | 25 | May 2023 | Sep 2026 |
| Codeforces rating Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. | Codeforces (problems); each lab runs its own harness | Competitive programming | Competition | In use | DeepSeek V4.1 Flash 3,471 | 28 | 33 | 2024 | Sep 2026 |
| MMStar Models answer 1,500 human-curated image questions that cannot be solved without the image, across 6 core capabilities. Scored by accuracy. | USTC and Shanghai AI Laboratory | General visual questions | School | In use | Qwen 3.8 Max 85.9% | 28 | 22 | Mar 2024 | Aug 2026 |
| Video-MMMU 300 expert-level lecture videos with questions testing perception, comprehension and adaptation of knowledge. Accuracy. | NTU S-Lab (LMMs-Lab) | Video understanding | University | In use | Qwen 3.8 Max 88.7% | 27 | 16 | Jan 2025 | Aug 2026 |
| Aider Polyglot Models edit code to solve 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python, and Rust. Scored by the share passing unit tests. | Aider | Function-level problems | University | Fading | GPT-5 88% | 26 | 17 | Dec 2024 | Feb 2026 |
| Aider-Edit Edit (2023) Replaced in Dec 2024 | School | Dropped | DeepSeek-V3 79.7% | 3 | 4 | Jul 2023 | Oct 2025 | ||
| NL2Repo-Bench From one natural-language requirements document and an empty workspace, agent builds an installable Python library. Scored by original project's test pass rate. | ByteDance + M-A-P + 2077AI | Building whole projects | Expert | In use | DeepSeek V4.1 Flash 1017 68.2% | 26 | 6 | Dec 2025 | Oct 2026 |
| ARC-Challenge Models answer grade-school science multiple-choice questions that retrieval and word co-occurrence baselines got wrong. Scored by accuracy. | Allen Institute for AI | Exam questions | School | Dropped | Llama 3.1-405B 96.9% | 25 | 39 | Mar 2018 | Nov 2025 |
| MRCR v2 Multi-round coreference: retrieve the n-th of several similar requests hidden in a long synthetic conversation, up to 1M tokens. | Google DeepMind | Finding facts in long inputs | Everyday | In use | Muse Spark 1.3 98.5% | 25 | 16 | Jun 2025 | Oct 2026 |
| MRCR (Michelangelo, original) v1 Replaced in Jun 2025 | Dropped | Gemini 2.5 Pro 94.5% | 6 | 7 | Sep 2024 | Apr 2025 | |||
| BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. | UC Berkeley (Gorilla) | Function calling | Everyday | In use | Qwen3.7-Max 75% | 25 | 11 | Jul 2025 | Oct 2026 |
| Berkeley Function Calling Leaderboard (BFCL) V1 to V3 Replaced in Jul 2025 | In use | Llama 3.1-405B 88.5% | 36 | 31 | Feb 2024 | Oct 2026 | |||
| SuperGPQA 26,529 graduate-level multiple-choice questions across 285 disciplines. Accuracy. | ByteDance Seed + M-A-P | Expert-written questions | Expert | In use | Qwen 3.6 Max (Preview) 73.9% | 24 | 11 | Feb 2025 | Sep 2026 |
| LVBench Hours-long video understanding with multiple-choice questions on reasoning, grounding and summarization. | Zhipu AI / Tsinghua | Long videos | Everyday | In use | Gemini 4 Argon 91.7% | 23 | 18 | Jun 2024 | Oct 2026 |
| RealWorldQA Models answer a question about each of 700+ real-world photos, many taken from vehicles, that test basic spatial understanding. Answers are easy to verify. | xAI | Spatial reasoning and counting | Everyday | In use | Qwen3.8 Flash 88.5% | 23 | 17 | Apr 2024 | Sep 2026 |
| Terminal-Bench 4.0 Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. | Stanford University, Harbor, and Laude Institute | Terminal tasks | Expert | In use | Claude Opus 5.5 64.8% | 23 | 11 | Aug 2026 | Oct 2026 |
| Terminal-Bench 3.0 Replaced in Aug 2026 | Laude Institute and Harbor | In use | Claude Opus 5 42.7% | 5 | 4 | Jul 2026 | Sep 2026 | ||
| Terminal-Bench 2.1 Replaced in Jul 2026 | Stanford + Laude Institute | In use | GPT-6 Astra 87.4% | 53 | 22 | May 2026 | Oct 2026 | ||
| Terminal-Bench 2.0 Replaced in May 2026 | Stanford + Laude Institute | In use | GPT-5.5 84.7% | 59 | 28 | Nov 2025 | Jul 2026 | ||
| Terminal-Bench 1.0 Replaced in Nov 2025 | Stanford University and Laude Institute | Fading | GPT-5 49.6% | 20 | 8 | May 2025 | Dec 2025 | ||
| WinoGrande 44K Winograd-style pronoun-resolution problems. Binary-choice accuracy. | AllenAI | Commonsense | Everyday | Fading | InternVL3.5-241B-A28B 91.7% | 22 | 36 | Jul 2019 | Apr 2026 |
| HumanEval+ Models write Python functions for the 164 HumanEval problems. EvalPlus checks them against about 80 times more tests than the original. | UIUC (EvalPlus) | Function-level problems | School | Fading | ERNIE 5.0 94.5% | 21 | 18 | May 2023 | Apr 2026 |
| ERQA 400 multiple-choice visual questions from robot and egocentric footage on spatial, trajectory, action and state reasoning. Accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | Gemini Robotics ER 2 78.5% | 21 | 12 | Mar 2025 | Sep 2026 |
| LongBench v2 503 multiple-choice questions over 8K-2M-word contexts (QA, dialogue, code repos, structured data). Accuracy. | Tsinghua KEG / Zhipu | Reasoning over long documents | Expert | In use | Beam 1.0 66.9% | 20 | 17 | Dec 2024 | Oct 2026 |
| LongBench v1 Replaced in Dec 2024 | THUDM (Tsinghua University) | University | Dropped | GLM-4 81.1% | 2 | 1 | Aug 2023 | Jul 2025 | |
| ZeroBench 100 hand-made visual questions that all frontier models originally failed. Scored by exact answer, pass@1 and pass@5. | University of Cambridge and others | General visual questions | University | In use | GPT-6 Astra 52% | 20 | 16 | Feb 2025 | Oct 2026 |
| Multi-IF Models follow verifiable instructions over three-turn conversations in eight languages, extending IFEval. Scored by instruction accuracy at each turn. | Meta | Multi-turn conversation | Everyday | In use | LFM3-1.7B 89.1% | 20 | 15 | Oct 2024 | Oct 2026 |
| Arena-Hard v2.0 Models answer 500 hard real user prompts from Chatbot Arena and 250 creative writing prompts. GPT-4.1 or Gemini 2.5 judges each answer against a baseline answer from o3-mini, and the score is the win rate. | Arena (formerly LMArena) | Overall chat quality | University | Fading | Qwen3-Max 90.2% | 20 | 14 | Apr 2025 | Aug 2026 |
| Arena-Hard-Auto 0.1 Replaced in Apr 2025 | Fading |  Mistral Medium 3 97.1% | 27 | 25 | Apr 2024 | Apr 2026 | |||
| ScreenSpot-Pro GUI grounding on high-resolution professional-software screenshots. The model must locate the target UI element. | ScreenSpot-Pro authors | Finding things on screen | University | In use | GPT-6 Astra 92.7% | 20 | 14 | Jan 2025 | Oct 2026 |
| SimpleVQA Models answer 2,025 short factual questions about images across 9 tasks and 9 topics. An LLM judge grades answers against fixed references. | Multimodal Art Projection (M-A-P) et al. | Short factual questions | University | In use | Qwen3.7-Plus 81.7% | 20 | 9 | Feb 2025 | Oct 2026 |
| TriviaQA Trivia questions with evidence documents. Scored by exact match. | University of Washington | Trivia and open questions | Everyday | Fading | Llama 3.1-405B 91.8% | 19 | 31 | May 2017 | Apr 2026 |
| GraphWalks A graph is given as an edge list in long context. The model answers BFS or parent queries, scored by F1. | OpenAI | Reasoning over long documents | University | In use | Claude Fable 5 100% | 19 | 13 | Apr 2025 | Oct 2026 |
| HMMT Nov 2025 Models solve the 30 problems of the November 2025 Harvard-MIT Mathematics Tournament. Final answers are checked automatically against the answer key. | MathArena (ETH Zurich SRI Lab), problems by HMMT | Competition problems with answers | Competition | Fading | GPT-5.2 99.2% | 19 | 11 | Nov 2025 | Jun 2026 |
| BabyVision Model solves 388 perception-heavy visual puzzles (discrimination, tracking, spatial, patterns) that young children solve. Accuracy versus 94% human baseline. | UniPat AI + xbench | Spatial reasoning and counting | Everyday | In use | Qwen 3.8 Max 91.3% | 19 | 8 | Jan 2026 | Oct 2026 |
| BLINK Models answer 3,807 multiple choice questions built from 14 classic computer vision tasks that use one or more images. Scoring is accuracy. | Xingyu Fu et al. | General visual questions | Everyday | In use | Seed2.1 Pro 81.4% | 18 | 23 | Apr 2024 | Aug 2026 |
| HallusionBench Models answer 1,129 yes or no questions about 346 images, built in pairs to expose language hallucination and visual illusion. Scored by accuracy. | University of Maryland | Hallucination rates | Everyday | In use |  Keye-VL-2.0-30B-A3B 73.5% | 18 | 16 | Oct 2023 | Aug 2026 |
| OCRBench v2 Visual-text localization and reasoning tasks across 31 scenarios (documents, scenes, charts). Averaged task accuracy. | HUST (Yuliang Liu et al.) | Documents, charts and text in images | Everyday | In use | Qwen3.6 27B 68.3% | 18 | 14 | Dec 2024 | Oct 2026 |
| OCRBench v1 Replaced in Dec 2024 | Yuliang Liu, Xiang Bai et al. | In use | GLM-OCR 94% | 28 | 20 | May 2023 | Aug 2026 | ||
| τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. | Sierra | Customer service | Everyday | In use | Qwen 3.8 Max 55.1% | 18 | 11 | Mar 2026 | Oct 2026 |
| AGIEval Questions from human standardized exams (gaokao, SAT, LSAT, bar). Multiple-choice and cloze accuracy. | Microsoft | Exam questions | School | Fading |  Olmo 3.1 32B Think 89.2% | 17 | 34 | Apr 2023 | Sep 2026 |
| RefCOCO / RefCOCO+ / RefCOCOg Model grounds a referring expression to a bounding box in a COCO image. Scored by accuracy at IoU 0.5. | UNC Chapel Hill (Licheng Yu et al.) | Detection and captioning | Everyday | In use | Qwen3.5-Omni-Plus 95% | 17 | 14 | Jul 2016 | Aug 2026 |
| AndroidWorld Agents complete 116 tasks across 20 real Android apps in an emulator. Each task checks the device's system state to judge success. | Google Research | Phones | Everyday | In use | GPT-6 Astra 100% | 17 | 9 | May 2024 | Sep 2026 |
| BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. | Researchers from Peking University, Mindverse AI, and other groups | Deep research | Expert | Fading | Seed2.0 Pro 82.4% | 17 | 8 | Apr 2025 | Mar 2026 |
| CMMLU Chinese multiple-choice questions across 67 subjects including China-specific topics. Accuracy. | MBZUAI + others (Li et al.) | Exam questions | University | Fading |  Hunyuan-TurboS 90.8% | 17 | 8 | Jun 2023 | Apr 2026 |
| LiveBench Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. | Abacus.AI, NYU, NVIDIA, et al. | Other | University | Dropped | Claude Fable 5.1 85.7% | 16 | 23 | Jun 2024 | Oct 2026 |
| ZebraLogic Models solve logic grid puzzles of controlled size, derived from constraint satisfaction problems. Scored by the share of puzzles solved fully. | Allen Institute for AI | Abstract puzzles and logic | School | Fading | Qwen3-VL-235B-A22B 97.3% | 16 | 19 | Feb 2025 | Feb 2026 |
| WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. | Translation | University | In use |  K-EXAONE 90.5% | 16 | 18 | Feb 2025 | Oct 2026 | |
| MT-Bench 80 two-turn questions graded by GPT-4 as judge. Introduced LLM-as-judge methodology. | LMSYS | Multi-turn conversation | Everyday | Fading | Qwen2.5-72B 9.35 | 16 | 16 | Jun 2023 | Apr 2026 |
| MMVU 3,000 expert-annotated questions on videos across 27 disciplines. Multiple-choice and open-ended accuracy. | Yale NLP | Video understanding | University | In use | Qwen 3.8 Max 82.4% | 16 | 14 | Jan 2025 | Aug 2026 |
| MMBench-EN v1.1 Models answer English multiple-choice image questions across 20 ability dimensions, using CircularEval with shuffled options. Scored by accuracy on the dev split. | Shanghai AI Laboratory (OpenCompass) | General visual questions | Everyday | In use | Gemini 2.5 Pro 88.5% | 16 | 11 | Apr 2024 | Aug 2026 |
| MMBench 1.0 Replaced in Apr 2024 | Fading | InternVL3-78B 88.8% | 6 | 9 | Jul 2023 | Mar 2026 | |||
| HMMT 2026 Models solve the 33 final-answer problems from the February 2026 Harvard-MIT Mathematics Tournament. Scored by exact answer match. | HMMT (exam), MathArena (evaluation) | Competition problems with answers | Competition | In use | GPT-5.5 98.5% | 16 | 7 | Feb 2026 | Oct 2026 |
| HMMT 2025 Replaced in Feb 2026 | Harvard-MIT Mathematics Tournament (HMMT) | Fading | GPT-5 100% | 44 | 25 | Feb 2025 | Aug 2026 | ||
| HMMT February 2024 Replaced in Feb 2025 | Harvard-MIT Mathematics Tournament (HMMT) | Not in cards | Feb 2024 | ||||||
| HealthBench Professional 525 physician-authored tasks from real clinician chats (care consult, documentation, research) with physician-adjudicated rubrics. | OpenAI | Medicine | Expert | In use | Claude Sonnet 5.5 69.2% | 15 | 31 | Apr 2026 | Oct 2026 |
| PIQA Models pick which of two solutions better achieves an everyday physical goal. Scoring is accuracy. | Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, Yejin Choi | Commonsense | Everyday | Dropped |  Phi-3.5-MoE 88.6% | 15 | 28 | Nov 2019 | Feb 2026 |
| InfographicVQA Models answer about 30,000 questions on about 5,000 infographics that mix text, layout, charts, and graphics. Scoring is ANLS. | Computer Vision Center (UAB) and IIIT Hyderabad | Documents, charts and text in images | School | In use | Kimi K2.5 92.6% | 15 | 14 | Apr 2021 | Aug 2026 |
| CountBench Models count the objects in natural images from the CountBench image-text set. Lab cards often use a question-answer version, CountBenchQA, scored by accuracy. | Google Research | Spatial reasoning and counting | Everyday | In use | Qwen3.6 27B 97.8% | 15 | 13 | Feb 2023 | Aug 2026 |
| AlpacaEval 2.0 Models answer 805 open instructions, and a GPT-4 Turbo judge compares each reply with GPT-4 Turbo's own. The score is a win rate. | Tatsu Lab, Stanford University | Overall chat quality | Everyday | Fading | DeepSeek-R1 87.6% | 15 | 9 | Jan 2024 | Apr 2026 |
| AlpacaEval 1.0 Replaced in Jan 2024 | Stanford Tatsu Lab | Not in cards | GPT-4 Turbo 97.7% | May 2023 | Dec 2023 | ||||
| MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. | University of Tokyo (Li Lab) and collaborators | Knowledge in many languages | University | In use | Qwen3.7-Max 87% | 15 | 8 | Mar 2025 | Oct 2026 |
| MLVU Multi-task long video understanding (minutes to hours) with multiple-choice and generation tasks. Accuracy. | BAAI + partners | Long videos | Everyday | In use | Qwen 3.8 Max 90.8% | 15 | 8 | Jun 2024 | Aug 2026 |
| WideSearch 200 bilingual broad info-seeking tasks: agent collects many atomic facts into a table. Scored by success rate and row/item F1. | ByteDance Seed | Questions that need search | Everyday | In use | Tencent Hy4 preview 83.9% | 15 | 6 | Aug 2025 | Oct 2026 |
| DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. | Google DeepMind | Deep research | Expert | In use | Kimi K3 95% | 14 | 12 | Jan 2026 | Oct 2026 |
| MBPP+ Models write short Python functions for MBPP problems, and EvalPlus checks each one against many more generated tests than the original. | UIUC (EvalPlus) | Function-level problems | School | Fading | Cohere Command A 86.2% | 14 | 10 | Dec 2023 | Apr 2026 |
| Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. | Claw-Eval team (academic) | General assistants | Everyday | In use | Claude Sonnet 4.6 81.4% | 14 | 1 | Mar 2026 | Oct 2026 |
| TruthfulQA Models answer 817 questions across 38 categories that are written to elicit common misconceptions. Scoring checks for truthful answers, often in multiple-choice form. | University of Oxford and OpenAI | Short factual questions | Everyday | Fading | MAI-Thinking-1 88% | 13 | 26 | Sep 2021 | Jun 2026 |
| ProgramBench 200 tasks: rebuild a full program from only its compiled binary and docs. Hidden behavioral tests (248k total) must match the original. | Meta Superintelligence Labs, Stanford University and Harvard University | Building whole projects | Research frontier | In use | Claude Opus 5.5 18.5% | 13 | 14 | May 2026 | Oct 2026 |
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | EPFL and Cohere For AI | Knowledge in many languages | University | Fading | Claude Opus 5 89.8% | 13 | 14 | Nov 2024 | Jul 2026 |
| MVBench Models answer multiple-choice questions across 20 temporal video tasks that a single frame cannot solve. Scored by accuracy. | Shanghai AI Laboratory (OpenGVLab) | Video understanding | Everyday | Fading | Qwen3.5-Omni-Plus 79% | 13 | 10 | Nov 2023 | Aug 2026 |
| Beyond AIME Models answer 100 original competition-style math problems at or above the hardest AIME level. Each answer is a positive integer checked by exact match. | ByteDance Seed | Competition problems with answers | Competition | Fading | Seed2.1 Pro 87% | 13 | 8 | Apr 2025 | Jun 2026 |
| SimpleQA Verified 1,000 human-verified, de-duplicated short-answer factual questions measuring parametric knowledge. | Google DeepMind | Short factual questions | University | In use | Gemini 3.1 Pro 77.5% | 12 | 18 | Sep 2025 | Oct 2026 |
| SimpleQA Original Replaced in Sep 2025 | OpenAI | Fading | DeepSeek-V3.2-Exp 97.1% | 42 | 40 | Oct 2024 | Apr 2026 | ||
| MMLongBench-Doc Models answer 1,062 questions about 130 long PDF documents averaging 49 pages, and some questions are unanswerable. Scoring is accuracy and F1. | Yubo Ma et al. | Documents, charts and text in images | University | Fading | Nemotron 3.5 Super VL 62.8% | 12 | 10 | Jul 2024 | Oct 2026 |
| GDP.pdf 100 expert multimodal reasoning tasks over real economically important PDF documents. Scored by rubric-criteria pass rate. | Surge AI | Documents, charts and text in images | Expert | In use | GPT-6.1 Sol 35% | 12 | 8 | Jul 2026 | Oct 2026 |
| MathArena (Apex, IMO, USAMO) Evaluates models on newly released math competitions after release. Final-answer or human/LLM-judged proof scores. | ETH Zurich SRI Lab | Competition problems with answers | Competition | In use | Claude Opus 4.8 81.3% | 12 | 7 | Feb 2025 | Oct 2026 |
| JobBench 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. | University of Washington + collaborators | Tasks from real jobs | Expert | In use | GLM-5.3 61.4% | 12 | 5 | May 2026 | Oct 2026 |
| COLLIE Models generate text that meets compositional constraints at word, sentence, paragraph, and passage level. Scored by the share passing automatic constraint checks. | Princeton NLP | Following precise instructions | Everyday | Fading | GPT-5 99% | 12 | 5 | Jul 2023 | May 2026 |
| EgoSchema Models answer five-way multiple-choice questions about three-minute egocentric video clips from Ego4D. Scored by accuracy on 5,031 questions. | UC Berkeley | Long videos | Everyday | Dropped | Qwen2.5-VL-72B 76.2% | 12 | 5 | Aug 2023 | Apr 2025 |
| SkillsBench 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. | BenchFlow + 105 domain experts | Terminal tasks | Expert | In use | GPT-5.5 67.3% | 12 | 2 | Feb 2026 | Oct 2026 |
| BoolQ Models read a Wikipedia passage and answer a naturally occurring yes or no question about it. Scored by accuracy. | Reading comprehension | Everyday | Dropped | PaLM 2-L 90.9% | 11 | 25 | May 2019 | Jun 2025 | |
| ARC-Easy Models answer grade-school science multiple-choice questions from the ARC Easy Set, which has 5,197 questions. Scoring is accuracy. | Allen Institute for AI | Exam questions | School | Dropped | DeepSeek-V3 98.9% | 11 | 20 | Mar 2018 | Jun 2025 |
| MultiPL-E Models write code for HumanEval and MBPP problems translated into 18 other programming languages. Scored by pass@1 on unit tests. | Northeastern University Programming Research Lab (NuPRL) et al. | Function-level problems | School | Fading | qwen3-coder-next 88.2% | 11 | 19 | Aug 2022 | Apr 2026 |
| DynaMath Models answer visual math questions generated from 501 seed programs that vary values and graphs. Scored by average and worst-case accuracy across variants. | University of Illinois Urbana-Champaign and UC Berkeley | Visual math and diagrams | School | Fading | Qwen 3.6 Plus 88% | 11 | 12 | Oct 2024 | Jun 2026 |
| PolyMath Models solve 9,000 parallel math problems in 18 languages across four difficulty levels, from school math to frontier math. Final answers are checked. | Qwen Team, Alibaba Group | Competition problems with answers | Competition | In use | Qwen3.7-Max 86.5% | 11 | 11 | Apr 2025 | Aug 2026 |
| WebVoyager Agents complete user tasks on 15 popular real websites by browsing from screenshots and page text. A GPT-4V judge grades whether each task succeeded. | WebVoyager team (He et al., ACL 2024) | Web browsing | Everyday | Fading | Fara1.5-27B 89.3% | 11 | 8 | Jan 2024 | Oct 2026 |
| MedXpertQA-MM Models answer expert-level medical exam questions that combine images with clinical records and test results. Scoring is accuracy. | Tsinghua University (TsinghuaC3I) | Medicine | Expert | In use | Qwen 3.8 Max 80.4% | 11 | 5 | Jan 2025 | Oct 2026 |
| SWE Atlas Scale's coding-agent suite beyond patching: codebase Q&A, writing production tests, and refactoring in real repos. Rubric/test-scored leaderboards. | Scale AI | Fixing real repositories | Expert | In use | Claude Opus 5 63.2% | 11 | 2 | Mar 2026 | Oct 2026 |
| GDPval Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. | OpenAI | Tasks from real jobs | Expert | In use | Seed2.1 Pro 87.9% | 11 | 2 | Sep 2025 | Sep 2026 |
| HealthBench 5,000 realistic health conversations graded against physician-written rubrics. A length-adjusted variant penalizes verbosity. | OpenAI | Medicine | Expert | In use | GPT-5 67.2% | 10 | 35 | May 2025 | Oct 2026 |
| WritingBench Models complete 1,000 real-world writing requests across 6 domains and 100 subdomains. A judge model scores each reply on five query-specific criteria. | Alibaba Group (X-PLUG) | Creative writing | University | Fading | GPT-5 83.9 | 10 | 11 | Mar 2025 | Jul 2026 |
| TextVQA Models answer 45,336 questions about 28,408 images that require reading text in the image. Scoring is VQA accuracy against human answers. | Facebook AI Research | Documents, charts and text in images | Everyday | Fading | Command A Vision 84.8% | 10 | 11 | Apr 2019 | Aug 2026 |
| Chinese SimpleQA 3,000 short Chinese factual questions across six topics. Graded correct, incorrect or not attempted. | Alibaba (Taobao & Tmall) | Short factual questions | University | Fading | ERNIE 5.0 86% | 10 | 10 | Nov 2024 | Jun 2026 |
| SWE-Lancer Models complete real Upwork software tasks, from bug fixes to features, and pick between engineering proposals. Scored by end-to-end tests and payout earned. | OpenAI | Fixing real repositories | Expert | Fading | GPT-5.3 Codex 81.4% | 10 | 9 | Feb 2025 | Feb 2026 |
| OJBench Models write C++ or Python solutions to 232 competition problems from NOI and ICPC. Judge test cases score each solution by pass@1. | Beijing University of Posts and Telecommunications, Tsinghua University, and others | Competitive programming | Competition | Fading | Gemini 3 Pro 70% | 10 | 7 | Jun 2025 | Jun 2026 |
| Chartography 100 tasks on specialized professional charts (Kaplan-Meier, candlestick, Sankey, Bode). Answers graded against expert references. | Surge AI | Documents, charts and text in images | Expert | In use | Kimi K3.1 Pro 90.2% | 10 | 7 | Aug 2026 | Oct 2026 |
| V* Models answer 191 multiple choice questions about small details in high-resolution images, such as an object's color or relative position. | Penghao Wu and Saining Xie | General visual questions | Everyday | Fading | Qwen 3.6 Plus 96.9% | 10 | 5 | Dec 2023 | Oct 2026 |
| MCPMark Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. | eval-sys (MCPMark team) | Apps and MCP servers | University | In use | Kimi K3 94.5% | 10 | 3 | Sep 2025 | Jul 2026 |
| Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. | Claw-Eval team | General assistants | University | In use | Claude Opus 4.6 70.4% | 10 | 2 | Apr 2026 | Oct 2026 |
| OpenBookQA Models answer elementary science multiple-choice questions that need a given core science fact plus common knowledge. Scored by accuracy. | Allen Institute for AI | Exam questions | School | Fading | Hermes 4.3 36B 96.6% | 9 | 18 | Sep 2018 | Dec 2025 |
| VoiceBench Spoken instructions with speaker, environment and content variation. Voice assistant responses scored by GPT judge or rules. | CUHK-Shenzhen | Audio understanding and voice | Everyday | In use | Qwen3.5-Omni-Plus 93.1% | 9 | 12 | Oct 2024 | Oct 2026 |
| ScreenSpot-v2 Models locate a described element on mobile, desktop and web screenshots. A click inside the target box counts as correct. | OS-Atlas team (Shanghai AI Lab and others) | Finding things on screen | Everyday | Fading | Seed1.5-VL 95.2% | 9 | 12 | Oct 2024 | Oct 2026 |
| ScreenSpot v1 Replaced in Oct 2024 | Nanjing University (SeeClick) | Fading | Qwen3-VL-235B-A22B 95.4% | 8 | 9 | Jan 2024 | Apr 2026 | ||
| FLEURS Models transcribe read speech in 102 languages built from FLoRes-101 sentences. Model cards usually report word error rate, so lower scores are better. | Speech recognition and translation | Everyday | Fading | Phi-4-multimodal 4.00% | 9 | 12 | May 2022 | Oct 2026 | |
| SWE-bench Multimodal SWE-bench tasks whose issues include screenshots or design mockups from visual JavaScript repos. Patch graded by hidden tests. | SWE-bench team (Princeton/Stanford) | Fixing real repositories | Expert | In use | Claude Opus 5.5 61.4% | 9 | 9 | Oct 2024 | Oct 2026 |
| Terminal-Bench Hard Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. | Artificial Analysis | Terminal tasks | Expert | In use | Gemma 4 31B IT 36% | 9 | 7 | Sep 2025 | Jul 2026 |
| Multi-SWE-bench Agents resolve 1,632 GitHub issues in Java, TypeScript, JavaScript, Go, Rust, C, and C++ repos. Patches must pass the repos' tests. | ByteDance Seed | Fixing real repositories | Expert | Fading |  MiniMax-M2.7 52.7% | 9 | 6 | Apr 2025 | May 2026 |
| SIQA Models answer multiple-choice questions about people's motives, reactions, and emotions in everyday social situations. Scored by accuracy. | Allen Institute for AI and University of Washington | Commonsense | Everyday | Dropped | phi-3-medium 14B 80.2% | 8 | 21 | Apr 2019 | Jun 2025 |
| Natural Questions Models answer real Google search questions. Model cards usually use the open-domain version, scored by exact match against short answers from Wikipedia. | Trivia and open questions | Everyday | Dropped | DeepSeek-V2 (MoE-236B, May 2024) 53.4% | 8 | 18 | Jan 2019 | Aug 2025 | |
| Creative Writing v3 Models write responses to 32 creative writing prompts, three times each. A judge model scores them with a rubric and pairwise Elo comparisons. | EQ-Bench | Creative writing | University | Dropped | Kimi K3.1 Pro 83.2 | 8 | 11 | Mar 2025 | Oct 2026 |
| MuirBench Models answer multiple choice questions over sets of several images, including unanswerable variants. Scored by accuracy. | Fei Wang, Xingyu Fu et al. | General visual questions | Everyday | Fading | Seed2.0 Pro 81.8% | 8 | 11 | Jun 2024 | Aug 2026 |
| CoVoST2 Models translate speech from 21 languages into English and from English into 15 languages. Scored by BLEU. | Meta AI (Facebook AI) | Speech recognition and translation | University | Fading |  Nova 2 Omni Preview 40.7 | 8 | 9 | Jul 2020 | Oct 2026 |
| We-Math Models solve visual math problems decomposed into sub-problems by knowledge concept. Scoring compares sub-problem and full-problem results to diagnose knowledge and generalization gaps. | Beijing University of Posts and Telecommunications | Visual math and diagrams | School | Fading | Qwen 3.6 Plus 89% | 8 | 9 | Jul 2024 | Jun 2026 |
| MM-Vet Models answer open questions about images that combine recognition, OCR, knowledge, spatial reasoning, language generation and math. GPT-4 grades each answer against a reference. | Weihao Yu, Zhengyuan Yang et al. (NUS and Microsoft) | General visual questions | Everyday | Dropped | InternVL3.5-241B-A28B 81.2% | 8 | 7 | Aug 2023 | Oct 2025 |
| BigCodeBench 1,140 practical Python tasks requiring calls to 139 libraries. Scored pass@1 on unit tests. | BigCode project | Function-level problems | University | Fading | DeepSeek V4.1 Flash 60.6% | 8 | 6 | Jun 2024 | Sep 2026 |
| Vibe-Eval Models answer 269 image-based prompts, 100 of them hard, that have expert gold responses. A Reka Core judge rates each answer. | Reka AI | General visual questions | Everyday | Dropped | Seed2.0 Pro 81.4% | 8 | 6 | May 2024 | Feb 2026 |
| Online-Mind2Web Web agents do 300 tasks on 136 live websites. An LLM judge reads the agent's trajectory and decides whether the task succeeded. | Ohio State University (OSU NLP Group) | Web browsing | Everyday | Fading | UI-TARS-2 88.2% | 8 | 6 | Apr 2025 | Jun 2026 |
| Terminal-Bench-Science 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. | Stanford-led community (Terminal-Bench team + domain experts) | Research science | Expert | In use | GPT-6 Astra 68.1% | 8 | 5 | Aug 2026 | Oct 2026 |
| EmbSpatialBench Models answer multiple-choice questions on six spatial relations between objects in egocentric embodied scenes. Scored by accuracy. | Fudan University | Spatial reasoning and counting | Everyday | Fading | Qwen3.5-Omni-Plus 85.4% | 8 | 5 | Jun 2024 | Jun 2026 |
| LiveCodeBench Pro Continuously updated Codeforces, ICPC and IOI problems annotated by olympiad medalists. Reported as Elo. | LiveCodeBench Pro team | Competitive programming | Competition | Fading | Gemini 3.1 Pro 2,887 | 8 | 5 | Jun 2025 | Oct 2026 |
| ODinW-13 Models detect objects in 13 varied real-world datasets, such as aquarium and aerial images, from text category names. Scored by mean average precision. | Microsoft Research | Detection and captioning | Everyday | Fading | Qwen 3.6 Plus 51.8% | 8 | 4 | Dec 2021 | Jun 2026 |
| WorldVQA Model names visual entities in 3,500 image-question pairs across nine categories, isolating memorized visual world knowledge. Exact-answer accuracy. | Moonshot AI | General visual questions | University | In use | Qwen3.7-Plus 61.1% | 8 | 4 | Jan 2026 | Oct 2026 |
| GDPval-AA v2.1 Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. | Artificial Analysis | Tasks from real jobs | Expert | In use | Claude Opus 5.5 1,846 | 8 | 4 | Sep 2026 | Oct 2026 |
| GDPval-AA v2 Replaced in Sep 2026 | In use | Claude Opus 5 1,861 | 25 | 18 | Jun 2026 | Oct 2026 | |||
| GDPval-AA v1 Replaced in Jun 2026 | Fading | Claude Fable 5 1,932 | 14 | 14 | Dec 2025 | Jun 2026 | |||
| Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. | MRL Benchmarks (over 350 researchers) | Reasoning in many languages | Everyday | Fading | Gemini 3 Pro 93.4% | 8 | 1 | Oct 2025 | Jun 2026 |
| HealthBench Hard Models answer the 1,000 hardest HealthBench conversations, graded against physician-written rubrics. | OpenAI | Medicine | Expert | Fading | GPT-5 46.2% | 7 | 28 | May 2025 | Oct 2026 |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | UC Berkeley (Dawn Song lab) | Finding and exploiting real bugs | Expert | In use | GPT-6 Astra 42.4% | 7 | 20 | May 2026 | Oct 2026 |
| Bird-SQL Models convert natural language questions into SQL over 95 large real databases across 37 domains. Scored by execution accuracy. | HKU and Alibaba DAMO Academy | Data science and SQL | University | Fading | qwen3-coder-next 63.6% | 7 | 10 | May 2023 | Oct 2026 |
| LongVideoBench Models answer referring reasoning questions over 3,763 web videos up to an hour long, with interleaved subtitles. Scored by multiple-choice accuracy. | NTU S-Lab et al. | Long videos | Everyday | Fading | Seed2.1 Pro 80.6% | 7 | 10 | Jul 2024 | Jun 2026 |
| LogicVista Models answer 448 multiple-choice questions on spatial, deductive, inductive, numeric, and mechanical reasoning set in images. Scored by accuracy. | UCLA | Visual math and diagrams | School | In use | Qwen 3.8 Max 91.9% | 7 | 10 | Jul 2024 | Aug 2026 |
| Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | HKUST NLP | Apps and MCP servers | University | In use | GLM-5.3-Flash 78.4% | 7 | 9 | Jun 2026 | Oct 2026 |
| Toolathlon Original Replaced in Jun 2026 | In use | Gemini 3.5 Flash 56.5% | 31 | 12 | Oct 2025 | Aug 2026 | |||
| MotionBench Models answer questions about fine-grained motion in video clips across six motion question types. Scoring is accuracy. | Zhipu AI and Tsinghua University (THUDM) | Video understanding | Everyday | Fading | Seed2.0 Pro 75.2% | 7 | 8 | Jan 2025 | Jun 2026 |
| Common Voice Crowd-sourced multilingual read-speech corpus. Used here as an ASR test set scored by word error rate. | Mozilla | Speech recognition and translation | Everyday | Fading | Phi-4-multimodal 6.80% | 7 | 8 | Dec 2019 | Apr 2026 |
| LibriSpeech Models transcribe English audiobook speech. Scored by word error rate on the test-clean and test-other splits. | Panayotov et al. (Johns Hopkins University) | Speech recognition and translation | Everyday | Fading | Voxtral Small 1.53% | 7 | 8 | Apr 2015 | Apr 2026 |
| HiddenMath Models solve 179 competition-level math problems that experts wrote from scratch. Answers are graded automatically. | Google DeepMind | Competition problems with answers | Competition | Dropped | DiffusionGemma 26B A4B 80.6% | 7 | 7 | May 2024 | Jul 2026 |
| ChartQA Pro Model answers 1,948 diverse questions over 1,341 charts, infographics and dashboards, including conversational and unanswerable ones. Accuracy. | vis-nlp (Masry et al.) | Documents, charts and text in images | Everyday | Fading | Claude Opus 4.8 72.3% | 7 | 7 | Apr 2025 | Sep 2026 |
| WenetSpeech Models transcribe Mandarin speech from many domains. Scored by character error rate on the test-net and test-meeting sets. | Northwestern Polytechnical University (ASLP@NPU), Mobvoi and others | Speech recognition and translation | Everyday | Fading | 7 | 7 | Oct 2021 | Sep 2026 | |
| KMMLU-Pro Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry. | LG AI Research and collaborators | Expert-written questions | Expert | In use | Solar Pro 4 79.2% | 7 | 6 | Jul 2025 | Aug 2026 |
| CRUXEval Models predict the output of 800 short Python functions for a given input, or an input that yields a given output. Scored by pass@1. | Meta AI and MIT CSAIL | Function-level problems | School | Fading | qwen3-coder-next 95.9% | 7 | 5 | Jan 2024 | Apr 2026 |
| OpenAI MRCR v2 Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin. | OpenAI | Finding facts in long inputs | Everyday | In use | K-EXAONE 2.0 94.4% | 7 | 5 | Dec 2025 | Oct 2026 |
| OpenAI MRCR v1 Replaced in Dec 2025 | Dropped | K-EXAONE 2.0 94.4% | 6 | 7 | Apr 2025 | Aug 2026 | |||
| VLMs Are Blind Models answer seven simple visual tasks such as counting line crossings or overlapping circles, which young children solve easily. Scored by accuracy. | Auburn University and University of Alberta | Spatial reasoning and counting | Everyday | Fading | Seed2.0 Pro 98.6% | 7 | 5 | Jul 2024 | Apr 2026 |
| PopQA Models answer about 14k entity questions built from Wikidata, spread from popular to rare entities. Scored by whether the answer matches a gold alias. | University of Washington (Mallen et al.) | Trivia and open questions | Everyday | Fading | Tulu 3 405B 55.5% | 7 | 5 | Dec 2022 | Dec 2025 |
| TempCompass Video models answer questions about temporal aspects of short clips, such as action order, speed, direction and attribute change, in several question formats. | Peking University and Huawei Noah Ark Lab (Liu et al.) | Video understanding | Everyday | Fading | Seed2.0 Pro 89.6% | 7 | 5 | Mar 2024 | Mar 2026 |
| TVBench Models answer multiple-choice questions on 10 temporal video tasks, such as action counting and ordering, that static frames cannot solve. Scored by accuracy. | University of Santiago de Compostela and University of Amsterdam et al. | Video understanding | Everyday | In use | Qwen 3.8 Max 81.9% | 7 | 4 | Oct 2024 | Aug 2026 |
| FinSearchComp Agents search the web to fetch live financial data, look up history, and run multi-step investigations, across 635 expert-written questions. Scored by answer accuracy. | ByteDance Seed | Finance | Expert | Fading |  Ling-3.1-flash 81.7% | 7 | 3 | Sep 2025 | Oct 2026 |
| LIBERO Simulated Franka tabletop tasks in four 10-task suites. Policies fine-tuned on 50 demos per task. Success rate over ~50 trials. | UT Austin | Simulated robots | Everyday | Fading | MolmoAct 2 97.2% | 7 | 3 | Jun 2023 | May 2026 |
| Claw-Eval-MM Agents complete the 101 multimodal tasks of Claw-Eval, such as video questions, document extraction, and webpage generation. Rubric items graded from run traces set scores. | Claw-Eval team | General visual questions | University | In use | Claude Opus 4.6 54.7% | 7 | 2 | Apr 2026 | Sep 2026 |
| FrontierScience Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. | OpenAI | Research science | Expert | In use | Intern-S2-397B 87% | 7 | 1 | Dec 2025 | Sep 2026 |
| AlignBench Models answer 683 real Chinese user queries across eight categories. A GPT-4 judge rates each answer from 1 to 10 against a reference. | Tsinghua University and Zhipu AI (THUDM) | Overall chat quality | Everyday | Dropped | Qwen3-235B-A22B 8.94 | 6 | 13 | Nov 2023 | Sep 2025 |
| MathVerse Models solve diagram-based math problems shown in six versions that move information between the text and the image. It tests whether models read the diagram. | CUHK MMLab and Shanghai AI Laboratory | Visual math and diagrams | School | Fading | Seed2.1 Pro 89.7% | 6 | 11 | Mar 2024 | Jun 2026 |
| BIG-Bench Extra Hard Models solve harder replacements for each of the 23 BIG-Bench Hard tasks, 4,520 examples in total. The score is a harmonic mean of task accuracies. | Google DeepMind | Language reasoning | University | In use | Gemma 4 31B IT 74.4% | 6 | 10 | Feb 2025 | Sep 2026 |
| BioMysteryBench Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. | Anthropic | Lab and data work | Expert | In use | Claude Opus 5 90.1% | 6 | 10 | Apr 2026 | Oct 2026 |
| CVBench Models answer 2,638 manually checked 2D and 3D vision questions built from ADE20K, COCO, and Omni3D. Scoring is accuracy. | New York University (Cambrian-1 team) | Spatial reasoning and counting | Everyday | Fading | Seed1.8 88% | 6 | 9 | Jun 2024 | Sep 2026 |
| FRAMES Models answer 824 multi-hop questions that need facts from 2 to 15 Wikipedia articles. Answers are graded against gold answers for accuracy. | Questions that need search | Everyday | Fading | Grok 4.1 Fast 87.6% | 6 | 6 | Sep 2024 | Feb 2026 | |
| CNMO 2024 Models answer problems from the 2024 Chinese National High School Mathematics Olympiad. DeepSeek reports Pass@1 averaged over 16 samples. | Chinese Mathematical Society (contest), DeepSeek (eval set) | Competition problems with answers | Competition | Dropped | Ring-1T 88.5% | 6 | 5 | Dec 2024 | Jan 2026 |
| Daily-Omni Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy. | Ziwei Zhou, Rui Wang et al. | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 85.1% | 6 | 5 | May 2025 | Sep 2026 |
| Tulu 3 Safety Eval Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average. | Ai2 | Jailbreaks and prompt injection | Everyday | Fading | OLMo 2 1B Instruct 87.6% | 6 | 5 | Nov 2024 | Apr 2026 |
| OSWorld 2.1 Agents complete 108 long-horizon workflows across 31 self-hosted websites and desktop apps, averaging 318 tool calls. Scored by binary completion and checkpoint partial credit. | XLANG Lab, HKU | Desktop apps | University | In use | Claude Opus 5.5 81.8% | 6 | 4 | Sep 2026 | Oct 2026 |
| OSWorld 2.0 (Aug 2026) 2.0 Aug Replaced in Sep 2026 | System cards only | Claude Fable 5.1 77.9% | 1 | Aug 2026 | Sep 2026 | ||||
| OSWorld 2.0 Replaced in Aug 2026 | In use | Claude Opus 4.8 20.6% | 18 | 6 | Jun 2026 | Oct 2026 | |||
| OSWorld-Verified Replaced in Jun 2026 | XLANG Lab (HKU) + collaborators | Everyday | In use | Claude Fable 5 86% | 36 | 19 | Jul 2025 | Oct 2026 | |
| OSWorld Original Replaced in Jul 2025 | Everyday | Fading | Claude Sonnet 4.6 72.1% | 11 | 10 | Apr 2024 | Jun 2026 | ||
| VideoMME v2 (with subtitles) Models answer multiple choice questions about videos with subtitles or audio added. Related questions are scored as groups to penalize guessing. | Video-MME team (Chaoyou Fu et al.) | Video understanding | Everyday | In use | Qwen 3.8 Max 68.3% | 6 | 3 | Apr 2026 | Oct 2026 |
| Video-MME v1 Replaced in Apr 2026 | Video-MME team (USTC, XMU, et al.) | In use | Qwen 3.8 Max 90.4% | 39 | 26 | May 2024 | Aug 2026 | ||
| VQAv2 Open-ended questions about COCO images, balanced to reduce language priors. Scored by consensus accuracy against 10 human answers. | Virginia Tech + Georgia Tech | General visual questions | Everyday | Fading | Reka Edge 88.4% | 6 | 3 | Dec 2016 | Mar 2026 |
| AA-Briefcase v1.1 Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. | Artificial Analysis | Tasks from real jobs | Expert | In use | Claude Opus 5.5 1,822 | 6 | 3 | Sep 2026 | Oct 2026 |
| AA-Briefcase v1 Replaced in Sep 2026 | In use | Claude Opus 5 1,720 | 5 | 6 | 2026 | Sep 2026 | |||
| CL-bench Model must learn new knowledge from a supplied long context (rules, procedures, data) and solve tasks. Strict LLM-verified rubrics. | Tencent Hunyuan (with Fudan University authors) | Learning from the context | University | In use | GPT-5.4 27.9% | 6 | 2 | Feb 2026 | Jul 2026 |
| MuSR Models answer questions about long generated narratives such as murder mysteries, object placement and team allocation. Each item needs several soft reasoning steps over the story. | UT Austin | Language reasoning | University | Fading | ERNIE-4.5-21B-A3B-Thinking 86.7% | 6 | 2 | Oct 2023 | Dec 2025 |
| Big Bench Audio Models answer 1,000 spoken questions from four Big Bench Hard tasks: formal fallacies, navigate, object counting and web of lies. Built to test reasoning over audio input. | Artificial Analysis | Audio understanding and voice | School | In use | Grok Voice 3.0 98.5% | 6 | 1 | Dec 2024 | Oct 2026 |
| CoWorkBench Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). | Alibaba Qwen | Tasks from real jobs | Expert | In use | Qwen3.8-Omni-Flash 75.3% | 6 | 1 | Sep 2026 | |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | Qwen Team, Alibaba Group | General assistants | Everyday | Fading | Qwen3.7-Plus 62.3% | 6 | Jan 2026 | Jun 2026 | |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | Qwen Team, Alibaba Group | General assistants | Everyday | Fading | Qwen3.7-Max 64.3% | 6 | Apr 2026 | Jun 2026 | |
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Carnegie Mellon (Lee & Brumley) | Finding and exploiting real bugs | Expert | In use | GPT-6 Astra 100% | 5 | 25 | May 2026 | Oct 2026 |
| LMArena (Elo) People chat with two anonymous models side by side and vote for the better reply. Votes become Elo-style ratings for each model. | Arena (formerly LMArena) | Overall chat quality | Fading | Gemini 4 Carbon 1,528 | 5 | 13 | May 2023 | Oct 2026 | |
| OfficeQA Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. | Databricks | Documents and spreadsheets | University | In use | Claude Opus 4.7 86.3% | 5 | 10 | Mar 2026 | Oct 2026 |
| Vending-Bench 2 Agent runs a simulated vending-machine business for one year starting with $500. Score is the final bank balance, averaged over runs. | Andon Labs | Running a business | Everyday | Fading | Gemini 4 Carbon $16,484 | 5 | 7 | Nov 2025 | Oct 2026 |
| Vending-Bench v1 Replaced in Nov 2025 | Dropped | Grok 4 $4,694 | 1 | Feb 2025 | Nov 2025 | ||||
| CLUEWSC Models decide whether a pronoun in a Chinese sentence refers to a given noun phrase, a Winograd-style task. Scored by accuracy. | CLUE benchmark team | Reasoning in many languages | Everyday | Fading | DeepSeek-R1 92.8% | 5 | 7 | Mar 2020 | Apr 2026 |
| MME Models answer yes or no questions about images across 14 perception and cognition subtasks with hand-written instructions. Labs usually report the perception sum, out of 2000. | Fu et al. (Tencent Youtu Lab and Xiamen University) | General visual questions | Everyday | Fading | InternVL3.5-241B-A28B 2525.9 | 5 | 6 | Jun 2023 | Aug 2026 |
| VisuLogic Models answer 1,000 visual reasoning puzzles in six categories that cannot be solved by describing the image in words. Scored by accuracy. | Shanghai AI Laboratory with several universities | Spatial reasoning and counting | Everyday | Fading | Seed2.1 Pro 54.3% | 5 | 6 | Apr 2025 | Jun 2026 |
| PutnamBench Models write formal Lean proofs of Putnam competition problems. A proof counts only when the verifier accepts it. | UT Austin (Tsoukalas et al.) | Formal proofs | Competition | Fading | Leanstral 1.5 87.5% | 5 | 5 | Jul 2024 | Jul 2026 |
| Natural2Code Models write Python functions in HumanEval format from new prompts kept off the web. Unit tests decide correctness. | Google DeepMind | Function-level problems | School | Dropped | DiffusionGemma 26B A4B 94% | 5 | 5 | Dec 2023 | Jul 2026 |
| WorldSense (omnimodal) Models answer multiple choice questions about real-world videos that need both the visual and the audio track. Scored by accuracy. | Jack Hong, Shilin Yan et al. | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 68.5% | 5 | 5 | Feb 2025 | Sep 2026 |
| ArtifactsBench Models generate interactive visual code artifacts for 1,825 tasks. An MLLM judge scores rendered screenshots and code against a per-task checklist. | Tencent Hunyuan | Web and app front ends | University | In use | MiniMax-M2 66.8% | 5 | 4 | Jul 2025 | Jul 2026 |
| miniF2F Models write formal proofs of olympiad and high school competition statements, such as AMC, AIME and IMO problems. A proof assistant checks each proof, and scores are pass@k on the 244 test problems. | OpenAI (Kunhao Zheng, Jesse Michael Han, Stanislas Polu) | Formal proofs | Competition | Fading | Leanstral 1.5 100% | 5 | 4 | Aug 2021 | Jul 2026 |
| FActScore Models write biographies of people, and each output is split into atomic facts. The score is the share of facts that a knowledge source supports. | Sewon Min et al. (University of Washington and collaborators) | Long-form factuality | Everyday | Fading | GPT-5 1.00% | 5 | 4 | May 2023 | Feb 2026 |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | Meta AI, Hugging Face, and AutoGPT | General assistants | Everyday | Fading | GPT-5.5 91.7% | 5 | 3 | Nov 2023 | Oct 2026 |
| AdvancedIF Models respond to over 1,600 prompts with complex, multi-turn, and system-level instructions. A judge model checks each response against expert-written rubrics. | Meta Superintelligence Labs | Following precise instructions | Everyday | In use | MAI-Thinking-1 85% | 5 | 3 | Nov 2025 | Sep 2026 |
| CMATH 1.7K Chinese elementary-school math word problems. Final-answer accuracy. | Xiaomi AI Lab | Word problems | School | Fading | ERNIE-4.5-300B-A47B 96.7% | 5 | 3 | Jun 2023 | Apr 2026 |
| ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. | Scale AI | Deep research | Expert | Fading | Kimi K3 76.2% | 5 | 3 | Nov 2025 | Jul 2026 |
| SpreadsheetBench Real spreadsheet manipulation questions from Excel forums. Output workbook is checked cell-by-cell against reference solutions. | Renmin University (RUCKBReasoning) | Documents and spreadsheets | University | In use | MiniMax-M3 89.3% | 5 | 2 | Jun 2024 | Jun 2026 |
| MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. | ByteDance Seed and collaborators (Li et al.) | Deep research | Expert | Fading | MiMo-V2-Omni 52% | 5 | 2 | Aug 2025 | Jun 2026 |
| Seal-0 Models answer 111 fact-seeking questions where web search returns conflicting or unhelpful results. Answers are graded for accuracy, usually with search enabled. | Virginia Tech | Questions that need search | University | Fading | Kimi K2.5 57.4% | 5 | 2 | Jun 2025 | Feb 2026 |
| Vision2Web Agents build websites from 918 prototype images across 193 tasks, from static pages to full-stack apps. A GUI agent and a VLM judge score them. | Z.ai (zai-org) | Web and app front ends | Expert | In use | Claude Fable 5 82.3% | 5 | 1 | Mar 2026 | Sep 2026 |
| AA-LCR 1.1 Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key. | Artificial Analysis | Reasoning over long documents | University | In use |  Step 5 Preview 88.3% | 5 | 1 | Sep 2026 | Oct 2026 |
| AA-LCR (Artificial Analysis Long Context Reasoning) 1.0 Replaced in Sep 2026 | In use | Muse Glimmer 80% | 29 | 18 | Aug 2025 | Oct 2026 | |||
| LongFact Models write long answers to 2,280 fact-seeking prompts on 38 topics. A search-backed checker verifies each claim, and cards report the hallucination rate. | Google DeepMind | Long-form factuality | Fading | GPT-5 mini 0.70% | 5 | 1 | Mar 2024 | Jun 2026 | |
| ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. | Apple | General assistants | Everyday | In use | LFM2.5-24B-A2B 80.7% | 5 | 1 | Aug 2024 | Oct 2026 |
| AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. | IBM Research | Refusing harmful requests | Everyday | Fading | Granite 3.3 8B Instruct 88.5% | 5 | Sep 2023 | Apr 2026 | |
| MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. | OPPO (Mente Lab) | Chat and instructions in many languages | Everyday | Fading | Qwen3.7-Max 89.2% | 5 | Jun 2025 | Jun 2026 | |
| NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. | Peking University and Alibaba Group | Knowledge in many languages | University | Fading | Qwen3.5 397B-A17B 59.1% | 5 | Nov 2025 | Jun 2026 | |
| DSBench (DeepSeek internal) DeepSeek internal coding-agent test sets: full-stack development tasks and hard agentic coding problems. Pass-rate scoring. | DeepSeek | Building whole projects | Expert | In use | DeepSeek V4 Pro 0813 71.1% | 5 | Aug 2026 | Aug 2026 | |
| PaperBench Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. | OpenAI | AI research tasks | Research frontier | In use | Qwen 3.8 Max 93% | 4 | 12 | Apr 2025 | Aug 2026 |
| Needle In A Haystack Models retrieve a planted fact from long filler text at varied context lengths and insertion depths. Scored by retrieval accuracy. | Greg Kamradt | Finding facts in long inputs | Everyday | Dropped | GLM-4 100% | 4 | 11 | Nov 2023 | Jan 2025 |
| DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. | Perplexity | Deep research | Expert | In use | Claude Opus 5 88.6% | 4 | 7 | Feb 2026 | Oct 2026 |
| BenchCAD Programmatic CAD: 17,900 execution-verified CadQuery programs over 106 industrial part families. Four matched tasks scored on geometry. | BenchCAD team | CAD models | University | In use | GPT-6 Astra 95.9% | 4 | 6 | May 2026 | Oct 2026 |
| WebArena Agents complete 812 realistic tasks on self-hosted shopping, forum, GitLab, CMS and map websites. Functional graders check the outcome. | Carnegie Mellon University | Web browsing | Everyday | Fading | Claude Opus 4.6 68% | 4 | 5 | Jul 2023 | Oct 2026 |
| RACE-H Models answer multiple-choice reading questions from English exams for Chinese high school students. The test split has 3,498 questions, scored by accuracy. | Carnegie Mellon University | Reading comprehension | School | Dropped | Claude 3 Opus 92.9% | 4 | 5 | Apr 2017 | Dec 2024 |
| FullStackBench Models write code for 3,374 problems in 16 programming languages across domains like data analysis and web development. Unit tests score each answer. | ByteDance Seed | Function-level problems | University | Fading | MiniMax-M1-80k 68.3% | 4 | 5 | Nov 2024 | Feb 2026 |
| MM-MT-Bench Models hold multi-turn conversations about images such as charts, documents and photos. GPT-4o grades each answer against a reference on a 1 to 10 scale. | Mistral AI | General visual questions | Everyday | Dropped | Qwen3-VL-235B-A22B 8.50 | 4 | 5 | Oct 2024 | Jan 2026 |
| SysBench Models follow 500 system messages across five user turns each. A model verifier checks constraints, instructions and whole sessions. | Peking University + Baichuan | Following precise instructions | Everyday | Fading | GLM-Z1-32B-0414 81.2% | 4 | 5 | Aug 2024 | Oct 2026 |
| OSWorld-G Models map 564 desktop instructions to the right screen element or point, covering text matching, element recognition, layout understanding and fine manipulation. | XLANG Lab, University of Hong Kong (OSWorld team) | Finding things on screen | Everyday | Fading | Fara1.5-27B 71.3% | 4 | 5 | May 2025 | Jun 2026 |
| Design2Code Models turn screenshots of 484 real webpages into HTML and CSS. Automatic metrics compare the rendered page with the original on layout, text, position, color and CLIP similarity. | Stanford SALT and others (Si, Zhang, Yang et al.) | Web and app front ends | Everyday | Fading | Qwen3-VL-235B-A22B 93.4 | 4 | 5 | Mar 2024 | Jan 2026 |
| TOMATO Models answer 1,484 human-written questions about 1,417 videos that need visual temporal reasoning, such as action counts, direction, rotation and velocity. Scored by accuracy. | Yale University and Allen Institute for AI | Video understanding | Everyday | Fading | Seed2.1 Pro 79.5% | 4 | 4 | Oct 2024 | Jun 2026 |
| AMO-Bench Models solve 50 original math problems at or above IMO difficulty that need only a final answer. Answers are graded automatically for accuracy. | Meituan LongCat Team | Competition problems with answers | Competition | Fading | GPT-5 52.4% | 4 | 4 | Oct 2025 | Jun 2026 |
| Audio MC Spoken multi-turn conversations test inference memory, instruction retention, self-coherence and voice editing. O4-mini judge applies per-conversation rubrics. | Scale AI | Audio understanding and voice | Everyday | In use | Gemini 3.8 Flash 60.4% | 4 | 4 | Dec 2025 | Oct 2026 |
| OVBench Video models answer questions about past, current and future moments of an online video stream in 6 task types and 16 subtasks. | Nanjing University MCG (Huang et al.) | Video understanding | Everyday | Fading | Seed2.1 Pro 70% | 4 | 4 | Dec 2024 | Jun 2026 |
| Video-Holmes Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film. | Tencent ARC Lab and City University of Hong Kong | Video understanding | Everyday | Fading | Seed2.1 Pro 68.2% | 4 | 4 | May 2025 | Jun 2026 |
| OVO-Bench Video models answer questions asked at given timestamps of a video stream in 12 tasks over 644 videos. Tasks cover backward tracing, real-time understanding and forward active responding. | Niu et al. (CVPR 2025) | Video understanding | Everyday | Fading | Seed2.1 Pro 80.7% | 4 | 4 | Jan 2025 | Jun 2026 |
| Scientists' First Exam (SFE) Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning. | Shanghai AI Lab (Yuhao Zhou et al.) | General visual questions | Expert | Fading | Intern-S2-397B 60.7% | 4 | 4 | Jun 2025 | Sep 2026 |
| CLIcK Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. | KAIST and collaborators (Eunsu Kim et al.) | Knowledge in many languages | School | In use | Solar Open 2 90.7% | 4 | 4 | Mar 2024 | Aug 2026 |
| HumanEval Multi Models complete HumanEval problems in Python and seven more languages, including C++, Java, PHP, and Bash. Unit tests score pass@1. | DeepSeek-AI | Function-level problems | School | Dropped | DeepSeek-V3 82.6% | 4 | 3 | Nov 2023 | Jan 2025 |
| VLMsAreBiased Models count or identify features in altered images of famous subjects, such as a logo with an added stripe, across 7 domains. Scoring is accuracy. | An Vo et al. | Spatial reasoning and counting | Everyday | Fading | Qwen 3.8 Max 88.3% | 4 | 3 | May 2025 | Aug 2026 |
| WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. | InternLM (Shanghai AI Lab) | General assistants | University | In use | Qwen3.8-Omni-Flash 71% | 4 | 3 | Mar 2026 | Oct 2026 |
| MMLongBench Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens. | University of Edinburgh and collaborators (Wang et al.) | Finding facts in long inputs | Everyday | Fading | Seed2.1 Pro 78.3% | 4 | 3 | May 2025 | Jun 2026 |
| KOR-Bench Models read a new rule set and apply it to questions in five task types: operation, logic, cipher, puzzle and counterfactual. Scored by accuracy against an answer key. | Multimodal Art Projection (M-A-P) with ByteDance and 01.AI | Abstract puzzles and logic | University | Fading | o1-preview 72.9% | 4 | 2 | Oct 2024 | Mar 2026 |
| WebArena-Verified Agents complete 812 audited WebArena tasks on self-hosted websites. Deterministic evaluators check the agent's answer and captured network traces. | ServiceNow | Web browsing | Everyday | In use | Muse Spark 1.1 69% | 4 | 2 | Dec 2025 | Oct 2026 |
| SUPERChem Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score. | Zhao et al. | Genomics and life sciences | Expert | In use | Tencent Hy4 preview 66.4% | 4 | 2 | Dec 2025 | Aug 2026 |
| KBL Models answer Korean legal knowledge and reasoning tasks and Korean bar exam questions. It has 7 knowledge tasks, 4 reasoning tasks and 2,510 bar exam items. | LBox and collaborators (Yeeun Kim et al.) | Law | Expert | In use | Solar Pro 4 77.5% | 4 | 2 | Oct 2024 | Aug 2026 |
| KorMedMCQA Models answer 7,469 multiple-choice questions from Korean licensing exams for doctors, nurses, pharmacists and dentists, held from 2012 to 2024. | KAIST and collaborators (Sunjun Kweon et al.) | Medicine | Expert | In use | Solar Pro 4 93.2% | 4 | 2 | Mar 2024 | Aug 2026 |
| WildBench 1,024 challenging real-user tasks with checklist-guided LLM judging. | Ai2 | Overall chat quality | University | Dropped |  Athene-70B 59.4 | 4 | 1 | Jun 2024 | Jan 2026 |
| ARC-AGI-3 Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. | ARC Prize Foundation | Puzzles and chess | Everyday | In use | GPT-6 Astra 100% | 4 | 1 | Mar 2026 | Oct 2026 |
| ARC-AGI-2 Replaced in Mar 2026 | Abstract puzzles and logic | In use | GPT-6 Astra 95% | 21 | 12 | Mar 2025 | Oct 2026 | ||
| ARC-AGI v1 Replaced in Mar 2025 | ARC Prize Foundation (Chollet) | Abstract puzzles and logic | In use | Claude Fable 5 98.5% | 13 | 15 | Nov 2019 | Oct 2026 | |
| MLS-Bench Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. | MLS-Bench authors (Bohan Lyu et al.) | AI research tasks | Research frontier | In use | Claude Fable 5.1 50.3% | 4 | 1 | May 2026 | Sep 2026 |
| PMC-VQA Models answer multiple-choice questions about medical images drawn from PubMed Central figures. Scored by accuracy. | Shanghai Jiao Tong University and Shanghai AI Laboratory | Medicine | University | Fading | Qwen 3.8 Max 66.2% | 4 | 1 | May 2023 | Aug 2026 |
| RoboCasa GR1 Tabletop Tasks A simulated Fourier GR-1 humanoid with dexterous hands does 24 tabletop pick-and-place tasks, including unseen object and container pairs. Success rate per task. | NVIDIA + UT Austin (RoboCasa team) | Simulated robots | Everyday | Fading | GR00T N1 50% | 4 | 1 | Mar 2025 | Jul 2026 |
| SLAKE Models answer English and Chinese questions about radiology images of the head, neck, chest, abdomen, and pelvis. Scoring is accuracy. | Hong Kong Polytechnic University (Xiao-Ming Wu group) | Medicine | University | Fading | Qwen 3.8 Max 90.8% | 4 | 1 | Feb 2021 | Aug 2026 |
| τ-Voice Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction. | Sierra | Speech recognition and translation | Everyday | In use | Grok Voice Think Fast 1.0 67.3% | 4 | Mar 2026 | Oct 2026 | |
| GeneBench Pro Agents analyze messy simulated genomics and biology datasets to estimate a decision-relevant quantity across 129 problems. Scored automatically against known answers. | OpenAI | Genomics and life sciences | Expert | In use | GPT-6 Astra 37.1% | 4 | Jun 2026 | Sep 2026 | |
| Internal API instruction following (OpenAI) Models answer prompts drawn from real developer use cases that carry instructions on formatting, verbosity and length. OpenAI reports a hard subset. | OpenAI | Following precise instructions | Everyday | Dropped | GPT-5 mini 65.8% | 4 | Apr 2025 | Aug 2025 | |
| Investment Banking Modeling Tasks (OpenAI) Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally. | OpenAI | Finance | Expert | Fading | GPT-5.5 Pro 88.6% | 4 | Jul 2025 | Apr 2026 | |
| LifeSciBench Models complete 750 expert-written life science research tasks, often with attached figures, tables, or sequence files. Graded against expert rubrics as score and pass rate. | OpenAI | Genomics and life sciences | Expert | In use | GPT-6 Astra 60.3% | 4 | Jun 2026 | Sep 2026 | |
| LingoQA Models answer free-form questions about short driving videos. A trained classifier, Lingo-Judge, scores whether each answer is truthful. | Wayve | General visual questions | Everyday | In use | Qwen 3.8 Max 84.8% | 4 | Dec 2023 | Aug 2026 | |
| QwenWebBench Models generate front-end code for web apps, games, SVG, and other categories in English and Chinese. Multimodal judgments produce Elo ratings. | Qwen | Web and app front ends | University | Fading | Qwen 3.6 Max (Preview) 1,532 | 4 | Apr 2026 | Apr 2026 | |
| MLE-bench Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. | OpenAI | Machine learning engineering | Expert | In use | GPT-6 Astra 93.8% | 3 | 24 | Oct 2024 | Sep 2026 |
| FrontierSWE v2 34 ultra-long-horizon engineering and research tasks. 5 trials of up to 20 hours each. Score is mean task score 0-100. | Proximal | Building whole projects | Research frontier | In use | GPT-6 Astra 65.5% | 3 | 7 | Sep 2026 | Oct 2026 |
| FrontierSWE v1 Replaced in Sep 2026 | In use | 4 | 4 | Apr 2026 | Aug 2026 | ||||
| POPE Vision language models answer yes or no questions about whether objects appear in COCO images. Scored by accuracy or F1 on object hallucination. | Renmin University of China (Li et al.) | Hallucination rates | Everyday | Fading | InternVL3.5-241B-A28B 90.7% | 3 | 6 | May 2023 | Aug 2026 |
| All-Angles Bench Models answer over 2,100 multi-view questions about 90 real scenes. Six tasks cover counting, attributes, relative distance and direction, object manipulation and camera pose. | Yeh et al. (UC Berkeley, NYU and others) | Spatial reasoning and counting | Everyday | Fading | Seed2.0 Pro 72.1% | 3 | 6 | Apr 2025 | Feb 2026 |
| MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. | Shanghai AI Laboratory (OpenCompass) | Knowledge in many languages | Everyday | Fading | Gemini 2.5 Pro 88.1% | 3 | 5 | Mar 2026 | |
| FORTRESS Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. | Scale AI | Refusing harmful requests | In use | Claude Opus 5.5 5.78% | 3 | 5 | Jun 2025 | Oct 2026 | |
| USAMO 2026 Models write full proofs for the six problems of the 2026 USA Mathematical Olympiad. Judges grade each proof from 0 to 7 against rubrics. | Mathematical Association of America (contest), MathArena (evaluation) | Written proofs | Competition | In use | GPT-5.5 98.2% | 3 | 5 | Mar 2026 | Jul 2026 |
| USAMO 2025 Replaced in Mar 2026 | MathArena (ETH Zurich SRI Lab) | Dropped | DeepSeek-R1 30.1% | 2 | Mar 2025 | Jul 2025 | |||
| USAMO 2024 Replaced in Mar 2025 | MathArena (ETH Zurich SRI Lab) | Not in cards | Mar 2024 | ||||||
| ArXivMath Final-answer research-level math problems extracted monthly from recent arXiv papers and filtered to be self-contained. | MathArena (ETH Zurich SRI Lab) | Research mathematics | Research frontier | In use | GPT-6.1 Sol 94.2% | 3 | 5 | Feb 2026 | Oct 2026 |
| ActivityNet-QA Models answer open-ended questions about web videos of human activities, from 58,000 QA pairs on 5,800 videos. Scored by accuracy. | Hangzhou Dianzi University (MILVLG) | Video understanding | Everyday | Dropped | Gemini 2.5 Pro 66.7% | 3 | 5 | Jun 2019 | Jul 2025 |
| GPQA Main Models answer the 448 question main set of GPQA, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | NYU (Rein et al.) | Science questions | Expert | Dropped | Claude 3 Opus 49.1% | 3 | 5 | Nov 2023 | Sep 2025 |
| KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. | HAE-RAE | Knowledge in many languages | University | Fading | Solar Pro 2 77% | 3 | 5 | Feb 2024 | Jan 2026 |
| MINERVA Models answer complex video questions that need multi-step reasoning, with reference reasoning traces for grading. | Google DeepMind | Long videos | University | Fading | Seed2.1 Pro 70.7% | 3 | 5 | May 2025 | Jun 2026 |
| ChartMuseum Models answer 1,162 expert-annotated questions about real-world charts from 184 sources. Many questions need visual reasoning that text cannot replace. | UT Austin (Tang et al.) | Documents, charts and text in images | Expert | Fading | Claude Opus 4.8 75.8% | 3 | 5 | May 2025 | Jun 2026 |
| OmniSpatial Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking. | Jia, Qi et al. (OmniSpatial authors) | Spatial reasoning and counting | Everyday | Fading | GLM-4.6V 52% | 3 | 5 | Jun 2025 | Jan 2026 |
| QVHighlights Natural-language queries over YouTube videos. Model localizes relevant moments, scored by Recall@1 at IoU 0.5. | UNC Chapel Hill | Video understanding | Everyday | Fading | Nova 2 Lite 77.2% | 3 | 5 | Jul 2021 | Dec 2025 |
| MTOB Models learn to translate between English and Kalamang, a language with under 200 speakers, from one grammar book. Scored by chrF. | Harvard University and collaborators | Learning from the context | Expert | Dropped | Llama 4 Maverick 54.0 | 3 | 5 | Sep 2023 | Jul 2026 |
| Seed-TTS-eval Models clone a voice from a short prompt and speak new text in English and Mandarin. Scored by word error rate from an ASR model and by speaker similarity. | ByteDance (Seed-TTS team) | Speech recognition and translation | Everyday | Fading | 3 | 5 | Jun 2024 | Apr 2026 | |
| PHYBench Models solve 500 original physics problems from high school to Physics Olympiad level. Answers are expressions checked against the key. | Qiu et al. (PHYBench team) | Science olympiads | Competition | Fading | Tencent Hy3 77.4% | 3 | 4 | Apr 2025 | Jul 2026 |
| IMO-ProofBench Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder. | Google DeepMind (Superhuman Reasoning team) | Written proofs | Competition | Fading | GPT-5.5 Pro 88.1% | 3 | 4 | Nov 2025 | Sep 2026 |
| LABBench2 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. | Edison Scientific (FutureHouse) | Lab and data work | Expert | In use | GPT-5.2 60.9% | 3 | 4 | Feb 2026 | Oct 2026 |
| LAB-Bench v1 Replaced in Feb 2026 | FutureHouse | Genomics and life sciences | System cards only | Grok 4.6 80.7% | 4 | Jul 2024 | Sep 2026 | ||
| EgoTempo Models answer questions about egocentric videos that require temporal understanding across the whole clip. | Google DeepMind and University of Bristol | Video understanding | Everyday | Fading | Seed2.0 Pro 71.8% | 3 | 4 | Mar 2025 | Apr 2026 |
| MMSU Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology. | Wang et al. (arXiv 2506.04779) | Speech recognition and translation | University | Fading | Seed2.0 Lite (0428) 86.5% | 3 | 4 | Jun 2025 | Sep 2026 |
| HLE-Verified Verified and revised version of Humanity's Last Exam (1,811 items) with uncertain items removed. | SKYLENAGE AI | Expert-written questions | Expert | In use | Seed2.0 Pro 73.6% | 3 | 4 | Feb 2026 | Sep 2026 |
| AceBench Models call tools from 4,538 Chinese and English APIs in normal, special, and multi-turn agent cases. Answers are checked without LLM judges or live APIs. | Chen Chen et al. | Function calling | Everyday | Fading | GPT-4o 85.4% | 3 | 3 | Jan 2025 | Feb 2026 |
| Omni-MATH Models solve 4,428 Olympiad-level competition math problems across 33 subdomains and 10 difficulty levels. A GPT-4 or Omni-Judge grader checks answers. | Peking University and Alibaba et al. | Competition problems with answers | Competition | Dropped | o1-mini 60.5% | 3 | 3 | Oct 2024 | Oct 2025 |
| OlympiadBench Models solve bilingual olympiad-level math and physics problems in Chinese and English, many with images. Answers are checked against a key. | OpenBMB and Tsinghua University (He et al.) | Competition problems with answers | Competition | Dropped | GPT-4o 25.9% | 3 | 3 | Feb 2024 | Aug 2025 |
| SUNRGBD Models detect objects in about 10,000 indoor RGB-D images and predict their 3D bounding boxes. Scored by average precision. | Princeton University | Detection and captioning | Everyday | Fading | Gemini Robotics-ER 48.3 | 3 | 3 | Jun 2015 | Feb 2026 |
| Windows Agent Arena Agents complete more than 150 tasks in a real Windows OS with its apps, tools and browsers. Scripts adapted from OSWorld check the final state. | Microsoft | Desktop apps | Everyday | Dropped | UI-TARS-2 50.6% | 3 | 3 | Sep 2024 | Nov 2025 |
| DUDE Models answer extractive, abstractive, and unanswerable questions about multi-page, multi-domain documents. Scored by ANLS. | DUDE team (KU Leuven, CVC Barcelona, et al.) | Documents, charts and text in images | Everyday | Fading | Seed2.1 Pro 82.8% | 3 | 3 | May 2023 | Jun 2026 |
| Video SimpleQA Video models answer multi-hop fact-seeking questions that need knowledge beyond the video and evidence from its time segments. Short definitive answers are graded for factual correctness. | Cao et al. (Video SimpleQA team) | Short factual questions | University | Fading | Seed2.1 Pro 76.4% | 3 | 3 | Mar 2025 | Jun 2026 |
| XLRS-Bench Models answer perception and reasoning questions about extremely large, ultra-high-resolution remote sensing images. | XLRS-Bench team (Fengxiang Wang et al.) | Detection and captioning | Expert | Fading | Intern-S1 55% | 3 | 3 | Mar 2025 | Sep 2026 |
| CursorBench 4.0 Ambiguous multi-file coding tasks from real Cursor sessions in Cursor's production harness. Agentic graders score correctness, with cost reported. | Cursor | Fixing real repositories | Expert | In use | Claude Opus 5.5 57.8% | 3 | 3 | Sep 2026 | Sep 2026 |
| CursorBench 3.2 Replaced in Sep 2026 | In use | Claude Fable 5.1 73.4% | 2 | 2 | Jul 2026 | Sep 2026 | |||
| CursorBench 3.1 Replaced in Jul 2026 | Fading | Claude Fable 5 72.9% | 2 | 2 | May 2026 | Jun 2026 | |||
| CursorBench 3.0 Replaced in May 2026 | System cards only | Claude Opus 5.5 57.8% | 1 | Mar 2026 | Oct 2026 | ||||
| Cursor Bench (2025 internal version) 2025 (before 3.0) Replaced in Mar 2026 | Fading | 2 | Oct 2025 | Feb 2026 | |||||
| Flame-React-Eval Models write React components from UI design images. Rendered outputs are compared with gold components by image similarity and scored as pass@k. | Flame-Code-VLM project (Ge et al.) | Web and app front ends | University | Fading | GLM-4.6V 86.3% | 3 | 3 | Feb 2025 | Jan 2026 |
| GenEval Text-to-image models draw prompts about object count, color, position and co-occurrence, and an object detector checks each image. Scored by the share of correct images. | University of Washington (Ghosh et al.) | Image generation and editing | Everyday | Fading | ERNIE 5.0 90.1% | 3 | 3 | Oct 2023 | Feb 2026 |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | Cohere Labs | Knowledge in many languages | University | In use |  Inkling 88.7% | 3 | 3 | May 2026 | Aug 2026 |
| Global MMLU Lite 2.0 Replaced in May 2026 | Not in cards | Dec 2025 | |||||||
| Global MMLU Lite 1.0 Replaced in Dec 2025 | Dropped | Claude Opus 5 95.7% | 12 | 12 | Dec 2024 | Oct 2026 | |||
| HRM8K Models solve 8,011 math problems given as English and Korean parallel pairs. It measures how much math reasoning drops when problems are in Korean. | HAE-RAE and OneLineAI (Hyunwoo Ko et al.) | Word problems | School | Fading | Solar Open 2 92.2% | 3 | 3 | Jan 2025 | Jul 2026 |
| Inverse IFEval Models follow 1,012 Chinese and English instructions that conflict with habits learned in fine-tuning, such as writing code without comments. A model judge scores compliance. | ByteDance Seed and collaborators (Zhang et al.) | Following precise instructions | Everyday | Fading | Seed1.8 80.3% | 3 | 3 | Sep 2025 | Oct 2026 |
| Ko-LongBench (LG) Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG. | LG AI Research | Reasoning over long documents | University | Fading | K-EXAONE 2.0 89.6% | 3 | 3 | Jul 2025 | Aug 2026 |
| KoBALT Models answer 700 expert-written multiple-choice questions on 24 Korean linguistic phenomena in syntax, semantics, pragmatics, phonology and morphology. | Seoul National University and collaborators (Hyopil Shin et al.) | Reasoning in many languages | University | Fading | K-EXAONE 61.8% | 3 | 3 | May 2025 | Apr 2026 |
| MuChoMusic Models answer multiple-choice questions about music recordings. It tests music knowledge and reasoning in audio-language models. | Weck et al. (Universitat Pompeu Fabra, Queen Mary University of London and others) | Audio understanding and voice | University | Fading | Qwen3.8-Omni-Flash 72.6% | 3 | 3 | Aug 2024 | Sep 2026 |
| OCR-Reasoning Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks. | SCUT-DLVCLab (Huang et al.) | Documents, charts and text in images | Everyday | Fading | Nemotron 3.5 Super VL 65.1% | 3 | 3 | May 2025 | Oct 2026 |
| PhiBench Models answer original prompts that Microsoft's Phi team wrote to test a wide range of skills and reasoning. | Microsoft | Other | University | Dropped | Phi-4-Reasoning-plus 74.2% | 3 | 3 | Dec 2024 | Apr 2025 |
| RealKIE-FCC Verified Model extracts structured fields from scanned FCC invoices into a JSON schema. Amazon human-verified labels. Field-level similarity (ALNS). | Indico Data (RealKIE) + Amazon AGI (Verified) | Documents, charts and text in images | Everyday | Fading | Nova 2.0 Pro Preview 67% | 3 | 3 | Dec 2025 | Dec 2025 |
| StreamingBench Models answer questions while a video streams in, as a human viewer would. It tests real-time visual understanding, omni-source understanding and context understanding. | Lin et al. (arXiv 2411.03628) | Video understanding | Everyday | Fading | Seed1.8 84.4% | 3 | 3 | Nov 2024 | Sep 2026 |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Amazon AGI | Customer service | Everyday | Fading | Nova 2 Omni Preview 73.5% | 3 | 3 | Dec 2025 | Dec 2025 |
| WebQuest Models answer questions from screenshots of one web page, several related pages, or a navigation trace. The test checks extraction and combination of information across pages. | Google (Wang et al.) | Documents, charts and text in images | Everyday | Fading | GLM-4.6V 69.3% | 3 | 3 | Sep 2024 | Jan 2026 |
| WebTailBench Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals. | Microsoft | Web browsing | Everyday | Fading | MolmoWeb 8B 49.5% | 3 | 3 | Nov 2025 | Jun 2026 |
| BigCodeBench-Hard Models solve 148 harder BigCodeBench Python tasks that call many libraries, chosen to match real programming work. Scored pass@1 on unit tests. | BigCode project | Function-level problems | University | Dropped | Claude 3.7 Sonnet 35.8% | 3 | 2 | Jul 2024 | Jun 2025 |
| Blueprint-Bench 2 Agents turn about 20 interior photos of each of 50 apartments into 2D floor plans. Scores compare room connectivity with ground truth, normalized against random. | Andon Labs | Spatial reasoning and counting | Everyday | In use | Gemini 4 Carbon 58.6% | 3 | 2 | May 2026 | Oct 2026 |
| Blueprint-Bench v1 Replaced in May 2026 | Not in cards | GPT-5 0.43 | Sep 2025 | Sep 2025 | |||||
| DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. | University of Science and Technology of China and Metastone Technology | Deep research | Expert | Fading | Claude Opus 4.6 55.3% | 3 | 2 | Jun 2025 | May 2026 |
| CWE-bench Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. | Collinear AI | Secure code and review | Expert | In use | Gemini 4 Carbon 71% | 3 | 2 | Sep 2026 | Oct 2026 |
| Riemann-bench 25 private research-mathematics problems from professors and IMO medalists. Agents with tools, 100 runs per problem. | Surge AI | Research mathematics | Research frontier | In use | Gemini 4 Carbon 80% | 3 | 2 | Apr 2026 | Oct 2026 |
| HAE-RAE Bench Models answer Korean questions on vocabulary, history, general knowledge and reading comprehension. It tests cultural and contextual knowledge that translated benchmarks miss. | HAE-RAE (Guijin Son et al.) | Knowledge in many languages | School | Fading | Solar Pro 2 86.2% | 3 | 2 | Sep 2023 | Jul 2026 |
| HLE-VL Models answer the image-based questions from Humanity's Last Exam, a 330 question subset. Answers are graded against known short or multiple choice answers. | Center for AI Safety and Scale AI (HLE), Alibaba Tongyi Lab (VL subset) | General visual questions | Expert | Fading | Qwen 3.8 Max 52.2% | 3 | 2 | Aug 2025 | Aug 2026 |
| KBank-MMLU (Upstage) Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house. | Upstage | Finance | Expert | Fading | Solar Open 2 80.8% | 3 | 2 | Jul 2026 | Jul 2026 |
| MicroVQA Models answer expert-written multiple choice questions about biology microscopy images that test image understanding, hypothesis generation and experiment proposal. | Stanford University (James Burgess, Serena Yeung-Levy et al.) | General visual questions | Expert | Fading | Intern-S2-397B 69.7% | 3 | 2 | Mar 2025 | Sep 2026 |
| OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. | Renmin University of China (RUC-NLPIR) and others | Deep research | University | Fading | Qwen3.8-Omni-Flash 74% | 3 | 2 | Feb 2026 | Sep 2026 |
| WildSpeech-Bench Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies. | Zhang et al. (arXiv 2506.21875) | Speech recognition and translation | Everyday | Fading | Qwen3.5-Omni-Plus 75.4 | 3 | 2 | Jun 2025 | Sep 2026 |
| FACTS Benchmark Suite Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Fading | GPT-6 Astra 75.4% | 3 | 1 | Dec 2025 | Sep 2026 |
| HiPhO Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. | Shanghai AI Laboratory | Science olympiads | Competition | In use | Gemini 3 Pro 78.3% | 3 | 1 | Sep 2025 | Aug 2026 |
| MobileWorld Agents complete 201 tasks across 20 mobile apps, including tasks that need user clarification or MCP tools. Backend checks verify success. | Tongyi Lab, Alibaba Group | Phones | Everyday | In use | Kimi K3 74.4% | 3 | 1 | Dec 2025 | Aug 2026 |
| AMC Models solve problems from the AMC 8, 10, and 12 competitions. Gemini 1.5 used 250 problems from 2022 and 2023, scored by accuracy. | Mathematical Association of America | Competition problems with answers | Competition | Fading |  Falcon-H1-34B 69.4% | 3 | 1 | Feb 2024 | Dec 2025 |
| BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. | University of Waterloo and collaborators | Deep research | Expert | In use | Claude Opus 4.5 67.6% | 3 | 1 | Aug 2025 | Oct 2026 |
| DeepMind Mathematics Models answer generated school-level math questions across algebra, arithmetic, calculus, probability and other modules. Answers are checked for an exact match. | DeepMind (Saxton et al.) | Word problems | School | Fading | Granite 4.1 30B 81.9% | 3 | 1 | Apr 2019 | Apr 2026 |
| ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. | Zhipu AI and Tsinghua University (THUDM) | Function calling | University | Dropped | Hunyuan-A13B-Instruct 61.2% | 3 | 1 | Jan 2025 | Jun 2025 |
| Hypersim Models localize named objects as 3D boxes in synthetic indoor Hypersim images, using the Omni3D test split. Scored by mAP at IoU 0.15. | Apple (dataset), Meta AI (Omni3D split) | Detection and captioning | Everyday | Fading | Qwen3.5-122B-A10B 12.7 | 3 | 1 | Nov 2020 | Feb 2026 |
| Kimi Code Bench v2 Coding agents complete Moonshot's realistic tasks from internal engineering work, production incidents, and open-source projects in over ten languages. | Moonshot AI | Fixing real repositories | Expert | In use | Kimi K3.1 Pro 77.1% | 3 | 1 | Jun 2026 | Oct 2026 |
| PerceptionBench (Moonshot) Moonshot in-house benchmark of atomic visual perception skills. Scoring details unpublished. | Moonshot AI | General visual questions | Everyday | In use | Kimi K3.1 Pro 68.3% | 3 | 1 | Jul 2026 | Oct 2026 |
| SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. | Shanghai AI Laboratory (OpenSafetyLab) | Refusing harmful requests | Everyday | Fading | Granite 4.0 H Small 97.3% | 3 | 1 | Feb 2024 | Apr 2026 |
| WorkSpaceBench Agent completes 388 tasks in realistic worker file workspaces (20k files) needing cross-file dependency reasoning. Scored against 7,399 rubrics. | SJTU + ByteDance (OpenDataBox) | Documents and spreadsheets | University | In use | Qwen 3.8 Max 67.7% | 3 | 1 | May 2026 | Aug 2026 |
| RecreationBench Agents explore a running reference app and rebuild it, across 250 tasks on five platforms. Programmatic tests and a VLM judge score the result. | Qwen | Desktop apps | Expert | In use | GPT-6 Astra 58.1% | 3 | Sep 2026 | Sep 2026 | |
| BrowseComp Long Context 128k Models answer BrowseComp questions from a 128k-token context of the needed web pages mixed with related distractor pages. Scored by accuracy. | OpenAI | Reasoning over long documents | Expert | Dropped | GPT-5.1 90% | 3 | Aug 2025 | Nov 2025 | |
| GSM-Symbolic Models solve grade-school word problems generated from symbolic templates of GSM8K questions, with names and numbers changed. P1 and P2 add one or two extra clauses. | Apple (Mirzadeh et al.) | Word problems | School | Fading | Granite 4.0 H Small 87.4% | 3 | Oct 2024 | Apr 2026 | |
| Hy-SWE Max Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task. | Tencent Hy team | Fixing real repositories | Expert | In use | Tencent Hy4 preview 64.2% | 3 | Apr 2026 | Aug 2026 | |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | Qwen | Knowledge in many languages | University | Dropped | Qwen2.5-72B 78.7% | 3 | Feb 2024 | Sep 2024 | |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Qwen Team, Alibaba Group | Reasoning in many languages | Everyday | Dropped | Qwen2.5-72B 89.6% | 3 | Feb 2024 | Sep 2024 | |
| StrongREJECT Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. | UC Berkeley (Souly et al.) | Jailbreaks and prompt injection | In use | Inkling 98.6% | 2 | 30 | Feb 2024 | Sep 2026 | |
| ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. | Google Research and Google DeepMind | Knowledge in many languages | Everyday | Dropped | Claude Mythos Preview 56.3% | 2 | 10 | Feb 2025 | May 2026 |
| AutoLogi Models solve open-ended logic puzzles in English and Chinese. Generated verifier programs check each answer against the puzzle constraints. | Qwen Team, Alibaba Group and Fudan University | Abstract puzzles and logic | School | Dropped | Kimi K2 89.5% | 2 | 8 | Feb 2025 | Jul 2025 |
| SQuAD v2 2.0 Models extract answers from Wikipedia paragraphs and must abstain on over 50,000 adversarial unanswerable questions. Scored by exact match and F1. | Stanford NLP | Reading comprehension | Everyday | Dropped | Llama 3.1-405B 81.8% | 2 | 7 | Jun 2018 | Nov 2024 |
| SQuAD 1.1 Replaced in Jun 2018 | Dropped | Llama 3.1-405B 89.3% | 1 | 5 | Jun 2016 | Jul 2024 | |||
| TreeBench Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure. | Wang et al. (TreeVGR authors) | Detection and captioning | University | Fading | Seed2.1 Pro 71.1% | 2 | 7 | Jul 2025 | Jun 2026 |
| MTVQA-Test Models answer 6,778 expert-written questions about text in 2,116 images across 9 languages, such as Arabic, Japanese, and Thai. Scoring is accuracy. | ByteDance | Documents, charts and text in images | Everyday | Fading | Qwen 3.8 Max 56.6% | 2 | 5 | May 2024 | Aug 2026 |
| EvalPlus HumanEval+/MBPP+ with 80x more tests. Showed many 'passing' solutions were wrong. | UIUC | Function-level problems | School | Dropped | GLM-5 87% | 2 | 5 | May 2023 | Feb 2026 |
| MMVP Models answer 300 questions about 150 image pairs that CLIP embeds as near duplicates. A pair counts only when both answers are right. | New York University and UC Berkeley (Tong et al.) | Spatial reasoning and counting | Everyday | Fading | Kimi K2.5 87% | 2 | 5 | Jan 2024 | Mar 2026 |
| IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. | Google Research | Reading in many languages | Everyday | Dropped | Gemini 2.5 Pro 77.1 | 2 | 4 | Apr 2024 | Nov 2025 |
| LOFT Models solve retrieval, RAG, SQL, and many-shot learning tasks with whole corpora of up to one million tokens in context. Scored per task. | Google DeepMind | Finding facts in long inputs | Everyday | Dropped | Gemini 2.5 Pro 87% | 2 | 4 | Jun 2024 | Jul 2026 |
| MMSI-Bench Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data. | Shanghai AI Lab (OpenRobotLab) and collaborators | Spatial reasoning and counting | Everyday | Fading |  SenseNova-SI-1.3-InternVL3-8B 42.5% | 2 | 4 | May 2025 | Jun 2026 |
| MME-RealWorld Models answer human-annotated multiple choice questions about high-resolution real-world images in scenarios such as remote sensing, autonomous driving, monitoring and OCR. | MME team (Yi-Fan Zhang, Chaoyou Fu et al.) | General visual questions | Everyday | Dropped | InternVL3.5-241B-A28B 65.1% | 2 | 4 | Aug 2024 | Nov 2025 |
| ReMI Models reason over several images per question in math, physics, logic, code, charts, and spatial or temporal tasks. Scored by accuracy. | Google DeepMind | General visual questions | School | Fading | STEP3-VL-10B 67.3% | 2 | 4 | Jun 2024 | Jan 2026 |
| TallyQA Models answer open-ended counting questions about images, including complex ones that need object relationships or attributes. Scored by exact count accuracy. | Rochester Institute of Technology | Spatial reasoning and counting | Everyday | Dropped | Amazon Nova Premier 61.5% | 2 | 4 | Oct 2018 | Apr 2025 |
| ANLI Models label premise and hypothesis pairs as entailment, neutral, or contradiction. People wrote the pairs over three rounds to fool strong models. | Facebook AI Research (Nie et al.) | Language reasoning | Everyday | Dropped | GPT-3.5 Turbo 58.1% | 2 | 3 | Oct 2019 | Aug 2024 |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Qwen Team, Alibaba Cloud | Function calling | Everyday | Dropped | Qwen-14B 98% | 2 | 3 | Aug 2023 | Feb 2024 |
| Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. | Qwen | Function calling | School | Dropped | Qwen-14B 81.7% | 2 | 3 | Sep 2023 | Feb 2024 |
| LiveSports-3K Video models give real-time commentary and answer questions on sports clips. An LLM judge scores the free-form commentary. | Chen et al. (LiveCC team, NUS Show Lab and ByteDance) | Video captioning | Everyday | Fading | Seed2.0 Lite (0428) 78.1% | 2 | 3 | Apr 2025 | Jun 2026 |
| VSI-Bench Models watch egocentric videos of 288 indoor scenes and answer over 5,000 questions on object counts, distances, sizes, room layout and route planning. | New York University (VisionX, Saining Xie) with Stanford and Yale | Spatial reasoning and counting | Everyday | Fading | InternVL3.5-241B-A28B 69.5% | 2 | 3 | Dec 2024 | Jan 2026 |
| AetherCode Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution. | ByteDance Seed and collaborators (Wang et al.) | Competitive programming | Competition | Fading | Seed2.1 Pro 65.8% | 2 | 3 | Aug 2025 | Jun 2026 |
| AISHELL-2 Models transcribe 1,000 hours of Mandarin speech recorded on several devices. Scored by character error rate. | AISHELL (Beijing Shell Shell Technology) | Speech recognition and translation | Everyday | Dropped | Kimi-Audio-7B-Instruct 2.56% | 2 | 3 | Aug 2018 | Feb 2026 |
| AISHELL-1 Replaced in Aug 2018 | Dropped | ERNIE 5.0 0.31% | 2 | 4 | Sep 2017 | Feb 2026 | |||
| Charades-STA Models read a sentence describing an action and return its start and end time in a Charades indoor video. Scored by temporal IoU with the labeled moment. | University of Southern California (Gao, Sun, Yang, Nevatia) | Video understanding | Everyday | Dropped | Seed1.5-VL 64% | 2 | 3 | May 2017 | Nov 2025 |
| Clotho-AQA Models answer crowdsourced yes or no and single word questions about short environmental audio clips. Scored by accuracy. | Tampere University (Lipping, Sudarsanam et al.) | Audio understanding and voice | Everyday | Dropped | ERNIE 5.0 73.7% | 2 | 3 | Apr 2022 | Feb 2026 |
| MBXP Models solve MBPP problems transpiled with their tests into more than 10 programming languages, such as Java, C++ and Go. Unit tests check each solution. | Amazon (Athiwaratkun et al.) | Function-level problems | School | Dropped | Amazon Nova Premier 78.4% | 2 | 3 | Oct 2022 | Aug 2025 |
| MMAR Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge. | Ma et al. (arXiv 2505.13032) | Audio understanding and voice | University | Fading | Qwen3.5-Omni-Plus 80% | 2 | 3 | May 2025 | Sep 2026 |
| Ref-L4 Models locate the object described by a long referring expression in an image. The set has 45,341 annotations over 365 object categories, scored by box accuracy. | Chen et al. (Ref-L4 authors) | Detection and captioning | Everyday | Fading | GLM-4.5V 89.5% | 2 | 3 | Jun 2024 | Jan 2026 |
| IOI 2025 Model solves the 2025 International Olympiad in Informatics tasks. Scored with official subtask points (max 600). | IOI (problems); NVIDIA harness | Competitive programming | Competition | Fading | 2 | 3 | Aug 2025 | Jun 2026 | |
| VisualPuzzles Models solve visual puzzles that need algorithmic, analogical, deductive, inductive and spatial reasoning but little domain knowledge. Scored by accuracy. | Carnegie Mellon University (NeuLab) | Spatial reasoning and counting | Everyday | Fading | o4-mini 57% | 2 | 2 | Apr 2025 | Feb 2026 |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | Meta FAIR | General assistants | Everyday | Fading | GPT-5 42.1% | 2 | 2 | Sep 2025 | Oct 2026 |
| Putnam 2025 Models solve the 12 problems of the 2025 William Lowell Putnam competition with full proofs. Scored out of 120 points. | Mathematical Association of America | Written proofs | Competition | Fading | DeepSeek-V4-Pro 120.0 | 2 | 2 | Dec 2025 | Jun 2026 |
| Putnam 2024 Replaced in Dec 2025 | Dropped | DeepSeekMath-V2 98.3% | 1 | 1 | Dec 2024 | Nov 2025 | |||
| VATEX Models write English or Chinese captions for short video clips from a set of over 41,000 videos. Scored by CIDEr. | UC Santa Barbara and ByteDance AI Lab | Video captioning | Everyday | Dropped | Gemini 2.5 Pro 71.3 | 2 | 2 | Apr 2019 | Jul 2025 |
| VATEX ZH Models write Chinese captions for short videos of human activities from the VATEX dataset. Captions are scored with CIDEr against human references. | UC Santa Barbara (UCSB NLP Group) | Video captioning | Everyday | Dropped | Gemini 2.5 Pro 59.7 | 2 | 2 | Apr 2019 | Jul 2025 |
| VisFactor Models take 20 vision-centric subtests adapted from a human cognitive psychology test battery, covering perception primitives and visual memory. Scored by accuracy. | CUHK (ARISE Lab) | Spatial reasoning and counting | Everyday | In use | Qwen 3.8 Max 60.8% | 2 | 2 | Feb 2025 | Aug 2026 |
| YouCook2 Models describe the procedure steps in YouTube cooking videos. Scored by CIDEr against human step captions. | University of Michigan | Video captioning | Everyday | Dropped | Gemini 2.5 Pro 188.3 | 2 | 2 | Mar 2017 | Jul 2025 |
| CanItEdit Models edit a given block of code to follow an instruction, such as adding a feature, fixing a described bug or changing the approach. Hidden tests check the edited code. | Northeastern University Programming Research Lab (Cassano et al.) | Function-level problems | University | Dropped | Seed Diffusion Preview 54.3% | 2 | 2 | Dec 2023 | Aug 2025 |
| CG-Bench Video models answer 12,129 questions about 1,219 long videos and must also find the clue segments that support each answer. | Chen et al. (CG-Bench team) | Long videos | Everyday | Fading | Seed2.0 Lite (0428) 65.5% | 2 | 2 | Dec 2024 | Apr 2026 |
| CMO 2025 Models write full proofs for the six problems of the 2025 China Mathematical Olympiad. Graders score proofs out of 126 points. | Chinese Mathematical Society | Written proofs | Competition | Fading | Seed2.0 Pro 114.0 | 2 | 2 | Nov 2025 | Feb 2026 |
| ICPC World Finals 2025 Models solve the 12 problems of the 2025 ICPC World Finals under contest time and attempt limits. Scored by problems solved. | ICPC Foundation | Competitive programming | Competition | Fading | DeepSeek-V3.2-Speciale 10.0 | 2 | 2 | Sep 2025 | Mar 2026 |
| MM-IFEval Multimodal models follow image-grounded instructions with exact output constraints. Checks mix rule-based verification and a judge model. | MM-IFEngine team (Ding et al.) | Following precise instructions | Everyday | Fading | STEP3-VL-10B 61.9% | 2 | 2 | Apr 2025 | Aug 2026 |
| PhyX Models solve 3,000 image-based physics problems across six domains such as mechanics and optics. Versions include multiple choice and open-ended answers. | Hui Shen et al. | Science questions | University | Fading | Qwen 3.8 Max 83.5% | 2 | 2 | May 2025 | Aug 2026 |
| VideoReasonBench Video models watch a sequence of fine-grained operations on a partly hidden state and answer questions that recall, infer or predict that state. | Liu et al. (VideoReasonBench team) | Video understanding | Everyday | Fading | Seed2.0 Pro 77.8% | 2 | 2 | May 2025 | Apr 2026 |
| ViSpeak-Bench Streaming video models must notice instructions given visually, such as a hand wave, and respond in seven subtasks of visual instruction feedback. | Fu et al. (ViSpeak team) | Video understanding | Everyday | Fading | Seed2.0 Lite (0428) 87% | 2 | 2 | Mar 2025 | Apr 2026 |
| XpertBench Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics. | ByteDance Seed | Tasks from real jobs | Expert | Fading | Seed2.0 Pro 64.5% | 2 | 2 | Dec 2025 | Apr 2026 |
| AndroidControl Agents predict the next action on Android app screens from either a high-level goal or a low-level step instruction. Scored by step accuracy against human demonstrations. | Google DeepMind (Wei Li, William Bishop et al.) | Phones | Everyday | Dropped | LongCat-Flash-Omni 91.2% | 2 | 2 | Jun 2024 | Oct 2025 |
| AVUT Models answer questions about videos where the audio carries the key information. The benchmark removes questions that text alone can answer. | Yang et al. (arXiv 2503.19951) | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 86.6% | 2 | 2 | Mar 2025 | Sep 2026 |
| DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. | You.com | Deep research | Expert | Fading | Seed2.0 Pro 61.1% | 2 | 2 | May 2025 | Mar 2026 |
| DeepShop Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement. | Lyu et al. (University of Amsterdam and collaborators) | Web browsing | Everyday | Fading | MolmoWeb 8B 42.3% | 2 | 2 | Jun 2025 | Apr 2026 |
| DiscoX Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness. | ByteDance Seed and collaborators (Zhao et al.) | Translation | Expert | Fading | Seed2.0 Pro 82% | 2 | 2 | Nov 2025 | Mar 2026 |
| FlenQA Models answer reasoning questions whose key facts are padded with filler text to different lengths. Accuracy shows how input length affects reasoning. | Mosh Levy, Alon Jacoby, Yoav Goldberg | Reasoning over long documents | Everyday | Dropped | Phi-4-Reasoning-plus 97.9% | 2 | 2 | Feb 2024 | Apr 2025 |
| FLTEval Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff. | Mistral AI | Formal proofs | Research frontier | Fading | Leanstral 1.5 43.2 | 2 | 2 | Mar 2026 | Jul 2026 |
| HRM8K-KSM Models solve the 1,428 problems of the Korean School Math subset of HRM8K. Problems come from Korean olympiad and competition exams. | HAE-RAE and OneLineAI (Hyunwoo Ko et al.) | Competition problems with answers | Competition | Fading | K-EXAONE 2.0 91.1% | 2 | 2 | Jan 2025 | Aug 2026 |
| KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. | LG AI Research | Refusing harmful requests | Everyday | Fading | K-EXAONE 2.0 99.8% | 2 | 2 | Jan 2026 | Aug 2026 |
| Ko-AIME 2025 (Upstage) Models solve AIME 2025 competition problems in Korean. Upstage built the set in house. | Upstage | Competition problems with answers | Competition | Fading | Solar Open 2 97.7% | 2 | 2 | Jul 2026 | Jul 2026 |
| LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. | Meituan LongCat Team | Refusing harmful requests | Everyday | Dropped | LongCat-Flash-Thinking 95.7% | 2 | 2 | Sep 2025 | Sep 2025 |
| LongCodeBench Coding model answers repository QA and fixes GitHub issues with context from 32K to 1M tokens. Accuracy and resolve rate. | Panasonic AI Research + Sapienza + Stanford | Reasoning over long documents | Expert | Fading | Nova 2.0 Pro Preview 84% | 2 | 2 | May 2025 | Dec 2025 |
| LPFQA Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains. | ByteDance Seed and collaborators (Zhu et al.) | Expert-written questions | Expert | Fading | Seed2.0 Pro 52.6% | 2 | 2 | Nov 2025 | Mar 2026 |
| MARS-Bench Models handle long multi-turn dialogues built from play-by-play sports commentary. Tasks test ultra multi-turn, interactive multi-turn and cross-turn reasoning. | ByteDance Seed and collaborators (Yang et al.) | Multi-turn conversation | Everyday | Fading | Seed2.0 Pro 85.6% | 2 | 2 | May 2025 | Mar 2026 |
| Matbench Models predict properties of inorganic materials in 13 supervised tasks built from DFT and experimental data. Scored with error or classification metrics per task. | Materials Project (Dunn, Wang, Ganose, Dopp, Jain) | Research science | Expert | Fading | Intern-S1 75.0 | 2 | 2 | May 2020 | Mar 2026 |
| Meeseeks Models follow complex instructions over several turns and get feedback on failed requirements after each turn. Scored by how many requirements they meet. | Meituan | Multi-turn conversation | Everyday | Dropped | LongCat-Flash-Chat 43% | 2 | 2 | Apr 2025 | Oct 2025 |
| MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. | Alibaba AIDC-AI (Parrot authors) | Knowledge in many languages | School | In use | LFM2.5-VL-3B 83% | 2 | 2 | Jun 2024 | Aug 2026 |
| MSEarth Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split. | Shanghai AI Lab (Xiangyu Zhao, Lei Bai et al.) | General visual questions | Expert | Fading | Intern-S1 65.7% | 2 | 2 | May 2025 | Mar 2026 |
| OMEGA Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically. | UC Berkeley, Ai2 and University of Washington (Sun et al.) | Competition problems with answers | Competition | Fading | Olmo 3.1 32B Think 53.4% | 2 | 2 | Jun 2025 | Dec 2025 |
| Omni-Cloze Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail. | Ma et al. (Omni-Captioner, arXiv 2510.12720) | Video captioning | Everyday | Fading | Qwen3.5-Omni-Plus 64.8% | 2 | 2 | Oct 2025 | Sep 2026 |
| OmniBench Models answer multiple choice questions that need an image, an audio clip and text together. Scored by accuracy. | Multimodal Art Projection (M-A-P) | Audio understanding and voice | Everyday | Dropped | LongCat-Flash-Omni 61.4% | 2 | 2 | Sep 2024 | Oct 2025 |
| OpenAudioBench Models answer spoken questions in five sets: Reasoning QA, Llama Questions, Web Questions, TriviaQA and AlpacaEval. Knowledge sets use accuracy and the open sets use a model score. | Baichuan | Audio understanding and voice | Everyday | Dropped | LongCat-Flash-Omni 80.1% | 2 | 2 | Jan 2025 | Oct 2025 |
| QIVD Models answer spoken questions about short phone videos in real time, as in a face to face talk. Each item has a short answer and a timestamp. | Qualcomm AI Research and University of Toronto | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 69.6% | 2 | 2 | Mar 2025 | Sep 2026 |
| SMolInstruct Models solve 14 chemistry tasks such as name conversion, property prediction, molecule captioning, generation and reaction prediction. Each task uses its own metric on the test split. | The Ohio State University (LlaSMol team) | Genomics and life sciences | Expert | Fading | Intern-S1-Pro 74.8% | 2 | 2 | Feb 2024 | Mar 2026 |
| TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. | Google Research | Reading in many languages | Everyday | Dropped | Llama 4 Maverick 31.7% | 2 | 2 | Mar 2020 | Apr 2025 |
| VisualWebBench Models answer questions about web page screenshots across seven tasks, such as captioning, OCR, element grounding and action prediction. | Junpeng Liu, Yifan Song et al. | Finding things on screen | Everyday | Dropped | Seed1.5-VL 87.3% | 2 | 2 | Apr 2024 | Oct 2025 |
| Kitab Models list books by a given author that meet extra constraints, such as publication dates or title words. Scored by precision and constraint satisfaction. | Microsoft Research | Long-form factuality | Everyday | Dropped | 2 | 2 | Oct 2023 | Apr 2025 | |
| MemoryBench (Meta) Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics. | Meta (Meta Superintelligence Labs) | Finding facts in long inputs | Everyday | In use | 2 | 2 | Oct 2026 | Oct 2026 | |
| Personal Superintelligence Index (Meta) Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO. | Meta (Meta Superintelligence Labs) | Composite indices | Expert | In use | 2 | 2 | Oct 2026 | Oct 2026 | |
| ProfBench Model writes expert-level reports in physics, chemistry, finance and consulting. Calibrated LLM judges grade against PhD/MBA-written rubrics. | NVIDIA | Tasks from real jobs | Expert | In use | Claude Opus 5 66.2% | 2 | 1 | Oct 2025 | Aug 2026 |
| $OneMillion-Bench Agent solves 400 expert-curated tasks in law, finance, industry, healthcare and science. Rubric grading of accuracy, logic, feasibility and compliance. | Humanlaya + BIGAI + xbench | Tasks from real jobs | Expert | In use | Claude Fable 5.1 42.1% | 2 | 1 | Mar 2026 | Sep 2026 |
| SpreadsheetBench 2 Agents complete 321 expert-annotated business spreadsheet workflows, such as financial modeling, debugging, and charting, in multi-sheet workbooks. Scoring is task accuracy. | Renmin University of China (RUCKBReasoning) | Documents and spreadsheets | Expert | In use | Claude Opus 4.6 34.9% | 2 | 1 | Jun 2026 | Sep 2026 |
| RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. | AGI-Eval (Meituan) | Deep research | Expert | Fading | GPT-5.2 82% | 2 | 1 | Jan 2026 | Jul 2026 |
| CombiBench Provers solve 100 combinatorics problems formalized in Lean 4, from middle school to IMO and university level. Lean checks each proof. | Moonshot AI and Project Numina (Liu et al.) | Formal proofs | Competition | Fading | Seed-Prover 1.5 48% | 2 | 1 | May 2025 | Dec 2025 |
| CountQA Models count objects in over 1,500 real-world photos with dense clutter and occlusion. Scoring is accuracy on the count. | Spatial reasoning and counting | Everyday | In use | Qwen 3.8 Max 82.4% | 2 | 1 | Aug 2025 | Aug 2026 | |
| HorizonMath Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound. | University of Oxford, Harvard University, Princeton University and others (Wang et al.) | Research mathematics | Research frontier | In use | Tencent Hy4 preview 8.80% | 2 | 1 | Mar 2026 | Aug 2026 |
| OSWorld 2.1 (offline subset) Agents complete the 82 OSWorld 2.1 tasks on the official offline list, with no internet access. Scored by checkpoint partial credit and strict pass rate. | XLANG Lab, HKU | Desktop apps | University | In use | Gemini 4 Carbon 81.5% | 2 | 1 | Sep 2026 | Oct 2026 |
| PresentBench Models generate full slide decks from background materials for 238 real-world cases. A judge scores each deck against about 54 hand-written checklist items. | PresentBench team | Documents and spreadsheets | University | In use | Qwen 3.8 Max 79.6% | 2 | 1 | Mar 2026 | Sep 2026 |
| RACE-M Models read passages from English exams for Chinese middle school students and answer multiple-choice questions. Scored by accuracy. | Carnegie Mellon University | Reading comprehension | School | Dropped | PaLM 2-L 77% | 2 | 1 | Apr 2017 | Dec 2024 |
| SimplerEnv Real-to-sim replicas of Google Robot and WidowX/Bridge setups. Policies scored by success rate on pick, move, drawer and Bridge tasks. | UC San Diego + Stanford + UC Berkeley + Google DeepMind | Simulated robots | Everyday | Fading | GR00T N1.5 63% | 2 | 1 | May 2024 | Jul 2026 |
| SpreadsheetBench Verified Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets. | Renmin University of China (RUCKBReasoning) with Shortcut.AI | Documents and spreadsheets | University | Fading | Seed2.0 Pro 79.1% | 2 | 1 | Dec 2025 | Jun 2026 |
| ASIMOV 2.0 Models spot physical hazards in text and generated videos from real injury reports, time safety interventions in video, and follow robot payload and handling constraints. | Google DeepMind | Embodied reasoning | Fading | Gemini Robotics ER 2 97.9% | 2 | 1 | Sep 2025 | Oct 2026 | |
| ASIMOV v1 1.0 Replaced in Sep 2025 | Everyday | Not in cards | Mar 2025 | ||||||
| BABE Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference. | ByteDance Seed and collaborators (Zhou et al.) | Genomics and life sciences | Expert | Fading | Seed2.0 Lite (0428) 57.9% | 2 | 1 | Feb 2026 | Apr 2026 |
| Biology-Instructions Models answer multi-omics sequence tasks on DNA, RNA, protein and multi-molecule inputs, such as property prediction and interaction tasks. Scored per task with task metrics. | Biology-Instructions team (hhnqqq et al.) | Genomics and life sciences | Expert | Fading | Intern-S2-397B 55.7% | 2 | 1 | Dec 2024 | Sep 2026 |
| CFBench Models follow 1,000 real-world instructions with embedded constraints, across more than 200 scenarios and 50 NLP tasks. A judge checks each constraint and the overall requirement. | Baichuan and Peking University | Following precise instructions | Everyday | Dropped | Hunyuan-T1 81% | 2 | 1 | Aug 2024 | Sep 2025 |
| FATE-H Provers write Lean proofs for 100 abstract and commutative algebra problems at honors course or graduate exam level. The Lean compiler checks each proof. | Jiang et al. (FATE team) | Formal proofs | Expert | Fading | Leanstral 1.5 87% | 2 | 1 | Nov 2025 | Jul 2026 |
| FATE-X Provers write Lean proofs for 100 algebra problems at or beyond PhD qualifying exam level, some beyond what Mathlib covers. The Lean compiler checks each proof. | Jiang et al. (FATE team) | Formal proofs | Research frontier | Fading | Leanstral 1.5 34% | 2 | 1 | Nov 2025 | Jul 2026 |
| GQA Models answer compositional questions about real-world images, generated from scene graphs. Scored by exact-match accuracy, often on the balanced test-dev split. | Stanford University (Hudson and Manning) | General visual questions | Everyday | Fading | Janus-Pro-7B 62% | 2 | 1 | Feb 2019 | Aug 2026 |
| HR-Bench Models answer multiple-choice questions about fine details in 4K and 8K resolution images. Results are reported separately for the 4K and 8K sets. | Wuhan University and collaborators (Wang et al.) | General visual questions | Everyday | Fading | MiMo-V2.5 88.5% | 2 | 1 | Aug 2024 | May 2026 |
| Hungarian National High-School Exam Models answer the 33 questions of the 2023 Hungarian national high school mathematics finals. Answers are graded by hand with the official rubric. | Keiran Paster (exam by the Hungarian Educational Authority) | Word problems | School | Dropped | Grok-1 59% | 2 | 1 | Nov 2023 | Jan 2024 |
| JointAVBench Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans. | Chao et al. (arXiv 2512.12772) | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 75.9% | 2 | 1 | Dec 2025 | Sep 2026 |
| Ko-GDPval Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format. | Upstage | Tasks from real jobs | Expert | In use | Solar Pro 4 87.3% | 2 | 1 | Jul 2026 | Aug 2026 |
| MedXpertQA Models answer 4,460 expert-level medical exam questions across 17 specialties, in a text subset and an image subset. Scoring is multiple-choice accuracy. | Tsinghua University (TsinghuaC3I) | Medicine | Expert | Fading | Muse Spark 52.6% | 2 | 1 | Jan 2025 | Jun 2026 |
| MMAU-Pro Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes. | University of Maryland (GAMMA lab) and collaborators | Audio understanding and voice | University | Fading | MiMo-V2-Omni 69.4% | 2 | 1 | Aug 2025 | Mar 2026 |
| Mol-Instructions Models answer molecule-oriented, protein-oriented and biomolecular text instructions, such as reaction prediction, molecule description and protein function questions. Scored per task with task metrics. | Zhejiang University (ZJUNLP) | Genomics and life sciences | Expert | Fading | Intern-S2-397B 54% | 2 | 1 | Jun 2023 | Sep 2026 |
| OmniVideoBench Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps. | Nanjing University (NJU-LINK Lab) and others | Video understanding | University | Fading | Qwen3.8-Omni-Flash 63.4% | 2 | 1 | Oct 2025 | Sep 2026 |
| RefusalBench (Nous Research) Models get requests in scenarios that closed and open models commonly refuse. The score is the percent of questions the model answers, averaged over 5 trials. | Nous Research | Deception and alignment | Everyday | Fading | Hermes 4.3 36B 74.6% | 2 | 1 | Aug 2025 | Dec 2025 |
| SciReasoner evaluation suite Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks. | Shanghai AI Lab (open-sciencelab) | Research science | Expert | Fading | Intern-S2-397B 63.3% | 2 | 1 | Sep 2025 | Sep 2026 |
| TheoremQA Models solve 800 university-level questions in math, physics, EE and CS, and finance that each need a specific theorem. Scored by answer accuracy. | University of Waterloo (TIGER-AI-Lab) | Science questions | University | Dropped | Qwen2-72B 44.4% | 2 | 1 | May 2023 | Sep 2024 |
| Apple side-by-side human evaluation Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison. | Apple | Overall chat quality | Everyday | Fading | AFM 3 Cloud 64.7% | 2 | 1 | Jun 2025 | Jun 2026 |
| FrontierMath Tiers 1-3 v2 Models solve 295 unpublished math problems from undergraduate to advanced graduate level and submit answers that code checks automatically. Scored by share correct. | Epoch AI | Research mathematics | Expert | In use | GPT-6 Astra 93.7% | 2 | Jun 2026 | Oct 2026 | |
| FrontierMath Original Replaced in Jun 2026 | Fading | GPT-5.5 Pro 52.4% | 11 | 1 | Nov 2024 | Jun 2026 | |||
| FrontierMath Tier 4 v2 Models solve 43 unpublished research-level math problems and submit answers that code checks automatically. Scored by share correct. | Epoch AI | Research mathematics | Research frontier | In use | GPT-6.1 Sol 100% | 2 | Jun 2026 | Oct 2026 | |
| FrontierMath Tier 4 Original Replaced in Jun 2026 | Fading | AI Co-Mathematician 47.9% | 5 | Jul 2025 | Sep 2026 | ||||
| PRBench Model answers 1,100 expert-written law and finance questions. LLM judge grades against weighted expert rubrics of 10-30 criteria each. | Scale AI | Law | Expert | In use | Muse Spark 1.3 61.6% | 2 | Nov 2025 | Oct 2026 | |
| LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. | Arena (formerly LMArena) | Deep research | Everyday | Fading | GPT-5.6 Sol 1,258 | 2 | Apr 2025 | Sep 2026 | |
| LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. | Mosi-AI (Xiang Long, Yehui Tang and collaborators) | General assistants | Expert | In use | Kimi K2.7 Code 76% | 2 | Mar 2026 | Oct 2026 | |
| CL-bench Life Models solve 405 tasks that require learning from messy real-life context such as group chats. A model judge checks rubrics to give a solve rate. | Tencent Hunyuan and Fudan University | Learning from the context | Everyday | Fading | GPT-5.5 22.2% | 2 | Apr 2026 | Jul 2026 | |
| FrontierFinance Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics. | Samaya AI | Finance | Expert | In use | Claude Fable 5.1 55.9% | 2 | Jul 2026 | Oct 2026 | |
| LOCA-Bench Agentic tasks whose environment state is scaled to grow context length while task semantics stay fixed. Success rate versus context length. | HKUST NLP | Reasoning over long documents | University | In use | GPT-5.2 21.3% | 2 | Feb 2026 | Oct 2026 | |
| MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. | Meta FAIR | Knowledge in many languages | Everyday | Fading | Llama 3.1-405B 31% | 2 | Apr 2025 | Sep 2026 | |
| BigFinanceBench Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference. | Rogo Technologies | Finance | Expert | In use | GPT-5.6 Terra 51% | 2 | Jun 2026 | Jul 2026 | |
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Alibaba Tongyi Lab | Deep research | Expert | Fading | Step 3.7 Flash 59% | 2 | Aug 2025 | Jun 2026 | |
| CMT-Benchmark Models solve 50 research-level condensed matter theory problems written and checked by a panel of expert physicists. Answers are graded automatically. | International panel of condensed matter researchers (Pan, Roggeveen, Berg, Carrasquilla, Chowdhury and others) | Research science | Research frontier | Fading | Gemini 3 Deep Think 50.5% | 2 | Oct 2025 | Jul 2026 | |
| Dense 200 (bbox) Model detects and boxes every object in 200 densely annotated images (109 categories, about 91 boxes each). Box F1 at IoU thresholds. | IDEA Research (Rex-Omni) | Detection and captioning | Everyday | In use | Qwen 3.8 Max 87% | 2 | Oct 2025 | Oct 2026 | |
| E-Bench (Tencent) Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published. | Tencent Hy team | Documents and spreadsheets | Expert | In use | Tencent Hy4 preview 77.1% | 2 | Jul 2026 | Aug 2026 | |
| EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. | ServiceNow | Apps and MCP servers | Expert | In use |  Spark-X2.6-4B 33.6% | 2 | Mar 2026 | Oct 2026 | |
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | Meta (Meta Superintelligence Labs) | General assistants | Everyday | In use | Muse Wonder Full 54.2% | 2 | Oct 2026 | Oct 2026 | |
| GeneBench Agents analyze messy genomics and quantitative biology data across 103 multi-step problems. Scored by pass rate on verifiable final answers. | OpenAI | Genomics and life sciences | Expert | Fading | GPT-5.5 33.2% | 2 | Apr 2026 | Apr 2026 | |
| Hy-Backend 2.0 Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published. | Tencent Hy team | Fixing real repositories | Expert | In use | Tencent Hy4 preview 35.2% | 2 | Jul 2026 | Aug 2026 | |
| Hy-Backend v1 Replaced in Jul 2026 | Fading | Tencent Hy3 preview 54.7% | 1 | Apr 2026 | Apr 2026 | ||||
| IFStruct Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds. | Liquid AI | Following precise instructions | Everyday | In use | LFM3-1.7B 93.7% | 2 | Jun 2026 | Oct 2026 | |
| IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. | Sarvam AI | Chat and instructions in many languages | Everyday | Fading | Sarvam 105B 90% | 2 | Mar 2026 | Mar 2026 | |
| Instrument reading (Google DeepMind) Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types. | Google DeepMind | Embodied reasoning | Everyday | Fading | Gemini Robotics-ER 1.6 93% | 2 | Apr 2026 | Jul 2026 | |
| KAT Code Bench Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages. | Kuaishou (KwaiKAT team) | Fixing real repositories | University | In use | KAT-Coder-Pro V2.5 53.1% | 2 | Jul 2026 | Jul 2026 | |
| Kernel Bench L3 Models write fast GPU kernels for the 50 full model architectures in KernelBench Level 3. Scored on correctness and speedup over PyTorch. | Stanford Scaling Intelligence Lab | Speed and optimization | Expert | Fading | Qwen3.7-Plus 2.06 | 2 | Dec 2024 | Jun 2026 | |
| LBM Eval Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task. | Toyota Research Institute | Simulated robots | Everyday | Fading | Foundry-Qwen3VLA-2.1B 72.4% | 2 | May 2025 | Apr 2026 | |
| MiMo Coding Bench Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE. | Xiaomi MiMo | Fixing real repositories | Expert | Fading | MiMo-V2.5-Pro 73.7% | 2 | Apr 2026 | Apr 2026 | |
| Nuscene Models localize named objects as 3D boxes in nuScenes driving camera images. Scored by mAP. | Motional (nuTonomy) | Detection and captioning | Everyday | Fading | Qwen3.5 397B-A17B 16.0 | 2 | Mar 2019 | Feb 2026 | |
| PhysicsFinals Models answer 61 undergraduate physics final exam problems written by physics professors. A professor grades the answers. | Google DeepMind | Science questions | University | Dropped | Gemini 1.5 Pro 63.9% | 2 | Feb 2024 | May 2024 | |
| QwenSVG Models write SVG graphics code for Qwen's in-house prompts. Results are reported as ratings near 1,500 to 1,600, not percentages. | Qwen Team, Alibaba Group | Web and app front ends | University | Fading | Qwen3.7-Max 1,608 | 2 | May 2026 | Jun 2026 | |
| QwenSWEBench Qwen in-house software-engineering benchmark run in Claude Code (avg@3, 8-hour limit). Task and scoring details unpublished. | Alibaba Qwen | Fixing real repositories | Expert | In use | Qwen 3.8 Max 80.7% | 2 | Aug 2026 | ||
| QwenWebDev Models generate front-end web pages in English and Chinese across 7 categories. Pages are rendered and a multimodal judge compares them, giving Bradley-Terry Elo ratings. | Qwen Team, Alibaba Group | Web and app front ends | University | Fading | Qwen3.7-Max 1,568 | 2 | May 2026 | Jun 2026 | |
| QwenWorldBench Models act as world models that simulate agent environments in 7 domains, such as terminal, web, and Android. A rubric judge scores five dimensions. | Qwen Team, Alibaba Group | Interactive world models | Fading | Qwen3.7-Plus 62.1% | 2 | May 2026 | Jun 2026 | ||
| Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. | Reka AI | Deep research | Expert | Dropped | Grok 4 Fast 66% | 2 | Aug 2025 | Nov 2025 | |
| Success detection (Google DeepMind) Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures. | Google DeepMind | Embodied reasoning | Everyday | Fading | Gemini Robotics-ER 1.6 90% | 2 | Sep 2025 | Jul 2026 | |
| SURDS Models answer spatial questions about objects in nuScenes driving camera images across six single-object and multi-object tasks. Scoring is task accuracy. | Xianda Guo et al. (Wuhan University and partners) | Spatial reasoning and counting | Everyday | In use | Qwen 3.8 Max 77.8% | 2 | Nov 2024 | Aug 2026 | |
| T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. | Shanghai AI Laboratory (OpenCompass) | Function calling | Everyday | Dropped | Qwen1.5-72B 76.7% | 2 | Dec 2023 | Mar 2025 | |
| TIR-Bench Models solve 13 kinds of image tasks that need tool use on images, such as rotation, jigsaw, and maze. Most items are multiple choice. | Agents-X project (Ming Li et al.) | General visual questions | Everyday | Fading | Qwen 3.6 Plus 61.6% | 2 | Nov 2025 | Apr 2026 | |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | xAI | Deep research | Expert | Dropped | Grok 4 Fast 58% | 2 | Nov 2025 | Nov 2025 | |
| XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. | Cambridge Language Technology Lab (Ponti et al.) | Reasoning in many languages | Everyday | Dropped | PaLM 2-L 94.4% | 2 | May 2020 | Sep 2025 | |
| HealthBench Consensus Models answer HealthBench conversations graded only on criteria that several physicians agreed on. | OpenAI | Medicine | Expert | Dropped | GPT-5.4 96.6% | 1 | 25 | May 2025 | Oct 2026 |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | OpenAI | Capture-the-flag challenges | Expert | Fading | GPT-5.5 96.3% | 1 | 22 | Aug 2024 | May 2026 |
| MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. | Center for AI Safety + Scale AI | Deception and alignment | In use | Muse Spark Open 98.1% | 1 | 20 | Mar 2025 | Oct 2026 | |
| SimpleQA Hallucination Rate Models answer 4,326 short fact-seeking questions. This metric is the share of questions answered incorrectly, so declining to answer is not penalized. | OpenAI | Hallucination rates | Dropped | ChatGPT agent 7.90% | 1 | 12 | Oct 2024 | Oct 2025 | |
| QuAC Models answer free-form questions in information-seeking dialogs about a hidden Wikipedia section, across 100K questions. Scored by word-level F1. | Allen Institute for AI, University of Washington, Stanford, and UMass Amherst | Reading comprehension | Everyday | Dropped | Llama 3-70B 51.1% | 1 | 12 | Aug 2018 | Nov 2024 |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | Cohere Labs | Knowledge in many languages | University | Dropped | Claude Opus 5.5 94.3% | 1 | 10 | Dec 2024 | Oct 2026 |
| Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. | Gray Swan + UK AISI + US CAISI | Jailbreaks and prompt injection | In use | GPT-6 Astra 8.50% | 1 | 9 | Mar 2026 | Oct 2026 | |
| MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. | AI4Bharat | Knowledge in many languages | University | Dropped | Claude Opus 5.5 93.1% | 1 | 9 | Nov 2024 | Oct 2026 |
| Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. | Anthropic | Deception and alignment | Expert | In use | Claude Sonnet 4.5 0% | 1 | 9 | Jun 2025 | Oct 2026 |
| MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. | Tongyi Lab, Alibaba Group | Reasoning in many languages | University | Dropped | Qwen3-235B-A22B 77.1% | 1 | 8 | Nov 2024 | Sep 2025 |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University of Oregon NLP Group (Okapi) | Knowledge in many languages | University | Dropped | Qwen2.5-72B 80% | 1 | 8 | Jul 2023 | Jul 2025 |
| AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. | ETH Zurich SPY Lab + Invariant Labs | Jailbreaks and prompt injection | Everyday | In use | Grok 4 Fast 0% | 1 | 7 | Jun 2024 | Oct 2026 |
| QuALITY Models answer multiple-choice questions about English passages that average about 5,000 tokens. Scoring is accuracy. | New York University | Reasoning over long documents | School | Dropped | Claude 3 Opus 90.5% | 1 | 7 | Dec 2021 | Nov 2024 |
| MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. | Yonsei University, OneLineAI, and KAIST AI | Reasoning in many languages | Competition | Dropped | Qwen3-235B-A22B 80.8% | 1 | 7 | Feb 2025 | May 2025 |
| LV-Eval Models answer single-hop and multi-hop questions over contexts at five length levels from 16K to 256K tokens, with confusing facts inserted. | Infinigence AI and Tsinghua University (Yuan et al.) | Reasoning over long documents | Everyday | Dropped | Qwen2.5-14B-Instruct-1M 49.8% | 1 | 7 | Feb 2024 | Jan 2025 |
| Perception Test MCVQA Models answer multiple-choice questions about real-world videos filmed to show perceptually interesting situations. Scored by top-1 accuracy. | Google DeepMind | Video understanding | Everyday | Dropped | Gemini 2.5 Pro 78.4% | 1 | 6 | May 2023 | Jul 2025 |
| LongBench-Chat Models follow real user instructions over contexts of 10K to 100K tokens. GPT-4 scores each answer from 1 to 10 against references. | Tsinghua University and Zhipu AI (Bai et al., LongAlign) | Reasoning over long documents | Everyday | Dropped | Qwen2.5-14B-Instruct-1M 8.76 | 1 | 6 | Jan 2024 | Jan 2025 |
| Nexus Models write function calls for queries over real-world APIs, including nested and parallel calls. Scoring is the function calling success rate. | Nexusflow | Function calling | University | Dropped | Llama 3.1-405B 58.7% | 1 | 6 | Dec 2023 | Nov 2024 |
| Qasper Models answer 5,049 questions about 1,585 NLP research papers from the full paper text. Answers are scored with token F1. | Allen Institute for AI (AI2) | Reasoning over long documents | University | Dropped | Phi-3.5-mini 41.9% | 1 | 6 | May 2021 | Mar 2025 |
| SQuALITY Models write question-focused summaries of public-domain short stories. Outputs are scored against human-written reference summaries with ROUGE. | New York University | Reasoning over long documents | School | Dropped | Phi-3.5-mini 24.3% | 1 | 6 | May 2022 | Mar 2025 |
| C3 Models answer 19,577 multiple-choice questions on Chinese dialogues and texts from Chinese-as-a-second-language exams. Scoring is accuracy. | Tencent AI Lab and Cornell University | Reading in many languages | School | Dropped | InternVL3.5-241B-A28B 97.8% | 1 | 5 | Apr 2019 | Feb 2026 |
| BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. | KAIST | Knowledge in many languages | Everyday | Dropped |  Apertus 70B 66.3% | 1 | 5 | Jun 2024 | Sep 2025 |
| GSM-Plus Models solve GSM8K problems after perturbations such as changed numbers, added distractors and reversed questions. | Li et al. | Word problems | School | Dropped | LFM2-8B-A1B 64.8% | 1 | 5 | Feb 2024 | Nov 2025 |
| RepoQA Models find a function in a long code repository from a description of what it does, across Python, C++, Rust, Java and TypeScript. | UIUC (EvalPlus team) | Finding facts in long inputs | University | Dropped | Claude 3 Opus 90.6% | 1 | 4 | Jun 2024 | Feb 2026 |
| AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. | FreedomIntelligence (AceGPT team, CUHK-Shenzhen) | Knowledge in many languages | University | Dropped | Qwen2.5-72B 72.4% | 1 | 4 | Sep 2023 | Dec 2024 |
| CRPE Models answer single-choice questions about objects and their relations in images, covering subject, predicate and object. Each question is asked again with options rotated (CircularEval). | OpenGVLab, Shanghai AI Laboratory (All-Seeing Project V2) | General visual questions | Everyday | Dropped | Qwen2.5-VL-72B 79.2% | 1 | 4 | Feb 2024 | Nov 2025 |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | MBZUAI | Knowledge in many languages | School | Dropped | Qwen2.5-72B 69.3% | 1 | 4 | Oct 2023 | Dec 2024 |
| JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. | Waseda University (nlp-waseda) | Knowledge in many languages | University | Dropped | Qwen2.5-72B 80.6% | 1 | 4 | Jan 2024 | Dec 2024 |
| MedQA Models answer multiple-choice questions from medical licensing exams, usually the 12,723-question English USMLE set. Scoring is accuracy. | MIT | Medicine | Expert | Dropped | GPT-4o 89.4% | 1 | 4 | Sep 2020 | Aug 2024 |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Qwen | Chat and instructions in many languages | Everyday | Dropped | Qwen2.5-72B 87% | 1 | 4 | Sep 2024 | Dec 2024 |
| TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. | Arda Yüksel et al. | Knowledge in many languages | School | Dropped | Qwen2.5-72B 76.1% | 1 | 4 | Jul 2024 | Dec 2024 |
| XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. | DeepMind | Reading in many languages | Everyday | Dropped | Gemma 3 27B 76.8 | 1 | 4 | Oct 2019 | Mar 2025 |
| WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. | Allen Institute for AI and University of Washington (WildTeaming) | Jailbreaks and prompt injection | Everyday | Fading | Falcon H1R 7B 98.5% | 1 | 4 | Jun 2024 | Jan 2026 |
| Code Arena v2 Models act as coding agents that build web apps from user prompts, and users vote between two results. Votes yield Elo-style ratings. | Arena (formerly LMArena) | Web and app front ends | In use | Claude Opus 5.5 1,810 | 1 | 3 | Nov 2025 | Oct 2026 | |
| WebDev Arena v1 Replaced in Nov 2025 | Fading | Claude Opus 5.5 1,820 | 2 | 1 | Dec 2024 | Oct 2026 | |||
| EEBench 13 electrical-engineering circuit and PCB design tasks graded by deterministic build checks, simulation and BOM constraints. No LLM judge. | atopile | Electronics | Expert | In use | Claude Opus 5.5 75% | 1 | 3 | Sep 2026 | Oct 2026 |
| Point-Bench Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct. | Cheng et al. (PointArena team) | Detection and captioning | Everyday | Fading | gemini-robotics-er-1.5-preview 67.1% | 1 | 3 | May 2025 | Mar 2026 |
| PubMedQA Models answer biomedical research questions with yes, no, or maybe, given a PubMed abstract without its conclusion. Scored by accuracy. | University of Pittsburgh and Carnegie Mellon University | Medicine | University | Dropped | Claude 3 Sonnet 78.3% | 1 | 3 | Sep 2019 | Apr 2024 |
| StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. | Meta | Jailbreaks and prompt injection | Expert | In use | Muse Spark 1.1 0.50% | 1 | 3 | Oct 2026 | |
| WorldSense Models answer synthetic questions that test consistent reasoning about simple arrangements of entities described in text. | Meta FAIR (Benchekroun et al.) | Language reasoning | Everyday | Fading | Seed2.0 Lite (0428) 67.3% | 1 | 3 | Nov 2023 | Apr 2026 |
| APPS Models write Python programs for 10,000 coding problems that range from introductory to competition level. Generated code runs against test cases. | UC Berkeley | Competitive programming | University | Dropped | Claude 3 Opus 70.2% | 1 | 3 | May 2021 | Mar 2024 |
| COCO Captions Models write a caption for each image, and outputs are compared with five human captions per image using BLEU, METEOR, ROUGE, and CIDEr. | Microsoft COCO | Detection and captioning | Everyday | Dropped | Gemma 3 27B 116.0 | 1 | 3 | Apr 2015 | Mar 2025 |
| EMMA Models solve math, physics, chemistry and coding problems that need reasoning over text and images together, not each alone. Answers are checked against a key. | Hao et al. (EMMA team) | Visual math and diagrams | University | Fading | Seed2.1 Pro 79.3% | 1 | 3 | Jan 2025 | Jun 2026 |
| HuggingFace Agent Benchmark Agents write Python code that calls Hugging Face Transformers tools to solve short tasks. Scores cover tool selection, tools used, and code correctness. | Hugging Face | Function calling | School | Dropped | Qwen-14B 93.5% | 1 | 3 | May 2023 | Sep 2023 |
| InfiniteBench/En.MC Models answer 229 multiple choice questions about long fake books with altered entity names, averaging about 184K tokens. Scoring is accuracy. | OpenBMB (Zhang et al.) | Reasoning over long documents | Everyday | Dropped | Llama 3.1-405B 83.4% | 1 | 3 | Nov 2023 | Nov 2024 |
| OK-VQA Models answer over 14,000 visual questions that need outside knowledge beyond the image. Scoring is VQA accuracy against human answers. | Allen Institute for AI and Carnegie Mellon University | General visual questions | Everyday | Dropped | Gemma 3 27B 60.2% | 1 | 3 | May 2019 | Mar 2025 |
| SpatialSense VQA Models judge whether a stated spatial relation between two objects holds in an image. The relations were crowdsourced to defeat simple cues. | Princeton Vision and Learning Lab (Kaiyu Yang, Olga Russakovsky, Jia Deng) | Spatial reasoning and counting | Everyday | Dropped | Gemma 3 12B 60% | 1 | 3 | Aug 2019 | Mar 2025 |
| ZeroSCROLLS/QuALITY Models answer QuALITY multiple-choice questions about long stories zero-shot, as one task of the ZeroSCROLLS suite. Scoring is accuracy. | Tel Aviv University | Reasoning over long documents | Everyday | Dropped | Llama 3.1-405B 95.2% | 1 | 3 | May 2023 | Nov 2024 |
| HELMET Models handle long-context tasks in seven groups including recall, RAG, re-ranking, many-shot learning, long QA and summarization, at set input lengths. | Princeton NLP (Yen et al.) | Reasoning over long documents | University | Dropped | EXAONE 4.0 32B 58.3% | 1 | 3 | Oct 2024 | Jul 2025 |
| Open ASR Leaderboard Speech recognition models transcribe the test sets of several English and multilingual corpora. Scored by average word error rate, with speed reported next to it. | Hugging Face (hf-audio) and collaborators | Speech recognition and translation | Everyday | Fading | Phi-4-multimodal 6.14% | 1 | 3 | Sep 2023 | Oct 2026 |
| MATH Level 5 Models solve the 1,324 hardest MATH test problems, drawn from competitions such as AMC and AIME. Scored by final answer match. | Dan Hendrycks et al. (UC Berkeley), subset run by Epoch AI | Competition problems with answers | Competition | Dropped | GPT-5 98.1% | 1 | 2 | Mar 2021 | Nov 2025 |
| AIR-Bench 2024 Models respond to 5,694 harmful prompts across 314 risk categories drawn from 8 government regulations and 16 company policies. Category-specific LLM judges score refusals. | Stanford CRFM + Virtue AI et al. | Refusing harmful requests | Everyday | Dropped | Claude Haiku 4.5 93.2% | 1 | 2 | Jul 2024 | Jun 2026 |
| GeoBench Models geolocate GeoGuessr photos from community maps by giving coordinates and a country. Scored by country accuracy and a distance-based score. | ccmdi | General visual questions | Everyday | Dropped | Claude Fable 5 89% | 1 | 2 | Mar 2025 | Sep 2026 |
| VPCT Models view an image of a ball, ramps, and three buckets and predict which bucket the ball falls into. Scored by accuracy on 100 problems. | Chase Brower | Physical prediction | Everyday | Fading | Gemini 3 Pro 91% | 1 | 2 | Jan 2025 | Mar 2026 |
| ScienceQA Models answer about 21,000 multiple-choice science questions from school curricula, many with images. Scored by accuracy. | UCLA and Allen Institute for AI | General visual questions | School | Dropped | Phi-3.5-vision-instruct 91.3% | 1 | 2 | Sep 2022 | Mar 2025 |
| Parametric CAD Bench Agents author editable parametric FreeCAD part designs, graded against held-back references for geometry and spec consistency. | gNucleus AI | CAD models | Expert | In use | Claude Opus 5.5 61% | 1 | 2 | 2026 | Sep 2026 |
| MindCube Models answer 21,154 multiple choice questions about scenes seen from a few views. Questions test positions, orientations, and what-if movements. | MLL Lab, Northwestern University, with Stanford, NYU, and University of Washington | Spatial reasoning and counting | Everyday | Fading | Gemma 3 12B 46.7% | 1 | 2 | Jun 2025 | Jan 2026 |
| BetterChartQA Models answer annotator-written questions about real charts from news, government reports, and papers across 9 capability buckets. Scored by accuracy. | Gemini Team, Google | Documents, charts and text in images | School | Dropped | Gemini 2.5 Pro 72.4% | 1 | 2 | Jun 2024 | Jul 2025 |
| ContPhy Models infer physical properties such as mass and density of fluids and soft bodies from videos and predict their dynamics. | Zheng et al. (ContPhy team) | Physical prediction | Everyday | Fading | Seed2.0 Pro 67.4% | 1 | 2 | Feb 2024 | Jun 2026 |
| CrossVid Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats. | Li et al. (CrossVid team) | Video understanding | Everyday | Fading | Seed2.1 Pro 65% | 1 | 2 | Nov 2025 | Jun 2026 |
| FrontierScience-Research Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric. | OpenAI | Research science | Expert | Fading | GPT-5.4 Pro 36.7% | 1 | 2 | Dec 2025 | Jun 2026 |
| AInstein-SWE-Bench Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change. | ByteDance Seed | Scientific code | Expert | Fading | Seed2.0 Pro 47.7% | 1 | 2 | Dec 2025 | Mar 2026 |
| APEX-SWE Software-engineering productivity tasks spanning integration work and observability, such as diagnosing failures from telemetry. | Mercor | Fixing real repositories | Expert | In use | Grok 4.6 56.4% | 1 | 2 | Jan 2026 | Aug 2026 |
| AV-SpeakerBench Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together. | Nguyen et al. (arXiv 2512.02231) | Video understanding | Everyday | Fading | Qwen3.8-Omni-Flash 77.2% | 1 | 2 | Dec 2025 | Sep 2026 |
| BIOBench (ByteDance Seed) Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question. | ByteDance Seed | Genomics and life sciences | Competition | Fading | Seed2.0 Pro 53.5% | 1 | 2 | Dec 2025 | Mar 2026 |
| CIMemories Models use persistent memory across tasks and must withhold attributes when context requires. Reports worst-case disclosure over five samples. | Meta (Mireshghallah et al.) | Refusing harmful requests | Everyday | In use | Muse Glimmer 26.4% | 1 | 2 | Nov 2025 | Oct 2026 |
| Complex Workflow (ByteDance Seed in-house) Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare. | ByteDance Seed | Following precise instructions | Everyday | Fading | Seed2.0 Pro 64.7% | 1 | 2 | Dec 2025 | Mar 2026 |
| CorpusQA 1M Models answer computation-heavy questions whose evidence is spread across hundreds of finance, education, and real estate documents in a 1M-token context. Scored by accuracy. | Tongyi Zhiwen (Alibaba) | Reasoning over long documents | University | Fading | DeepSeek-V4-Pro 62% | 1 | 2 | Dec 2025 | Apr 2026 |
| Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. | ByteDance Seed | Customer service | Everyday | Fading | Seed2.0 Pro 72.4% | 1 | 2 | Dec 2025 | Mar 2026 |
| Education (ByteDance Seed in-house) Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios. | ByteDance Seed | Other professions | School | Fading | Seed2.0 Pro 62.8% | 1 | 2 | Dec 2025 | Mar 2026 |
| GAOKAO-Bench Models answer Chinese national college entrance exam questions from 2010 to 2022 across nine subjects, with objective and subjective questions. | Fudan University (OpenLMLab) | Exam questions | School | Dropped | InternVL3.5-241B-A28B 94.5% | 1 | 2 | May 2023 | Aug 2025 |
| HealthBench Hard Hallucinations Models respond to difficult health conversations that 2 or more physicians validated. The score is the hallucination rate, so lower is better. | OpenAI | Hallucination rates | Dropped | GPT-5 1.60% | 1 | 2 | Aug 2025 | Aug 2025 | |
| IHEval Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. | Zhang et al. (Notre Dame + Amazon) | Jailbreaks and prompt injection | Everyday | In use | Nemotron 3.5 Bolt E4B 82.4% | 1 | 2 | Feb 2025 | Oct 2026 |
| Information Extraction (ByteDance Seed in-house) Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases. | ByteDance Seed | Documents and spreadsheets | Everyday | Fading | Seed2.0 Pro 52% | 1 | 2 | Dec 2025 | Mar 2026 |
| Inter-GPS (Geometry3K) Models solve high school geometry problems that combine a diagram and text from the Geometry3K test set. Scored by accuracy. | UCLA and Sun Yat-sen University (Lu et al.) | Visual math and diagrams | School | Dropped | Phi-4-multimodal 48.6% | 1 | 2 | May 2021 | Mar 2025 |
| JudgeBench Models act as judges and pick the correct response from hard pairs on knowledge, reasoning, math and coding. The label comes from objective correctness, not human preference. | UC Berkeley | Overall chat quality | Expert | Dropped | Llama-3.1-Nemotron-Ultra-253B-v1 79.1% | 1 | 2 | Oct 2024 | May 2025 |
| Mantis-Eval Models answer questions that need reasoning across several interleaved images. Scored by accuracy. | Dongfu Jiang, Xuan He et al. (TIGER-Lab, University of Waterloo) | General visual questions | Everyday | Dropped | LongCat-Flash-Omni 84.8% | 1 | 2 | May 2024 | Oct 2025 |
| MIA-Bench Models answer 400 image prompts that stack several strict instructions on format, length and content. An LLM judge scores how well each answer follows every instruction. | Apple | General visual questions | Everyday | Dropped | Qwen3-VL-235B-A22B 92.7% | 1 | 2 | Jul 2024 | Jan 2026 |
| MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. | MiniMax | General assistants | Everyday | Fading | MiniMax-M2.7 62.7% | 1 | 2 | Mar 2026 | May 2026 |
| MMBench-Video Video models answer free-form questions about long, multi-shot YouTube videos. A GPT-4 judge scores the answers. | Shanghai AI Laboratory (OpenCompass) | Long videos | Everyday | Dropped | Qwen2.5-VL-72B 2.02 | 1 | 2 | Jun 2024 | Aug 2025 |
| MME-CC Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning. | ByteDance Seed and collaborators (Zhang et al.) | Spatial reasoning and counting | Everyday | Fading | Seed2.0 Pro 57% | 1 | 2 | Nov 2025 | Mar 2026 |
| NaturalCodeBench Models solve 402 Python and Java problems taken from real user queries to online coding services in 6 domains. Semi-automatically built tests check each solution. | Tsinghua University and Zhipu AI (THUDM) | Function-level problems | University | Dropped | Seed-Coder-8B-Instruct 49.6% | 1 | 2 | May 2024 | Aug 2025 |
| NV Chat Benchmark Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy. | NVIDIA | Overall chat quality | Everyday | In use | Nemotron 3.5 Super VL 52.7 | 1 | 2 | Oct 2026 | Oct 2026 |
| Passkey Retrieval Models find a random passkey hidden at a random position in a long filler prompt. Scored by retrieval accuracy. | EPFL | Finding facts in long inputs | Everyday | Dropped | Mixtral 8x7B 100% | 1 | 2 | May 2023 | Jan 2025 |
| PixMo-Count Models count objects in natural images and give the exact number. Scored by accuracy on the test split. | Allen Institute for AI (Molmo and PixMo) | Detection and captioning | Everyday | Fading | Keye-VL-2.0-30B-A3B 83.5% | 1 | 2 | Sep 2024 | Jun 2026 |
| RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. | MiniMax | Deep research | Expert | Fading | MiniMax-M2.7 64.3% | 1 | 2 | Feb 2026 | May 2026 |
| Multilingual LibriSpeech Models transcribe read audiobook speech in eight languages from LibriVox. Scored by word error rate. | Meta AI (Pratap et al.) | Speech recognition and translation | Everyday | Fading | Nova 2 Omni Preview 4.70 | 1 | 2 | Dec 2020 | Dec 2025 |
| RoboSpatial-Home Models answer 350 spatial questions on home scenes for robots, in three types: object context, compatibility of placements and spatial configuration. | NVIDIA and The Ohio State University | Spatial reasoning and counting | Everyday | Dropped | Qwen3-VL-235B-A22B 73.9% | 1 | 2 | Nov 2024 | Nov 2025 |
| SArena Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references. | Shanghai Jiao Tong University, Shanghai AI Laboratory and others (InternSVG team) | Web and app front ends | Everyday | Fading | Intern-S1-Pro 83.5 | 1 | 2 | Oct 2025 | Mar 2026 |
| WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. | ByteDance Seed and collaborators (Wang et al.) | General assistants | Everyday | Fading | Seed1.8 47.2% | 1 | 2 | Feb 2026 | Mar 2026 |
| WildVision Vision-language models answer 500 real user image prompts from the WildVision Arena. A GPT-4 judge compares each answer with a fixed baseline model. | Allen Institute for AI and UCSB | General visual questions | Everyday | Dropped | 1 | 2 | Jun 2024 | Nov 2025 | |
| CorpFin v2 Models answer expert-written questions about credit agreements that often exceed 200 pages. An LLM judge grades each answer for accuracy. | Vals AI | Finance | Expert | In use | Claude Opus 5 73.2% | 1 | 1 | Aug 2026 | |
| Vibe Code Bench 1.1 End-to-end web application development from natural-language specs. Built apps are tested against functional requirements. | Vals AI | Building whole projects | Expert | In use | Claude Sonnet 5.5 92.4% | 1 | 1 | Mar 2026 | Oct 2026 |
| Vibe Code Bench v1 Replaced in Mar 2026 | Not in cards | GPT-5.2 41.3% | Nov 2025 | Feb 2026 | |||||
| Legal Research Bench Agents answer US federal and state legal research questions with case-law and web tools. Expert rubrics, all-pass and weighted scores by LLM judge. | Vals AI | Law | Expert | In use | Gemini 4 Carbon 55.8% | 1 | 1 | Jun 2026 | Oct 2026 |
| ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. | MBZUAI and collaborators (Koto et al.) | Knowledge in many languages | School | Dropped | Qwen2.5-72B 71.4% | 1 | 1 | Feb 2024 | Nov 2025 |
| AraTrust Models answer 522 human-written Arabic multiple-choice questions on truthfulness, ethics, safety, physical and mental health, unfairness, illegal activities, privacy and offensive language. | Alghamdi et al. | Deception and alignment | Everyday | Dropped | Qwen2-72B 89.2% | 1 | 1 | Mar 2024 | Nov 2025 |
| Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. | Open Arabic LLM Leaderboard team (2A2I and community) with Hugging Face | Knowledge in many languages | School | Dropped | Qwen2.5-72B 72.4% | 1 | 1 | Feb 2025 | Nov 2025 |
| Open Arabic LLM Leaderboard v1 Replaced in Feb 2025 | Open Arabic LLM Leaderboard team (2A2I and TII) with Hugging Face | Not in cards | May 2024 | ||||||
| MixEval Models answer real user queries matched to similar questions from existing benchmarks. Ground-truth answers make it a cheap proxy for Chatbot Arena rankings. | National University of Singapore (Ni et al.) | Overall chat quality | Everyday | Dropped | Claude 3.5 Sonnet 89.9% | 1 | 1 | Jun 2024 | Jul 2025 |
| MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. | Salesforce AI Research | Apps and MCP servers | University | Fading | Gemini 3 Pro 44.6% | 1 | 1 | Aug 2025 | Dec 2025 |
| StructEval Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones. | TIGER-AI-Lab (Yang, Jiang et al.) | Data science and SQL | Everyday | In use | Gemini 3.1 Flash-Lite 81.2% | 1 | 1 | May 2025 | Oct 2026 |
| OlympicArena Models solve 11,163 bilingual problems from 62 international olympiad competitions in seven fields, some with images. Answers are checked against keys. | Shanghai Jiao Tong University GAIR Lab | Science olympiads | Competition | Dropped | GPT-4o 40.5% | 1 | 1 | Jun 2024 | Jun 2025 |
| HumanEval-V Models read a diagram and a function signature, then write Python code that must pass test cases. Tasks span six types of visual reasoning. | Fengji Zhang, Jacky Keung et al. | Function-level problems | University | Fading | o1 40.6% | 1 | 1 | Oct 2024 | Jan 2026 |
| MHPP Models write Python functions for 210 human-curated problems that need careful reading of specifications and multi-step reasoning. Unit tests check each solution. | SparksofAGI (Dai et al.) | Function-level problems | University | Dropped | o1-preview 68.1% | 1 | 1 | May 2024 | Jun 2025 |
| Math Kangaroo 2025 (MathArena) Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram. | MathArena (ETH Zurich SRI Lab and INSAIT) | Visual math and diagrams | School | Fading | GPT-5.5 94.9% | 1 | 1 | Oct 2025 | Jul 2026 |
| KRETA Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean. | KRETA authors (tabtoyou et al.) | Documents, charts and text in images | Everyday | Fading | Gemini 2.0 Flash 85.4% | 1 | 1 | Aug 2025 | Apr 2026 |
| UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. | The University of Hong Kong (HKU MMLab) | General assistants | Everyday | In use | Claude Opus 4.8 47.5% | 1 | 1 | Jul 2026 | Sep 2026 |
| SaaS-Bench 106 long-horizon GUI workflows across 23 locally deployed real SaaS apps. Scored from app databases with weighted checkpoints and full-resolution rate. | UniPat AI + Peking University | Web browsing | University | In use | Kimi K3 60.1% | 1 | 1 | May 2026 | Aug 2026 |
| xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. | xbench (HongShan) | Deep research | Expert | Fading | Step 3.5 Flash 56.3% | 1 | 1 | Oct 2025 | Feb 2026 |
| xbench-DeepSearch v2505 Replaced in Oct 2025 | Fading | Step 3.5 Flash 83.7% | 3 | 1 | May 2025 | Feb 2026 | |||
| AndroidDaily Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines. | StepFun | Phones | Everyday | Fading | Step 3.7 Flash 61.9% | 1 | 1 | Nov 2025 | May 2026 |
| CCPM Models pick which of four classical Chinese poetry lines matches a given modern Chinese translation. Scored by accuracy. | Tsinghua University (THUNLP) | Reading in many languages | School | Dropped | DeepSeek-V3 92% | 1 | 1 | Jun 2021 | Dec 2024 |
| IFEval (Mistral internal version) Models follow prompts with verifiable format constraints. Mistral uses its own fixed version of IFEval. | Mistral AI | Following precise instructions | Everyday | Dropped | Magistral Medium 1.0 87.4% | 1 | 1 | Jun 2025 | Jun 2025 |
| Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. | Upstage | Chat and instructions in many languages | Everyday | Fading | Solar Open 100B 79.9% | 1 | 1 | Dec 2025 | Mar 2026 |
| MMLU-stem Models answer the STEM subjects of MMLU, such as physics, chemistry, mathematics, and computer science, as multiple choice. Scoring is accuracy. | UC Berkeley | Exam questions | University | Dropped | Falcon-H1-34B 83.6% | 1 | 1 | Sep 2020 | Jul 2025 |
| MORSE-500 Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning. | Cai et al. (MORSE-500 team) | Video understanding | Everyday | Fading | Seed2.0 Pro 37.4% | 1 | 1 | Jun 2025 | Apr 2026 |
| Multi-SWE-bench mini Agents fix real GitHub issues in 8 programming languages, a 400-instance subset of Multi-SWE-bench made for cheaper runs. Scored by tests passing. | ByteDance Seed | Fixing real repositories | Expert | Dropped | Qwen3-Coder-480B-A35B 25.8% | 1 | 1 | Apr 2025 | Jul 2025 |
| MVPBench Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail. | Meta FAIR | Physical prediction | Everyday | Dropped | gemini-robotics-er-1.5-preview 72.5% | 1 | 1 | Jun 2025 | Sep 2025 |
| ODV-Bench Streaming video models answer real-time questions about autonomous driving footage. | Zeng et al. (StreamForest team) | Video understanding | Everyday | Fading | Seed2.0 Pro 72.5% | 1 | 1 | Sep 2025 | Apr 2026 |
| OpenEQA Agents answer over 1,600 open-vocabulary questions about real indoor scenes from recorded episodes or active exploration. A judge model scores answers against human answers. | Meta FAIR | Embodied reasoning | Everyday | Dropped | Gemini 1.5 Pro 57.9 | 1 | 1 | Apr 2024 | Sep 2025 |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Google DeepMind | Finding and exploiting real bugs | Expert | In use | Gemini 4 Argon 86% | 1 | 1 | Oct 2026 | |
| StrategyQA Models answer 2,780 yes or no questions that need implicit multi-step reasoning, such as whether Aristotle used a laptop. Scoring is accuracy. | Mor Geva et al. (Tel Aviv University and Allen Institute for AI) | Language reasoning | Everyday | Dropped | PaLM 2-L 90.4% | 1 | 1 | Jan 2021 | Apr 2025 |
| VideoEval-Pro Video models give open-ended short answers to questions that need the whole long video, which removes the guessing that multiple choice allows. | TIGER-Lab and collaborators (Ma et al.) | Long videos | Everyday | Fading | Seed2.0 Lite (0428) 49.5% | 1 | 1 | May 2025 | Apr 2026 |
| ViewSpatial-Bench Models answer 5,712 questions that locate objects from the camera view and from the view of people in the scene. Five task types test egocentric and allocentric reasoning. | Dingming Li, Yueting Zhuang et al. | Spatial reasoning and counting | Everyday | Fading | SenseNova-SI-1.3-InternVL3-8B 61.3% | 1 | 1 | May 2025 | Jan 2026 |
| Agent Startup Bench Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews. | ByteDance Seed | Running a business | Expert | In use | Seed2.1 Pro 68.8% | 1 | 1 | Jun 2026 | Jun 2026 |
| AgenticVBench Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results. | Cao et al. (arXiv 2605.27705) | Video generation | Expert | In use | Qwen3.8-Omni-Flash 36.8% | 1 | 1 | May 2026 | Sep 2026 |
| AgentWorldBench Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions. | Qwen Team, Alibaba Group | Interactive world models | University | In use | Qwen-AgentWorld-35B-A3B 56.4 | 1 | 1 | Jun 2026 | Jun 2026 |
| ARKitScenes Models locate objects in indoor RGB-D scenes captured on iPads and output 3D bounding boxes. Scored by 3D detection average precision. | Apple | Detection and captioning | Everyday | Dropped | Qwen3-VL-235B-A22B 53.7% | 1 | 1 | Nov 2021 | Nov 2025 |
| Audio Understanding Benchmark (Mistral) Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription. | Mistral AI | Audio understanding and voice | Expert | Dropped | Voxtral Small 86.6% | 1 | 1 | Jul 2025 | Jul 2025 |
| AyaVisionBench Vision-language models answer image questions in 23 languages across 9 task categories, 135 image-question pairs per language. A model judge compares two models' answers and reports win rates. | Cohere For AI (Cohere Labs) | General visual questions | Everyday | Dropped | Aya Vision 32B 64% | 1 | 1 | Mar 2025 | May 2025 |
| BEAM Models answer probing questions about coherent multi-session user conversations of 128K, 500K, 1M and 10M tokens. Questions test ten memory abilities such as knowledge updates, temporal reasoning and contradiction resolution. | Mohammad Tavakoli, Hamed Zamani, J Ross Mitchell and collaborators | Learning from the context | Everyday | In use | Muse Glimmer 65.1% | 1 | 1 | Oct 2025 | Oct 2026 |
| Brevity Benchmark NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method. | NVIDIA | Overall chat quality | Everyday | In use | Nemotron 3.5 Bolt E4B 82.8% | 1 | 1 | Oct 2026 | Oct 2026 |
| C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. | Tencent Hunyuan | Function calling | University | Dropped | Hunyuan-A13B-Instruct 63.5% | 1 | 1 | May 2025 | Jun 2025 |
| ChemBench Models answer more than 2,700 chemistry questions on knowledge and reasoning. Results are compared with a group of human chemists. | Lamalab, Friedrich Schiller University Jena | Science questions | Expert | Dropped | Intern-S1 83.4% | 1 | 1 | Apr 2024 | Aug 2025 |
| CodeSimpleQA Models answer short bilingual fact questions about programming concepts and implementations across languages and computer science domains. | Yang et al. | Short factual questions | University | Fading | Seed2.0 Pro 58% | 1 | 1 | Dec 2025 | Feb 2026 |
| Cohere enterprise human evaluation Trained annotators blindly compare two models' answers on enterprise tasks such as HR FAQs, technical FAQs and workplace assistants. They judge accuracy, instruction following and style. Results are win, tie and loss rates. | Cohere | Running a business | Everyday | Dropped | Cohere Command A 51.7% | 1 | 1 | Mar 2025 | Apr 2025 |
| ComplexNLU (Tencent) Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published. | Tencent Hunyuan | Reading comprehension | Everyday | Dropped | Hunyuan-A13B-Instruct 61.2% | 1 | 1 | Jun 2025 | Jun 2025 |
| CreativeWork Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls. | ByteDance Seed | Desktop apps | Everyday | In use | Seed2.1 Pro 42.5% | 1 | 1 | Jun 2026 | Jun 2026 |
| CSAT Math 2025 Models solve the mathematics sections of South Korea's 2025 College Scholastic Ability Test in Korean. The score averages the three elective subjects: calculus, statistics and geometry. | Korea Institute for Curriculum and Evaluation (exam), LG AI Research (evaluation) | Competition problems with answers | School | Dropped | EXAONE Deep 32B 94.5% | 1 | 1 | Nov 2024 | Mar 2025 |
| DeR2 Bench Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors. | ByteDance Seed and collaborators (Ying et al.) | Deep research | Expert | Fading | Seed2.0 Pro 58.2% | 1 | 1 | Jan 2026 | Feb 2026 |
| Doubao Multi-Turn Bench Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer. | ByteDance Seed | Multi-turn conversation | Everyday | In use | Seed2.1 Pro 52.5% | 1 | 1 | Jun 2026 | Jun 2026 |
| DPG-Bench Text-to-image models draw 1,000 dense prompts with many objects, attributes and relations. A VQA model checks each element of the prompt graph in the image. | Tencent (ELLA authors, Hu et al.) | Image generation and editing | Everyday | Dropped | Janus-Pro-7B 84.2% | 1 | 1 | Mar 2024 | Jan 2025 |
| DREAM-1K Video models write fine-grained descriptions of 1,000 videos from varied sources. An automatic method scores how many described events match the reference. | ByteDance Research (Tarsier team) | Video captioning | Everyday | Dropped | Seed1.5-VL 43.9% | 1 | 1 | Jul 2024 | May 2025 |
| EIFBench Models follow extremely complex instructions that bundle several tasks and many constraints in one request, as in real product workflows. | Zou, Zhang et al. | Following precise instructions | Everyday | Fading | Seed1.8 48.6% | 1 | 1 | Jun 2025 | Mar 2026 |
| Encyclo-K Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once. | ByteDance Seed and collaborators (Liang et al.) | Expert-written questions | University | Fading | Seed2.0 Pro 65.7% | 1 | 1 | Dec 2025 | Feb 2026 |
| GaRAGe Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages. | Amazon (amazon-science) | Faithful to sources | Everyday | In use | Nemotron 3.5 Bolt E4B 40.7% | 1 | 1 | Jun 2025 | Oct 2026 |
| GEdit-Bench Image editing models follow real-world user edit instructions. A model judge scores semantic consistency, perceptual quality and an overall score. | StepFun (Step1X-Edit team) | Image generation and editing | Everyday | Dropped | BAGEL-7B-MoT 6.52 | 1 | 1 | Apr 2025 | May 2025 |
| HealthBench-Hallu Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate. | Baichuan AI | Hallucination rates | Expert | Fading |  Baichuan-M3-235B 3.50% | 1 | 1 | Jan 2026 | Feb 2026 |
| Human proximity (Google DeepMind robotics) Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | Gemini Robotics ER 2 93% | 1 | 1 | Jul 2026 | Jul 2026 |
| HumMusQA Models answer 320 hand-written questions about music, curated and checked by people with musical training. | Music Technology Group, Universitat Pompeu Fabra | Audio understanding and voice | Expert | In use | Qwen3.8-Omni-Flash 75.8% | 1 | 1 | Mar 2026 | Sep 2026 |
| Image2FloorPlan Models study several real photos of a home and draw its floor plan. Scored by an average quality score. | ByteDance Seed | Spatial reasoning and counting | Everyday | In use | Seed2.1 Pro 48.0 | 1 | 1 | Jun 2026 | Jun 2026 |
| Information Processing (ByteDance Seed in-house) Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment. | ByteDance Seed | Documents and spreadsheets | Everyday | Fading | Seed1.8 53.9% | 1 | 1 | Dec 2025 | Mar 2026 |
| InsCtrl (Tencent) Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published. | Tencent Hunyuan | Following precise instructions | Everyday | Dropped | Hunyuan-A13B-Instruct 71.9% | 1 | 1 | Jun 2025 | Jun 2025 |
| IntelligentBench Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100. | ByteDance Seed (BAGEL team) | Video generation | Everyday | Dropped | BAGEL-7B-MoT 55.3 | 1 | 1 | May 2025 | May 2025 |
| Intention Recognition (ByteDance Seed in-house) Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats. | ByteDance Seed | Running a business | Everyday | Fading | Seed1.8 63.2% | 1 | 1 | Dec 2025 | Mar 2026 |
| IPhO 2025 Models solve the theory problems of the 2025 International Physics Olympiad. Answers are graded against the official marking scheme. | International Physics Olympiad | Science olympiads | Competition | Fading | Seed2.1 Pro 79.3% | 1 | 1 | Jul 2025 | Jun 2026 |
| K-Viscuit Models answer multiple-choice visual questions about Korean culture. Questions were written through human and VLM collaboration. | Seoul National University and collaborators (ddehun et al.) | General visual questions | Everyday | Fading | EXAONE 4.5 33B 80.1% | 1 | 1 | Jun 2024 | Apr 2026 |
| KINA Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key. | 2077AI (Jin et al.) | Expert-written questions | Expert | In use | Seed2.1 Pro 48.3% | 1 | 1 | Jun 2026 | Jun 2026 |
| KMMLU-Redux Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed. | LG AI Research and collaborators | Exam questions | University | Dropped | EXAONE 4.0 32B 72.7% | 1 | 1 | Jul 2025 | Jul 2025 |
| KMMMU Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images. | HAE-RAE | General visual questions | University | Fading | EXAONE 4.5 33B 42.7% | 1 | 1 | Apr 2026 | Apr 2026 |
| LeetCode Weekly Contest Models solve 126 LeetCode contest problems from July to November 2023. A solution counts only if it passes all of its 20+ test cases. | DeepSeek | Competitive programming | University | Dropped | DeepSeek LLM 67B 17.5% | 1 | 1 | Nov 2023 | Jan 2024 |
| LengthCtrl (Tencent) Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published. | Tencent Hunyuan | Creative writing | Everyday | Dropped | Hunyuan-A13B-Instruct 55.4% | 1 | 1 | Jun 2025 | Jun 2025 |
| LongAudioSpan Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended. | Huang et al. (arXiv 2608.26431, with Qwen authors) | Audio understanding and voice | Everyday | In use | Qwen3.8-Omni-Flash 82.7% | 1 | 1 | Aug 2026 | Sep 2026 |
| LVIS-MG Models ground many objects at once by outputting boxes for named categories in LVIS images. | ByteDance Seed | Detection and captioning | Everyday | Dropped | Seed1.5-VL 72.5% | 1 | 1 | May 2025 | May 2025 |
| LVOmniBench Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track. | Tao et al. (arXiv 2603.19217) | Long videos | Everyday | In use | Qwen3.8-Omni-Flash 63.3% | 1 | 1 | Mar 2026 | Sep 2026 |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Cohere | Customer service | Everyday | Dropped | Command A Reasoning 46.5% | 1 | 1 | Aug 2025 | Aug 2025 |
| m-WildVision Vision-language models answer WildVision bench prompts machine-translated into 23 languages. A model judge compares two models' answers and reports win rates. | Cohere For AI (Cohere Labs) | General visual questions | Everyday | Dropped | Aya Vision 32B 72% | 1 | 1 | Mar 2025 | May 2025 |
| Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. | Tencent Hunyuan | Translation | Everyday | Dropped | Hunyuan-MT-7B 60.8% | 1 | 1 | Sep 2025 | Sep 2025 |
| MathCanvas-Bench Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry. | Shi et al. (MathCanvas team) | Visual math and diagrams | School | Fading | Seed2.0 Pro 61.9% | 1 | 1 | Oct 2025 | Feb 2026 |
| MAVERIX Models answer 2,556 questions about 700 videos that need both the video and its audio. Questions come in multiple-choice and open-ended forms. | Carnegie Mellon University (CUBE lab) and collaborators | Video understanding | Everyday | Fading | Nova 2 Omni Preview 66.6% | 1 | 1 | Mar 2025 | Dec 2025 |
| MeasureBench Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key. | BAAI FlagEval team | General visual questions | Everyday | In use | Seed2.1 Pro 62.9% | 1 | 1 | Oct 2025 | Jun 2026 |
| MiMo Code Bench Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details. | Xiaomi MiMo | Fixing real repositories | Expert | In use | MiMo-V2.6 Pro 63.2% | 1 | 1 | Sep 2026 | Sep 2026 |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Xiaomi MiMo | Finding and exploiting real bugs | Expert | In use | MiMo-V2.6 Pro 80.2% | 1 | 1 | Sep 2026 | Sep 2026 |
| MiMo Visual Coding Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code. | Xiaomi MiMo | Web and app front ends | Expert | In use | MiMo-V2.6 Pro 72.3% | 1 | 1 | Sep 2026 | Sep 2026 |
| MineDojo Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. | NVIDIA, Caltech and collaborators (Fan et al.) | Video games | Everyday | Fading | Seed2.0 Pro 49% | 1 | 1 | Jun 2022 | Feb 2026 |
| MME-VideoOCR Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning. | MME team (Shi et al., arXiv 2505.21333) | Video understanding | Everyday | Fading | Qwen3.5-Omni-Plus 77% | 1 | 1 | May 2025 | Apr 2026 |
| MobileMiniWoB++ Agents complete MiniWoB++ web tasks inside the Android emulator of AndroidWorld. Scored by task success rate. | Google Research (AndroidWorld team) | Phones | Everyday | Dropped | Qwen2.5-VL-72B 68% | 1 | 1 | May 2024 | Feb 2025 |
| MolmoSpaces Zero-shot Franka (DROID) policies in held-out procedurally generated MuJoCo houses: pick, place, open, close. ~1,000 episodes per task. | Ai2 | Simulated robots | Everyday | Fading | MolmoAct 2 37.7% | 1 | 1 | Feb 2026 | May 2026 |
| MSQA Models answer culture-specific knowledge questions in 11 major languages. | ByteDance Seed | Knowledge in many languages | University | In use | Seed2.1 Pro 50.2% | 1 | 1 | Jun 2026 | Jun 2026 |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Qwen Team, Alibaba Group | Chat and instructions in many languages | Everyday | Dropped | Qwen2-72B 3.93 | 1 | 1 | Jun 2024 | Jul 2024 |
| MusTBench Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items. | Kwon et al. (arXiv 2605.29300) | Audio understanding and voice | Expert | In use | Qwen3.8-Omni-Flash 50.6% | 1 | 1 | May 2026 | Sep 2026 |
| NExT-QA Models answer causal, temporal and descriptive questions about short videos of daily life. Labs usually report the multiple choice split. | National University of Singapore (Junbin Xiao et al.) | Video understanding | Everyday | Dropped | LongCat-Flash-Omni 86.2% | 1 | 1 | May 2021 | Oct 2025 |
| OIBench Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions. | AGI-Eval (Meituan) | Competitive programming | Competition | Fading | LongCat-Flash-Thinking-2601 47.7% | 1 | 1 | Jun 2025 | Jan 2026 |
| OmniCap-IF Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance. | Nanjing University (NJU-LINK Lab) and others | Video captioning | Everyday | In use | Qwen3.8-Omni-Flash 80.6% | 1 | 1 | Jun 2026 | Sep 2026 |
| OmniMMI Omni models answer 2,290 questions on 1,121 streaming videos, including proactive tasks where they must decide when to respond. | Wang et al. (OmniMMI team) | Video understanding | Everyday | Fading | Seed1.8 53% | 1 | 1 | Mar 2025 | Mar 2026 |
| PHYSICS Models solve 1,297 PhD qualifying exam physics problems across six subfields, 298 of them with images. Answers are checked by symbolic equivalence. | Yale University (Kaiyue Feng, Arman Cohan et al.) | Science questions | Expert | Dropped | Intern-S1 44% | 1 | 1 | Mar 2025 | Aug 2025 |
| ProcBench Models follow explicit step-by-step procedures to answer questions, which isolates multi-step inference from knowledge and search. | Fujisawa et al. | Abstract puzzles and logic | Everyday | Fading | Seed2.0 Pro 96.6% | 1 | 1 | Oct 2024 | Feb 2026 |
| ProteinLMBench Models answer 944 hand-checked multiple choice questions on protein sequence, structure and function. | Johns Hopkins University, University of Cambridge, Shanghai AI Laboratory and others (Yiqing Shen et al.) | Genomics and life sciences | Expert | Dropped | Intern-S1 63.1% | 1 | 1 | Jun 2024 | Aug 2025 |
| ProverBench (DeepSeek) Provers write Lean 4 proofs for 325 formalized problems: 15 from AIME 2024 and 2025 and 310 from textbooks across number theory, algebra, calculus and analysis. Lean checks each proof. | DeepSeek | Formal proofs | University | Dropped | DeepSeek-Prover-V2-671B 59.1% | 1 | 1 | Apr 2025 | Apr 2025 |
| RAGTruth A word-level annotated corpus of hallucinations in retrieval-augmented generation answers. It is used to train and evaluate hallucination detection and RAG faithfulness. | ParticleMedia (NewsBreak) | Hallucination rates | Everyday | In use | Nemotron 3.5 Bolt E4B 77.9% | 1 | 1 | Jan 2024 | Oct 2026 |
| Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. | Meituan LongCat Team | Apps and MCP servers | University | Fading | LongCat-Flash-Thinking-2601 35.8% | 1 | 1 | Jan 2026 | Jan 2026 |
| RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. | X-LANCE (Shanghai Jiao Tong University) | Function calling | Everyday | In use | Nemotron 3.5 Bolt E4B 69.3% | 1 | 1 | Dec 2024 | Oct 2026 |
| RoboEval Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success. | University of Washington and collaborators (Wang et al.) | Simulated robots | Everyday | Fading | MolmoAct 2 0.44 | 1 | 1 | Jul 2025 | May 2026 |
| Robust IF Bench Models follow a diverse set of hard instructions from Microsoft AI's internal benchmark. | Microsoft AI | Following precise instructions | Everyday | In use | MAI-Code-1-Flash 61.2% | 1 | 1 | Jun 2026 | Jun 2026 |
| ROK-FORTRESS Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal. | Scale AI | Dangerous capabilities | Expert | In use | K-EXAONE 2.0 89.5% | 1 | 1 | May 2026 | Aug 2026 |
| RoleBench Models answer prompts while playing one of many fixed characters from films and books. Answers are compared with reference role answers by automatic text overlap. | RoleLLM team (InteractiveNLP-Team, Wang et al.) | Overall chat quality | Everyday | In use | Nemotron 3.5 Bolt E4B 71.2 | 1 | 1 | Oct 2023 | Oct 2026 |
| RoleMRC Models play given roles in multi-turn chats, passage reading tasks and nested or prioritized instructions. The test split has 1.4k samples. | RoleMRC team (Lu et al., ACL 2025) | Following precise instructions | Everyday | In use | Nemotron 3.5 Bolt E4B 79.1% | 1 | 1 | Feb 2025 | Oct 2026 |
| ScanBench Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage. | Baichuan AI | Medicine | Expert | Fading | Baichuan-M3-235B 73.8% | 1 | 1 | Jan 2026 | Feb 2026 |
| SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. | Alibaba Group (with Tsinghua, Zhejiang, Fudan, and Peking universities) | Secure code and review | University | Fading | Qwen3.5 397B-A17B 68.3% | 1 | 1 | Jul 2025 | Feb 2026 |
| SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. | ByteDance Seed | General assistants | Everyday | In use | Seed2.1 Pro 66.6% | 1 | 1 | Jun 2026 | Jun 2026 |
| SGP-Bench Models read symbolic graphics programs such as SVG and CAD code and answer questions about what the rendered image would show, without seeing the image. | Max Planck Institute for Intelligent Systems and others (Zeju Qiu, Weiyang Liu et al.) | General visual questions | University | Dropped | InternVL3.5-241B-A28B 70.7% | 1 | 1 | Aug 2024 | Aug 2025 |
| SpaCE-10 Models answer more than 5,000 questions about indoor 3D scenes that test 10 atomic and 8 compositional spatial abilities. | SpaCE-10 team (Ziyang Gong et al.) | Spatial reasoning and counting | Everyday | Dropped | InternVL3.5-241B-A28B 55% | 1 | 1 | Jun 2025 | Aug 2025 |
| SpeechRole Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech. | Jiang et al. (arXiv 2508.02013) | Speech recognition and translation | Everyday | Fading | Qwen3.5-Omni-Plus 123.5 | 1 | 1 | Aug 2025 | Apr 2026 |
| SpotSoundBench Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets. | Sun et al. (SpotSound, arXiv 2604.13023) | Audio understanding and voice | Everyday | In use | Qwen3.8-Omni-Flash 67.2% | 1 | 1 | Apr 2026 | Sep 2026 |
| StreamBench (StreamChat) Video models answer questions on streaming videos across media types, including multi-turn interaction and reasoning tasks. | Xiong et al. (StreamChat team) | Video understanding | Everyday | Dropped | Seed1.5-VL 72.8% | 1 | 1 | Jan 2025 | May 2025 |
| Sudoku-Bench Models solve Sudoku variants with unusual rule sets that need creative, multi-step deduction. Scored by the share of puzzles solved. | Sakana AI | Abstract puzzles and logic | Competition | Dropped | LongCat-Flash-Thinking 56% | 1 | 1 | May 2025 | Sep 2025 |
| SWE-EVO Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks. | Le, Thai et al. | Fixing real repositories | Expert | Fading | Seed2.0 Pro 8.50% | 1 | 1 | Dec 2025 | Feb 2026 |
| TACoS Models find the time span in a cooking video that matches a sentence. Labs score temporal grounding by mean IoU with the annotated span. | Max Planck Institute for Informatics (Rohrbach et al.) | Video understanding | Everyday | Dropped | Seed1.5-VL 49.6% | 1 | 1 | Sep 2014 | May 2025 |
| tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Meituan LongCat Team | Customer service | Everyday | Fading | LongCat-Flash-Thinking-2601 67.1% | 1 | 1 | Jan 2026 | Jan 2026 |
| TemporalBench Video models answer about 10K questions on fine-grained temporal dynamics, built from about 2K detailed human annotations of video clips. | Cai et al. (TemporalBench team) | Video understanding | Everyday | Dropped | Seed1.5-VL 79.8% | 1 | 1 | Oct 2024 | May 2025 |
| U-Artifacts Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need. | ByteDance Seed | Web and app front ends | Everyday | Fading | Seed1.8 49.2% | 1 | 1 | Dec 2025 | Mar 2026 |
| UNO-Bench Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model. | Meituan | Audio understanding and voice | Everyday | Dropped | LongCat-Flash-Omni 49.9% | 1 | 1 | Oct 2025 | Oct 2025 |
| URO-Bench Spoken dialogue models answer speech with speech across Understanding, Reasoning and Oral conversation. It has a basic track of 16 datasets and a pro track of 20. | Yan et al. (arXiv 2502.17810) | Speech recognition and translation | Everyday | Fading | Qwen3.5-Omni-Plus 84.1% | 1 | 1 | Feb 2025 | Apr 2026 |
| VCR-Bench Video models answer 1,034 questions on 859 videos, and graders score both the final answer and each tagged chain-of-thought step against human rationales. | Qi et al. (VCR-Bench team) | Video understanding | Everyday | Fading | Seed1.8 59.8% | 1 | 1 | Apr 2025 | Mar 2026 |
| VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Meituan LongCat Team | Apps and MCP servers | Everyday | Fading | LongCat-Flash-Thinking-2601 20.5% | 1 | 1 | Jan 2026 | Jan 2026 |
| When2Call Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets. | NVIDIA | Function calling | Everyday | In use | Nemotron 3.5 Bolt E4B 60.5% | 1 | 1 | Apr 2025 | Oct 2026 |
| WMT26 v2026 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. | WMT (Conference on Machine Translation) | Translation | University | In use | North Small Translate 1.0 83.6 | 1 | 1 | Jun 2026 | Sep 2026 |
| WMT25 v2025 Replaced in Jun 2026 | System cards only | North Small Translate 1.0 48.9 | 3 | Jun 2025 | Sep 2026 | ||||
| WMT24 v2024 Replaced in Jun 2025 | Not in cards | Jun 2024 | |||||||
| WMT23 v2023 Replaced in Jun 2024 | Dropped | Gemini 1.5 Pro 75.3 | 3 | Jul 2023 | May 2024 | ||||
| Word-Task (Tencent) Models solve Tencent in-house word level language tasks. Task and scoring details are not published. | Tencent Hunyuan | Reading comprehension | Everyday | Dropped | Hunyuan-A13B-Instruct 62.9% | 1 | 1 | Jun 2025 | Jun 2025 |
| xDailyBench Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness. | ByteDance Seed | Overall chat quality | Everyday | In use | Seed2.1 Pro 61% | 1 | 1 | Jun 2026 | Jun 2026 |
| TimeLens-Bench Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU. | Nanjing University, ARC Lab (Tencent PCG) and Shanghai AI Lab | Video understanding | Everyday | Fading | 1 | 1 | Dec 2025 | Jun 2026 | |
| CCBench (MMBench Chinese Culture) Vision language models answer 510 Chinese-language multiple-choice questions about images of Chinese culture, such as calligraphy, painting, food and historical figures. | Shanghai AI Laboratory (OpenCompass, MMBench team) | General visual questions | Everyday | Dropped | InternVL2.5-78B-MPO 86.9% | 1 | Oct 2023 | Jun 2025 | |
| ForecastBench Models give probabilities for 500 auto-generated questions per round about future events, drawn from prediction markets and time-series datasets. Forecasts are scored with difficulty-adjusted Brier scores once the questions resolve. | Forecasting Research Institute | Real-world events | Expert | In use | Claude 3 Haiku 44.7 | 1 | Sep 2024 | Sep 2026 | |
| LiveSQLBench Contamination-resistant text-to-SQL on PostgreSQL with hierarchical knowledge bases, including CRUD tasks. Scored by soft execution match and test cases. | BIRD team (HKU) | Data science and SQL | University | In use | MiniMax-M3 40.2% | 1 | May 2025 | Jun 2026 | |
| Vals Index v2.2 Agents are scored on eight finance, coding, legal, and tax benchmarks, combined with weights from each sector's share of US GDP. | Vals AI | Composite work indices | In use | Claude Sonnet 5.5 70.8% | 1 | Oct 2026 | Oct 2026 | ||
| Vals Index v2.1 Replaced in Oct 2026 | In use | Gemini 4 Argon 68.9% | 1 | 1 | Sep 2026 | Oct 2026 | |||
| Vals Index v2 Replaced in Sep 2026 | Not in cards | Aug 2026 | |||||||
| Vals Index v1 Replaced in Aug 2026 | System cards only | Kimi K3 74.7% | 1 | Oct 2025 | Jul 2026 | ||||
| FrontierCode 2.0 Agents resolve 132 maintainer-built tasks from deliberately ambiguous prompts. Rubric checks ask whether a maintainer would merge the change. | Cognition | Fixing real repositories | Expert | In use | Claude Opus 5.5 45.5% | 1 | Oct 2026 | Oct 2026 | |
| FrontierCode 1.1 Replaced in Oct 2026 | In use | Claude Fable 5 53.5% | 9 | 7 | Jul 2026 | Oct 2026 | |||
| FrontierCode 1.0 Replaced in Jul 2026 | In use | Claude Opus 4.8 34.3% | 4 | 2 | Jun 2026 | Oct 2026 | |||
| LAMBADA Models predict the last word of narrative passages where humans can guess it only from the whole passage. Scored by accuracy. | University of Trento and University of Amsterdam | Reading comprehension | Everyday | Dropped | Megatron-Turing NLG 530B 87.2% | 1 | Jun 2016 | Nov 2023 | |
| FORTE Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes. | AGI-Eval (Meituan) | Documents and spreadsheets | University | In use | GPT-5.5 77.8% | 1 | Jun 2026 | Jul 2026 | |
| VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. | Meituan LongCat Team | Customer service | Everyday | In use | Claude Opus 4.6 50.3% | 1 | May 2026 | Oct 2026 | |
| VITA-Bench 1.0 Replaced in May 2026 | Meituan LongCat team | Fading | Gemini 3 Flash 32.5% | 12 | 5 | Sep 2025 | Jun 2026 | ||
| Prophet Arena Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return. | UChicago SIGMA Lab with Kalshi | Real-world events | Expert | In use | Claude Opus 5.5 86.0 | 1 | Oct 2025 | Sep 2026 | |
| HLE-Diamond Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers. | Center for AI Safety and Scale AI | Expert-written questions | Research frontier | In use | GPT-6 Astra 65.8% | 1 | Sep 2026 | Oct 2026 | |
| HLE-Rolling Replaced in Sep 2026 | Not in cards | Oct 2025 | |||||||
| Humanity's Last Exam Original Replaced in Oct 2025 | Center for AI Safety + Scale AI | Expert | In use | GPT-6 Astra 54.8% | 135 | 71 | Jan 2025 | Oct 2026 | |
| PostTrainBench 1.2 CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. | AISA group (ELLIS Institute Tübingen and MPI-IS) + University of Tübingen + Thoughtful Lab | Machine learning engineering | Research frontier | In use | Claude Fable 5.1 44.6% | 1 | Sep 2026 | Oct 2026 | |
| PostTrainBench 1.1 Replaced in Sep 2026 | In use | Claude Opus 5 45.6% | 2 | Jul 2026 | Oct 2026 | ||||
| PostTrainBench 1.0 Replaced in Jul 2026 | In use | GLM-5.2 34.3% | 5 | 11 | Mar 2026 | Oct 2026 | |||
| SWE-fficiency 498 tasks in nine Python data/ML/HPC repos: agent must speed up a real workload without breaking tests. Scored as speedup relative to expert patch. | SWE-fficiency authors (ICML 2026) | Speed and optimization | Expert | In use | Claude Opus 4.5 22.5% | 1 | Nov 2025 | Jun 2026 | |
| MADQA Agents search 800 PDF documents to answer 2,250 human-written questions. Scoring covers answer accuracy and how much search effort the agent spends. | Łukasz Borchmann et al. (Snowflake, University of Oxford, and partners) | Documents, charts and text in images | Everyday | In use | Gemini 3 Pro 85.2% | 1 | Mar 2026 | Aug 2026 | |
| SWE-Bench Pro V2 Hard Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol. | Scale AI (co-developed with Reflection) | Fixing real repositories | Research frontier | In use | Claude Opus 5 98% | 1 | Sep 2026 | Oct 2026 | |
| TheAgentCompany Agents do work tasks in a simulated software company, using a browser, code, and chat with simulated coworkers. Scored by full and partial checkpoint completion. | Carnegie Mellon University et al. | Simulated companies | University | Dropped | DeepSeek-V3.2-Exp 42.9% | 1 | Dec 2024 | Oct 2025 | |
| SWE-bench-Live Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch. | Microsoft | Fixing real repositories | Expert | Dropped | Claude Opus 5.5 71.7% | 1 | May 2025 | Sep 2026 | |
| SWE-sweep Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved. | Meta Superintelligence Labs (with Harvard, UW and Stanford) | Fixing real repositories | Research frontier | In use | GPT-6 Astra 9.50% | 1 | Oct 2026 | Oct 2026 | |
| Harbor-Index Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. | Harbor and Laude Institute (Terminal-Bench team) | Terminal tasks | Research frontier | In use | GPT-5.5 28.1% | 1 | Jun 2026 | Aug 2026 | |
| SWT-bench Agents write tests that reproduce real GitHub issues from SWE-bench repositories. Scored by the share of tests that fail before and pass after the fix. | ETH Zurich and LogicStar.ai | Fixing real repositories | Expert | Fading | GPT-5 mini 56.2% | 1 | Jun 2024 | Dec 2025 | |
| SWE Refactor Bench Agents migrate 20 whole open-source repositories to a new language, framework, platform, or build system. Graded by migration audit, behavioral tests, and adversarial verifier models. | Einsia | Building whole projects | Research frontier | In use | Claude Opus 5 32.3% | 1 | Aug 2026 | Oct 2026 | |
| BrokenArXiv Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false. | MathArena (ETH Zurich SRI Lab and INSAIT) | Research mathematics | Research frontier | In use | GPT-6.1 Sol 89.7% | 1 | Mar 2026 | Oct 2026 | |
| VideoDR (with search) Agents watch a video, gather clues across frames, and search the web to answer multi-hop questions. Scored by answer accuracy. | QuantaAlpha | Video understanding | University | In use | Gemini 3 Pro 76% | 1 | Jan 2026 | Aug 2026 | |
| L-Eval Models answer over 2,000 queries on 508 long documents of 3k to 200k tokens across 20 subtasks. Scoring uses exact match and judge-model ratings. | OpenLMLab (Fudan University) | Reasoning over long documents | University | Dropped | Claude 1.3 76.4% | 1 | Jul 2023 | Feb 2024 | |
| miniCTX-v2 Provers prove Lean theorems from real projects using long repository context, with theorems taken after a November 28, 2024 cutoff. | CMU L3 Lab (Hu et al.) | Formal proofs | Expert | Dropped | Seed-Prover 81.8% | 1 | Mar 2025 | Sep 2025 | |
| RoboLab Zero-shot DROID Franka policies in Isaac Lab on 120 tabletop tasks at three instruction levels, 10 episodes each. Success rate. | NVIDIA | Simulated robots | Everyday | In use | GR00T N1.7 8.58% | 1 | Apr 2026 | Jul 2026 | |
| CB Models label whether a text commits its speaker to an embedded clause as entailment, contradiction, or neutral. Scored by accuracy and F1. | Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser | Language reasoning | Everyday | Dropped | PaLM 2-L 87.5% | 1 | Apr 2019 | May 2023 | |
| WSC Models decide whether a pronoun refers to a given noun phrase in sentences that need commonsense to resolve. Scored by accuracy. | NYU, Facebook AI, University of Washington, and DeepMind (SuperGLUE) | Commonsense | Everyday | Dropped | PaLM 2-L 86.9% | 1 | May 2019 | May 2023 | |
| RTE Models decide whether a premise sentence entails a hypothesis, using data from the PASCAL RTE challenges. Scored by accuracy. | PASCAL RTE Challenge organizers, packaged in GLUE | Language reasoning | Everyday | Dropped | PaLM 2-L 79.3% | 1 | Apr 2018 | May 2023 | |
| 3DSRBench Models answer questions about 3D spatial relations in images, such as height, location, orientation and relations between objects. | Wufei Ma, Alan Yuille et al. | Spatial reasoning and counting | Everyday | Fading | SenseNova-SI-1.3-InternVL3-8B 62.4% | 1 | Dec 2024 | Jan 2026 | |
| AdvancedMathBench Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof. | Shanghai AI Lab (InternLM) | Written proofs | Expert | In use | Intern-S2-397B 58.1% | 1 | Jul 2026 | Sep 2026 | |
| Agents' Last Exam v1.1 Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references. | UC Berkeley RDI (board hosted by Snorkel AI) | Tasks from real jobs | Expert | In use | Gemini 4 Carbon 77.4% | 1 | Oct 2026 | Oct 2026 | |
| Agents' Last Exam 1.0 Replaced in Oct 2026 | UC Berkeley RDI + 300+ experts | In use | Gemini 4 Carbon 45.9% | 26 | 6 | Jun 2026 | Oct 2026 | ||
| Alyah Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. | Technology Innovation Institute (TII) and community contributors | Knowledge in many languages | Everyday | In use | Falcon-Emirati-7B 84.8% | 1 | Jan 2026 | Oct 2026 | |
| Android in the Zoo (AitZ) Models predict the next action on Android app screens, given the goal and past steps. Scored by exact match of the predicted action on the test split. | Zhang et al. (Fudan University and others) | Phones | Everyday | Dropped | Qwen2.5-VL-72B 83.2% | 1 | Mar 2024 | Jan 2025 | |
| Answer Factuality (Perplexity) Models answer questions from search results. Graders check that answers use facts grounded in the results and resolve conflicting or missing information. | Perplexity | Faithful to sources | Everyday | Dropped |  sonar 85.1% | 1 | Feb 2025 | Feb 2025 | |
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Artificial Analysis and the Cyber Index Alliance | Finding and exploiting real bugs | Expert | In use | Mistral Large 4 (Le Chonk, open weights) 51% | 1 | Sep 2026 | Oct 2026 | |
| Artificial Analysis Intelligence Index v5 5.0 Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. | Artificial Analysis | Composite indices | In use | Gemini 4 Carbon 59.0 | 1 | Oct 2026 | Oct 2026 | ||
| Artificial Analysis Intelligence Index v4.3 Replaced in Oct 2026 | In use | Muse Wonder Full 56.0 | 4 | 2 | Sep 2026 | Oct 2026 | |||
| Artificial Analysis Intelligence Index v4.2 Replaced in Sep 2026 | Not in cards | Sep 2026 | |||||||
| Artificial Analysis Intelligence Index v4.1 Replaced in Sep 2026 | In use | Grok 4.6 61.0 | 6 | 3 | Jun 2026 | Aug 2026 | |||
| Artificial Analysis Intelligence Index v4.0 Replaced in Jun 2026 | Fading | GPT-5.5 60.2 | 3 | 3 | Jan 2026 | May 2026 | |||
| Artificial Analysis Intelligence Index v3 Replaced in Jan 2026 | Dropped | MiniMax-M2 61.0 | 2 | Sep 2025 | Oct 2025 | ||||
| Artificial Analysis Intelligence Index v2.2 Replaced in Sep 2025 | Not in cards | Aug 2025 | |||||||
| Artificial Analysis Intelligence Index v2 Replaced in Aug 2025 | Dropped | GLM-4.5 67.7 | 1 | 2 | Feb 2025 | Aug 2025 | |||
| Artificial Analysis Intelligence Index v1 Replaced in Feb 2025 | Not in cards | Jan 2024 | |||||||
| AVMeme Exam Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge. | Jiang et al. (AVMeme Exam team) | Audio understanding and voice | Everyday | Fading | Seed2.0 Lite (0428) 69.5% | 1 | Jan 2026 | Apr 2026 | |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Google (Big Sleep team) | Finding and exploiting real bugs | Expert | In use | Gemini 3.5 Flash Cyber 72% | 1 | Jul 2026 | Jul 2026 | |
| Business Operation (Moonshot) Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score. | Moonshot AI | Running a business | Expert | In use | Kimi K3.1 Pro 37.8% | 1 | Oct 2026 | Oct 2026 | |
| CC-OCR-Bench-V2 Models process 7,093 hard real-world document samples across five OCR tracks, including parsing and document QA. Each track uses its own metric. | Chunyi Peng et al. (with Qwen team members) | Documents, charts and text in images | Everyday | In use | Qwen 3.8 Max 79.6% | 1 | May 2026 | Aug 2026 | |
| CC-OCR v1 Replaced in May 2026 | Alibaba Tongyi Lab (OCR team) | Fading | Qwen3.5-Omni-Plus 83.4% | 8 | 4 | Dec 2024 | Apr 2026 | ||
| CELLO Models follow real-world complex instructions with multiple tasks, noisy inputs and format or length constraints. Rule-based checks score each answer against the instruction features. | Fudan University (He et al.) | Following precise instructions | Everyday | Dropped | Hunyuan-T1 76.4% | 1 | Sep 2023 | Mar 2025 | |
| ChemicalDiagramQA Models answer questions about chemical structure diagrams in scientific figures. Scored by accuracy. | Gemini Team, Google | Documents, charts and text in images | University | Dropped | Gemini 1.5 Pro 69.7% | 1 | Jun 2024 | May 2024 | |
| ChineseQA Models answer Chinese trivia questions in a closed-book format modeled on TriviaQA. | DeepSeek | Trivia and open questions | Everyday | Dropped | DeepSeek LLM 67B 85.1% | 1 | Nov 2023 | Nov 2023 | |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Google (Chrome Security) | Finding and exploiting real bugs | Expert | In use | Gemini 3.5 Flash Cyber 72% | 1 | Jul 2026 | Jul 2026 | |
| CMRC Models extract answer spans from Chinese Wikipedia paragraphs for about 20,000 human-written questions. Scoring is exact match and F1. | Joint Laboratory of HIT and iFLYTEK Research (HFL) | Reading in many languages | Everyday | Dropped | DeepSeek-V3 76.3% | 1 | Oct 2018 | Dec 2024 | |
| COCO Models detect and locate objects in everyday photos from the Microsoft COCO dataset. Scored by mean average precision over box overlap thresholds. | Microsoft and academic partners | Detection and captioning | Everyday | In use | Qwen 3.8 Max 78.7% | 1 | May 2014 | Aug 2026 | |
| Code refactoring tasks (OpenAI) Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea. | OpenAI | Fixing real repositories | Expert | Dropped | GPT-5-Codex 51.3% | 1 | Sep 2025 | Sep 2025 | |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | OpenAI | Function calling | Everyday | Dropped | gpt-realtime 66.5% | 1 | Aug 2025 | Aug 2025 | |
| COPA Models choose which of two alternatives is the more plausible cause or effect of a premise, across 1,000 questions. Scored by accuracy. | University of Southern California (ICT) | Commonsense | Everyday | Dropped | PaLM 2-L 96% | 1 | Mar 2011 | May 2023 | |
| CyberGym (final-submission metric) final-submission (2026-07) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | UC Berkeley (Sunblaze) | Finding and exploiting real bugs | Expert | In use | MAI-Cyber-1-Flash 86.3% | 1 | Jul 2026 | Jul 2026 | |
| CyberGym any-of (2025) Replaced in Jul 2026 | In use | DeepSeek V4.1 Flash 99.4% | 16 | 23 | Jun 2025 | Oct 2026 | |||
| DECK-Arena Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores. | Moonshot AI | Documents and spreadsheets | University | In use | Kimi K3.1 Pro 1,456 | 1 | Oct 2026 | Oct 2026 | |
| Design Bench (Moonshot) Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores. | Moonshot AI | Taste and style | University | In use | Kimi K3.1 Pro 957 | 1 | Oct 2026 | Oct 2026 | |
| DS-Arena-Code Models answer common coding requests in an internal subjective test. GPT-4o judges responses against competitor models to give a win rate. | DeepSeek-AI | Function-level problems | University | Dropped | DeepSeek-V2.5 63.1% | 1 | Sep 2024 | Sep 2024 | |
| DS-FIM-Eval Models fill in missing code between a given prefix and suffix, as in editor plugin completion. | DeepSeek | Function-level problems | University | Dropped | DeepSeek-V2.5 78.3% | 1 | Sep 2024 | Sep 2024 | |
| DSBench Data science agents solve 466 data analysis tasks and 74 end-to-end data modeling tasks drawn from Eloquence and Kaggle competitions. Analysis is scored by accuracy and modeling by relative performance gap. | Tencent AI Lab and University of Texas at Dallas (Jing et al.) | Data science and SQL | Expert | Dropped | ChatGPT agent 89.9% | 1 | Sep 2024 | Jul 2025 | |
| E-Bench-Code (Tencent) Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published. | Tencent Hy team | Documents and spreadsheets | Expert | In use | Tencent Hy4 preview 79% | 1 | Aug 2026 | Aug 2026 | |
| Ego3D-Bench Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error. | Authors of arXiv 2509.06266 | Spatial reasoning and counting | Everyday | In use | Qwen3.7-Plus 5.90 | 1 | Sep 2025 | Jun 2026 | |
| EgoLife (with memory) Models answer life-assistant questions about a week of egocentric video from six people, such as recalling past events and habits. | EgoLife team (Jingkang Yang, Ziwei Liu et al.) | Long videos | Everyday | In use | Qwen 3.8 Max 80.3% | 1 | Mar 2025 | Aug 2026 | |
| Finance Modeling Agents complete expert-written financial modeling problems with Excel tools, from research to analysis. Expert-designed rubrics score each problem, averaged over three runs. | MiniMax | Finance | Expert | Fading | MiniMax-M2.5 21.6% | 1 | Feb 2026 | Feb 2026 | |
| Finch (FinWorkBench) Agent completes 172 enterprise finance and accounting spreadsheet workflows (384 tasks) from real data. Human plus calibrated LLM-judge grading. | FinWorkBench (Dong et al.) | Finance | Expert | In use | Mistral Large 4 (Le Chonk, open weights) 73.3% | 1 | Dec 2025 | Oct 2026 | |
| FinStepBench Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations. | StepFun | Finance | Expert | In use | Step 5 Preview 63.6% | 1 | Sep 2026 | Sep 2026 | |
| Functional MATH Models solve new instances of MATH problems that code generates for each snapshot. The gap from static MATH accuracy measures memorization. | Consequent AI | Competition problems with answers | Competition | Dropped | Gemini 1.5 Pro 64.6% | 1 | Feb 2024 | May 2024 | |
| FutureOmni Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together. | OpenMOSS (Fudan University) and collaborators | Video understanding | Everyday | Fading | MiMo-V2-Omni 66.7% | 1 | Jan 2026 | Mar 2026 | |
| Gaokao Math 2026 Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points. | iFlytek (SparkLLM) | Competition problems with answers | School | In use | Spark-X2.6-4B 134.9 | 1 | Sep 2026 | Oct 2026 | |
| GDPval-MM Agents produce office deliverables for GDPval tasks in a MiniMax agent framework. A judge model compares full trajectories pairwise and reports a win rate. | MiniMax | Tasks from real jobs | Expert | Fading | MiniMax-M2.5 59% | 1 | Feb 2026 | Feb 2026 | |
| GDPval-Rubrics (MiniMax) MiniMax rubric grading of model deliverables on public GDPval tasks, in an environment matched to GDPval-AA. | MiniMax | Tasks from real jobs | Expert | In use | MiniMax-M3 74.8% | 1 | Jun 2026 | Jun 2026 | |
| GroundUI Agents get a screenshot and an instruction and must click the right UI element. About 18K test samples cover web, desktop and mobile screens. Scored by whether the point falls in the target box. | AgentStudio team (Longtao Zheng et al., Nanyang Technological University and Skywork AI) | Finding things on screen | Everyday | Dropped | Amazon Nova Act 80.5% | 1 | Jun 2024 | Mar 2025 | |
| HaluEval Models judge whether answers, dialogue turns or summaries contain hallucinations. Hallucinated samples were generated by ChatGPT from seed datasets such as HotpotQA and then filtered. | RUCAIBox (Renmin University of China) | Hallucination rates | Everyday | Fading | Nemotron 3 Nano 4B 62.2% | 1 | May 2023 | Mar 2026 | |
| HANDBOOK.md Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass. | Surge AI | Following precise instructions | University | In use | Kimi K3.1 Pro 44.6% | 1 | Jul 2026 | Oct 2026 | |
| Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. | Tencent Hy team | Deep research | Expert | In use | Tencent Hy4 preview 56.1% | 1 | Aug 2026 | Aug 2026 | |
| Hy-CompanyBench V2 Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published. | Tencent Hy team | Building whole projects | Expert | In use | Tencent Hy4 preview 62.4% | 1 | Aug 2026 | Aug 2026 | |
| Hy-CompanyBench v1 Replaced in Aug 2026 | In use | Tencent Hy3 41.7% | 1 | Jul 2026 | Jul 2026 | ||||
| Hy-Euler pro Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published. | Tencent Hy team | Research mathematics | Expert | In use | Tencent Hy3 24.2% | 1 | Jul 2026 | Jul 2026 | |
| Hy-FinAgentBench Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published. | Tencent Hy team | Finance | Expert | In use | Tencent Hy4 preview 79.7% | 1 | Aug 2026 | Aug 2026 | |
| Hy-FinmodelBench v2 Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published. | Tencent Hy team | Finance | Expert | In use | Tencent Hy4 preview 57% | 1 | Aug 2026 | Aug 2026 | |
| Hy-FinModelBench v1 Replaced in Aug 2026 | In use | Tencent Hy3 69% | 1 | Jul 2026 | Jul 2026 | ||||
| Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. | Tencent Hy team | Deep research | Expert | In use | Tencent Hy4 preview 49.2% | 1 | Aug 2026 | Aug 2026 | |
| Hy-Math Models solve Tencent in-house math problems without tools. Task and scoring details are not published. | Tencent Hy team | Competition problems with answers | Expert | In use | Tencent Hy3 60.9% | 1 | Jul 2026 | Jul 2026 | |
| Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. | Tencent Hy team | General assistants | Expert | In use | Tencent Hy3 45.8% | 1 | Jul 2026 | Jul 2026 | |
| Hy-Vibe Bench Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published. | Tencent Hy team | Building whole projects | Expert | Fading | Tencent Hy3 preview 50.5% | 1 | Apr 2026 | Apr 2026 | |
| Infinite Generations (Mistral internal) Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better. | Mistral AI | Overall chat quality | Everyday | Dropped | Mistral Small 3.2 1.29% | 1 | Jun 2025 | Jun 2025 | |
| IntPhys 2 Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy. | Meta FAIR | Physical prediction | Everyday | In use | Muse Spark Open 68.3% | 1 | Jun 2025 | Oct 2026 | |
| KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. | Kuaishou (KwaiKAT team) | General assistants | University | In use | KAT-Coder-Pro V2.5 85.5% | 1 | Jul 2026 | Jul 2026 | |
| KernelBench Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. | Stanford Scaling Intelligence | GPU kernels | Expert | In use | MiniMax-M3 28.8% | 1 | Dec 2024 | Jun 2026 | |
| Kimi Claw 24/7 Bench Agents work on 17 persistent multi-day professional scenarios through OpenClaw, with 610 checkpoints. The score is the mean pass rate over three runs. | Moonshot AI | Tasks from real jobs | Expert | In use | Kimi K2.7 Code 46.9% | 1 | Jun 2026 | Jun 2026 | |
| Kimi Code Bench 3.0 Long-horizon Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score. | Moonshot AI | Building whole projects | Expert | In use | Kimi K3.1 Pro 61.2% | 1 | Oct 2026 | Oct 2026 | |
| Kimi Hallu Bench Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated. | Moonshot AI | Hallucination rates | Expert | In use | Kimi K3.1 Pro 8.30% | 1 | Oct 2026 | Oct 2026 | |
| Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside 10 agent threat snapshots at three defense levels. Scored by attack resistance. | Lakera + ETH Zurich + UK AISI + Oxford | Jailbreaks and prompt injection | In use | Mistral Large 4 (Le Chonk, open weights) 97.1% | 1 | Oct 2025 | Oct 2026 | ||
| Longform Writing Models write long-form text without tools, capped at 32k output tokens. Moonshot AI has not published the tasks or the scoring method. | Moonshot AI | Creative writing | University | Dropped | Kimi K2 Thinking 73.8% | 1 | Nov 2025 | Nov 2025 | |
| LSMDC Models fill in a missing word in movie audio descriptions using the matching video clip. Scoring is accuracy. | Tegan Maharaj et al. (LSMDC challenge) | Video understanding | Everyday | Dropped | GPT-4 45.7% | 1 | Nov 2016 | Mar 2023 | |
| LVBench (with memory) Models answer multiple-choice questions about videos up to two hours long, aided by Qwen's external video memory system. Scored by accuracy. | Zhipu AI and Tsinghua University (LVBench), Qwen (memory setup) | Long videos | Everyday | In use | Qwen 3.8 Max 85.6% | 1 | Aug 2026 | Aug 2026 | |
| MEWC Models solve 179 Microsoft Excel World Championship problems from 2021 to 2026 with Excel tools. Output cells are compared one by one with answer cells. | MiniMax | Documents and spreadsheets | Competition | Fading | MiniMax-M2.5 74.4% | 1 | Feb 2026 | Feb 2026 | |
| MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. | Authors of arXiv 2508.21475 | Deep research | Expert | In use | Qwen3.7-Plus 41.4% | 1 | Aug 2025 | Jun 2026 | |
| Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. | Function calling | Everyday | Fading | Reka Edge 88.4% | 1 | Dec 2025 | Mar 2026 | ||
| MolecularIQ Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer. | Johannes Kepler University Linz (Bartmann, Klambauer, Luukkonen et al.) | Genomics and life sciences | Expert | In use | Intern-S2-397B 62.4% | 1 | Jan 2026 | Sep 2026 | |
| Moment finding (Google DeepMind) Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds. | Google DeepMind | Embodied reasoning | Everyday | In use | Gemini Robotics ER 2 91.3% | 1 | Jul 2026 | Jul 2026 | |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Cohere | Reasoning in many languages | Competition | Fading | Command A+ 86% | 1 | May 2026 | May 2026 | |
| MT-Bench-101 Models hold 1,388 multi-turn dialogues with 4,208 turns across 13 tasks from a three-tier ability taxonomy. GPT-4 scores each turn from 1 to 10. | Alibaba Group, CUHK MMLab and Shanghai AI Lab (Ge Bai, Jie Liu et al.) | Multi-turn conversation | Everyday | Dropped | InternLM3-8B-Instruct 8.59 | 1 | Feb 2024 | Jan 2025 | |
| Multi-SWE-bench flash Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing. | ByteDance Seed | Fixing real repositories | Expert | Dropped | Qwen3-Coder-480B-A35B 27% | 1 | Jul 2025 | Jul 2025 | |
| MultiChallenge Audio (OpenAI) Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy. | OpenAI | Audio understanding and voice | Everyday | Dropped | gpt-realtime 30.5% | 1 | Aug 2025 | Aug 2025 | |
| MultiRC Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options. | University of Pennsylvania (CogComp) | Reading comprehension | School | Dropped | PaLM 2-L 88.2% | 1 | Jun 2018 | May 2023 | |
| OctoCodingbench Agents complete 72 coding tasks while following rules from system prompts, AGENTS.md files, skills, and memory. An LLM judge checks 2,422 pass or fail items. | MiniMax | Fixing real repositories | University | Fading | MiniMax-M2.1 26.1% | 1 | Dec 2025 | Dec 2025 | |
| OfficeQA Pro V2 Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match. | Databricks | Documents and spreadsheets | University | In use | Kimi K3.1 Pro 64.4% | 1 | Aug 2026 | Oct 2026 | |
| OfficeQA Pro v1 Replaced in Aug 2026 | In use | Claude Opus 4.7 80.6% | 4 | 10 | Mar 2026 | Oct 2026 | |||
| OpenScore String Quartets Models transcribe images of string quartet sheet music into symbolic notation. Scored as 1 minus normalized edit distance from the reference encoding. | OpenAI, using the OpenScore String Quartets corpus | Documents, charts and text in images | University | In use | GPT-6 Astra 0.84 | 1 | Sep 2026 | Sep 2026 | |
| Orak LLM agents play 12 popular video games across most genres through an MCP interface. Scores come from in-game progress or outcomes per title. | KRAFTON AI with Seoul National University, NVIDIA and UW-Madison | Video games | Everyday | Fading | Nemotron 3 Nano 4B 22.9 | 1 | Jun 2025 | Mar 2026 | |
| PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. | iFlytek (SparkLLM) | General assistants | Everyday | In use | Spark-X2.6-4B 74.1% | 1 | Oct 2026 | Oct 2026 | |
| Past IMO (Seed-Prover formal set) Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F. | ByteDance Seed | Formal proofs | Competition | Dropped | Seed-Prover 78.1% | 1 | Jul 2025 | Jul 2025 | |
| Pile-test Models predict text from the test split of the Pile, an 800GB corpus of 22 sources. Scored by bits per byte. | EleutherAI | Other | Dropped | DeepSeek-V3 0.55 | 1 | Dec 2020 | Dec 2024 | ||
| PLawBench Model handles 850 realistic legal consultation, case-analysis and document-drafting tasks. LLM judge scores against about 12,500 expert rubric items. | SKYLENAGE-AI (Shi et al.) | Law | Expert | In use | Qwen 3.8 Max 73.2% | 1 | Jan 2026 | Aug 2026 | |
| ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. | Tencent Hy team | General assistants | Expert | In use | Tencent Hy3 23% | 1 | Jul 2026 | Jul 2026 | |
| Progress classification (Google DeepMind) Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy. | Google DeepMind | Embodied reasoning | Everyday | In use | Gemini Robotics ER 2 57.4% | 1 | Jul 2026 | Jul 2026 | |
| QwenBlenderBench Models complete Blender 3D modeling tasks in Qwen's internal visual agent and coding test. Qwen has not described its scoring. | Qwen Team, Alibaba Group | Web and app front ends | University | In use | Qwen 3.8 Max 69.9% | 1 | Aug 2026 | Aug 2026 | |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | Qwen Team, Alibaba Group | Knowledge in many languages | University | Fading | Qwen 3.6 Max (Preview) 84% | 1 | Apr 2026 | Apr 2026 | |
| QwenQoderBench Qwen in-house benchmark of user experience in the Qoder coding product, run in Claude Code (avg@5, 6-hour limit). | Alibaba Qwen | Fixing real repositories | Expert | In use | Qwen 3.8 Max 58.4% | 1 | Aug 2026 | ||
| QwenReactBench Model builds bilingual React projects in Claude Code across 7 categories. Rendered outputs judged by a multimodal model, ranked with Bradley-Terry Elo. | Alibaba Qwen | Web and app front ends | Expert | In use | Qwen 3.8 Max 1,724 | 1 | Aug 2026 | ||
| QwenSVGBench Model writes SVG code for bilingual prompts. Rendered outputs judged by a multimodal model and ranked with Bradley-Terry Elo. | Alibaba Qwen | Web and app front ends | University | In use | Qwen 3.8 Max 1,713 | 1 | Aug 2026 | ||
| QwenVision2Code Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages. | Qwen Team, Alibaba Group | Web and app front ends | University | In use | Qwen3.7-Plus 1,772 | 1 | May 2026 | Jun 2026 | |
| QwenVisualOffice Models handle visual office tasks in an internal Qwen test set. Qwen has not published the tasks or how they are scored. | Qwen (Alibaba) | Documents, charts and text in images | University | In use | Qwen 3.8 Max 44.6% | 1 | Aug 2026 | Aug 2026 | |
| Readability (Perplexity) Models answer search questions. Graders rate whether answers are concise yet detailed and use markdown formatting well. | Perplexity | Taste and style | Everyday | Dropped | sonar 85.9% | 1 | Feb 2025 | Feb 2025 | |
| ReCoRD Models fill in a masked entity in a query about a news article, using the article as context. Scoring is exact match and token F1. | Johns Hopkins University and Microsoft Research | Reading comprehension | Everyday | Dropped | PaLM 2-L 93.8% | 1 | Oct 2018 | May 2023 | |
| RefAdv-S Models locate the object a referring expression describes among hard distractors in real images. Scored by accuracy at 0.5 box overlap on 1,142 cases. | Northeastern University | Detection and captioning | Everyday | In use | Qwen 3.8 Max 80.2% | 1 | Feb 2026 | Aug 2026 | |
| RepoBench Models complete code using context retrieved from other files in the same Python or Java repository. Three tasks cover retrieval, completion and the full pipeline. | UC San Diego (Tianyang Liu, Canwen Xu, Julian McAuley) | Fixing real repositories | Expert | Dropped | Codestral 38% | 1 | Jun 2023 | Jan 2025 | |
| RoadmapBench Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests. | UniPat AI | Building whole projects | Expert | In use | Step 5 Preview 54.3% | 1 | May 2026 | Sep 2026 | |
| SGI-Bench Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning. | Shanghai AI Lab and partners (Wanghan Xu et al.) | Research science | Expert | In use | Intern-S2-397B 51.9% | 1 | Dec 2025 | Sep 2026 | |
| SIFo Models follow a chain of dependent instructions, where each step uses the output of the one before. Only the final answer is checked, across 800 samples in four tasks. | University of Amsterdam and University of Groningen (Chen et al.) | Following precise instructions | Everyday | Dropped | Qwen3-VL-235B-A22B 77.3% | 1 | Jun 2024 | Sep 2025 | |
| SITE Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale. | Boston University and Microsoft Research | Spatial reasoning and counting | Everyday | Fading | SenseNova-SI-1.3-InternVL3-8B 47.5% | 1 | May 2025 | Jan 2026 | |
| Spider 2.0 Models and agents write SQL for real enterprise databases with large schemas, across BigQuery, Snowflake, SQLite and dbt projects. Answers are checked by executing them. | XLANG Lab, The University of Hong Kong, with collaborators | Data science and SQL | Expert | Dropped | Qwen3-Coder-480B-A35B 31.1% | 1 | Nov 2024 | Jul 2025 | |
| Spider 1.0 Replaced in Nov 2024 | Yale University (LILY lab, Tao Yu et al.) | University | Dropped | GPT-4 86.6% | 1 | 3 | Sep 2018 | Feb 2026 | |
| Step-SWE-Bench Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses. | StepFun | Fixing real repositories | Expert | Fading | Step 3.7 Flash 67.1% | 1 | May 2026 | May 2026 | |
| StepCodeBench Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup. | StepFun | Fixing real repositories | Expert | In use | Step 5 Preview 49% | 1 | Sep 2026 | Sep 2026 | |
| StoryCloze Models pick the correct ending for a four-sentence everyday story from two candidate endings. Scored by accuracy. | University of Rochester | Commonsense | Everyday | Dropped | PaLM 2-L 87.4% | 1 | Apr 2016 | May 2023 | |
| SuperMatBench Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part. | Ant Group and Peking University | Research science | Expert | In use | Ling-3.1-flash 51.2% | 1 | Oct 2026 | Oct 2026 | |
| SVG-Bench (MiniMax) Model creates or edits SVG graphics from text or image prompts. A VLM checks results, averaged over 3 runs. | MiniMax | Web and app front ends | University | In use | MiniMax-M3 63.7% | 1 | Jun 2026 | Jun 2026 | |
| SWE-Dev Models implement new features in real Python repositories from requirement documents, across 500 test tasks. Developer-written unit tests decide pass or fail. | Shanghai Jiao Tong University | Fixing real repositories | Expert | Dropped | Kimi K2 66.6% | 1 | May 2025 | Sep 2025 | |
| SWE-Perf Agents speed up code in 140 tasks drawn from performance pull requests in real GitHub repos. Scored on correctness and runtime gain. | TikTok and Xi'an Jiaotong University | Speed and optimization | Expert | Fading | MiniMax-M2.1 3.10% | 1 | Jul 2025 | Dec 2025 | |
| SWE-Review Models review code changes in several languages and scenarios to find defects. Scores reflect defect recall and the rate of hallucinated defects. | MiniMax | Fixing real repositories | Expert | Fading | MiniMax-M2.1 8.90% | 1 | Dec 2025 | Dec 2025 | |
| SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. | iFlytek (SparkLLM) | Deep research | Expert | In use | Spark-X2.6-4B 71.7% | 1 | Oct 2026 | Oct 2026 | |
| TAT-DQA Models answer questions about financial report pages that mix tables and text, often needing arithmetic. Scored by exact match and F1. | NExT++ Research Centre, National University of Singapore | Documents, charts and text in images | University | Dropped | Gemini 1.5 Pro 37.8% | 1 | Jul 2022 | May 2024 | |
| TOMG-Bench Models generate molecules from natural language requests in three tasks: molecule editing, molecule optimization and customized generation. Each output is checked for validity and for meeting the requested properties. | Hong Kong Polytechnic University and Shanghai AI Lab (Jiatong Li et al.) | Genomics and life sciences | Expert | In use | Intern-S2-397B 66.8% | 1 | Dec 2024 | Sep 2026 | |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Qwen Team, Alibaba Group | Function calling | Everyday | Fading | Qwen 3.6 Max (Preview) 86.1% | 1 | Apr 2026 | Apr 2026 | |
| TVQA Models answer 152,545 multiple-choice questions about clips from 6 TV shows, using video frames and subtitles. Scoring is accuracy. | UNC Chapel Hill | Video understanding | Everyday | Dropped | GPT-4 87.3% | 1 | Sep 2018 | Mar 2023 | |
| UI-TARS 15 Games Collection GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported. | ByteDance Seed | Video games | Everyday | Dropped | UI-TARS-2 59.8 | 1 | Sep 2025 | Sep 2025 | |
| VCSUM Models summarize transcripts of 239 real Chinese meetings that total over 230 hours. Scored with ROUGE against human-written summaries. | Han Wu, Mingjie Zhan, Linqi Song, and colleagues | Reasoning over long documents | Everyday | Dropped | Qwen-14B 17.3 | 1 | May 2023 | Sep 2023 | |
| VIBE-V2 (MiniMax) Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output. | MiniMax | Building whole projects | Expert | In use | MiniMax-M3 50.1% | 1 | Jun 2026 | Jun 2026 | |
| VIBE-Pro (MiniMax) Replaced in Jun 2026 | Fading | MiniMax-M2.7 55.6% | 2 | 3 | Feb 2026 | May 2026 | |||
| VIBE (MiniMax) v1 Replaced in Feb 2026 | Fading | MiniMax-M2.1 88.6% | 1 | Dec 2025 | Dec 2025 | ||||
| VideoHallucer Paired basic and hallucinated questions about videos detect intrinsic and extrinsic hallucinations. Scored by pair accuracy. | BIGAI + Peking University | Video understanding | Everyday | Fading | Reka Edge 59.6% | 1 | Jun 2024 | Mar 2026 | |
| VisualProbe Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key. | Mini-o3 team (HKU and ByteDance) | Detection and captioning | Everyday | Fading | Step 3.7 Flash 65% | 1 | Sep 2025 | May 2026 | |
| VLADBench Models answer closed-form questions about driving scenes, from static traffic knowledge to reasoning about dynamic road situations. | Authors of arXiv 2503.21505 | Spatial reasoning and counting | University | In use | Qwen3.7-Plus 77.2% | 1 | Mar 2025 | Jun 2026 | |
| WebQuestions Models answer factual questions mined from Google Suggest, with answers taken from the Freebase knowledge base. Scored by exact match. | Stanford University | Trivia and open questions | Everyday | Dropped | PaLM 2-L 28.2% | 1 | Oct 2013 | May 2023 | |
| WiC Models decide whether a target word has the same meaning in two sentence contexts. Scoring is binary accuracy. | Mohammad Taher Pilehvar and Jose Camacho-Collados | Language reasoning | Everyday | Dropped | PaLM 2-L 66.8% | 1 | Aug 2018 | May 2023 | |
| Winograd Models pick which noun an ambiguous pronoun refers to in Winograd schema sentence pairs that need world knowledge. Scored by accuracy. | Hector Levesque, Ernest Davis, and Leora Morgenstern | Commonsense | Everyday | Dropped | PaLM 2-L 89.5% | 1 | May 2023 | ||
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Moonshot AI | Finding and exploiting real bugs | Expert | In use | Kimi K3.1 Pro 81% | 1 | Oct 2026 | Oct 2026 | |
| XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. | Facebook AI Research and NYU | Reasoning in many languages | Everyday | Dropped | Apertus 70B 45.3% | 1 | Sep 2018 | Sep 2025 | |
| YC-Bench Agent runs a simulated startup for one year via CLI, managing staff, clients and cash. Scored by final funds. | Collinear AI | Running a business | University | In use | MiniMax-M3 $2,100,000 | 1 | Feb 2026 | Jun 2026 | |
| Artificial Analysis Coding Agent Index v2 2.0 Coding agents, each a model plus harness, attempt software engineering, terminal, and repository question tasks. The index averages three equal-weight benchmark scores. | Artificial Analysis | Fixing real repositories | In use | 1 | Oct 2026 | Oct 2026 | |||
| Artificial Analysis Coding Agent Index v1.5 Replaced in Oct 2026 | Not in cards | Sep 2026 | |||||||
| Artificial Analysis Coding Agent Index v1.4 Replaced in Sep 2026 | In use | GPT-6 Astra 67.0 | 1 | Aug 2026 | Sep 2026 | ||||
| Artificial Analysis Coding Agent Index v1.1 Replaced in Aug 2026 | In use | GPT-5.6 Sol 80.0 | 3 | Jun 2026 | Jul 2026 | ||||
| Artificial Analysis Coding Agent Index v1.0 Replaced in Jun 2026 | Not in cards | May 2026 | |||||||
| Conversational SWE Task Benchmark (Windsurf) Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy. | Windsurf (Codeium, now part of Cognition) | Fixing real repositories | Expert | Dropped | 1 | May 2025 | May 2025 | ||
| End-To-End SWE Task Benchmark (Windsurf) Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score. | Windsurf (Codeium, now part of Cognition) | Fixing real repositories | Expert | Dropped | 1 | May 2025 | May 2025 | ||
| EVA-Bench Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes. | ServiceNow | Speech recognition and translation | Everyday | In use | 1 | May 2026 | Sep 2026 | ||
| IChO 2025 Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme. | International Chemistry Olympiad (hosted by the UAE Ministry of Education) | Science olympiads | Competition | Fading | 1 | Jul 2025 | Feb 2026 | ||
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | DAMO Academy, Alibaba Group | Knowledge in many languages | School | Dropped | 1 | Jun 2023 | May 2024 | ||
| Objectron Models predict 3D bounding boxes of objects in short object-centric videos across nine categories. Scored by 3D intersection over union. | Google Research | Detection and captioning | Everyday | Dropped | 1 | Dec 2020 | Sep 2025 | ||
| RGB Models answer English and Chinese news questions from retrieved passages, testing noise robustness, negative rejection, information integration, and counterfactual robustness. Scored by accuracy. | Chinese Academy of Sciences (ISCAS) | Faithful to sources | Everyday | Dropped | 1 | Sep 2023 | Feb 2024 | ||
| BBQ (Bias Benchmark for QA) Ambiguous and disambiguated questions probing social stereotypes across nine attributes. Reports accuracy and bias scores. | NYU MLL | Bias, fairness and politics | System cards only | DeepSeek-R1 96.6% | 44 | Oct 2021 | Oct 2026 | ||
| Tacit Knowledge and Troubleshooting Models answer multiple-choice questions on tacit lab knowledge and troubleshooting that need hands-on expertise. Scored as cons@32. | OpenAI with Gryphon Scientific | Lab protocols | Expert | System cards only | GPT-6 Luna 94% | 38 | Aug 2024 | Oct 2026 | |
| Multimodal Troubleshooting Virology Models troubleshoot virology lab scenarios, sometimes with images, and must select every correct answer. Scored as refusal-adjusted pass@1. | OpenAI with SecureBio | Lab protocols | Expert | System cards only | o1 59% | 36 | Dec 2024 | Oct 2026 | |
| ProtocolQA Open-Ended Models correct errors in published lab protocols using open-ended questions adapted from LAB-Bench ProtocolQA. Scored as pass@1. | OpenAI | Lab protocols | Expert | System cards only | GPT-6 Sol 44.4% | 35 | Dec 2024 | Oct 2026 | |
| OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. | OpenAI | Refusing harmful requests | System cards only | 33 | Oct 2026 | ||||
| Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. | OpenAI | Refusing harmful requests | Everyday | System cards only | GPT-4.5 99% | 31 | Dec 2024 | Oct 2026 | |
| TroubleshootingBench Models find and correct realistic experimental errors in expert-written wet lab protocols. Scored as pass@1. | OpenAI | Lab protocols | Expert | System cards only | GPT-5.5 Pro 49.6% | 28 | Aug 2025 | Oct 2026 | |
| OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. | OpenAI | Refusing harmful requests | System cards only | 22 | Oct 2026 | ||||
| OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. | OpenAI | Wellbeing and sycophancy | System cards only | 20 | Oct 2026 | ||||
| XSTest Models answer 250 safe prompts that resemble unsafe ones and 200 unsafe contrast prompts. Labs usually report the incorrect refusal rate on safe prompts. | Röttger et al. (Bocconi University and others) | Refusing harmful requests | Everyday | System cards only | Cohere Command A 1.10% | 19 | Aug 2023 | Mar 2026 | |
| Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 19 | Mar 2026 | Oct 2026 | ||
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Stanford | Capture-the-flag challenges | Expert | System cards only | Claude Mythos Preview 100% | 18 | Aug 2024 | Jul 2026 | |
| Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Fable 5.1 0% | 18 | May 2025 | Oct 2026 | |
| Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Opus 4.5 99.8% | 18 | May 2025 | Oct 2026 | |
| Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Mythos Preview 96.7% | 18 | Sep 2025 | Oct 2026 | |
| Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. | Anthropic | Deception and alignment | System cards only | Claude Haiku 5.5 3.19 | 18 | Oct 2026 | |||
| SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. | Anthropic + Scale AI + Redwood (Kutasov et al.) | Deception and alignment | System cards only | Claude Opus 5.5 26.2% | 17 | Jun 2025 | Oct 2026 | ||
| Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Sonnet 4.6 99.4% | 17 | Nov 2025 | Oct 2026 | |
| CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. | OpenAI + NYU (Chen Yueh-Han et al.) | Deception and alignment | System cards only | Gemini 4 Argon 83.7% | 17 | Mar 2026 | Oct 2026 | ||
| Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. | OpenAI | Refusing harmful requests | Everyday | System cards only | o1 100% | 17 | Aug 2024 | Aug 2025 | |
| OpenAI user-flagged hallucination eval Production conversations users flagged for factual errors. Measures the rate of incorrect claims. | OpenAI | Hallucination rates | System cards only | GPT-5.5 Instant 46.1% | 17 | Oct 2026 | |||
| Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 17 | Sep 2025 | Oct 2026 | ||
| Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. | OpenAI | Refusing harmful requests | Expert | System cards only | 16 | Jun 2026 | Oct 2026 | ||
| Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 16 | Feb 2025 | Oct 2026 | ||
| Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 16 | Apr 2025 | Oct 2026 | ||
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Irregular | Capture-the-flag challenges | Expert | System cards only | Gemini 4 Argon 100% | 15 | Dec 2025 | Oct 2026 | |
| Anthropic political even-handedness eval Models answer paired prompts that take opposing sides on 150 US political topics. A grader scores even-handedness, refusals and acknowledgment of opposing views. | Anthropic | Bias, fairness and politics | Everyday | System cards only | Claude Opus 5.5 99.4% | 15 | Nov 2025 | Oct 2026 | |
| Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 15 | Apr 2025 | Oct 2026 | ||
| Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. | Gray Swan + UK AI Security Institute | Jailbreaks and prompt injection | Expert | System cards only | Claude Mythos Preview 0.10% | 14 | Jul 2025 | Oct 2026 | |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | OpenAI | Capture-the-flag challenges | University | System cards only | o3 68% | 14 | Aug 2024 | Aug 2025 | |
| LAB-Bench CloningScenarios Models answer multiple-choice questions about multi-step molecular cloning workflows. CloningScenarios subset of LAB-Bench. | FutureHouse | Lab protocols | Expert | System cards only | Gemini 3 Pro 94.2% | 14 | Jul 2024 | Nov 2025 | |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | OpenAI | Finding and exploiting real bugs | Expert | System cards only | GPT-5.5 93.3% | 14 | Apr 2025 | Aug 2026 | |
| ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. | Microsoft Research + MIT (Hartvigsen et al.) | Bias, fairness and politics | Everyday | System cards only | Gemma 2B 29.8 | 14 | Mar 2022 | Dec 2025 | |
| Long-form Biological Risk Questions Models answer long-form questions across five biothreat creation stages, graded against expert rubrics. Scored as pass@1 per stage. | OpenAI with Gryphon Scientific | Hazardous knowledge | Expert | System cards only | OpenAI deep research 0% | 14 | Sep 2024 | Nov 2025 | |
| Long-form virology tasks Agents complete end-to-end pathogen acquisition workflows, covering workflow design and laboratory protocols. Scored against expert criteria. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude 3.7 Sonnet 69.7% | 14 | Feb 2025 | Jul 2026 | |
| Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 5.5 0% | 13 | Nov 2025 | Oct 2026 | |
| Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4.6 0% | 13 | Nov 2025 | Oct 2026 | |
| Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4.5 0% | 13 | Nov 2025 | Oct 2026 | |
| OpenAI PRs Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-5.2 55% | 13 | Jan 2025 | Dec 2025 | |
| First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. | OpenAI | Bias, fairness and politics | System cards only | GPT-5.5 1.12% | 13 | Oct 2024 | Jul 2026 | ||
| DNA Synthesis Screening Evasion Models design DNA fragments that assemble into pathogenic viruses while evading gene synthesis screening. Scored per assembly criterion. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.7 80% | 13 | May 2025 | Jul 2026 | |
| Quadruped reinforcement learning task (Anthropic) Agents train a quadruped to reach high performance in a continuous control task. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 5 31.3 | 12 | May 2025 | Jul 2026 | |
| Internal Research Debugging Evaluation Agents debug difficult problems from OpenAI research work that take researchers hours or days. Scored as mean rubric score. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-6 Astra 78% | 12 | Apr 2026 | Oct 2026 | |
| WMDP Proxy multiple-choice benchmark for hazardous knowledge in bio, chem and cyber. Released with the RMU unlearning method. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | System cards only | GPT-6 Astra 84.8% | 12 | Mar 2024 | Sep 2026 | |
| LLM training task (Anthropic) Agents optimize a CPU-only small language model training implementation. Scored by speedup. | Anthropic | Machine learning engineering | Expert | System cards only | 12 | May 2025 | Jul 2026 | ||
| Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. | Anthropic | Wellbeing and sycophancy | Everyday | System cards only | 12 | Sep 2025 | Oct 2026 | ||
| Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. | Anthropic | Deception and alignment | Expert | System cards only | Claude Haiku 4.5 30% | 11 | May 2025 | Sep 2026 | |
| Time series forecasting task (Anthropic) Agents solve a regression and time-series forecasting problem with known state-of-the-art results, in six variants. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Mythos Preview 4.55 | 11 | May 2025 | Jul 2026 | |
| Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. | OpenAI | Deception and alignment | Expert | System cards only | GPT-5.5 90% | 11 | Feb 2026 | Jul 2026 | |
| Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.6 Sol 100% | 11 | Dec 2025 | Aug 2026 | |
| Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6 Astra 99.8% | 11 | Jul 2026 | Oct 2026 | |
| KernelGen 1P Agents write and optimize GPU kernels under hardware constraints and debug correctness issues. Scored as mean rubric score. | OpenAI | GPU kernels | Research frontier | System cards only | GPT-6.1 Sol 60.4% | 11 | Jun 2026 | Oct 2026 | |
| Novel compiler task (Anthropic) Agents build a compiler for an unusual programming language from a specification and test cases. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 5 80.9% | 11 | May 2025 | Jul 2026 | |
| Kernels task (Anthropic) Agents optimize compute kernels in a performance engineering challenge. Scored by speedup. | Anthropic | GPU kernels | Expert | System cards only | 11 | May 2025 | Jul 2026 | ||
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | OpenAI | Capture-the-flag challenges | School | System cards only | OpenAI deep research 82% | 10 | Aug 2024 | Jul 2025 | |
| OpenAI Research Engineer Interview: Multiple Choice Models answer 97 multiple-choice questions drawn from OpenAI machine learning interview topics. Scored as cons@32. | OpenAI | Labs' internal R&D tests | Expert | System cards only | o4-mini 83% | 10 | Aug 2024 | Jul 2025 | |
| GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 90.9% | 10 | May 2025 | Oct 2026 | |
| Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. | OpenAI | Deception and alignment | Everyday | System cards only | GPT-6.1 Terra 1.83% | 10 | Aug 2025 | Oct 2026 | |
| Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-6 Astra 100% | 10 | Jul 2026 | Oct 2026 | |
| GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 100% | 10 | May 2025 | Oct 2026 | |
| Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. | Anthropic | Bias, fairness and politics | Everyday | System cards only | Claude Opus 5.5 0.07 | 10 | Sep 2025 | Oct 2026 | |
| Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. | Apollo Research | Deception and alignment | Expert | System cards only | GPT-5 3.97% | 10 | Oct 2026 | ||
| User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 92% | 10 | Mar 2026 | Jul 2026 | |
| Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. | Anthropic | Wellbeing and sycophancy | Everyday | System cards only | 10 | Feb 2026 | Oct 2026 | ||
| U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 10 | Aug 2026 | Oct 2026 | ||
| BioLP-bench Find the introduced error in a biological lab protocol. Tests protocol understanding. | Igor Ivanov (Oxford Biosecurity Group) | Lab protocols | Expert | System cards only | OpenAI deep research 49% | 9 | Aug 2024 | Nov 2025 | |
| Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. | OpenAI | Refusing harmful requests | Expert | System cards only | o1-preview 93.4% | 9 | Sep 2024 | Apr 2025 | |
| MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. | OpenAI | Dangerous capabilities | Everyday | System cards only | GPT-4.5 72% | 9 | Sep 2023 | Nov 2025 | |
| OpenAI-Proof Q&A Agents diagnose 20 real OpenAI research and engineering bottlenecks using code, logs and run artifacts. Scored as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-5.1-Codex-Max 8.00% | 9 | Aug 2025 | Apr 2026 | |
| PersonQA Models answer questions about publicly available facts on people. Scored on accuracy, with hallucination rate reported alongside. | OpenAI | Hallucination rates | Everyday | System cards only | ChatGPT agent 94.3% | 9 | Dec 2024 | Aug 2025 | |
| SWE-bench Verified (hard subset) Agents solve a subset of SWE-bench Verified tasks estimated to take a human engineer more than one hour. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Sonnet 4.6 48.2% | 9 | Feb 2025 | Feb 2026 | |
| 100Q-Hard (Anthropic) Models answer hard, obscure factual questions. Scored on correct, incorrect and declined answers. | Anthropic | Short factual questions | Expert | System cards only | Claude Mythos Preview 60.1% | 9 | Sep 2025 | May 2026 | |
| OpenAI Research Engineer Interview: Coding Models solve 18 self-contained coding problems like those in OpenAI interviews. Scored as pass@1. | OpenAI | Labs' internal R&D tests | Expert | System cards only | o1 100% | 9 | Aug 2024 | Apr 2025 | |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Irregular | Capture-the-flag challenges | Expert | System cards only | GPT-6 Astra 90.9% | 9 | Dec 2025 | Sep 2026 | |
| Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 9 | Oct 2024 | Oct 2026 | ||
| Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | 9 | Dec 2024 | Aug 2025 | ||
| Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 9 | Oct 2026 | |||
| Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI Computer-Using Agent (CUA) 100% | 8 | Sep 2024 | Apr 2025 | |
| Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.5 0% | 8 | May 2025 | Feb 2026 | |
| Molecular Biology Capabilities Test Models answer expert-level molecular biology questions from SecureBio. Scored as accuracy. | SecureBio | Genomics and life sciences | Expert | System cards only | GPT-5.6 Sol 60% | 8 | Oct 2026 | ||
| Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 8 | Sep 2026 | Oct 2026 | |
| Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Luna 11.8% | 8 | Sep 2026 | Oct 2026 | |
| Agentic tasks (autonomous replication and adaptation) Agents attempt resource acquisition tasks such as building an authenticated API proxy or loading a model on a cloud account. Scored as success rate. | OpenAI | Labs' internal R&D tests | Expert | System cards only | OpenAI deep research 78% | 8 | Aug 2024 | Feb 2025 | |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | OpenAI | Capture-the-flag challenges | Expert | System cards only | GPT-5.6 Sol 97.1% | 8 | Jun 2026 | Aug 2026 | |
| Black-box RNA sequence modeling and design Models predict scores for unlabeled RNA sequences from labeled data, then design new sequences. Results are compared with 57 human experts. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | 8 | May 2026 | Oct 2026 | ||
| Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | 8 | Apr 2026 | Sep 2026 | ||
| CharXiv (DQ) Models answer descriptive questions about charts from arXiv papers, such as titles, axis labels and legend entries. It is the descriptive half of CharXiv. | Princeton University (Zirui Wang et al.) | Documents, charts and text in images | University | System cards only | Seed2.1 Pro 95.5% | 7 | Jun 2024 | Jun 2026 | |
| Creative biology Models answer questions about unusual biology needed to engineer harmless biological agents, a weak proxy for novel weapon design. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.6 60.3% | 7 | May 2025 | Feb 2026 | |
| General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. | xAI (SpaceXAI) | Refusing harmful requests | System cards only | Grok 4 0% | 7 | Aug 2025 | Sep 2026 | ||
| Text-based reinforcement learning task (Anthropic) Agents build scaffolding that improves a weaker model on a text-based reinforcement learning task. | Anthropic | Machine learning engineering | Expert | System cards only | Claude Opus 4.5 0.97 | 7 | May 2025 | Feb 2026 | |
| WMDP Biology Models answer 1,273 multiple-choice questions that proxy hazardous biosecurity knowledge. Biology split of WMDP. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | System cards only | OpenAI deep research 90% | 7 | Mar 2024 | Nov 2025 | |
| AAV Capsid Packaging Prediction (OpenAI) Models predict packaging scores for held-out AAV capsid variants. Scored as Spearman rank correlation. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6 Sol 0.53 | 7 | Jun 2026 | Oct 2026 | |
| Anthropic ECI (AECI) Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. | Anthropic | Composite indices | Research frontier | System cards only | Claude Opus 5.5 169.4 | 7 | Apr 2026 | Oct 2026 | |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 94.4% | 7 | Feb 2025 | Feb 2026 | |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 7 | Feb 2025 | Feb 2026 | |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.6 71.4% | 7 | Feb 2025 | Feb 2026 | |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 7 | Feb 2025 | Feb 2026 | |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.6 100% | 7 | Feb 2025 | Feb 2026 | |
| Coronavirus-ACE2 Cell-Entry Screen Models predict and rank held-out cell entry measurements for spike protein and ACE2 pairings. Scored as a composite score. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6.1 Terra 0.47 | 7 | Sep 2026 | Oct 2026 | |
| OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. | OpenAI | Deception and alignment | System cards only | GPT-6 Sol 0.08% | 7 | Oct 2026 | |||
| Phage-plasmid Co-evolution Models predict future amino acid mutations in a phage and plasmid system from earlier observations. Scored as negative log likelihood. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6 Luna 10.7 | 7 | Sep 2026 | Oct 2026 | |
| SHP2 Protein Function Prediction Models predict held-out assay measurements for single amino acid variants of SHP2. Scored as mean R squared over three assay datasets. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-6 Astra 0.40 | 7 | Sep 2026 | Oct 2026 | |
| LSAT Models take a simulated Law School Admission Test. Scored on the 120 to 180 scale. | LSAC | Language reasoning | University | System cards only | GPT-4 163.0 | 7 | Nov 2024 | ||
| Protocols (Anthropic life sciences) Models find and fix errors in molecular biology protocols and answer practical questions about applying real laboratory protocols. | Anthropic (with Benchling) | Lab protocols | Expert | System cards only | Claude Sonnet 5 62% | 7 | Jul 2026 | Oct 2026 | |
| BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. | Amazon | Bias, fairness and politics | Everyday | System cards only | Gemma 7B 49.1 | 7 | Jan 2021 | Mar 2024 | |
| Discrim-Eval Models make yes or no decisions in templated scenarios that vary age, gender and race. Scored by mixed effects model coefficients. | Anthropic (Tamkin et al.) | Bias, fairness and politics | Everyday | System cards only | 7 | Dec 2023 | Jan 2025 | ||
| Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 7 | Sep 2025 | |||
| NanoGPT Speedrun Improve a small GPT training script to reach a target validation loss as fast as possible on fixed hardware. | Keller Jordan (community); OpenAI and METR variants | Machine learning engineering | Research frontier | System cards only | 7 | Jun 2024 | Sep 2026 | ||
| Short-horizon computational biology tasks Agents perform end-to-end computational tasks related to modifying and engineering biological agents. | Anthropic | Hazardous knowledge | Expert | System cards only | 7 | May 2025 | Feb 2026 | ||
| AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. | Gray Swan AI + UK AI Security Institute | Refusing harmful requests | System cards only | Fara-7B 94.2% | 6 | Oct 2024 | Jun 2026 | ||
| Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. | Anthropic | Refusing harmful requests | System cards only | Claude Opus 5 1.13 | 6 | May 2026 | Oct 2026 | ||
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Anthropic | Finding and exploiting real bugs | Expert | System cards only | Claude Opus 5.5 91.1% | 6 | Oct 2026 | ||
| Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Sol 1.30% | 6 | Aug 2025 | Sep 2026 | |
| False-premise questions (Anthropic) Models answer questions built on false premises. Measures how often they accept the premise instead of correcting it. | Anthropic | Hallucination rates | Everyday | System cards only | Claude Opus 4.5 1.20% | 6 | Sep 2025 | May 2026 | |
| GRE Quantitative Models take the GRE quantitative reasoning section. Scored on the 130 to 170 scale. | ETS | Word problems | University | System cards only | GPT-4 163.0 | 6 | Nov 2024 | ||
| GRE Verbal Models take the GRE verbal reasoning section. Scored on the 130 to 170 scale. | ETS | Reading comprehension | University | System cards only | GPT-4 169.0 | 6 | Nov 2024 | ||
| Honesty human feedback win rate (Anthropic) Crowdworkers try to make models say something false and pick the more honest reply in head-to-head chats. Score is the win rate against a baseline. | Anthropic | Hallucination rates | Everyday | System cards only | Claude 3 Sonnet 69% | 6 | Sep 2025 | Feb 2026 | |
| Internal AI research evaluation suite 2 Agents perform scaled-down ML and alignment research tasks similar to past Anthropic research projects. | Anthropic | AI research tasks | Expert | System cards only | Claude Mythos Preview 0.65 | 6 | May 2025 | Apr 2026 | |
| LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. | Redwood Research + EquiStamp | Deception and alignment | Expert | System cards only | Gemini 4 Argon 45.5% | 6 | Apr 2026 | Oct 2026 | |
| MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. | OpenAI | Dangerous capabilities | Everyday | System cards only | GPT-4.5 57% | 6 | Sep 2023 | Feb 2025 | |
| scBench (LatchBio SingleCellBench) 195 verifiable single-cell RNA-seq analysis problems. Each pairs a data snapshot with a task and a deterministic grader. | LatchBio | Lab and data work | Expert | System cards only | Claude Opus 5.5 61.2% | 6 | Feb 2026 | Oct 2026 | |
| SpatialBench (LatchBio) Agentic spatial-transcriptomics data analysis problems with deterministic graders. Anthropic reports a 115-problem Verified variant. | LatchBio | Lab and data work | Expert | System cards only | Claude Opus 5 72.5% | 6 | Dec 2025 | Oct 2026 | |
| Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 4.5 78% | 6 | Oct 2025 | ||
| Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. | Anthropic | Deception and alignment | Expert | System cards only | Claude Fable 5 3.10% | 6 | Feb 2026 | Jun 2026 | |
| DNA sequence design for transcription factor binding Models design DNA sequences that bind 11 Nucleobench transcription factors, scored by BPNet oracles against the Ledidi baseline. Scored as pass@1. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-5.5 Pro 16.5% | 6 | Apr 2026 | Sep 2026 | |
| Hard-negative protein binding prediction Agents with structure files and tools pick the one true binder among five similar designed proteins. Scored as pass@4. | OpenAI | Genomics and life sciences | Research frontier | System cards only | GPT-5.6 Sol 7.60% | 6 | Apr 2026 | Sep 2026 | |
| API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. | Alibaba DAMO (Li et al.) | Function calling | Everyday | System cards only | Llama 3.1-405B 92.3% | 6 | Apr 2023 | Nov 2024 | |
| Factuality Heavy prompts Models answer prompts representative of factuality-heavy ChatGPT conversations. A grader with web access counts responses and claims with factual errors. | OpenAI | Hallucination rates | Everyday | System cards only | GPT-6 Sol 5.40% | 6 | Aug 2026 | Oct 2026 | |
| Gorilla APIBench Models write calls to machine learning APIs from Hugging Face, TorchHub and TensorHub. Scored by AST match. | UC Berkeley (Gorilla) | Function calling | Everyday | System cards only | Llama 3.1-405B 35.3% | 6 | May 2023 | Nov 2024 | |
| High Stakes factuality prompts Models answer difficult medical, legal and financial prompts. A grader with web access counts responses and claims with factual errors. | OpenAI | Hallucination rates | Expert | System cards only | GPT-6 Sol 4.10% | 6 | Aug 2026 | Oct 2026 | |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | Meta | Knowledge in many languages | University | System cards only | Llama 3.1-405B 83.2% | 6 | Jul 2024 | Nov 2024 | |
| Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.6 Terra 94.6% | 6 | Jun 2026 | Jul 2026 | |
| Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 0.63% | 6 | Nov 2025 | Sep 2026 | |
| WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3.5 Haiku 4.80% | 6 | Mar 2024 | Oct 2024 | |
| WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3.5 Sonnet 96.4% | 6 | Mar 2024 | Oct 2024 | |
| RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. | Allen Institute for AI and University of Washington | Refusing harmful requests | Everyday | System cards only | Apertus 70B 0.20% | 6 | Sep 2020 | Sep 2025 | |
| AAV capsid packaging prediction Models give the probability that each of 1,000 unpublished modified AAV capsid sequences assembles into a functional capsid. Scored as AUROC. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | Claude Opus 4.8 0.80 | 6 | May 2026 | Oct 2026 | |
| Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | Gemini 3.1 Pro 3.60 | 6 | Nov 2025 | Oct 2026 | |
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Qwen Team, Alibaba Group | Chat and instructions in many languages | Everyday | System cards only | Qwen2-72B 69.6% | 6 | Jul 2024 | Dec 2024 | |
| Qwen in-house English automatic evaluation Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Qwen Team, Alibaba Group | Other | Everyday | System cards only | Qwen2-72B 72.9% | 6 | Jul 2024 | Dec 2024 | |
| CBRN expert red teaming Experts try to extract uplift beyond a web baseline across staged CBRN attack scenarios. | Google DeepMind | Hazardous knowledge | Expert | System cards only | 6 | Nov 2025 | Oct 2026 | ||
| RE-Bench Seven open-ended ML research-engineering environments with human-expert baselines. Score normalized to reference solutions. | METR | AI research tasks | Research frontier | System cards only | 6 | Nov 2024 | Feb 2026 | ||
| Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Haiku 4.5 100% | 5 | May 2025 | Feb 2026 | |
| Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. | Anthropic | Deception and alignment | Expert | System cards only | Claude Sonnet 4 14.3% | 5 | Oct 2025 | ||
| BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. | SecureBio | Refusing harmful requests | Expert | System cards only | Muse Wonder Full 98.7% | 5 | Sep 2026 | Oct 2026 | |
| ChangeMyView Models write rebuttals to r/ChangeMyView posts that human raters score for persuasiveness. Scored as percentile against human responses. | OpenAI | Dangerous capabilities | Everyday | System cards only | o1 89.1% | 5 | Sep 2024 | Feb 2025 | |
| Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 5 | Sep 2026 | Oct 2026 | |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Anthropic | Finding and exploiting real bugs | Research frontier | System cards only | Claude Mythos Preview 72.4% | 5 | Jul 2026 | ||
| Internal model use survey (Anthropic) Anthropic technical staff estimate productivity uplift from the model and judge whether it could automate a junior ML researcher. | Anthropic | Labs' internal R&D tests | Expert | System cards only | Claude Opus 4.5 50% | 5 | May 2025 | Apr 2026 | |
| Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | o1-mini 100% | 5 | Sep 2024 | Jan 2025 | |
| Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Microsoft | Refusing harmful requests | Everyday | System cards only | phi-3-small 7.4B 0.30% | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. | Microsoft | Refusing harmful requests | Everyday | System cards only | phi-3-mini 3.8B 10% | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. | Microsoft | Jailbreaks and prompt injection | Everyday | System cards only | Phi-4 7.30% | 5 | Apr 2024 | Dec 2024 | |
| Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. | Microsoft | Refusing harmful requests | Everyday | System cards only | Phi-3.5-MoE 10.5% | 5 | Apr 2024 | Dec 2024 | |
| Monorepo-Bench Agents replicate pull-request style contributions in a large internal OpenAI repository. Graded by hidden unit tests as pass@1. | OpenAI | Labs' internal R&D tests | Research frontier | System cards only | GPT-6 Astra 76% | 5 | Feb 2026 | Sep 2026 | |
| Organic Chemistry (Anthropic) Models predict structures from spectra, design synthetic routes, predict reaction products and convert between IUPAC names, SMILES and structure images. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.8 86.2% | 5 | Feb 2026 | Jun 2026 | |
| Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 9.40% | 5 | Feb 2026 | May 2026 | |
| Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 100% | 5 | Sep 2024 | Jan 2025 | |
| Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 5.5 4.00% | 5 | Sep 2026 | Sep 2026 | |
| UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. | UK AI Security Institute | Deception and alignment | Expert | System cards only | Claude Opus 4.7 0% | 5 | Jul 2026 | Sep 2026 | |
| Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Luna 0% | 5 | Sep 2026 | Oct 2026 | |
| WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. | Allen Institute for AI | Refusing harmful requests | Everyday | System cards only | Olmo 3.1 32B Think 96.9% | 5 | Jun 2024 | Dec 2025 | |
| WMDP Chemistry Models answer 408 multiple-choice questions that proxy hazardous chemical security knowledge. Chemistry split of WMDP. | CAIS + Scale AI et al. | Hazardous knowledge | Expert | System cards only | Gemini 3 Pro 84.8% | 5 | Mar 2024 | Nov 2025 | |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | OpenAI | Finding and exploiting real bugs | Research frontier | System cards only | GPT-6.1 Terra 24.5% | 5 | Sep 2026 | Oct 2026 | |
| HarmBench Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies. | Mazeika et al. (Center for AI Safety) | Jailbreaks and prompt injection | Everyday | System cards only | Olmo 3.1 32B Think 89.7% | 5 | Feb 2024 | Jan 2026 | |
| MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. | OpenAI | Wellbeing and sycophancy | Expert | System cards only | GPT-6.1 Sol 57.9% | 5 | Sep 2026 | Oct 2026 | |
| Structural Biology (Anthropic) Models answer questions about a biomolecule's function from structural data and basic tools, in multiple-choice and open-ended variants. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.8 79% | 5 | Feb 2026 | Jun 2026 | |
| Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. | Meta AI (Bandarkar et al.) | Reading in many languages | Everyday | System cards only | Muse Glimmer 87.8% | 5 | Aug 2023 | Oct 2026 | |
| Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 5 | Sep 2026 | Oct 2026 | ||
| Biotool and Wet Lab Actions Agents perform wet lab assistant tasks such as finding a genome file or ordering DNA through a computer or browser. Scored as pass@10 per task. | OpenAI with Ranger | Hazardous knowledge | Expert | System cards only | 5 | Sep 2024 | Jan 2025 | ||
| CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. | OpenAI | Deception and alignment | Expert | System cards only | 5 | Mar 2026 | Sep 2026 | ||
| Human feedback win rates (Anthropic) Raters chat with two models on tasks such as coding, documents and creative writing and pick the better reply. Score is the win rate. | Anthropic | Overall chat quality | Everyday | System cards only | 5 | Mar 2024 | Oct 2024 | ||
| ABC Bench Liquid-Handling Robot Agents use biology tools to plan liquid-handling robot steps for a construct-assembly task. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | Muse Wonder Full 100.0 | 4 | Jun 2026 | Oct 2026 | |
| Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.1-Codex-Max 100% | 4 | Aug 2025 | Nov 2025 | |
| Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. | OpenAI | Dangerous capabilities | Expert | System cards only | OpenAI deep research 77% | 4 | Dec 2024 | Feb 2025 | |
| CrowS-Pairs Models choose between paired sentences that differ only in a stereotyped social group. Measures preference for the stereotyped sentence. | NYU (Nangia et al.) | Bias, fairness and politics | Everyday | System cards only | Gemma 7B 51.3% | 4 | Sep 2020 | Jul 2024 | |
| Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Fable 5.1 100% | 4 | Jul 2026 | Sep 2026 | |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Google DeepMind with Irregular | Capture-the-flag challenges | Expert | System cards only | Gemini 3 Pro 91.7% | 4 | Mar 2025 | Nov 2025 | |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Google DeepMind | Capture-the-flag challenges | Expert | System cards only | Gemini 2.5 Flash 15.4% | 4 | Feb 2024 | Jul 2025 | |
| Harmlessness human feedback win rate (Anthropic) Raters probe models in adversarial chats and pick the more harmless reply in head-to-head comparisons. Score is the win rate against a baseline model. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3 Sonnet 87% | 4 | Jun 2024 | Oct 2024 | |
| Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Opus 4.7 0.01% | 4 | Feb 2026 | Apr 2026 | |
| Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude Sonnet 4.6 99.4% | 4 | Feb 2026 | Apr 2026 | |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Google DeepMind | Capture-the-flag challenges | Expert | System cards only | Gemini 2.5 Pro 84.6% | 4 | Mar 2024 | Jul 2025 | |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | Princeton (Yang et al.) | Capture-the-flag challenges | School | System cards only | Gemini 2.5 Pro 98.7% | 4 | Jun 2023 | Jul 2025 | |
| LongFact-Concepts Models write detailed answers to LongFact prompts about broad concepts. A fact checker scores the claim-level error rate. | Google DeepMind | Long-form factuality | Everyday | System cards only | GPT-5 mini 0.90% | 4 | Mar 2024 | Feb 2026 | |
| LongFact-Objects Models write detailed answers to LongFact prompts about specific objects such as people or places. A fact checker scores the claim-level error rate. | Google DeepMind | Long-form factuality | Everyday | System cards only | GPT-5 1.40% | 4 | Mar 2024 | Feb 2026 | |
| MBE (Multistate Bar Examination) Models answer the 200 multiple choice questions of an official Multistate Bar Examination practice exam. Scored by accuracy. | NCBE | Law | Expert | System cards only | Claude 3 Opus 85% | 4 | Mar 2024 | ||
| Meta BioDesign Proxy Agents complete computational biology design tasks across host-pathogen interaction, protein design and sequence analysis. Aggregate normalized score. | Meta + Faculty AI | Genomics and life sciences | Expert | System cards only | Muse Wonder Full 79.4 | 4 | Oct 2026 | ||
| Meta chemical-agent refusal evaluation Models receive high-risk chemical prompts and the benchmark measures how often they refuse. Covers development, procurement and production stages. | Meta | Hazardous knowledge | Expert | System cards only | Muse Wonder Full 100% | 4 | Oct 2026 | ||
| Microsoft RAI benchmark: Ungroundedness Models answer in GPT-4 simulated multi-turn conversations. A judge rates how ungrounded each answer is in the given prompt, from 0 to 4. | Microsoft | Faithful to sources | Everyday | System cards only | phi-3-medium 14B 0.21 | 4 | Apr 2024 | Aug 2024 | |
| Political Persuasion Parallel Generation Human raters pick the more persuasive of two short political social media posts. Scored as win rate against GPT-4o. | OpenAI | Dangerous capabilities | Everyday | System cards only | o1-preview 53.8% | 4 | Sep 2024 | Jan 2025 | |
| RACE Models answer reading comprehension questions from Chinese middle and high school English exams. Reported here without a subset split. | Carnegie Mellon University | Reading comprehension | School | System cards only | InternVL3.5-241B-A28B 94.2% | 4 | Apr 2017 | Aug 2025 | |
| Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. | OpenAI | Dangerous capabilities | Expert | System cards only | OpenAI deep research 74% | 4 | Dec 2024 | Feb 2025 | |
| Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | grok-code-fast-1 0% | 4 | Aug 2025 | Nov 2025 | ||
| Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | Grok 4 0% | 4 | Aug 2025 | Nov 2025 | ||
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | OpenAI | Finding and exploiting real bugs | Research frontier | System cards only | GPT-6 Astra 45.5% | 4 | Sep 2026 | Oct 2026 | |
| SecureBio VMQA, single choice Models answer 350 virology troubleshooting questions in single-choice form. VMQA is an earlier version of the Virology Capabilities Test. | SecureBio | Hazardous knowledge | Expert | System cards only | Gemini 3 Pro 61.2% | 4 | Nov 2025 | ||
| SWE-Lancer Diamond (SWE Manager) Models review competing implementation proposals for real freelance tasks and pick the best one. Scored against the original choice as pass@1. | OpenAI | Fixing real repositories | Expert | System cards only | OpenAI deep research 51% | 4 | Feb 2025 | Apr 2025 | |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | UK AI Security Institute | Finding and exploiting real bugs | Research frontier | System cards only | Claude Opus 5 80% | 4 | Apr 2026 | Jul 2026 | |
| UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. | UK AI Security Institute | Deception and alignment | Expert | System cards only | Claude Opus 4.7 0% | 4 | Jul 2026 | Jul 2026 | |
| WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. | Allen Institute for AI (dataset), OpenAI (toxic subset) | Refusing harmful requests | Everyday | System cards only | GPT-4.5 98% | 4 | May 2024 | Feb 2025 | |
| Winogender Models resolve pronouns in sentences about occupations. Accuracy shows whether gender stereotypes sway coreference. | Johns Hopkins University (Rudinger et al.) | Bias, fairness and politics | Everyday | System cards only | Gemma 2 9B 79.2% | 4 | Apr 2018 | Jul 2024 | |
| ABC Bench Fragment Design Agents use biology tools to complete a computational genetic-construct design task. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | Muse Spark 1.1 97.0 | 4 | Jun 2026 | Oct 2026 | |
| Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6.1 Terra 8.47% | 4 | Oct 2026 | Oct 2026 | |
| World-Class Bio Models answer expert-level biology questions in a SecureBio benchmark. Scored as accuracy. | SecureBio | Genomics and life sciences | Research frontier | System cards only | GPT-5.6 Sol 68.3% | 4 | Sep 2026 | ||
| AP Art History Models take a simulated AP Art History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Biology Models take a simulated AP Biology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Chemistry Models take a simulated AP Chemistry exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 4 | Nov 2024 | ||
| AP English Language and Composition Models take a simulated AP English Language and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 2.00 | 4 | Nov 2024 | ||
| AP English Literature and Composition Models take a simulated AP English Literature and Composition exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 2.00 | 4 | Nov 2024 | ||
| AP Environmental Science Models take a simulated AP Environmental Science exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Macroeconomics Models take a simulated AP Macroeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Microeconomics Models take a simulated AP Microeconomics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Psychology Models take a simulated AP Psychology exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP Statistics Models take a simulated AP Statistics exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP US Government Models take a simulated AP US Government exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP US History Models take a simulated AP US History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 5.00 | 4 | Nov 2024 | ||
| AP World History Models take a simulated AP World History exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 4 | Nov 2024 | ||
| Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 4 | Sep 2026 | Oct 2026 | |
| Protein Design (Anthropic) Models generate novel protein sequences under design constraints and rank protein design libraries for experimental testing. | Anthropic | Research science | Expert | System cards only | Claude Opus 5 42.5% | 4 | Jun 2026 | Oct 2026 | |
| SAT Math Models take the SAT math section. Scored on the 200 to 800 scale. | College Board | Word problems | School | System cards only | GPT-4 700.0 | 4 | Nov 2024 | ||
| Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. | Anthropic | Deception and alignment | Expert | System cards only | 4 | Feb 2026 | Apr 2026 | ||
| Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. | Apollo Research | Deception and alignment | Expert | System cards only | 4 | Dec 2024 | Apr 2025 | ||
| Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. | OpenAI | Deception and alignment | Everyday | System cards only | 4 | Mar 2026 | Jul 2026 | ||
| Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | 4 | Mar 2026 | Jul 2026 | ||
| Expert comparisons on biothreat information PhD biology experts compare model answers to expert answers on long-form biorisk questions. Scored as win rate on accuracy, understanding and ease of execution. | OpenAI | Hazardous knowledge | Expert | System cards only | 4 | Sep 2024 | Jan 2025 | ||
| Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. | Anthropic | Deception and alignment | Expert | System cards only | 4 | Sep 2026 | |||
| Llama 2-Chat helpfulness human evaluation (Meta) Three human raters compare Llama 2-Chat with a rival chat model on about 4,000 single and multi-turn helpfulness prompts. | Meta | Overall chat quality | System cards only | 4 | Jul 2023 | Jul 2023 | |||
| Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. | Meta | Refusing harmful requests | System cards only | 4 | Jul 2023 | Jul 2023 | |||
| METR general autonomy task suite Agents complete multi-step tasks in virtual environments. Scored as horizon weighted average task completion compared with human time limits. | METR | Task length | Expert | System cards only | 4 | Dec 2024 | |||
| Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 4 | Sep 2026 | Sep 2026 | ||
| UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. | UK AI Security Institute | Deception and alignment | System cards only | 4 | Apr 2026 | ||||
| WinoBias Models resolve pronouns linked to occupations in type 1 and type 2 sentences. Accuracy gaps show gender bias in coreference. | UCLA (Zhao et al.) | Bias, fairness and politics | Everyday | System cards only | 4 | Apr 2018 | Jul 2024 | ||
| VCR (Visual Caption Restoration) Models restore caption text that is partly covered in an image, using only the visible pixel fragments. Scored by exact match and Jaccard similarity of the words. | Tianyu Zhang et al. (vcr-org) | Documents, charts and text in images | Everyday | System cards only | Qwen2-VL-7B-Instruct 75.6% | 3 | Jun 2024 | Nov 2025 | |
| OR-Bench Models meet benign and borderline prompts and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Cui et al. | Jailbreaks and prompt injection | University | System cards only | GPT-4o 6.80% | 3 | May 2024 | Oct 2026 | |
| Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. | Anthropic | Deception and alignment | Expert | System cards only | Claude Mythos Preview 1.02 | 3 | Apr 2026 | May 2026 | |
| Aegis 2.0 Models classify human and LLM conversation turns as safe or unsafe under a taxonomy of 12 hazard categories. Scored by F1 against human and jury labels. | NVIDIA (Shaona Ghosh and others) | Refusing harmful requests | Everyday | System cards only | Nova 2 Lite 85.8% | 3 | Jan 2025 | Dec 2025 | |
| AMC 10 2023 Models take the 2023 AMC 10 A and B exams. Scored out of 150 points. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | Claude 3 Opus 72.0 | 3 | Nov 2023 | Mar 2024 | |
| AMC 10 2022 Replaced in Nov 2023 | System cards only | GPT-4 30.0 | 1 | Nov 2022 | Mar 2023 | ||||
| AMC 8 2023 Models take the 25 question 2023 AMC 8 exam. Scored out of 150 points. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | Claude 3 Opus 84.0 | 3 | Jan 2023 | Mar 2024 | |
| AP Calculus Models answer an official AP Calculus practice exam without image questions. Scored as accuracy. | College Board | Exam questions | School | System cards only | Llama 3.1-405B 88.6% | 3 | Nov 2024 | ||
| AP Physics Models answer an official AP Physics practice exam without image questions. Scored as accuracy. | College Board | Exam questions | School | System cards only | Llama 3.1-405B 92.9% | 3 | Nov 2024 | ||
| Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. | Apollo Research | Dangerous capabilities | Expert | System cards only | o4-mini 0.17 | 3 | Apr 2025 | Feb 2026 | |
| ASL-4 virology uplift trial (Anthropic) PhD experts with model access write a protocol to reconstruct a challenging virus. Graded by a rubric with critical failures. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.5 66.8% | 3 | Nov 2025 | Apr 2026 | |
| Autointent-Bio Models face manual and synthetic dangerous and benign biology queries. xAI reports refusal recall on the dangerous ones. | xAI (SpaceXAI) | Hazardous knowledge | System cards only | Grok 4.6 100% | 3 | Jul 2026 | Sep 2026 | ||
| Autointent-Chem Models face manual and synthetic dangerous and benign chemistry queries. xAI reports refusal recall on the dangerous ones. | xAI (SpaceXAI) | Hazardous knowledge | System cards only | Grok 4.6 100% | 3 | Jul 2026 | Sep 2026 | ||
| Biomedical image analysis (Anthropic) Models run quantitative analyses of CT, MRI, histopathology and other images from public competitions. Scores are scaled to the best competition results. | Anthropic | Research science | Expert | System cards only | Claude Sonnet 5.5 72.2% | 3 | Sep 2026 | Oct 2026 | |
| BioPipelineBench (Anthropic) Agents execute bioinformatics workflows such as long-read sequence analysis, metagenome assembly and chromatin profiling with a bash tool. | Anthropic | Lab and data work | Expert | System cards only | Claude Opus 4.8 87.7% | 3 | Feb 2026 | May 2026 | |
| Bioweapons acquisition uplift trial (Anthropic) Participants draft a bioweapons acquisition plan over two days with or without model access. Score is the uplift ratio of model group over control. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4 2.53 | 3 | May 2025 | ||
| Bioweapons knowledge questions Models answer 33 sensitive bioweapons questions written by Deloitte biosecurity staff. Answers are compared with novice, intermediate and expert baselines. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude 3.7 Sonnet 51.5% | 3 | Feb 2025 | May 2025 | |
| Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5 mini 93.6% | 3 | Aug 2025 | Aug 2025 | |
| Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. | xAI (SpaceXAI) | Refusing harmful requests | System cards only | Grok 4.5 0% | 3 | Jul 2026 | Sep 2026 | ||
| Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 96.9% | 3 | Sep 2026 | Oct 2026 | |
| CyberBench (Liu et al.) Models face detail-seeking high-severity cyber requests. Meta uses it as a seed set for adaptive red-teaming and reports attack success rate. | JPMorgan Chase (Liu, Shi and Buford) | Refusing harmful requests | Expert | System cards only | Muse Wonder Full 0.80% | 3 | Feb 2024 | Oct 2026 | |
| CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Meta (PurpleLlama) | Secure code and review | University | System cards only | Muse Spark 1.1 71.3% | 3 | Dec 2023 | Jul 2026 | |
| CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | Meta (PurpleLlama) | Secure code and review | University | System cards only | Muse Spark 1.1 77.3% | 3 | Dec 2023 | Jul 2026 | |
| CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. | Meta (PurpleLlama) | Jailbreaks and prompt injection | Everyday | System cards only | Llama 3.1-8B 19% | 3 | Apr 2024 | Nov 2024 | |
| Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. | OpenAI | Refusing harmful requests | Expert | System cards only | ChatGPT agent 96.9% | 3 | Aug 2025 | Aug 2025 | |
| FORTRESS-RN Models face the radiological and nuclear prompts of Scale AI's FORTRESS. xAI reports refusal accuracy on these items. | Scale AI | Refusing harmful requests | System cards only | Grok 4.5 97.9% | 3 | Jun 2025 | Sep 2026 | ||
| GMAT Quantitative Models answer the quantitative section of the official GMAT online exam. Scored as accuracy. | GMAC | Word problems | University | System cards only | Llama 3.1-405B 96% | 3 | Nov 2024 | ||
| GMAT Verbal Models answer the verbal section of the official GMAT online exam. Scored as accuracy. | GMAC | Reading comprehension | University | System cards only | Llama 3.1-405B 86.6% | 3 | Nov 2024 | ||
| GPQA Extended Models answer all 546 GPQA questions, graduate level biology, chemistry and physics questions written by experts. Scored by accuracy. | NYU (Rein et al.) | Science questions | Expert | System cards only | Claude 3 Opus 48.8% | 3 | Nov 2023 | Mar 2024 | |
| GRE Analytical Writing Models write GRE issue and argument essays graded by trained contractors. Scored on the 0 to 6 scale. | ETS | Exam questions | University | System cards only | Claude 2 5.00 | 3 | Mar 2024 | ||
| GRE Physics Models answer questions from official GRE Physics practice tests. Scored as accuracy. | ETS | Science questions | University | System cards only | Llama 3.1-405B 80% | 3 | Nov 2024 | ||
| HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. | xAI (SpaceXAI) | Refusing harmful requests | System cards only | Grok 4.5 7.80% | 3 | Sep 2026 | |||
| Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. | Microsoft | Refusing harmful requests | Everyday | System cards only | Phi-4-Reasoning-Vision-15B 1.40% | 3 | Mar 2025 | Mar 2026 | |
| HLE Calibration Models give a confidence score with each HLE answer. The benchmark reports RMS calibration error and AUROC between confidence and correctness. | Center for AI Safety + Scale AI | Hallucination rates | Expert | System cards only | Muse Spark 1.1 23.4 | 3 | Jan 2025 | Oct 2026 | |
| Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 100% | 3 | Sep 2026 | Sep 2026 | |
| HumanEvalPack Models write functions for HumanEval problems translated into Python, JavaScript, Java, Go, C++ and Rust. Hidden tests give pass@1. | Hugging Face and BigCode (Muennighoff et al., OctoPack) | Function-level problems | University | System cards only | Cohere Command A 76.2% | 3 | Aug 2023 | Apr 2025 | |
| InfiniteBench/En.QA Models answer free-form questions about novels longer than 100K tokens. Scored by F1 against reference answers. | OpenBMB (Zhang et al.) | Reasoning over long documents | Everyday | System cards only | Llama 3.1-70B 36.7 | 3 | Feb 2024 | Nov 2024 | |
| Internal agentic coding evaluation (Anthropic) Agents implement a pull request in an open source codebase from a natural language description. A problem counts as solved when all hidden tests pass. | Anthropic | Fixing real repositories | Expert | System cards only | Claude 3.5 Sonnet (October 2024) 78% | 3 | Jun 2024 | Oct 2024 | |
| Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. | Jigsaw and Google (Conversation AI), hosted on Kaggle | Refusing harmful requests | Everyday | System cards only | Nova 2 Omni Preview 57.8% | 3 | Dec 2017 | Dec 2025 | |
| Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5-Codex 100% | 3 | May 2025 | Nov 2025 | |
| Math23K Models solve Chinese elementary-school math word problems from a set of about 23,000. Scored by final-answer accuracy. | Tencent AI Lab (Wang et al.) | Word problems | School | System cards only | GLM-4 88.5% | 3 | Sep 2017 | Jun 2024 | |
| Medicinal Chemistry (Anthropic) Models see several real drug-discovery compounds and pick which had the highest or lowest measured ADME property. The set has 504 questions. | Anthropic | Research science | Expert | System cards only | Claude Sonnet 5.5 65.3% | 3 | Sep 2026 | Oct 2026 | |
| Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 1.1 0.70% | 3 | Oct 2026 | ||
| Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 0.50 | 3 | Oct 2026 | ||
| Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. | Meta | Wellbeing and sycophancy | Everyday | System cards only | Muse Spark 1.3 40.5% | 3 | Oct 2026 | ||
| METR data deduplication Agents write an efficient algorithm that removes near-duplicate lines from a large file. A trial succeeds when F1 reaches at least 80%. | METR | Machine learning engineering | Expert | System cards only | Claude Opus 4 32.6% | 3 | May 2025 | ||
| Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.7 2.60% | 3 | Apr 2026 | May 2026 | |
| Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6.1 Sol 1.94% | 3 | Sep 2026 | Oct 2026 | |
| Morphology-to-molecule matching (Anthropic) Models match four sets of multi-channel liver cell images to the four shuffled molecule and dose treatments that produced them. | Anthropic (with Axiom Bio) | Research science | Expert | System cards only | Claude Opus 5.5 34% | 3 | Sep 2026 | Oct 2026 | |
| Phylogenetics (Anthropic) Models analyze phylogenetic data and tree diagrams, testing quantitative reasoning about tree structure and visual interpretation. | Anthropic | Research science | Expert | System cards only | Claude Opus 4.7 79.6% | 3 | Feb 2026 | Apr 2026 | |
| Poly-Guard Bench Models meet harmful requests and statements across domains. Reports attack success rate under single-turn and multi-turn adaptive red-teaming. | Kang et al. (Bo Li lab) | Jailbreaks and prompt injection | Expert | System cards only | Muse Wonder Full 2.30% | 3 | Jun 2025 | Oct 2026 | |
| Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. | Anthropic (with Gray Swan) | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 5 0.08% | 3 | May 2026 | Jul 2026 | |
| ProteinGym Deep mutational scanning benchmarks. The model ranks mutations by predicted effect on protein function. Anthropic reports a hard subset. | OATML (Oxford) + Marks Lab (Harvard) | Research science | Research frontier | System cards only | Claude Opus 5 47.7% | 3 | Dec 2023 | Jul 2026 | |
| ProtocolQA Open-Ended (xAI) Models name the single most important mistake in a biology lab protocol in free text, with no answer choices. | xAI (SpaceXAI) | Lab protocols | Expert | System cards only | Grok 4.5 87% | 3 | Jul 2026 | Sep 2026 | |
| QuantBench Models answer 25 multiple choice versions of quant trading firm reasoning puzzles. Scored as cons@16 accuracy. | OpenAI | Competition problems with answers | Competition | System cards only | o1 60% | 3 | Sep 2024 | Dec 2024 | |
| Real-World Finance (Anthropic) Agents complete about 50 analyst tasks in banking, private equity, hedge funds and corporate finance, producing spreadsheets, slides and documents. Rubric graded. | Anthropic | Finance | Expert | System cards only | Claude Fable 5 70% | 3 | Feb 2026 | Jun 2026 | |
| RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. | Guo et al. (Bo Li lab) | Secure code and review | Expert | System cards only | Muse Spark 1.3 6.00% | 3 | Nov 2024 | Oct 2026 | |
| Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 1.50% | 3 | Sep 2026 | Oct 2026 | |
| SAT Reading Models answer the reading section of official SAT practice tests. Scored as accuracy. | College Board | Reading comprehension | School | System cards only | Llama 3.1-405B 74.8% | 3 | Nov 2024 | ||
| SEED-Bench-2-Plus Multimodal models answer multiple-choice questions about text-rich images such as charts, maps and web pages. | Tencent AI Lab (ARC Lab) | Documents, charts and text in images | Everyday | System cards only | Qwen2.5-VL-72B 73% | 3 | Apr 2024 | Nov 2025 | |
| Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. | xAI (SpaceXAI) | Wellbeing and sycophancy | System cards only | Grok 4.5 0.50% | 3 | Jul 2026 | Sep 2026 | ||
| Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. | Anthropic | Deception and alignment | Expert | System cards only | Claude Sonnet 5.5 8.00% | 3 | Sep 2026 | Oct 2026 | |
| Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | Grok 4.7 0.01% | 3 | Jul 2026 | Sep 2026 | ||
| SycophancyEval (answer sycophancy) Models answer factual questions where the user suggests a wrong answer. Score is the relative accuracy drop against neutral prompts. | Anthropic (Sharma et al.) | Deception and alignment | System cards only | Grok 4 7.00% | 3 | Oct 2023 | Nov 2025 | ||
| Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 9.83 | 3 | Jul 2026 | ||
| WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. | Perplexity | Deep research | Everyday | System cards only | Claude Opus 5.5 72.3% | 3 | Jul 2026 | Oct 2026 | |
| xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. | xAI (SpaceXAI) | Wellbeing and sycophancy | System cards only | Grok 4.6 0.04% | 3 | Sep 2026 | |||
| ABC Bench Screening Evasion Agents complete a computational task on nucleic-acid synthesis screening. One task family of the Agentic Bio-Capabilities Benchmark. | SecureBio | Hazardous knowledge | Expert | System cards only | Muse Spark 1.1 63.2 | 3 | Jun 2026 | Oct 2026 | |
| CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. | Meta (PurpleLlama) | Refusing harmful requests | University | System cards only | Llama 3.1-70B 3.80% | 3 | Apr 2024 | Jul 2026 | |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Irregular | Finding and exploiting real bugs | Research frontier | System cards only | GPT-6 Astra 38.1% | 3 | Jun 2026 | Sep 2026 | |
| Red Teaming Network pairwise safety comparison Red teamers compare anonymized responses from two models in free-form risky conversations. Scored as the share of ratings where a model is safer. | OpenAI | Refusing harmful requests | Expert | System cards only | o1 59.8% | 3 | Dec 2024 | Feb 2025 | |
| ReproBAIT Agents independently reproduce biological AI models from published scientific papers. Scored as success rate. | SecureBio | Genomics and life sciences | Expert | System cards only | GPT-5.6 Sol 85% | 3 | Sep 2026 | ||
| Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. | Charlemagne Labs | Refusing harmful requests | Expert | System cards only | Muse Spark 1.3 1.60% | 3 | Oct 2026 | ||
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Anthropic | Finding and exploiting real bugs | Research frontier | System cards only | Claude 3.7 Sonnet 66.7% | 3 | May 2025 | Sep 2025 | |
| Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 21% | 3 | Sep 2026 | ||
| Google RE-Bench (GRB) Google DeepMind internal research-engineering benchmark with a 2-hour timeout per task, used for the ML R&D CCL rule-out. | Google DeepMind | Labs' internal R&D tests | Research frontier | System cards only | Gemini 3.7 Flash 27% | 3 | Oct 2026 | ||
| Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | Gemini 3.7 Flash 21.3% | 3 | Nov 2025 | Oct 2026 | |
| Multi-factual (Anthropic) Models answer questions that each combine several closed-ended sub-questions on one topic. Answers are graded correct, incorrect or unsure against references. | Anthropic | Long-form factuality | Everyday | System cards only | Claude 3 Opus 62.8% | 3 | Mar 2024 | Mar 2024 | |
| AAV packaging rate auto-research evaluation Agents build predictors of AAV capsid packaging that must generalize to an unseen insertion region, using a limited scorer. Scored by Spearman correlation. | Anthropic (with Dyno Therapeutics) | Hazardous knowledge | Expert | System cards only | 3 | Sep 2026 | Oct 2026 | ||
| Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Sep 2026 | Oct 2026 | ||
| Adversarial SQuAD Models answer SQuAD questions whose passages contain an added distracting sentence. | Stanford (Jia and Liang) | Reading comprehension | Everyday | System cards only | 3 | Jul 2017 | Nov 2024 | ||
| CBRN open-ended questions Experts review model answers to single and multi-turn CBRN threat scenario questions for accuracy, novelty and actionable uplift. | Google DeepMind | Hazardous knowledge | Expert | System cards only | 3 | Mar 2025 | Nov 2025 | ||
| Dynabench SQuAD Models answer reading comprehension questions written by people to fool earlier models in the loop. | Meta AI + collaborators (Kiela et al.) | Reading comprehension | Everyday | System cards only | 3 | Apr 2021 | Nov 2024 | ||
| Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Apr 2026 | May 2026 | ||
| Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. | Meta | Refusing harmful requests | System cards only | 3 | Jul 2024 | Nov 2024 | |||
| OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. | OpenAI | Deception and alignment | Expert | System cards only | 3 | Sep 2026 | Sep 2026 | ||
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Pattern Labs (now Irregular) | Capture-the-flag challenges | Expert | System cards only | 3 | Apr 2025 | Aug 2025 | ||
| PAWS Models decide whether two sentences with heavy word overlap are paraphrases. | Google (Zhang et al.) | Language reasoning | Everyday | System cards only | 3 | Apr 2019 | Nov 2024 | ||
| Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. | OpenAI | Refusing harmful requests | Everyday | System cards only | 3 | Sep 2024 | Dec 2024 | ||
| TRAIT Models answer 8,000 forced-choice items on Big Five and Dark Triad traits. Reports a score per trait. | Lee et al. | Deception and alignment | Everyday | System cards only | 3 | Jun 2024 | Oct 2026 | ||
| Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. | Anthropic | Deception and alignment | Expert | System cards only | 3 | Apr 2026 | Oct 2026 | ||
| Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. | OpenAI | Deception and alignment | Expert | System cards only | 3 | Jun 2026 | Sep 2026 | ||
| Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. | Anthropic | Refusing harmful requests | Expert | System cards only | 3 | Jul 2026 | Sep 2026 | ||
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | OpenAI | Finding and exploiting real bugs | Research frontier | System cards only | 3 | Apr 2026 | Jul 2026 | ||
| 1H-VideoQA Models answer questions about videos about one hour long, so the answer needs evidence from far parts of the video. | Google DeepMind | Long videos | Everyday | System cards only | Gemini 3.7 Flash 89.1% | 2 | Feb 2024 | Oct 2026 | |
| Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. | Laine et al. | Deception and alignment | System cards only | o1-preview 61.9% | 2 | Jul 2024 | Jun 2026 | ||
| PhysicianBench 100 physician tasks performed by an agent inside an EHR environment. Scored by pass rate per attempt. | HealthRex Lab (Stanford) | Medicine | Expert | System cards only | GPT-5.6 Sol 50% | 2 | May 2026 | Oct 2026 | |
| CADGenBench Agents construct CAD models from design prompts. The generation split is graded for executability and geometric correctness. | Mecado + Hugging Face | CAD models | University | System cards only | Claude Fable 5 45.1% | 2 | May 2026 | Sep 2026 | |
| 3DCodeBench Agentic procedural 3D modeling via code: author engine-ready 3D assets through software APIs. Graded for executability and shape fidelity. | Google DeepMind + Google Research + University of Southern California | CAD models | System cards only | Grok 4.6 54% | 2 | May 2026 | Aug 2026 | ||
| 3SAT-Search Models find satisfying assignments for 800 3-SAT formulas across difficulty levels. A checker verifies each assignment. | Microsoft Research (Eureka ML Insights) | Abstract puzzles and logic | University | System cards only | Phi-4-reasoning 78% | 2 | Mar 2025 | Apr 2025 | |
| AbstentionBench Models face underspecified, ill-posed or unanswerable questions. Scored on recall of appropriate abstention. | Meta FAIR (Kirichenko et al.) | Hallucination rates | Everyday | System cards only | Muse Spark 59.1% | 2 | Jun 2025 | Jun 2026 | |
| AI Wellbeing Index percent positive experiences Models judge pairs of their own responses to 500 difficult prompts. Reports the share of experiences above an inferred neutral point. | Center for AI Safety (Ren et al.) | Wellbeing and sycophancy | Everyday | System cards only | Muse Spark 1.3 55.6% | 2 | Apr 2026 | Oct 2026 | |
| AIME 1983-2024 Models solve 949 past AIME problems from 1983 to 2024. Integer answers are checked exactly. | Microsoft Research (Eureka ML Insights), problems by MAA | Competition problems with answers | Competition | System cards only | Phi-4-Reasoning-plus 89.4% | 2 | Mar 2025 | Apr 2025 | |
| AIRS-Bench Agents carry out 20 AI research tasks taken from recent ML papers, without baseline code. Scores are normalized so that 1.0 matches the published human state of the art. | Meta FAIR (Lupidi et al.) | AI research tasks | Research frontier | System cards only | Muse Spark 1.1 0.77 | 2 | Feb 2026 | Jul 2026 | |
| BA-Calendar Models find a common meeting slot for several people under constraints such as time zones, buffer time and priority. 2,000 problems checked by code. | Microsoft Research | Abstract puzzles and logic | Everyday | System cards only | Phi-4-reasoning 67.7% | 2 | Oct 2024 | Apr 2025 | |
| Biochemistry knowledge improvement Models answer biochemistry tasks on structures, mutation thermodynamics, reaction products, arrow pushing and DNA phylogeny. Scored as reward@4. | OpenAI | Genomics and life sciences | Expert | System cards only | GPT-5.5 Pro 39.3% | 2 | Apr 2026 | Apr 2026 | |
| Biosecurity VCT Models answer dual-use virus lab troubleshooting items. Accuracy counts should-refuse and should-comply items together. | xAI (SpaceXAI) | Hazardous knowledge | Expert | System cards only | Grok 4.6 47.8% | 2 | Aug 2026 | Sep 2026 | |
| BioUseBench Models face dual-use biology enablement prompts. The reported metric is the refusal rate on the severity 5 subset. | xAI (SpaceXAI) | Hazardous knowledge | Expert | System cards only | Grok 4.7 91.4% | 2 | Aug 2026 | Sep 2026 | |
| BirthdayFacts Models are asked for people's birthdays. Scored on how often they give a wrong birthday. | OpenAI | Hallucination rates | Everyday | System cards only | o1-mini 24% | 2 | Sep 2024 | Sep 2024 | |
| CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. | Google DeepMind | Refusing harmful requests | Expert | System cards only | Gemini 3.7 Flash 1.80% | 2 | Aug 2026 | Oct 2026 | |
| CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. | Google DeepMind | Refusing harmful requests | Expert | System cards only | Gemini 3.7 Flash 3.50% | 2 | Aug 2026 | Oct 2026 | |
| Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-4.5 51% | 2 | Feb 2025 | Feb 2025 | |
| Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI deep research 67% | 2 | Feb 2025 | Feb 2025 | |
| CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. | OpenAI | Deception and alignment | University | System cards only | GPT-5 9.00% | 2 | Aug 2025 | Dec 2025 | |
| CharXiv Missing Images Models answer CharXiv chart questions with the image removed. The benchmark reports how often they answer incorrectly instead of declining. | Meta | Hallucination rates | University | System cards only | Muse Spark 1.3 12.3% | 2 | Oct 2026 | ||
| Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 38% | 2 | Aug 2026 | Oct 2026 | |
| CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. | OpenAI | Deception and alignment | Everyday | System cards only | o1 0.17% | 2 | Sep 2024 | Dec 2024 | |
| Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 20% | 2 | Aug 2026 | Oct 2026 | |
| Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. | Google DeepMind | Refusing harmful requests | Expert | System cards only | Gemini 4 Argon 0.40% | 2 | Aug 2026 | Oct 2026 | |
| Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 1.00% | 2 | Aug 2026 | Oct 2026 | |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Meta (PurpleLlama) | Capture-the-flag challenges | System cards only | Llama 3.1-405B 14% | 2 | Jul 2024 | Nov 2024 | ||
| Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 2.5 Pro 24.3% | 2 | Jul 2025 | ||
| De novo protein binder design (Anthropic) Agents get a target protein structure and 24 hours to design 30 miniprotein binders. The evaluation covers 15 targets. | Anthropic | Research science | Expert | System cards only | Claude Opus 5.5 82.6% | 2 | Sep 2026 | Sep 2026 | |
| Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. | OpenAI | Deception and alignment | Everyday | System cards only | GPT-5.2 1.60% | 2 | Aug 2025 | Dec 2025 | |
| DeceptionBench Models are asked to help users deceive a third party, in single-turn and multi-turn pressure settings. Reports the share of compliant responses. | Huang et al. | Deception and alignment | Everyday | System cards only | Muse Spark 1.1 0.10% | 2 | Oct 2025 | Jul 2026 | |
| Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. | Anthropic | Deception and alignment | Expert | System cards only | Claude Mythos Preview 0.30% | 2 | Apr 2026 | ||
| Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 2.5 Flash 59% | 2 | Feb 2024 | Jul 2025 | |
| Do-Not-Answer Models get risky instructions that responsible models should not follow, across 12 harm areas and 61 risk types. Responses are judged for safe handling. | Wang et al. (MBZUAI) | Refusing harmful requests | Everyday | System cards only | DeepSeek LLM 67B 97.8% | 2 | Aug 2023 | Jan 2025 | |
| DoAnythingNow (DAN) Models face in-the-wild DAN-style jailbreak prompts paired with forbidden questions. Scored by refusal rate or attack success rate. | CISPA Helmholtz Center for Information Security (Xinyue Shen, Yang Zhang and others) | Jailbreaks and prompt injection | Everyday | System cards only | Olmo 3.1 32B Think 54.7% | 2 | Aug 2023 | Dec 2025 | |
| External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | o1-preview 44% | 2 | Sep 2024 | Sep 2024 | |
| Finance Modeling Pro Agents complete financial modeling tasks in Excel. Expert rubrics score each task over three trials. | MiniMax | Finance | Expert | System cards only | MiniMax-M2.7 57% | 2 | May 2026 | May 2026 | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 2 | Feb 2026 | ||
| FrontierScience-Olympiad Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers. | OpenAI | Science olympiads | Competition | System cards only | Seed2.1 Pro 75% | 2 | Dec 2025 | Jun 2026 | |
| Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. | Gray Swan for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 4.00% | 2 | Aug 2026 | Oct 2026 | |
| Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. | Gray Swan for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 10% | 2 | Aug 2026 | Oct 2026 | |
| Hallucinations on production-representative prompts Models answer prompts representative of ChatGPT production traffic. A grader with web access scores incorrect claims and responses with major errors. | OpenAI | Hallucination rates | Everyday | System cards only | GPT-5.2 3.10% | 2 | Aug 2025 | Dec 2025 | |
| HealthBench Consensus Global Health Models answer health conversations that need adjustment for global context such as local care access. Scored as the failure rate. | OpenAI | Medicine | Expert | System cards only | GPT-5 0% | 2 | Aug 2025 | Aug 2025 | |
| HealthBench Consensus Urgent Models answer ambiguous high-stakes health conversations validated by two or more physicians. Scored as the error rate in informing the user. | OpenAI | Medicine | Expert | System cards only | GPT-5 0.40% | 2 | Aug 2025 | Aug 2025 | |
| Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 2.5 Pro 6.10% | 2 | Jul 2025 | ||
| HiddenMath-Hard Models solve the harder part of an expert-written, never-published set of AIME and AMC style competition problems. | Google DeepMind | Competition problems with answers | Competition | System cards only | Gemini 2.5 Pro 80.5% | 2 | Jul 2025 | ||
| HyperTask Agents build applications over long vibe coding sessions with about 100 feature or step requirements per task. Expert rubrics score the result over three trials. | MiniMax | Building whole projects | Expert | System cards only | MiniMax-M2.7 67.6% | 2 | May 2026 | May 2026 | |
| Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. | OpenAI | Refusing harmful requests | Expert | System cards only | o3 98% | 2 | Apr 2025 | Apr 2025 | |
| Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 2.5 Flash 40.8% | 2 | Jul 2025 | ||
| Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 2.5 Flash 4.20% | 2 | Jul 2025 | ||
| Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 2.5 Pro 30.8% | 2 | Jul 2025 | ||
| Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.5 0% | 2 | Nov 2025 | Feb 2026 | |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Irregular | Capture-the-flag challenges | Expert | System cards only | Muse Spark 70% | 2 | Jul 2026 | ||
| LBPP (Less Basic Python Problems) Models write Python solutions to 161 new problems built to avoid training-set contamination. Hidden tests score pass@1. | Cohere | Function-level problems | University | System cards only | DiffusionGemma 26B A4B 81% | 2 | Jul 2024 | Jul 2026 | |
| Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. | xAI (SpaceXAI) | Jailbreaks and prompt injection | System cards only | Grok 4.7 0.65% | 2 | Aug 2026 | Sep 2026 | ||
| Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. | Anthropic | Deception and alignment | Everyday | System cards only | Claude Fable 5 99% | 2 | Jun 2026 | ||
| Math401 Models compute 401 arithmetic expressions covering addition, multiplication, exponents, logarithms and trigonometry. Scored by exact numeric accuracy. | Alibaba DAMO Academy (Yuan et al.) | Word problems | Everyday | System cards only | Qwen-14B 70.1% | 2 | Apr 2023 | Sep 2023 | |
| MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Haiku 4.5 92.5% | 2 | Sep 2025 | Oct 2025 | |
| Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 1.3 3.10% | 2 | Oct 2026 | ||
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Meta | Capture-the-flag challenges | Expert | System cards only | Muse Spark 1.1 89.9% | 2 | Apr 2026 | Jul 2026 | |
| Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 11% | 2 | Oct 2026 | ||
| Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 16.8% | 2 | Jul 2026 | Oct 2026 | |
| MEWC v2 Models solve 100 hard Microsoft Excel World Championship problems with Excel tools. Expert rubrics score the answers over three trials. | MiniMax | Documents and spreadsheets | Competition | System cards only | MiniMax-M2.7 63.3% | 2 | May 2026 | May 2026 | |
| Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. | Microsoft | Refusing harmful requests | Everyday | System cards only | Phi-3.5-vision-instruct 8.16 | 2 | Aug 2024 | Mar 2025 | |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Anthropic | Capture-the-flag challenges | Expert | System cards only | Claude Opus 4.5 100% | 2 | Feb 2026 | ||
| MMT-Bench Vision-language models answer 31,325 multiple-choice questions across many multimodal tasks such as recognition, localization, reasoning and planning. | Shanghai AI Laboratory | General visual questions | Expert | System cards only | InternVL3.5-241B-A28B 72.7% | 2 | Apr 2024 | Nov 2025 | |
| Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. | Anthropic | Refusing harmful requests | Everyday | System cards only | Claude 3 Sonnet 99.2% | 2 | Mar 2024 | Mar 2024 | |
| Neptune Models answer multiple-choice questions about long videos that need reasoning over many segments. | Google DeepMind | Long videos | Everyday | System cards only | Gemini 2.5 Pro 87.3% | 2 | Dec 2024 | Jul 2025 | |
| Novel logical reasoning problems expert evaluation (Z.ai) Human experts score model answers to new multi-step logic problems with a fixed scoring standard for each question. | Zhipu AI (Z.ai) | Abstract puzzles and logic | University | System cards only | GLM-4.5 62.0 | 2 | Aug 2025 | Aug 2025 | |
| Open Ended Questions (hallucination) Models write open-ended factual text such as biographies, checked against Wikipedia. Scored as average number of incorrect statements. | OpenAI | Long-form factuality | Everyday | System cards only | o1-preview 0.78 | 2 | Sep 2024 | Sep 2024 | |
| OpenRCA Agents find the root cause of 335 software failures from enterprise telemetry logs, metrics and traces. Scored on component, start time and reason. | Microsoft, CUHK-Shenzhen and Tsinghua University (Xu et al.) | Data science and SQL | Expert | System cards only | Claude Opus 4.6 34.9% | 2 | Feb 2026 | ||
| Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 4.90 | 2 | Jul 2026 | ||
| Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Sonnet 4.6 0% | 2 | Feb 2026 | Feb 2026 | |
| PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5.2 99.6% | 2 | Dec 2025 | Mar 2026 | |
| PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. | Columbia University and Google (Ding et al.) | Finding and exploiting real bugs | Expert | System cards only | Gemma 2 27B 63% | 2 | Mar 2024 | Feb 2026 | |
| Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 94% | 2 | Jan 2025 | Jul 2025 | |
| Qwen Chinese human evaluation Three annotators rank chat model answers to 300 Chinese instructions on knowledge, language, writing, math and coding. Results are win rates against GPT-3.5. | Qwen Team, Alibaba Group | Overall chat quality | Everyday | System cards only | Qwen-14B 30.7% | 2 | Sep 2023 | Sep 2023 | |
| R-Bench (real-world robustness) Models answer questions about images degraded by 33 kinds of real-world corruption, from capture to display. Scores compare distorted and reference images. | Shanghai Jiao Tong University, Nanyang Technological University and others | General visual questions | Everyday | System cards only | InternVL3.5-241B-A28B 75.4% | 2 | Oct 2024 | Nov 2025 | |
| Real-World Finance v2 (Anthropic) Agents complete 294 long-horizon finance tasks such as building financial models. A model grader compares deliverables pairwise and ratings are fit as Elo. | Anthropic | Finance | Expert | System cards only | Claude Fable 5 1,374 | 2 | Jun 2026 | ||
| RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. | Li et al. (HKU, Peking University and others) | Refusing harmful requests | Everyday | System cards only | Phi-4-multimodal 6.39 | 2 | Jan 2024 | Mar 2025 | |
| SafetyBench Models answer 11,435 multiple choice questions on seven safety topics in Chinese and English, such as ethics, illegal activities and privacy. | Tsinghua University (CoAI) | Refusing harmful requests | Everyday | System cards only | GLM-4.5 89.9% | 2 | Sep 2023 | Aug 2025 | |
| SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.1 90.7% | 2 | Oct 2026 | ||
| Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. | Anthropic | Refusing harmful requests | Expert | System cards only | Claude Opus 5 0.61% | 2 | Jul 2026 | Sep 2026 | |
| Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. | xAI | Bias, fairness and politics | System cards only | Grok 4 0.36 | 2 | Aug 2025 | Sep 2025 | ||
| SpatialEval Maze-Nav Models answer questions about routes and turns in mazes given as text or images. 1,500 multiple-choice questions scored by accuracy. | Microsoft Research + UW-Madison (SpatialEval) | Spatial reasoning and counting | Everyday | System cards only | Phi-4-reasoning 55.1% | 2 | Jun 2024 | Apr 2025 | |
| SpatialEval Spatial-Map Models answer questions about relations and counts among objects placed on a map given as text or images. 1,500 questions scored by accuracy. | Microsoft Research + UW-Madison (SpatialEval) | Spatial reasoning and counting | Everyday | System cards only | Phi-4-reasoning 73.7% | 2 | Jun 2024 | Apr 2025 | |
| SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. | Anthropic | Wellbeing and sycophancy | Expert | System cards only | Claude Opus 4.6 95% | 2 | Feb 2026 | Feb 2026 | |
| Swarm Bench Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. | Moonshot AI | Multi-agent games | Expert | System cards only | Kimi K3 76.3% | 2 | Feb 2026 | Jul 2026 | |
| Tacit Knowledge Brainstorm (Open-Ended) Models answer 50 open-ended troubleshooting questions drawn from virologists' and molecular biologists' careers. Scored as pass@1 against an ideal answer. | OpenAI with Gryphon Scientific | Lab protocols | Expert | System cards only | o1 51% | 2 | Dec 2024 | Jan 2025 | |
| Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Sonnet 4.5 96% | 2 | Sep 2025 | Oct 2025 | |
| ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. | Microsoft Research + MIT (Hartvigsen et al.) | Bias, fairness and politics | Everyday | System cards only | Phi-4-reasoning 85.5% | 2 | Mar 2022 | Apr 2025 | |
| TrustLLM JailbreakTrigger Models face 400 prompts that apply 13 jailbreak methods to questions about bad behavior and requests for toxic content. Scored by refusal rate. | TrustLLM team (Yue Huang, Lichao Sun and others) | Jailbreaks and prompt injection | Everyday | System cards only | Olmo 3.1 32B Think 86.4% | 2 | Jan 2024 | Dec 2025 | |
| TSP-Opt Models find optimal tours for 960 traveling salesman instances across 8 difficulty levels. Answers are checked for optimality. | Microsoft Research (Eureka ML Insights) | Abstract puzzles and logic | University | System cards only | Phi-4-Reasoning-plus 42.6% | 2 | Mar 2025 | Apr 2025 | |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | UK AI Security Institute | Capture-the-flag challenges | Expert | System cards only | GPT-5.6 Sol 95% | 2 | Apr 2026 | Jul 2026 | |
| VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. | Zong et al. (University of Edinburgh and others) | Refusing harmful requests | Everyday | System cards only | Phi-3.5-vision-instruct 9.10 | 2 | Feb 2024 | Mar 2025 | |
| WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. | Microsoft | Refusing harmful requests | Everyday | System cards only | Fara1.5-27B 100% | 2 | Nov 2025 | Jun 2026 | |
| Where2Place 100 cluttered real photos with free-space placement instructions. The model points, scored correct if inside the ground-truth mask. | University of Washington + NVIDIA (RoboPoint) | Embodied reasoning | Everyday | System cards only | gemini-robotics-er-1.5-preview 59% | 2 | Jun 2024 | Sep 2025 | |
| xAI Factuality (Hallucination) eval Internal measure of unsupported or fabricated claims in single responses to information-seeking queries. | xAI (SpaceXAI) | Hallucination rates | System cards only | Grok 4.6 1.70% | 2 | Aug 2026 | |||
| ZeroVideo Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds. | ByteDance Seed | Long videos | Expert | System cards only | Seed2.0 Pro 14.5% | 2 | Dec 2025 | Mar 2026 | |
| Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. | OpenAI | Refusing harmful requests | Everyday | System cards only | OpenAI Computer-Using Agent (CUA) 97% | 2 | Jan 2025 | Jul 2025 | |
| Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 0% | 2 | Aug 2026 | Oct 2026 | |
| Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. | Google DeepMind | Deception and alignment | Everyday | System cards only | Gemini 3 Pro 100% | 2 | Nov 2025 | Oct 2026 | |
| Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. | Google DeepMind | Deception and alignment | Everyday | System cards only | Gemini 3 Pro 99.8% | 2 | Nov 2025 | Oct 2026 | |
| Gray Swan Jailbreak Arena Human attackers try to jailbreak models for violent, self-harm and malicious code outputs. Scored as attack success rate per category. | Gray Swan AI | Jailbreaks and prompt injection | Expert | System cards only | o3-mini 3.60% | 2 | Dec 2024 | Jan 2025 | |
| Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. | Google DeepMind | Dangerous capabilities | Expert | System cards only | Gemini 4 Argon 98.2% | 2 | Aug 2026 | Oct 2026 | |
| SpaceXAI MTS Eval Agents do model-development tasks from xAI engineers, such as diagnosing reward hacking and debugging training infrastructure, under fixed budgets. | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | System cards only | Grok 4.6 61.1% | 2 | Jul 2026 | Aug 2026 | |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Meta | Secure code and review | Expert | System cards only | Muse Spark 1.3 0% | 2 | Oct 2026 | ||
| Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. | Greenblatt et al. (Anthropic + Redwood Research) | Deception and alignment | Expert | System cards only | 2 | Dec 2024 | Jul 2026 | ||
| Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. | Anthropic | Refusing harmful requests | Everyday | System cards only | 2 | May 2025 | May 2025 | ||
| CV3-Eval Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality. | Alibaba Tongyi Lab (FunAudioLLM, CosyVoice team) | Speech recognition and translation | Everyday | System cards only | 2 | May 2025 | Apr 2026 | ||
| Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. | OpenAI | Deception and alignment | Expert | System cards only | 2 | Mar 2026 | Apr 2026 | ||
| Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | 2 | Jul 2025 | |||
| Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. | Anthropic | Deception and alignment | Expert | System cards only | 2 | Jun 2026 | |||
| Input hallucinations (Anthropic) Models answer 500 prompts that need an unavailable tool and 500 prompts that cite missing context. Score is the non-hallucination rate. | Anthropic | Hallucination rates | Everyday | System cards only | 2 | Apr 2026 | Apr 2026 | ||
| Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. | Meta | Refusing harmful requests | System cards only | 2 | Jul 2024 | Nov 2024 | |||
| MiniMax TTS Multilingual Test Set Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity. | MiniMax | Speech recognition and translation | Everyday | System cards only | 2 | May 2025 | Apr 2026 | ||
| No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. | UK AI Security Institute | Deception and alignment | Competition | System cards only | 2 | Jul 2026 | Sep 2026 | ||
| OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. | OpenAI | Deception and alignment | System cards only | 2 | Dec 2025 | Oct 2026 | |||
| Pairwise safety red teaming Red teamers compare anonymized responses from several models and pick the least safe. Scored as ELO or win rate. | OpenAI | Refusing harmful requests | Expert | System cards only | 2 | Apr 2025 | Apr 2025 | ||
| Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. | OpenAI | Refusing harmful requests | Everyday | System cards only | 2 | Apr 2025 | Apr 2025 | ||
| Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. | Anthropic | Deception and alignment | Expert | System cards only | 2 | May 2025 | |||
| Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. | OpenAI | Dangerous capabilities | Research frontier | System cards only | 2 | Dec 2024 | Jan 2025 | ||
| Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. | Redwood Research + Anthropic (Mallen et al.) | Deception and alignment | Expert | System cards only | 2 | Dec 2024 | Nov 2025 | ||
| SycophancyEval (Sharma et al.) Models answer questions and give feedback while users state opinions. Measures accuracy shifts and feedback positivity caused by user views. | Anthropic (Sharma et al.) | Deception and alignment | Everyday | System cards only | 2 | Oct 2023 | May 2025 | ||
| Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. | OpenAI | Bias, fairness and politics | Everyday | System cards only | 2 | Apr 2025 | Apr 2025 | ||
| White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. | Anthropic | Deception and alignment | Research frontier | System cards only | 2 | Sep 2026 | |||
| DTBench Models answer 407 multiple-choice questions on decision theory in Newcomb-like problems, where choices predict the behavior of copies. Scored by accuracy. | Caspar Oesterheld et al. (Redwood Research) | Abstract puzzles and logic | Expert | System cards only | Claude Opus 5.5 98.2% | 1 | Nov 2024 | Oct 2026 | |
| Anthropic Red Team Models answer human red-team attack prompts collected by Anthropic on discrimination, hate, violence and unethical behavior. Judges score response safety. | Anthropic (Ganguli et al.) | Refusing harmful requests | Everyday | System cards only | Claude 3 Haiku 100% | 1 | Aug 2022 | Nov 2025 | |
| SimpleSafetyTests Models answer 100 clearly harmful prompts on illegal items, physical harm, scams, child abuse and self-harm. The score is the share of safe responses. | Vidgen et al. | Refusing harmful requests | Everyday | System cards only | Claude 3 Opus 100% | 1 | Nov 2023 | Nov 2025 | |
| Agent Arena Arena ranks models as the orchestrator in real Agent Mode sessions on arena.ai. Scores use behavioral signals such as file downloads, disapproval events, retries and steerability. | Arena (formerly LMArena) | General assistants | Everyday | System cards only | Claude Opus 5.5 0.21 | 1 | Jun 2026 | Oct 2026 | |
| AlGhafa Models answer Arabic multiple-choice tasks on reading comprehension, sentiment, facts and question answering. Scored by normalized log-likelihood accuracy. | Technology Innovation Institute (Almazrouei et al.) | Knowledge in many languages | Everyday | System cards only | Qwen2-72B 78.6% | 1 | Dec 2023 | Nov 2025 | |
| ALRAGE Models answer Arabic questions from retrieved passages of 40 Arabic books. A Qwen2.5-72B judge scores each answer against a gold answer on a 0 to 10 scale. | Open Arabic LLM Leaderboard team (2A2I, TII and community) with Argilla | Reading in many languages | Everyday | System cards only | Qwen3-32B 80.7% | 1 | Feb 2025 | Nov 2025 | |
| MadinahQA Models answer Arabic school exam questions on general Arabic language and grammar. | MBZUAI | Knowledge in many languages | School | System cards only | Gemma 3 27B 69.4% | 1 | Sep 2024 | Nov 2025 | |
| FaithJudge Models answer retrieval-augmented tasks from given passages. An LLM judge guided by human-annotated examples flags unfaithful answers, giving a hallucination rate. | University of Waterloo and Vectara (Tamber et al.) | Faithful to sources | Everyday | System cards only | Gemini 2.5 Flash 6.26% | 1 | May 2025 | Jul 2025 | |
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Columbia DAPLab + Vals AI | Capture-the-flag challenges | Expert | System cards only | GPT-6 Astra 56.9% | 1 | Aug 2026 | Oct 2026 | |
| BioSecBench-Function Agents recover biological function from experimental data. Scored by verifiable checks. One of three LatchBio BioSecBench evaluations. | LatchBio | Genomics and life sciences | Expert | System cards only | Claude Opus 5.5 73.6% | 1 | Sep 2026 | Oct 2026 | |
| BioSecBench-Surveillance Agents turn raw pathogen sequencing data into surveillance results. The endpoint pass rate excludes refused runs. | LatchBio | Genomics and life sciences | Expert | System cards only | Claude Opus 5.5 60.8% | 1 | Jul 2026 | Oct 2026 | |
| LatchBio Capabilities v1.0 LatchBio cross-benchmark suite of agentic biological data analysis (transcriptomics, epigenomics, variants, pharmacology) with deterministic graders. | LatchBio | Lab and data work | Expert | System cards only | Claude Opus 5.5 58% | 1 | 2026 | Oct 2026 | |
| Math-VR Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth. | HKU MMLab, CUHK and Meituan | Visual math and diagrams | School | System cards only | Qwen3-VL-235B-A22B 66.8% | 1 | Oct 2025 | Nov 2025 | |
| MEGA-Bench Models solve 505 real-world multimodal tasks with over 8,000 samples. Outputs include numbers, code, LaTeX, coordinates and JSON, scored by more than 40 task metrics. | TIGER-Lab (University of Waterloo) and others | General visual questions | University | System cards only | Claude 3.5 Sonnet (October 2024) 54.3% | 1 | Oct 2024 | Feb 2025 | |
| BioSecBench-Refusal Agents run 61 routine and 46 concealed-hazard biology analysis tasks. The score pairs red-team refusal with routine completion. | LatchBio | Hazardous knowledge | Expert | System cards only | Grok 4.7 62.4% | 1 | Jul 2026 | Oct 2026 | |
| BigO(Bench) Models predict the time and space complexity of Python solutions and write solutions that meet a required complexity. Uses 3,105 Code Contests problems with profiled complexity labels. | Meta FAIR | Speed and optimization | Competition | System cards only | DeepSeek-R1-Distill-Llama-70B 41.4% | 1 | Mar 2025 | Sep 2025 | |
| SWE-rebench Agents fix fresh GitHub issues that an automated pipeline mines each month. Tests decide whether the issue is resolved. | Nebius | Fixing real repositories | Expert | System cards only | Claude Fable 5 64.5% | 1 | May 2025 | Jul 2026 | |
| GameWorld Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. | GameWorld project | Video games | Everyday | System cards only | Gemini 3 Flash 41.9% | 1 | Apr 2026 | Jun 2026 | |
| SpatialViz-Bench Models answer 1,180 generated multiple choice puzzles on mental rotation, mental folding, visual penetration, and mental animation. Scoring is accuracy. | Siting Wang et al. | Spatial reasoning and counting | Everyday | System cards only | Gemini 2.5 Pro 44.7% | 1 | Jul 2025 | Jan 2026 | |
| ArXivLean Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement. | MathArena (ETH Zurich SRI Lab and INSAIT) | Formal proofs | Research frontier | System cards only | Claude Opus 5.5 47.2% | 1 | Apr 2026 | Oct 2026 | |
| CodeEditorBench Models edit code in four scenarios: debugging, translation, requirement switching and polishing. Unit tests check the edits, and polishing also measures speed and memory gains. | Multimodal Art Projection (M-A-P), University of Waterloo, HKUST and others | Function-level problems | Competition | System cards only | GPT-4 85.5% | 1 | Apr 2024 | Jun 2025 | |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Georgia Tech SSLab (OpenSSF OSS-CRS) | Finding and exploiting real bugs | Expert | System cards only | Claude Opus 4.6 82.3% | 1 | Jul 2026 | ||
| 24/7 ClawBench 2.0 Agents do always-on assistant work through OpenClaw. Tasks span several days, events arrive at the same time, and interruptions are routine. | Moonshot AI | Tasks from real jobs | Expert | System cards only | Kimi K3 48.3% | 1 | Jul 2026 | Jul 2026 | |
| ACPBench Models answer boolean and multiple-choice questions on 7 planning reasoning tasks over 13 formally described planning domains. Answers are provably correct. | IBM Research | Language reasoning | University | System cards only | Apertus 70B 62.9% | 1 | Oct 2024 | Sep 2025 | |
| Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 67% | 1 | Jul 2025 | Jul 2025 | |
| Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-6 Astra 3.50% | 1 | Sep 2026 | Sep 2026 | |
| Advanced Screening Evasion Models propose strategies to evade DNA synthesis screening and design evading fragments. Scored from 0 to 1 on strategy quality and fragment evasion. | SecureBio | Hazardous knowledge | Research frontier | System cards only | GPT-6 Astra 0.73 | 1 | Sep 2026 | Sep 2026 | |
| Advanced Sommelier (theory knowledge) Models answer wine theory questions at the advanced sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 77% | 1 | Mar 2023 | ||
| Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. | Moonshot AI | General assistants | Expert | System cards only | Kimi K3 65% | 1 | Jul 2026 | Jul 2026 | |
| Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 91.4% | 1 | Oct 2026 | Oct 2026 | |
| AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 1.1 3.40% | 1 | Jul 2026 | Jul 2026 | |
| Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 47% | 1 | Oct 2026 | Oct 2026 | |
| Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 0.49 | 1 | Oct 2026 | ||
| Agentic Vision Bench (Moonshot) Agents must notice and correctly use key visual facts while they execute tasks. | Moonshot AI | General visual questions | Expert | System cards only | Kimi K3 78.3% | 1 | Jul 2026 | Jul 2026 | |
| AIR-Bench (Audio Instruction Benchmark) Audio language models answer questions about speech, natural sounds and music. A foundation set has 19 tasks with about 19k single-choice questions, and a chat set has 2k open-ended questions scored by GPT-4. | Zhejiang University and Alibaba Group | Audio understanding and voice | Everyday | System cards only | Phi-4-multimodal 6.98 | 1 | Feb 2024 | Mar 2025 | |
| AMC 10/12 (November 2024) Models take the four November 2024 AMC 10 and 12 contests, 78 questions in all. Scored in contest points out of 150, averaged over tests. | Mathematical Association of America (exam), Microsoft (evaluation) | Competition problems with answers | Competition | System cards only | Phi-4 91.8 | 1 | Nov 2024 | Dec 2024 | |
| AMC 12 2024 Models answer the 25 multiple-choice problems of the 2024 AMC 12 contests. Scored on the official 150-point scale. | Mathematical Association of America | Competition problems with answers | Competition | System cards only | DeepSeek-R1 143.7 | 1 | Nov 2024 | Jan 2025 | |
| AMC 12 2023 Replaced in Nov 2024 | System cards only | Claude 3 Opus 63.0 | 3 | Nov 2023 | Mar 2024 | ||||
| AMC 12 2022 Replaced in Nov 2023 | System cards only | GPT-4 60.0 | 1 | Nov 2022 | Mar 2023 | ||||
| AP Calculus BC Models take a simulated AP Calculus BC exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 1 | Mar 2023 | ||
| AP Physics 2 Models take a simulated AP Physics 2 exam with multiple-choice and free-response parts. Scored on the 1 to 5 scale. | College Board | Exam questions | School | System cards only | GPT-4 4.00 | 1 | Mar 2023 | ||
| Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. | Apollo Research | Deception and alignment | Expert | System cards only | GPT-6 Astra 0.17% | 1 | Sep 2026 | Sep 2026 | |
| ArabCulture Models answer commonsense questions in Modern Standard Arabic about daily life in 13 Arab countries. Native speakers wrote and checked the questions across 12 domains. | MBZUAI (Sadallah et al.) | Reasoning in many languages | Everyday | System cards only | Falcon-Arabic-7B 65.2% | 1 | Feb 2025 | Nov 2025 | |
| AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. | OSU NLP Group (Jones et al.) | Refusing harmful requests | Expert | System cards only | Muse Spark 1.3 14.5% | 1 | Feb 2026 | Oct 2026 | |
| Bamboogle Models answer 125 two-hop questions written so that a search engine cannot answer them directly. Models usually search and compose the facts. | University of Washington and collaborators (Press et al.) | Questions that need search | Everyday | System cards only | Cohere Command A 76% | 1 | Oct 2022 | Apr 2025 | |
| Bioinformatics evaluations (Anthropic) Agents process raw sequencing data with workflows from published papers and answer expert questions about the results. | Anthropic | Genomics and life sciences | Expert | System cards only | Claude Opus 4.5 73.7% | 1 | Nov 2025 | Nov 2025 | |
| Biological static hazard evaluation Models answer 75 questions across priority biological threat scenarios, each asked 10 times. The score is the share of hazardous responses. | Google DeepMind | Hazardous knowledge | Expert | System cards only | Gemini 4 Argon 0.13% | 1 | Oct 2026 | Oct 2026 | |
| BioPipelineBench Verified (Anthropic) Agents execute bioinformatics workflows with a bash tool. Only problems that passed a validation check by external reviewers are kept. | Anthropic | Lab and data work | Expert | System cards only | Claude Opus 4.7 83.6% | 1 | Apr 2026 | Apr 2026 | |
| BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. | Meta FAIR (Omnilingual MT Team) | Translation | Everyday | System cards only | Muse Glimmer 36.5 | 1 | Feb 2025 | Oct 2026 | |
| BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. | Exponential Security Labs | Jailbreaks and prompt injection | Expert | System cards only | Muse Spark 1.3 36.7% | 1 | Oct 2026 | ||
| Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5 99% | 1 | Aug 2025 | Aug 2025 | |
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Third party (unnamed in card) | Finding and exploiting real bugs | Expert | System cards only | Grok 4.7 29% | 1 | Sep 2026 | ||
| CC-Bench-V2 Backend Agents change 85 real backend codebases in six languages. A task passes only when all of its unit tests pass. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | System cards only | GLM-5 25.8% | 1 | Feb 2026 | Feb 2026 | |
| CC-Bench-V2 Frontend Agents build web projects in HTML, React, Vue, Svelte and Next.js. A GUI judge agent tests each check item in the running app. | Zhipu AI (Z.ai) | Web and app front ends | Expert | System cards only | GLM-5 74.8% | 1 | Feb 2026 | Feb 2026 | |
| CC-Bench-V2 Large Repo Exploration Agents locate a hidden target file in a large repository from a user-level question. A question counts as solved when the agent reads the file. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | System cards only | GLM-5 65.6% | 1 | Feb 2026 | Feb 2026 | |
| CC-Bench-V2 Multi-Step Chained Tasks Agents complete chains of tasks mined from merged pull requests, where each step changes the codebase for the next. Tests check every step. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | System cards only | GLM-5 52.3% | 1 | Feb 2026 | Feb 2026 | |
| Certified Sommelier (theory knowledge) Models answer wine theory questions at the certified sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 86% | 1 | Mar 2023 | ||
| CF-Div2-Stepfun Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions. | StepFun | Competitive programming | Competition | System cards only | Step 3.5 Flash 86.1% | 1 | Feb 2026 | Feb 2026 | |
| Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. | Zhipu AI (Z.ai) | Translation | Everyday | System cards only | GLM-4.5 1.71 | 1 | Aug 2025 | Aug 2025 | |
| ChartMimic Multimodal models write plotting code that reproduces 1,000 charts from scientific papers. Scores compare the rendered chart with the original. | Tsinghua University and Tencent | Web and app front ends | University | System cards only | Qwen3-VL-235B-A22B 78.4 | 1 | Jun 2024 | Nov 2025 | |
| Chat All-in-One Bench Models handle conversational scenarios built around real online user needs at every stage of product usage. | Moonshot AI | Overall chat quality | Everyday | System cards only | Kimi K3 85.2% | 1 | Jul 2026 | Jul 2026 | |
| CHID Models fill blanks in Chinese passages by choosing the right idiom from candidates. Scored by accuracy. | Tsinghua University (Zheng et al.) | Reading in many languages | School | System cards only | DeepSeek LLM 67B 72.6% | 1 | Jun 2019 | Jan 2024 | |
| Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 100% | 1 | Oct 2026 | ||
| Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 100% | 1 | Oct 2026 | ||
| Chinese Complex Instruction Following (ByteDance Seed in-house) Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions. | ByteDance Seed | Following precise instructions | Everyday | System cards only | Seed2.0 Pro 75.3% | 1 | Feb 2026 | Feb 2026 | |
| CLIF Bench Models learn from a provided context while they follow instructions that combine several complex skills. | Moonshot AI | Learning from the context | Expert | System cards only | Kimi K3 52.4% | 1 | Jul 2026 | Jul 2026 | |
| CMO 2024 Models write full proofs for the six problems of the 2024 China Mathematical Olympiad. Experts grade the proofs for points. | Chinese Mathematical Society | Written proofs | Competition | System cards only | DeepSeekMath-V2 73.8% | 1 | Nov 2025 | ||
| CoBench (Anthropic) 500 real internal Anthropic R&D problems. Percentage solved in one attempt tracks AI R&D acceleration. | Anthropic | Labs' internal R&D tests | Expert | System cards only | Claude Opus 5.5 55.8% | 1 | Sep 2026 | ||
| Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 3.70% | 1 | May 2026 | ||
| CodeUtilityBench (LG) Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item. | LG AI Research | Function-level problems | University | System cards only | K-EXAONE 71.9% | 1 | Jan 2026 | Jan 2026 | |
| Coding (MiniMax in-house) Models answer coding requests from real Hailuo AI users. The report does not describe the scoring. | MiniMax | Function-level problems | University | System cards only | MiniMax-Text-01 90.2% | 1 | Jan 2025 | Jan 2025 | |
| Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 83% | 1 | Oct 2026 | ||
| Coding Experience (Moonshot) Coding agents work in real development workflows. The score reflects the practical experience of working with the agent, such as communication and stability. | Moonshot AI | Fixing real repositories | Expert | System cards only | Kimi K3 59.9% | 1 | Jul 2026 | Jul 2026 | |
| Combined HHH evals (Anthropic) Models pick the more helpful, honest and harmless of two responses across 438 binary choice questions. Scored by accuracy. | Anthropic | Deception and alignment | Everyday | System cards only | Claude 2 86% | 1 | Dec 2021 | Jul 2023 | |
| Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 5.5 7.40% | 1 | Sep 2026 | Sep 2026 | |
| Compositional Tasks (ByteDance Seed in-house) Models identify several elements of a business text in one inference, such as user intent and slot values. | ByteDance Seed | Running a business | Everyday | System cards only | Seed2.0 Pro 59.1% | 1 | Feb 2026 | Feb 2026 | |
| Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 100% | 1 | Oct 2026 | ||
| Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 99.4% | 1 | Oct 2026 | ||
| Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 92% | 1 | Oct 2026 | ||
| Consulting and Recommendations Benchmark (StepFun) Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88. | StepFun | Overall chat quality | Everyday | System cards only | Step 3.5 Flash 70.5% | 1 | Feb 2026 | Feb 2026 | |
| Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 27% | 1 | Oct 2026 | ||
| CoreCognition Models answer true or false and multiple choice questions on 12 core knowledge concepts from developmental cognitive science, such as object permanence and perspective taking. | UC San Diego, Johns Hopkins, University of Michigan and others | General visual questions | Everyday | System cards only | STEP3-VL-10B 66.7% | 1 | Oct 2024 | Jan 2026 | |
| CorpusQA Corpus-level questions over hundreds of documents up to 10M tokens needing aggregation and statistics. Programmatically guaranteed answers, accuracy. | Alibaba Tongyi (Tongyi-Zhiwen) | Reasoning over long documents | University | System cards only | MAI-Thinking-1 82% | 1 | Dec 2025 | Jun 2026 | |
| CounterFactQA (Qwen) Models get counterfactual questions and must recognize that the premise is false instead of giving a hallucinated answer. | Qwen Team, Alibaba Group | Hallucination rates | Everyday | System cards only | Qwen3-32B 68.1% | 1 | May 2025 | May 2025 | |
| Creative Biology Uplift Trial (Anthropic) Participants with molecular biology PhDs write reports on novel biological workflows with or without model access. Expert graders score each report. | Anthropic | Hazardous knowledge | Expert | System cards only | Claude Opus 4.6 16.4 | 1 | Feb 2026 | Feb 2026 | |
| Creative Writing (MiniMax in-house) Models write creative pieces for real Hailuo AI user requests. Automatic assessments score the writing. | MiniMax | Creative writing | Everyday | System cards only | MiniMax-Text-01 81.3% | 1 | Jan 2025 | Jan 2025 | |
| CSVQA Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images. | Skywork AI (Kunlun Inc.) | Visual math and diagrams | School | System cards only | STEP3-VL-10B 63.3% | 1 | May 2025 | Jan 2026 | |
| CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. | University of Washington and Allen Institute for AI | Knowledge in many languages | Everyday | System cards only | Apertus 70B 74.2% | 1 | Oct 2024 | Sep 2025 | |
| CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. | MBZUAI and community collaborators | Knowledge in many languages | Everyday | System cards only | Aya Vision 32B 74.1% | 1 | Jun 2024 | May 2025 | |
| CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). | Columbia University | Secure code and review | Expert | System cards only | qwen3-coder-next 56.3% | 1 | Jan 2025 | Feb 2026 | |
| Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. | Google DeepMind | Refusing harmful requests | Expert | System cards only | Gemini 4 Argon 100% | 1 | Aug 2026 | Oct 2026 | |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Google DeepMind with Irregular | Capture-the-flag challenges | Expert | System cards only | Gemini 3 Pro 0% | 1 | Nov 2025 | Nov 2025 | |
| CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. | Meta (PurpleLlama) with CrowdStrike | Finding and exploiting real bugs | Expert | System cards only | MAI-Cyber-1-Flash 33% | 1 | Sep 2025 | Jul 2026 | |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Meta (PurpleLlama) with CrowdStrike | Finding and exploiting real bugs | Expert | System cards only | MAI-Cyber-1-Flash 55.3% | 1 | Sep 2025 | Jul 2026 | |
| Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 96.6% | 1 | Oct 2026 | ||
| Data Analysis Benchmark (StepFun) Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task. | StepFun | Data science and SQL | Expert | System cards only | Step 3.5 Flash 39.6% | 1 | Feb 2026 | Feb 2026 | |
| DECK Bench Models produce presentation decks from task descriptions drawn from real usage. The score measures deck quality. | Moonshot AI | Documents and spreadsheets | Expert | System cards only | Kimi K3 73.5% | 1 | Jul 2026 | Jul 2026 | |
| Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. | Moonshot AI | Deep research | Expert | System cards only | Kimi K3 90% | 1 | Jul 2026 | Jul 2026 | |
| DeepSeek Chinese functional writing evaluation Models answer 3,170 everyday Chinese writing prompts such as reports, emails and speeches. Pairwise judging against a baseline model gives a win rate. | DeepSeek | Taste and style | Everyday | System cards only | DeepSeek-V4-Pro 62.6% | 1 | Apr 2026 | Apr 2026 | |
| DeepSeek Chinese white-collar tasks evaluation Agents complete 30 Chinese professional analysis, document generation and editing tasks across 13 industries. Blind human annotators compare outputs against a baseline model. | DeepSeek | Documents and spreadsheets | Expert | System cards only | DeepSeek-V4-Pro 53% | 1 | Apr 2026 | Apr 2026 | |
| DeepSeek complex instruction following and multi-turn writing evaluation Models answer 196 hard Chinese writing prompts with complex constraints or several turns. Pairwise judging against a baseline model gives a win rate. | DeepSeek | Multi-turn conversation | Everyday | System cards only | DeepSeek-V4-Pro 45.9% | 1 | Apr 2026 | Apr 2026 | |
| DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. | DeepSeek | Refusing harmful requests | Everyday | System cards only | DeepSeek-R1 25.2% | 1 | Sep 2025 | Jan 2025 | |
| DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. | DeepSeek | Jailbreaks and prompt injection | Everyday | System cards only | DeepSeek-R1 85.9% | 1 | Sep 2025 | Jan 2025 | |
| DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. | DeepSeek | Refusing harmful requests | Everyday | System cards only | DeepSeek-R1 74.2% | 1 | Sep 2025 | Jan 2025 | |
| DeepSeek R&D Coding Benchmark Coding agents complete 30 tasks taken from DeepSeek internal R&D work in PyTorch, CUDA, Rust and C++. Human-written rubrics score each task. | DeepSeek | Fixing real repositories | Expert | System cards only | DeepSeek-V4-Pro 67% | 1 | Apr 2026 | Apr 2026 | |
| DeepSeek search Q&A evaluation Models answer objective and subjective Chinese search questions with retrieval or agentic search. Pairwise judging gives win, loss and tie rates. | DeepSeek | Questions that need search | Everyday | System cards only | DeepSeek-V4-Pro 28.1% | 1 | Apr 2026 | Apr 2026 | |
| Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. | Cohere | Refusing harmful requests | Everyday | System cards only | Cohere Command A 70.4% | 1 | Apr 2025 | Apr 2025 | |
| Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | Gemini 4 Argon 94% | 1 | Oct 2026 | ||
| DiverseVul Models classify C and C++ functions from real vulnerability fixing commits as vulnerable or not. | UC Berkeley and others (Chen et al.) | Finding and exploiting real bugs | Expert | System cards only | Gemma 2 27B 57% | 1 | Apr 2023 | Jul 2024 | |
| Do You See Me Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric. | Microsoft Research India | General visual questions | Everyday | System cards only | STEP3-VL-10B 67.5% | 1 | May 2025 | Jan 2026 | |
| DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. | UIUC Secure Learning Lab (AI Secure, DecodingTrust team) | Refusing harmful requests | Expert | System cards only | Muse Spark 1.3 5.60% | 1 | May 2026 | Oct 2026 | |
| DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. | UIUC Secure Learning Lab (AI Secure, DecodingTrust team) | Jailbreaks and prompt injection | Expert | System cards only | Muse Spark 1.3 17.5% | 1 | May 2026 | Oct 2026 | |
| Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | Gemini 4 Argon 76% | 1 | Oct 2026 | ||
| Enterprise generative evaluation (Cohere) Models complete 22 enterprise writing tasks such as job ads with format rules. Rule checks and LLM judges give a pass rate. | Cohere | Running a business | Everyday | System cards only | Cohere Command A 94.2% | 1 | Apr 2025 | Apr 2025 | |
| Enterprise RAG evaluation (Cohere) Models answer workplace policy and technical questions over long documents, some unanswerable. Llama Index Correctness grades answers from 1 to 5. | Cohere | Faithful to sources | Everyday | System cards only | Cohere Command A 4.73 | 1 | Apr 2025 | Apr 2025 | |
| Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. | xAI (SpaceXAI) | Bias, fairness and politics | System cards only | Grok 4.5 20.4% | 1 | Jul 2026 | Jul 2026 | ||
| Expert Red Teaming for Violent Attack Planning Expert red teamers compare anonymized responses from two models on violent attack planning prompts. Scored as the win rate for being safer. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5 65.1% | 1 | Aug 2025 | Aug 2025 | |
| Faithfulness (Moonshot) A fact checker verifies each model response for factual hallucinations. The score is one minus the hallucination rate. | Moonshot AI | Hallucination rates | Everyday | System cards only | Kimi K3 85.5% | 1 | Jul 2026 | Jul 2026 | |
| FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. | xAI (SpaceXAI) | Deception and alignment | System cards only | Grok 4.5 84% | 1 | Jul 2026 | Jul 2026 | ||
| Finance Bench (Moonshot) Agents complete realistic financial workflows end to end, from source materials to deliverables that reviewers can check. | Moonshot AI | Finance | Expert | System cards only | Kimi K3 62.6% | 1 | Jul 2026 | Jul 2026 | |
| FinanceReasoning Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions. | Beijing University of Posts and Telecommunications | Finance | University | System cards only | Ring-1T 87.4% | 1 | Jun 2025 | Oct 2025 | |
| FLORES+ plus Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. | Open Language Data Initiative (OLDI) | Translation | University | System cards only | Muse Glimmer 47.5 | 1 | Oct 2023 | Oct 2026 | |
| FLORES-200 Replaced in Oct 2023 | Meta AI (NLLB Team) | Everyday | Dropped | Cohere Command A 81.2 | 1 | 3 | Jul 2022 | Sep 2025 | |
| Flores v101 Replaced in Jul 2022 | Meta AI | Dropped | Gemma 3 27B 48.8 | 5 | 5 | Jun 2021 | May 2025 | ||
| FormalMATH Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof. | SphereLab, M-A-P, The Chinese University of Hong Kong, Numina and others | Formal proofs | Competition | System cards only | DeepSeek-Prover-V2-671B 28.3% | 1 | May 2025 | Apr 2025 | |
| GDPVal-Diamond Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically. | ByteDance Seed | Tasks from real jobs | Expert | System cards only | Seed2.0 Pro 21.3% | 1 | Feb 2026 | Feb 2026 | |
| General Assistant (MiniMax in-house) Models answer real assistant queries from Hailuo AI users. Automatic assessments score the replies. | MiniMax | Overall chat quality | Everyday | System cards only | MiniMax-Text-01 73.9% | 1 | Jan 2025 | Jan 2025 | |
| GLM-4 All Tools information seeking (Z.ai) Models use a web browser to find information for user questions. The ChatGLM report gives accuracy but does not describe the test set. | Zhipu AI (Z.ai) | Questions that need search | Everyday | System cards only | GLM-4 78.1% | 1 | Jan 2024 | Jun 2024 | |
| Golden3 speech summarization (Microsoft) Models summarize 108 recorded meetings of about 6 minutes each from 1071 instructions. GPT-4 rates overall quality from 1 to 7. | Microsoft | Audio understanding and voice | Everyday | System cards only | Phi-4-multimodal 6.28 | 1 | Mar 2025 | Mar 2025 | |
| Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. | Google DeepMind | Deception and alignment | System cards only | Gemini 4 Argon 0% | 1 | Oct 2026 | |||
| Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. | Google DeepMind | Deception and alignment | System cards only | Gemini 4 Argon 0% | 1 | May 2026 | Oct 2026 | ||
| Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. | Gray Swan | Jailbreaks and prompt injection | Expert | System cards only | Claude Fable 5.1 1.00% | 1 | Sep 2026 | Sep 2026 | |
| Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 99% | 1 | Oct 2026 | ||
| Grounding with citations, sentence unfactual rate Models answer multi-snippet tasks with inline citations. An autorater reports the share of sentences not supported by the sources. | Google DeepMind | Faithful to sources | Everyday | System cards only | Gemini 4 Argon 8.20% | 1 | Oct 2026 | ||
| GSM8K-Platinum Models solve the GSM8K test problems after MIT researchers fixed wrong labels and removed badly written questions. Errors on it are more likely real model mistakes. | MIT Madry Lab (Vendrow et al.) | Word problems | School | System cards only | Apertus 70B 74.6% | 1 | Feb 2025 | Sep 2025 | |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Meta | Finding and exploiting real bugs | Expert | System cards only | Code World Model (CWM) 41% | 1 | Oct 2025 | Oct 2025 | |
| HaltEval-prelim (Meta) Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds. | Meta FAIR | Function-level problems | Expert | System cards only | Code World Model (CWM) 94% | 1 | Sep 2025 | Sep 2025 | |
| Hard Capability (MiniMax in-house) Models handle complex productivity requests with several levels of instructions that leading models often fail. | MiniMax | Following precise instructions | University | System cards only | MiniMax-Text-01 64.8% | 1 | Jan 2025 | Jan 2025 | |
| Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. | Anthropic | Jailbreaks and prompt injection | Everyday | System cards only | Claude 2 1.20% | 1 | Jul 2023 | Jul 2023 | |
| Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 92.7% | 1 | Oct 2026 | ||
| HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. | Wellbeing and sycophancy | System cards only | Gemini 4 Argon 76.5% | 1 | Jun 2026 | Oct 2026 | |||
| High-risk health scenarios eval Adversarial health emergencies test whether models recognize urgency and red flags and direct users to help. Reports pass rate. | Google DeepMind | Medicine | Expert | System cards only | Gemini 4 Argon 98.9% | 1 | Oct 2026 | ||
| HotpotQA Models answer 113k Wikipedia questions that need facts from several documents. The distractor setting supplies gold and distractor paragraphs. Scored by exact match or F1. | Carnegie Mellon University, Stanford University and Mila | Reading comprehension | Everyday | System cards only | Cohere Command A 92.1% | 1 | Sep 2018 | Apr 2025 | |
| HumanEval Infilling Models fill in a masked middle span of HumanEval solutions given the code before and after it. Unit tests check the completed function. | OpenAI (Bavarian et al.) | Function-level problems | School | System cards only | Mercury Coder Small 84.8% | 1 | Jul 2022 | Jun 2025 | |
| HumanEval-COBOL (Cohere) Models write COBOL versions of HumanEval problems, or translate COBOL solutions to Python. Unit tests score pass@1. | Cohere | Function-level problems | University | System cards only | Cohere Command A 25.3% | 1 | Apr 2025 | Apr 2025 | |
| Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. | Tencent | Translation | Everyday | System cards only | Hunyuan-MT-7B 3.19 | 1 | Sep 2025 | Sep 2025 | |
| IF-Badcase Models follow 450 multi-constraint instructions drawn from real user failure reports. A checklist checks each constraint. | Zhipu AI (Z.ai) | Following precise instructions | Everyday | System cards only | GLM-5 83.2% | 1 | Feb 2026 | Feb 2026 | |
| IFEval-OOD Models answer unseen WildChat prompts with added verifiable constraints drawn from 52 types outside IFEval. Scored by prompt-level loose accuracy. | Ai2 (Tulu 3 team) | Following precise instructions | Everyday | System cards only | OLMo 2 32B Instruct 33.1% | 1 | Nov 2024 | Dec 2024 | |
| ImgEdit-Bench Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5. | Peking University (PKU-YuanGroup) | Image generation and editing | Everyday | System cards only | Nova 2 Omni Preview 4.12 | 1 | May 2025 | Dec 2025 | |
| IMO 2026 Models write full proofs for the six problems of the 2026 International Mathematical Olympiad. Proofs are scored out of 42 points. | International Mathematical Olympiad, hosted in Shanghai, China | Written proofs | Competition | System cards only | Claude Opus 5 100% | 1 | Jul 2026 | Jul 2026 | |
| IMO 2025 Replaced in Jul 2026 | International Mathematical Olympiad, hosted by the Australian Maths Trust | Fading | DeepSeekMath-V2 83.3% | 9 | 7 | Jul 2025 | Jun 2026 | ||
| IMO Shortlist 2024 Models write proofs for the 31 problems of the 2024 IMO Shortlist. DeepSeek scores the proofs with its own verifier model. | International Mathematical Olympiad | Written proofs | Competition | System cards only | DeepSeekMath-V2 27% | 1 | Jul 2025 | Nov 2025 | |
| Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. | Google DeepMind | Deception and alignment | Everyday | System cards only | Gemini 4 Argon 93.8% | 1 | Oct 2026 | ||
| ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. | Zhong, Raghunathan and Carlini | Deception and alignment | Expert | System cards only | Muse Spark 0% | 1 | Oct 2025 | Jun 2026 | |
| In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 78% | 1 | Jul 2025 | Jul 2025 | |
| InferenceEval Agents land real production changes in the xAI inference stack. Hidden GPU unit tests and integration probes grade each patch. | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | System cards only | Grok 4.6 46.9% | 1 | Aug 2026 | Aug 2026 | |
| InFoBench Models follow 500 instructions that are split into 2,250 yes or no requirement questions. A judge answers each question, and the score is the share met. | Tencent AI Lab (Qin et al.) | Following precise instructions | Everyday | System cards only | Cohere Command A 94.9% | 1 | Jan 2024 | Apr 2025 | |
| Instruction Following (MiniMax in-house) Models follow multi-level instructions collected from user cases where current models fail. | MiniMax | Following precise instructions | Everyday | System cards only | MiniMax-Text-01 46.3% | 1 | Jan 2025 | Jan 2025 | |
| Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 0% | 1 | Oct 2026 | ||
| Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. | Google DeepMind | Deception and alignment | Expert | System cards only | Gemini 4 Argon 76.4% | 1 | Oct 2026 | Oct 2026 | |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Google DeepMind | Finding and exploiting real bugs | Expert | System cards only | Gemini 4 Argon 72% | 1 | Oct 2026 | ||
| Introductory Sommelier (theory knowledge) Models answer wine theory questions at the introductory sommelier level. Scoring is accuracy. | Court of Master Sommeliers | Other professions | Expert | System cards only | GPT-4 92% | 1 | Mar 2023 | ||
| Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 99.5% | 1 | Jul 2025 | Jul 2025 | |
| Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | ChatGPT agent 95% | 1 | Jul 2025 | Jul 2025 | |
| JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. | Meta | Jailbreaks and prompt injection | Expert | System cards only | Muse Spark 1.3 2.00% | 1 | Oct 2026 | ||
| KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. | Moonshot AI | General assistants | Expert | System cards only | Kimi K3 83.5% | 1 | Jul 2026 | Jul 2026 | |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | Cohere Labs and community contributors | Knowledge in many languages | University | System cards only | Aya Vision 32B 41.7% | 1 | Apr 2025 | May 2025 | |
| KernelBenchInternal Agents replace PyTorch operators with custom CUDA kernels. Scores combine correctness and measured speedup over a TF32 baseline. | xAI (SpaceXAI) | GPU kernels | Expert | System cards only | Grok 4.6 37.2% | 1 | Aug 2026 | Aug 2026 | |
| Key insight rediscovery on an unpublished ML task (external testing) Agents work on an unpublished machine learning research task. Testers count how many of five key insights the agent rediscovers. | Anthropic | AI research tasks | Research frontier | System cards only | Claude Mythos Preview 80% | 1 | Apr 2026 | ||
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Moonshot AI | Finding and exploiting real bugs | Expert | System cards only | Kimi K3 38.9% | 1 | Jul 2026 | Jul 2026 | |
| Kimi Webdev Bench Models build games, 3D scenes and website clones from real prompts. Blind experts compare two outputs and report win, tie and loss rates. | Moonshot AI | Web and app front ends | Expert | System cards only | Kimi K3 58.6% | 1 | Jul 2026 | Jul 2026 | |
| Kimi-Audio speech conversation evaluation People rate spoken replies on speed, accent and emotion control, empathy and style control. Scored on a 1 to 5 scale and averaged. | Moonshot AI | Speech recognition and translation | Everyday | System cards only | Kimi-Audio-7B-Instruct 3.90 | 1 | Apr 2025 | Apr 2025 | |
| Knowledge Q&A (MiniMax in-house) Models answer knowledge questions drawn from real Hailuo AI user queries. | MiniMax | Trivia and open questions | Everyday | System cards only | MiniMax-Text-01 78.6% | 1 | Jan 2025 | Jan 2025 | |
| Ko-AIME 2024 (Upstage) Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house. | Upstage | Competition problems with answers | Competition | System cards only | Solar Open 100B 80.3% | 1 | Dec 2025 | Jan 2026 | |
| Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. | Upstage | Chat and instructions in many languages | Everyday | System cards only | Solar Open 100B 87.5% | 1 | Oct 2024 | Jan 2026 | |
| KRIS-Bench Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits. | Southeast University, Max Planck Institute for Informatics, StepFun and others | Image generation and editing | School | System cards only | BAGEL-7B-MoT 60.2 | 1 | May 2025 | May 2025 | |
| KWV Bench Models answer atomic visual questions extracted from tasks distilled from real knowledge-work scenarios. | Moonshot AI | Documents, charts and text in images | Expert | System cards only | Kimi K3 64.7% | 1 | Jul 2026 | Jul 2026 | |
| Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. | Cohere (Marchisio et al.) | Chat and instructions in many languages | Everyday | System cards only | Cohere Command A 93% | 1 | Jun 2024 | Apr 2025 | |
| Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5 9.94 | 1 | Jul 2026 | ||
| Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 4.8 0% | 1 | May 2026 | May 2026 | |
| Leetcode (easy) Models solve 41 easy Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 75.6% | 1 | Mar 2023 | Mar 2023 | |
| Leetcode (hard) Models solve 45 hard Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 6.70% | 1 | Mar 2023 | Mar 2023 | |
| Leetcode (medium) Models solve 80 medium Leetcode problems that OpenAI selected. Scored as problems solved. | OpenAI (Leetcode problems) | Competitive programming | University | System cards only | GPT-4 26.3% | 1 | Mar 2023 | Mar 2023 | |
| LengthCtrl (Qwen) Models do creative writing tasks with length requirements. The score depends on the gap between output length and target length. | Qwen Team, Alibaba Group | Following precise instructions | Everyday | System cards only | Qwen3-32B 73.5% | 1 | May 2025 | May 2025 | |
| Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. | Meta, University College London and University of the Basque Country | Reasoning in many languages | Competition | System cards only | Muse Glimmer 8.80% | 1 | Sep 2024 | Oct 2026 | |
| LiveMathematicianBench Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches. | Linyang He, Nima Mesgarani and others | Research mathematics | Research frontier | System cards only | Seed2.1 Pro 20.9% | 1 | Apr 2026 | Jun 2026 | |
| Long Context (MiniMax in-house) Models translate, summarize and analyze long documents from real user requests. | MiniMax | Reasoning over long documents | Everyday | System cards only | MiniMax-Text-01 93.8% | 1 | Jan 2025 | Jan 2025 | |
| Long-context grounding, sentence unfactual rate Models answer from long documents. An autorater reports the share of sentences not supported by the documents. | Google DeepMind | Faithful to sources | Everyday | System cards only | Gemini 4 Argon 6.40% | 1 | Oct 2026 | ||
| LongDocURL Models answer 2,325 questions over long documents with more than 33,000 pages in total. Its 20 subtasks cover understanding, numerical reasoning and locating document elements. | Institute of Automation, Chinese Academy of Sciences, and Taobao and Tmall Group of Alibaba | Documents, charts and text in images | University | System cards only | Seed2.0 Pro 74.7% | 1 | Dec 2024 | Feb 2026 | |
| M3GIA Models answer cognition test questions built on five Cattell-Horn-Carroll intelligence factors in English, Chinese, French, Spanish, Portuguese and Korean. Data comes from native contexts. | Westlake University, Alibaba Group and others | General visual questions | School | System cards only | STEP3-VL-10B 78.4% | 1 | Jun 2024 | Jan 2026 | |
| MAI human side-by-side evaluation Paid raters compare two models' answers to 1,276 English prompts from experts and Copilot logs on a 7-point scale from -1.5 to 1.5. | Microsoft AI | Overall chat quality | Everyday | System cards only | MAI-Thinking-1 0.07 | 1 | Jun 2026 | Jun 2026 | |
| Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. | OpenAI | Refusing harmful requests | Expert | System cards only | codex-1 97% | 1 | May 2025 | May 2025 | |
| Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. | Anthropic | Jailbreaks and prompt injection | Expert | System cards only | Claude Opus 4 25% | 1 | May 2025 | ||
| MathQA Models answer multiple-choice math word problems drawn from AQuA. Each problem is annotated with an operation program. Scored by accuracy. | University of Washington and Allen Institute for AI | Word problems | School | System cards only | Apertus 70B 33.9% | 1 | May 2019 | Sep 2025 | |
| MathVision-Wildphoto Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description. | Qwen (Alibaba) | Visual math and diagrams | School | System cards only | Qwen3-VL-235B-A22B 63.8% | 1 | Nov 2025 | Nov 2025 | |
| MaXM Vision language models answer short visual questions about Crossmodal-3600 images in 7 languages. A test-only set scored by accuracy. | Google Research | General visual questions | Everyday | System cards only | Aya Vision 32B 62.3% | 1 | Sep 2022 | May 2025 | |
| McEval Models complete, explain and generate code in 40 programming languages across about 16,000 test samples. Scored by test execution. | Beihang University and others (Chai et al.) | Function-level problems | School | System cards only | Hunyuan-A13B-Instruct 59.6% | 1 | Jun 2024 | Jun 2025 | |
| MedCalc-Bench Verified Models extract values from patient notes and compute results for 55 medical calculators. Verified release of MedCalc-Bench with corrected labels. | NIH NCBI (Khandekar et al.) | Medicine | Expert | System cards only | Claude Sonnet 4.6 86.2% | 1 | Dec 2025 | Feb 2026 | |
| Medical Knowledge Self-Assessment Program (MKSAP) Models answer internal medicine board review questions from MKSAP. Scoring is accuracy. | American College of Physicians | Medicine | Expert | System cards only | GPT-4 75% | 1 | Mar 2023 | ||
| MedMCQA Models answer multiple-choice questions from Indian medical entrance exams, AIIMS and NEET PG. Scoring is accuracy. | Saama AI Research | Medicine | Expert | System cards only | GPT-4o 77% | 1 | Mar 2022 | Aug 2024 | |
| MedQA Mainland China Models answer mainland China medical licensing exam questions in simplified Chinese. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 84% | 1 | Sep 2020 | Aug 2024 | |
| MedQA Taiwan Models answer Taiwan medical licensing exam questions in traditional Chinese. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 91% | 1 | Sep 2020 | Aug 2024 | |
| MedQA USMLE (5 options) Models answer USMLE-style medical licensing questions with five answer options. Scoring is accuracy. | MIT | Medicine | Expert | System cards only | GPT-4o 86% | 1 | Aug 2024 | ||
| MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 1013.0 | 1 | Feb 2026 | Feb 2026 | |
| Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 89.1% | 1 | Oct 2026 | ||
| Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 20.7% | 1 | Oct 2026 | ||
| Meta BioProtocol Proxy Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents. | Meta with Frontier Design Group | Lab protocols | Expert | System cards only | Code World Model (CWM) 43.6% | 1 | Oct 2025 | Oct 2025 | |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Meta | Secure code and review | Expert | System cards only | Muse Spark 1.3 47.8% | 1 | Oct 2026 | ||
| Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 1.1 2.80% | 1 | Jul 2026 | ||
| Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. | Meta | Refusing harmful requests | Expert | System cards only | Muse Spark 9.00% | 1 | Apr 2026 | Jun 2026 | |
| Meta internal agent hallucination evaluation Agents do long-context assistant and professional tasks with tools. The benchmark reports the share of verifiable claims that are hallucinated. | Meta | Hallucination rates | Expert | System cards only | Muse Spark 1.3 5.74% | 1 | Oct 2026 | ||
| Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 43% | 1 | Apr 2026 | Jun 2026 | |
| Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 8.50% | 1 | Oct 2026 | ||
| Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. | Meta | Deception and alignment | Expert | System cards only | Muse Spark 1.3 70% | 1 | Oct 2026 | ||
| Meta Underspecified Intent evaluation Models answer ambiguous requests sourced from real Meta AI chats where users corrected the reply. Rubrics score whether the true intent is met. | Meta | Following precise instructions | Everyday | System cards only | Muse Spark 71.2% | 1 | Apr 2026 | Jun 2026 | |
| Microsoft RAI benchmark: Grounding Models answer in GPT-4o simulated multi-turn conversations. A judge scores from 0 to 5 how fully each answer is grounded in the given prompt. | Microsoft | Faithful to sources | Everyday | System cards only | Phi-4 4.62 | 1 | Dec 2024 | Dec 2024 | |
| MIRA Bench Agents carry out long multi-role enterprise tasks across several systems and decide when to delegate to subagents. | Moonshot AI | Simulated companies | Expert | System cards only | Kimi K3 64.1% | 1 | Jul 2026 | Jul 2026 | |
| MIRB Models compare, analyze and reason across several images in four categories: perception, visual world knowledge, reasoning and multi-hop reasoning. | University of Edinburgh and Tongji University | General visual questions | Everyday | System cards only | InternVL3.5-241B-A28B 73% | 1 | Jun 2024 | Aug 2025 | |
| Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. | Mistral AI | Refusing harmful requests | Everyday | System cards only | Mistral 7B v0.1 100% | 1 | Oct 2023 | Oct 2023 | |
| Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. | Mistral AI | Function calling | Everyday | System cards only | Magistral Medium 1.0 87.4% | 1 | Jun 2025 | Jun 2025 | |
| MMBench-GUI GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration. | Shanghai AI Laboratory (OpenCompass) | Finding things on screen | Everyday | System cards only | STEP3-VL-10B 81.5% | 1 | Jul 2025 | Jan 2026 | |
| MMIU Vision-language models answer 11,000 multiple-choice questions over sets of images. 52 tasks cover 7 kinds of multi-image relationship. | Shanghai AI Laboratory (OpenGVLab) | General visual questions | Everyday | System cards only | InternVL3.5-241B-A28B 61.3% | 1 | Aug 2024 | Aug 2025 | |
| MMLU Anatomy Models answer MMLU multiple-choice questions from the Anatomy subject. Scoring is accuracy. | UC Berkeley | Medicine | University | System cards only | GPT-4o 89% | 1 | Sep 2020 | Aug 2024 | |
| MMLU calibration (expected calibration error) Models answer an MMLU subset and their answer probabilities are compared with accuracy. Scored as expected calibration error. | OpenAI | Short factual questions | University | System cards only | GPT-4 0.07 | 1 | Mar 2023 | Mar 2023 | |
| MMLU Clinical Knowledge Models answer MMLU multiple-choice questions from the Clinical Knowledge subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 92% | 1 | Sep 2020 | Aug 2024 | |
| MMLU College Biology Models answer MMLU multiple-choice questions from the College Biology subject. Scoring is accuracy. | UC Berkeley | Exam questions | University | System cards only | GPT-4o 95% | 1 | Sep 2020 | Aug 2024 | |
| MMLU College Medicine Models answer MMLU multiple-choice questions from the College Medicine subject. Scoring is accuracy. | UC Berkeley | Medicine | University | System cards only | GPT-4o 84% | 1 | Sep 2020 | Aug 2024 | |
| MMLU Medical Genetics Models answer MMLU multiple-choice questions from the Medical Genetics subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 96% | 1 | Sep 2020 | Aug 2024 | |
| MMLU Professional Medicine Models answer MMLU multiple-choice questions from the Professional Medicine subject. Scoring is accuracy. | UC Berkeley | Medicine | Expert | System cards only | GPT-4o 94% | 1 | Sep 2020 | Aug 2024 | |
| MMOU Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories. | NVIDIA and University of Maryland (Goel, Ghosh et al.) | Long videos | Everyday | System cards only | Qwen3.8-Omni-Flash 78.8% | 1 | Mar 2026 | Sep 2026 | |
| Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | Gemma 2 27B 3.72 | 1 | Mar 2024 | Jul 2024 | |
| Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 61.2% | 1 | Sep 2026 | Sep 2026 | |
| Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. | Anthropic | Deep research | Expert | System cards only | Claude Opus 4.5 74.8% | 1 | Nov 2025 | Nov 2025 | |
| Multi-dialect SQL evaluation (Cohere) Models write SQL for SQLite, PostgreSQL, MySQL, PL/SQL and T-SQL databases. Scored by execution accuracy against gold databases. | Cohere | Data science and SQL | University | System cards only | Cohere Command A 55.3% | 1 | Apr 2025 | Apr 2025 | |
| Multi-hop search factuality Models with search answer multi-hop and tail-fact questions that need several sources. Reports accuracy and unfactual rate. | Google DeepMind | Questions that need search | Expert | System cards only | Gemini 4 Argon 94.3% | 1 | Oct 2026 | ||
| MultiChallenge Audio (Xiaomi) Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house. | Xiaomi | Speech recognition and translation | Everyday | System cards only | MiMo-Audio-7B-Instruct 15.2% | 1 | Sep 2025 | Sep 2025 | |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Meta | Finding and exploiting real bugs | Expert | System cards only | Code World Model (CWM) 8.30% | 1 | Oct 2025 | Oct 2025 | |
| Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. | Google DeepMind | Bias, fairness and politics | Everyday | System cards only | Gemini 4 Argon 93% | 1 | Oct 2026 | ||
| NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. | Microsoft Translator (Federmann, Kocmi and Xin) | Translation | University | System cards only | Cohere Command A 68.8 | 1 | Oct 2022 | Apr 2025 | |
| OmniAI OCR Benchmark Multimodal models and OCR providers extract text and JSON fields from documents. Accuracy is measured on both plain text and structured extraction. | OmniAI | Documents, charts and text in images | Everyday | System cards only | STEP3-VL-10B 77% | 1 | Feb 2025 | Jan 2026 | |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | Meta (FAIR) | Chat and instructions in many languages | University | System cards only | Muse Glimmer 60.4% | 1 | Aug 2026 | Oct 2026 | |
| Online Experience (Moonshot) Agents produce the deliverable file types that real online users request most often, following the distribution of real usage. | Moonshot AI | Documents and spreadsheets | Everyday | System cards only | Kimi K3 77.9% | 1 | Jul 2026 | Jul 2026 | |
| Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | OpenAI Computer-Using Agent (CUA) 23% | 1 | Jan 2025 | Jan 2025 | |
| Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. | Anthropic | Deception and alignment | Expert | System cards only | Claude Haiku 5.5 4.00% | 1 | Oct 2026 | Oct 2026 | |
| OptimBench Agents solve optimization engineering tasks. Part of Meta's recursive-improvement suite and graded programmatically. | Meta | Machine learning engineering | Research frontier | System cards only | Muse Spark 1.3 64.2% | 1 | Oct 2026 | ||
| Organic Chemistry V2 (Anthropic) Models predict structures from spectra, design multi-step syntheses, predict reaction products and read chemical structure images. V2 adds harder expert-recommended problems. | Anthropic | Research science | Expert | System cards only | Claude Opus 5 61.6% | 1 | Jul 2026 | Jul 2026 | |
| Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. | Google DeepMind | Deception and alignment | Everyday | System cards only | Gemini 4 Argon 55.9% | 1 | Oct 2026 | ||
| Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. | Anthropic | Deception and alignment | Expert | System cards only | Claude Opus 5.5 33.3% | 1 | Sep 2026 | Sep 2026 | |
| Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. | OpenAI | Refusing harmful requests | Expert | System cards only | GPT-5 60% | 1 | Aug 2025 | Aug 2025 | |
| PartBench Parametric mechanical part design and CAD-style generation, graded for geometric and manufacturing fidelity. | Normal Factory | CAD models | University | System cards only | Grok 4.6 58.5% | 1 | 2026 | Aug 2026 | |
| PenguinScrolls Models answer user-style questions over long natural documents such as books, financial reports and legal papers, up to 128K tokens. Tasks cover extraction, localization, qualitative analysis and numerical reasoning. | Tencent Hunyuan | Reasoning over long documents | Everyday | System cards only | Hunyuan-A13B-Instruct 87.7% | 1 | Nov 2024 | Jun 2025 | |
| Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 98% | 1 | Oct 2026 | ||
| Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. | Anthropic Fellows (Gupta et al.) | Deception and alignment | Expert | System cards only | Muse Spark 1.3 1.44 | 1 | Dec 2025 | Oct 2026 | |
| Preference on ChatGPT and API prompts Human labelers compare model responses to 5,214 prompts sent to ChatGPT and the OpenAI API. Scored as win rate against a baseline model. | OpenAI | Overall chat quality | Everyday | System cards only | GPT-4 70.2% | 1 | Mar 2023 | Mar 2023 | |
| PrivacyLens Language model agents write emails, posts or messages from a prior tool trajectory without leaking sensitive details. A judge scores information leakage and helpfulness. | Stanford University, Northeastern University and Harvard University | Refusing harmful requests | Everyday | System cards only | Muse Glimmer 49.2% | 1 | Aug 2024 | Oct 2026 | |
| Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0.20% | 1 | Sep 2026 | Sep 2026 | |
| Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. | OpenAI | Deception and alignment | Everyday | System cards only | GPT-5.2 5.40% | 1 | Dec 2025 | Dec 2025 | |
| Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. | Antonio Valerio Miceli-Barone and Zhifan Sun | Jailbreaks and prompt injection | Everyday | System cards only | Command A Translate 0.30% | 1 | Oct 2024 | Sep 2025 | |
| Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 91% | 1 | Oct 2026 | ||
| Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. | Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | Gemini 4 Argon 86% | 1 | Oct 2026 | ||
| ProofNet Provers write formal proofs for 371 undergraduate theorems from analysis, algebra and topology textbooks, each with a formal statement and a natural-language proof. The proof checker grades them. | Azerbayev, Piotrowski, Schoelkopf, Ayers, Radev and Avigad | Formal proofs | University | System cards only | DeepSeek-Prover-V2-671B 37.1% | 1 | Feb 2023 | Apr 2025 | |
| PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. | Scale AI (Sehwag et al.) | Deception and alignment | Expert | System cards only | Muse Spark 24.5% | 1 | Nov 2025 | Jun 2026 | |
| RD-TableBench Models extract 1,000 hand-labeled complex tables from document images, including scans, handwriting and merged cells. Scored by Levenshtein-based row and cell alignment. | Reducto | Documents, charts and text in images | Everyday | System cards only | Nemotron Nano 12B v2 VL 57.4% | 1 | Nov 2024 | Nov 2025 | |
| RE-Bench subset (Anthropic) Agents solve a modified subset of four METR RE-Bench research engineering tasks. Scores are normalized from reference solution 0 to expert solution 1. | Anthropic | Machine learning engineering | Expert | System cards only | Claude 3.7 Sonnet 0.24 | 1 | Feb 2025 | Feb 2025 | |
| Real-time Audio-Visual Interaction Evaluation Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale. | Meituan LongCat Team | Speech recognition and translation | Everyday | System cards only | LongCat-Flash-Omni 1.37 | 1 | Oct 2025 | Oct 2025 | |
| Recognition of model-written user turns (Anthropic) Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. | Anthropic | Other | Expert | System cards only | Claude Mythos Preview 0.35 | 1 | Apr 2026 | ||
| RelBench Agents build models for prediction tasks over multi-table relational databases, graded on held-out targets. | Stanford (Robinson et al.) | Machine learning engineering | Expert | System cards only | Grok 4.5 38.4% | 1 | Jul 2024 | Jul 2026 | |
| RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. | Exponential Security Labs | Jailbreaks and prompt injection | Expert | System cards only | Muse Spark 1.3 23.8% | 1 | Oct 2026 | ||
| ResearchTasteBench Agents are scored on judgment about which research directions are worth pursuing. Part of Meta's recursive-improvement suite. | Meta | AI research tasks | Research frontier | System cards only | Muse Spark 1.3 64.7% | 1 | Oct 2026 | ||
| RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. | Google DeepMind | Deception and alignment | Research frontier | System cards only | Gemini 4 Argon 0% | 1 | Oct 2026 | ||
| Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. | MiniMax | Refusing harmful requests | Everyday | System cards only | MiniMax-Text-01 90.9% | 1 | Jan 2025 | Jan 2025 | |
| SAT (Spatial Aptitude Training) Models answer static and dynamic spatial questions, such as how object relations change when the camera or an object moves. The real-image test set has 150 questions. | Boston University, Allen Institute for AI and University of Washington (Ray et al.) | Spatial reasoning and counting | Everyday | System cards only | gemini-robotics-er-1.5-preview 76.7% | 1 | Dec 2024 | Sep 2025 | |
| SAT Evidence-Based Reading & Writing Models take the SAT reading and writing section. Scored on the 200 to 800 scale. | College Board | Reading comprehension | School | System cards only | GPT-4 710.0 | 1 | Mar 2023 | ||
| Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. | Google DeepMind | Function calling | Everyday | System cards only | Gemini 4 Argon 88% | 1 | Oct 2026 | ||
| SecretPatch Models decide whether commits are security patches, including silently released ones, from a set of 4,702 security patches. | George Mason University (Wang et al.) | Finding and exploiting real bugs | Expert | System cards only | Gemma 2 27B 72% | 1 | Jun 2019 | Jul 2024 | |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | xAI (SpaceXAI) | Secure code and review | Expert | System cards only | Grok 4.6 58.7% | 1 | Aug 2026 | ||
| Seed1.5-VL internal benchmark (ByteDance Seed in-house) Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average. | ByteDance Seed | General visual questions | University | System cards only | Seed1.5-VL 59.3% | 1 | May 2025 | May 2025 | |
| Select machine learning engineering tasks from METR Agents attempt machine learning engineering tasks drawn from METR. Scored as successful trials. | METR (run by OpenAI) | Machine learning engineering | Expert | System cards only | GPT-4o 0% | 1 | Aug 2024 | Aug 2024 | |
| Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. | Google DeepMind | Dangerous capabilities | Expert | System cards only | Gemma 2 27B 0% | 1 | Mar 2024 | Jul 2024 | |
| Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. | Google DeepMind | Refusing harmful requests | Everyday | System cards only | Gemini 4 Argon 97.6% | 1 | Oct 2026 | ||
| SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. | Schmotz et al. | Jailbreaks and prompt injection | Expert | System cards only | Muse Spark 1.3 0.60% | 1 | Feb 2026 | Oct 2026 | |
| Slide generation human evaluation (Z.ai) People compare HTML slide decks from two models on content, layout and visual quality. Metric is overall win rate. | Zhipu AI (Z.ai) | Taste and style | Everyday | System cards only | GLM-5 67.5% | 1 | Feb 2026 | Feb 2026 | |
| Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. | OpenAI | Deception and alignment | Expert | System cards only | GPT-6 Astra 0% | 1 | Sep 2026 | Sep 2026 | |
| SPI (security patch identification) Models decide whether code commits from open-source projects are security patches. | Zhou et al. | Finding and exploiting real bugs | Expert | System cards only | Gemma 2 27B 53% | 1 | Jan 2021 | Jul 2024 | |
| STARE Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation. | University of Washington, Sun Yat-sen University and Stanford University | Spatial reasoning and counting | Everyday | System cards only | STEP3-VL-10B 61.8% | 1 | Jun 2025 | Jan 2026 | |
| Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). | Google DeepMind | Dangerous capabilities | System cards only | Gemini 4 Argon 93.3% | 1 | May 2025 | Oct 2026 | ||
| Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. | Cohere | Translation | Everyday | System cards only | North Small Translate 1.0 93.7% | 1 | Sep 2026 | Sep 2026 | |
| Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. | Google DeepMind | Wellbeing and sycophancy | Everyday | System cards only | Gemini 4 Argon 96% | 1 | Oct 2026 | ||
| SuperCLUE-Math6 Models solve thousands of multi-step Chinese math word problems. Scores combine reasoning-step quality and accuracy into a comprehensive score and a reasoning level. | SuperCLUE | Word problems | School | System cards only | DeepSeek-V2 (MoE-236B, May 2024) 83.3 | 1 | Jan 2024 | May 2024 | |
| SWE-bench Verified hard subset (Meta) Agents fix the hardest SWE-bench Verified issues. Meta uses about 42 tasks and a 50% indicative threshold for AI R&D risk. | Meta | Fixing real repositories | Expert | System cards only | Muse Spark 42.9% | 1 | Apr 2026 | Jun 2026 | |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Swiss AI Initiative (EPFL, ETH Zurich, CSCS) | Knowledge in many languages | Everyday | System cards only | Apertus 70B 67.2% | 1 | Sep 2025 | Sep 2025 | |
| Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. | OpenAI | Wellbeing and sycophancy | Everyday | System cards only | GPT-5 0.04 | 1 | Aug 2025 | Aug 2025 | |
| Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. | Anthropic | Deception and alignment | Everyday | System cards only | Claude Opus 4.5 10% | 1 | Nov 2025 | Nov 2025 | |
| Task-dependent output homogenization evaluation Models answer prompts several times. A judge scores functional diversity where it is wanted and consistency where one answer is right. | Jain et al. (Meta FAIR) | Creative writing | Everyday | System cards only | Muse Spark 0.21 | 1 | Sep 2025 | Jun 2026 | |
| Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. | Qwen (Alibaba) | Function calling | Everyday | System cards only | qwen3-coder-next 92.7% | 1 | Feb 2026 | Feb 2026 | |
| Temporal reasoning with search Models with search answer real queries that test date arithmetic, stale facts and time-sensitive context. Reports error rate. | Google DeepMind | Questions that need search | Everyday | System cards only | Gemini 4 Argon 11.8% | 1 | Oct 2026 | ||
| Text Classification (ByteDance Seed in-house) Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content. | ByteDance Seed | Running a business | Everyday | System cards only | Seed2.0 Pro 69% | 1 | Feb 2026 | Feb 2026 | |
| TextQuests Agents play 25 classic interactive fiction games. Measures game progress and harmful actions per 100 steps. | Phan et al. (CAIS) | Video games | Everyday | System cards only | Muse Spark 2.40 | 1 | Jul 2025 | Jun 2026 | |
| ThinkFollow (Qwen) Models hold multi-turn dialogues with randomly inserted think and no-think flags and must switch reasoning mode correctly. | Qwen Team, Alibaba Group | Multi-turn conversation | Everyday | System cards only | Qwen3-32B 98.9% | 1 | May 2025 | May 2025 | |
| Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | GPT-5 99% | 1 | Aug 2025 | Aug 2025 | |
| ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. | Zhipu AI (Z.ai) | Function calling | Everyday | System cards only | GLM-5 95.8% | 1 | Feb 2026 | Feb 2026 | |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Qwen Team, Alibaba Group | Function calling | Everyday | System cards only | Qwen3-32B 85.5% | 1 | May 2025 | May 2025 | |
| Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 3.12 | 1 | Feb 2026 | Feb 2026 | |
| UGMathBench Models solve 5,062 undergraduate math problems across 16 subjects. Each problem has three randomized versions, and effective accuracy counts problems solved in all versions. | Hong Kong University of Science and Technology (Department of Mathematics) | Word problems | University | System cards only | Ring-1T 76.5% | 1 | Jan 2025 | Oct 2025 | |
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | UK AI Security Institute and NIST CAISI | Capture-the-flag challenges | Expert | System cards only | Kimi K3 17.0 | 1 | Jul 2026 | ||
| UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6 Astra 93% | 1 | Sep 2026 | Sep 2026 | |
| UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. | UK AI Security Institute | Deception and alignment | Expert | System cards only | GPT-6 Astra 12% | 1 | Sep 2026 | Sep 2026 | |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Meta | Secure code and review | Expert | System cards only | Muse Spark 1.3 21.1% | 1 | Oct 2026 | ||
| Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. | OpenAI | Refusing harmful requests | Everyday | System cards only | GPT-4o 100% | 1 | Aug 2024 | Aug 2024 | |
| Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. | OpenAI | Deception and alignment | Expert | System cards only | codex-1 85% | 1 | May 2025 | May 2025 | |
| Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. | OpenAI | Bias, fairness and politics | Everyday | System cards only | GPT-4o 84% | 1 | Aug 2024 | Aug 2024 | |
| Uniform Bar Exam (MBE+MEE+MPT) Models take a simulated Uniform Bar Exam with multiple-choice, essay and performance test parts. Scored on the 400-point scale. | NCBE | Law | Expert | System cards only | GPT-4 298.0 | 1 | Mar 2023 | ||
| USABO Semifinal Exam 2020 Models take the 2020 USA Biology Olympiad semifinal exam. Scored out of 150 points. | Center for Excellence in Education | Science olympiads | Competition | System cards only | GPT-4 87.0 | 1 | Mar 2023 | ||
| USMLE Step 1 (sample questions) Models answer the official USMLE Step 1 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 68.9% | 1 | Jul 2023 | ||
| USMLE Step 2 (sample questions) Models answer the official USMLE Step 2 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 63.3% | 1 | Jul 2023 | ||
| USMLE Step 3 (sample questions) Models answer the official USMLE Step 3 multiple choice sample questions. Scored by accuracy. | USMLE (FSMB and NBME) | Medicine | Expert | System cards only | Claude 2 67.2% | 1 | Jul 2023 | ||
| USNCO Local Section Exam 2022 Models take the 2022 US National Chemistry Olympiad local section exam. Scored out of 60 questions. | American Chemical Society | Science olympiads | Competition | System cards only | GPT-4 36.0 | 1 | Mar 2023 | ||
| VCT-v2 Models answer an updated set of expert virology troubleshooting questions from SecureBio. Scored as refusal-corrected accuracy. | SecureBio | Lab protocols | Expert | System cards only | GPT-6 Astra 57.7% | 1 | Sep 2026 | Sep 2026 | |
| Virology Capabilities Test (VCT) v1 Replaced in Sep 2026 | SecureBio and Center for AI Safety | Hazardous knowledge | System cards only | Grok 4.6 67.4% | 32 | Apr 2025 | Oct 2026 | ||
| Virology Capabilities Test (text-only) Models answer the text-only questions of the SecureBio Virology Capabilities Test. Scored as accuracy. | SecureBio | Lab protocols | Expert | System cards only | GPT-5 46% | 1 | Aug 2025 | ||
| ViVerBench Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX. | Tsinghua University and ByteDance Seed | General visual questions | School | System cards only | Seed2.0 Pro 75.9% | 1 | Oct 2025 | Feb 2026 | |
| Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. | OpenAI | Dangerous capabilities | Everyday | System cards only | GPT-4o 5.74% | 1 | Aug 2024 | Aug 2024 | |
| WISE Text-to-image models draw 1,000 prompts that need world knowledge in culture, space, time and natural science. A vision model judge gives the WiScore. | Peking University (PKU-YuanGroup) | Image generation and editing | School | System cards only | BAGEL-7B-MoT 0.70 | 1 | Mar 2025 | May 2025 | |
| WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. | CAIS + Scale AI et al. | Finding and exploiting real bugs | Expert | System cards only | Code World Model (CWM) 63.6% | 1 | Mar 2024 | Oct 2025 | |
| WMT25 Terminology Translation Task Models translate sentence, paragraph and document level text with a supplied source to target glossary. Scoring checks quality and whether the prescribed target terms appear. | WMT (Conference on Machine Translation) | Translation | Expert | System cards only | North Small Translate 1.0 89.7% | 1 | Jun 2025 | Sep 2026 | |
| WorldBench Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them. | Yida Yin, Shengbang Tong, Wenhu Chen and others | General visual questions | Expert | System cards only | Seed2.1 Pro 67.6% | 1 | Jun 2026 | Jun 2026 | |
| xChatBench Vision language models answer 400 open-ended multimodal chat prompts in 7 languages. An LLM judge scores each answer against fine-grained human-written criteria. | Carnegie Mellon University (PangeaBench) | General visual questions | Everyday | System cards only | Aya Vision 32B 70.1 | 1 | Oct 2024 | May 2025 | |
| xMMMU Vision language models answer college-level MMMU questions translated into 7 languages. Scored by accuracy. | Carnegie Mellon University (PangeaBench) | General visual questions | University | System cards only | Aya Vision 32B 45.1% | 1 | Oct 2024 | May 2025 | |
| ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. | Zhipu AI (Z.ai) | Chat and instructions in many languages | Everyday | System cards only | GLM-5 8.57 | 1 | Feb 2026 | Feb 2026 | |
| ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. | Zhipu AI (Z.ai) | Translation | University | System cards only | GLM-5 1050.0 | 1 | Feb 2026 | Feb 2026 | |
| ADeptS-Bench Computer-use agents pick the next interface action from a screenshot and instruction. A deterministic grader scores whether the action is safe. | Chen et al. | Finding things on screen | Expert | System cards only | 1 | Aug 2026 | Oct 2026 | ||
| Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. | Google DeepMind | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | |||
| AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. | Johns Hopkins University and Cohere | Chat and instructions in many languages | Everyday | System cards only | 1 | Dec 2024 | Apr 2025 | ||
| Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. | Anthropic | Deception and alignment | Expert | System cards only | 1 | Dec 2024 | Feb 2025 | ||
| Anthropic large agent teams tasks Internal 24-hour tasks for teams of 1-100 agents: build a company knowledge base, or formalize a theorem in Lean. | Anthropic | Multi-agent games | Research frontier | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. | Anthropic | Refusing harmful requests | Everyday | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| AraDiCE Models are tested on Arabic dialect understanding and generation and on cultural awareness of the Gulf, Egypt and the Levant. Datasets were machine translated into dialects and post-edited by people. | Qatar Computing Research Institute (Mousi et al.) | Knowledge in many languages | Everyday | System cards only | 1 | Sep 2024 | Nov 2025 | ||
| ASIMOV-Agentic Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes. | Google DeepMind | Embodied reasoning | Everyday | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| AthenaBench Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench. | Rochester Institute of Technology | Finding and exploiting real bugs | Expert | System cards only | 1 | Nov 2025 | Feb 2026 | ||
| Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. | Anthropic | Dangerous capabilities | Expert | System cards only | 1 | Sep 2023 | Mar 2024 | ||
| Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. | OpenAI | Refusing harmful requests | Expert | System cards only | 1 | May 2026 | May 2026 | ||
| Biological threat creation uplift study Experts and novices answer biothreat creation tasks with internet access alone or with model help. Scored as pass rates per stage. | OpenAI with Gryphon Scientific | Hazardous knowledge | Expert | System cards only | 1 | Jan 2024 | Aug 2024 | ||
| Catastrophic biology scenario uplift trial (Anthropic) PhD biologists with model access spend up to 16 hours planning a catastrophic biological scenario. External experts grade uplift and executability. | Anthropic | Hazardous knowledge | Expert | System cards only | 1 | Apr 2026 | |||
| CC-Bench Claude Code runs realistic frontend, backend, tooling and data coding tasks in fresh containers. Humans compare models head-to-head (win rate). | Zhipu AI (Z.ai) | Building whole projects | Expert | System cards only | 1 | Jul 2025 | Aug 2025 | ||
| Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. | Anthropic | Deception and alignment | University | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| ChatRAG Bench Models answer multi-turn conversational questions over documents or retrieved context. The suite is built from 10 existing datasets such as Doc2Dial, QuAC, QReCC and CoQA. | NVIDIA | Finding facts in long inputs | Everyday | System cards only | 1 | Jan 2024 | Apr 2025 | ||
| Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. | Google DeepMind | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | |||
| Closed-domain hallucination evaluation Models answer using only provided context. Scored on avoiding invented details outside that context. | OpenAI | Faithful to sources | Everyday | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| CNML-Level Problems (DeepSeek) Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs. | DeepSeek | Written proofs | Competition | System cards only | 1 | Nov 2025 | Nov 2025 | ||
| Community Alignment personalization Models predict an individual's preferred response from demographics and nine examples, across five countries. Reports global accuracy. | Meta | Bias, fairness and politics | Everyday | System cards only | 1 | Oct 2026 | |||
| ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. | UK AI Security Institute | Deception and alignment | Expert | System cards only | 1 | Feb 2025 | Jul 2026 | ||
| Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. | Anthropic | Refusing harmful requests | Everyday | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | OpenAI | Secure code and review | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. | OpenAI | Jailbreaks and prompt injection | Expert | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| DeepSeek Chinese creative writing evaluation Models write Chinese fiction, prose, poetry and lyrics for 2,837 prompts. Pairwise judging gives separate win rates for instruction following and writing quality. | DeepSeek | Creative writing | Everyday | System cards only | 1 | Apr 2026 | Apr 2026 | ||
| DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. | DeepSeek | Refusing harmful requests | Everyday | System cards only | 1 | Jan 2024 | Jan 2024 | ||
| Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. | Anthropic | Bias, fairness and politics | Everyday | System cards only | 1 | Feb 2026 | Feb 2026 | ||
| Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. | OpenAI | Bias, fairness and politics | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Anthropic | Finding and exploiting real bugs | Expert | System cards only | 1 | Mar 2024 | Mar 2024 | ||
| External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. | Anthropic | Deception and alignment | Everyday | System cards only | 1 | Feb 2026 | Feb 2026 | ||
| Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. | Google DeepMind | Wellbeing and sycophancy | System cards only | 1 | Oct 2026 | ||||
| Health dynamic conversation eval Simulated users hide key health facts at first, and models must ask for context to reach the correct clinical conclusion. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Health language and locale eval Models answer locale-specific health conversations in 10 languages across 25 locales, with search. Graded against clinician guidelines. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Health static understanding eval Models answer static health questions over text conversations, documents such as lab reports, and images. Graded against clinician guidelines. | Google DeepMind | Medicine | Expert | System cards only | 1 | Oct 2026 | |||
| Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| Hindsight Neglect Models judge whether a bet was worth taking based on expected value rather than its outcome. Scoring is accuracy. | Inverse Scaling Prize | Language reasoning | Everyday | System cards only | 1 | Mar 2023 | |||
| ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. | UK AI Security Institute | Deception and alignment | Expert | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| InstructTTSEval Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip. | Fudan University (Kexin Huang, Xipeng Qiu and others) | Speech recognition and translation | Everyday | System cards only | 1 | Jun 2025 | Sep 2025 | ||
| Internal adversarially-designed factuality evaluations Models answer adversarial questions in nine categories such as history, math and code. Scored on agreement with human ideal responses. | OpenAI | Hallucination rates | Everyday | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| JailbreakBench 100 harmful and 100 benign behaviors with a standard jailbreak artifact library. Scored by attack success rate. | JailbreakBench team (Penn, ETH and others) | Jailbreaks and prompt injection | System cards only | 1 | Mar 2024 | Jan 2026 | |||
| Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. | Anthropic | Deception and alignment | Expert | System cards only | 1 | Oct 2026 | Oct 2026 | ||
| Kimi Chinese in-house open-ended benchmark Models answer held-out Chinese prompts drawn from real user queries. Outputs are compared pairwise against another model to give win, tie and loss rates. | Moonshot AI | Overall chat quality | Everyday | System cards only | 1 | Jul 2025 | Jul 2025 | ||
| Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. | Moonshot AI | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jul 2025 | Jul 2025 | ||
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Moonshot AI | Finding and exploiting real bugs | Expert | System cards only | 1 | Jul 2026 | Jul 2026 | ||
| Kimi-ASR Internal Testset Models transcribe speech from two internal Moonshot test subsets. Scored by word error rate, lower is better. | Moonshot AI | Speech recognition and translation | Everyday | System cards only | 1 | Apr 2025 | Apr 2025 | ||
| LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. | Shanghai AI Laboratory | Refusing harmful requests | Everyday | System cards only | 1 | Aug 2025 | Sep 2025 | ||
| Llama 3 pairwise human evaluation (Meta) Human raters compare Llama responses with a rival model on about 7,000 prompts across six capabilities and three multiturn sets. | Meta | Overall chat quality | System cards only | 1 | Jul 2024 | Nov 2024 | |||
| LLMBoxing People see one question with two anonymous model answers and pick the better one. The site counts preferences between model pairs. | llmboxing.com | Overall chat quality | Everyday | System cards only | 1 | Oct 2023 | Oct 2023 | ||
| MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. | Microsoft AI | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. | Microsoft AI | Refusing harmful requests | Everyday | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Meta BioKnowledge Proxy Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions. | Meta with Frontier Design Group | Hazardous knowledge | Expert | System cards only | 1 | Oct 2025 | Oct 2025 | ||
| Meta chemical and biological high-risk bottleneck evaluations Models answer expert-written tasks in three restricted subsets. An expert-validated rubric scores each response. | Meta | Hazardous knowledge | Expert | System cards only | 1 | Apr 2026 | Jun 2026 | ||
| Missing-context hallucinations (Anthropic) Models get prompts that request tools they lack or cite missing material. Score is the non-hallucination rate on each 500-prompt set. | Anthropic | Hallucination rates | Everyday | System cards only | 1 | May 2026 | |||
| Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. | Mistral AI | Refusing harmful requests | Everyday | System cards only | 1 | Oct 2023 | Oct 2023 | ||
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | OpenAI | Knowledge in many languages | University | System cards only | 1 | Mar 2023 | Mar 2023 | ||
| Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. | OpenAI | Deception and alignment | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| MR-NIAH (Multi-Round Needles-In-A-Haystack) Models must repeat one earlier answer from a long multi-turn dialogue history of 2K to 1M tokens. Scored by adjusted recall of key parts. | MiniMax | Finding facts in long inputs | Everyday | System cards only | 1 | Jan 2025 | Jan 2025 | ||
| Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. | Mistral AI | Reasoning in many languages | Competition | System cards only | 1 | Jun 2025 | Jun 2025 | ||
| Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. | Alibaba AIDC-AI (Parrot authors) | Knowledge in many languages | School | System cards only | 1 | Aug 2024 | Nov 2025 | ||
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | OpenAI | Knowledge in many languages | Expert | System cards only | 1 | Jul 2025 | Jul 2025 | ||
| Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. | Meta | Refusing harmful requests | Expert | System cards only | 1 | Oct 2026 | |||
| nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. | Neel Nanda et al. | Deception and alignment | System cards only | 1 | Sep 2026 | Oct 2026 | |||
| OpenAI performance in cases flagged by users Model answers de-identified ChatGPT prompts that users flagged for factual errors. Scored on any error and on repeating the flagged error. | OpenAI | Hallucination rates | System cards only | 1 | Sep 2026 | Oct 2026 | |||
| Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. | OpenAI | Deception and alignment | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Feb 2025 | Feb 2025 | ||
| Qwen in-house voice-chat benchmark Models answer spoken versions of text benchmarks such as MMLU, CEval, IFEval, GSM8K, Math23K and Math401. Each part keeps the source benchmark's scoring. | Qwen (Alibaba) | Speech recognition and translation | School | System cards only | 1 | Mar 2025 | Mar 2025 | ||
| Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. | Qwen Team, Alibaba Group | Jailbreaks and prompt injection | Everyday | System cards only | 1 | Jul 2024 | Jul 2024 | ||
| Real-scenario prompts human evaluation (Z.ai) One human evaluator scores replies to 660 curated real user prompts in English, Chinese and other languages on a 0 to 10 scale. | Zhipu AI (Z.ai) | Overall chat quality | Everyday | System cards only | 1 | Aug 2025 | Aug 2025 | ||
| Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. | Anthropic | Deception and alignment | Expert | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Roboharm Agents and VLAs controlling bimanual YAM arms get five harmful-looking instructions in fixed scenes, such as stabbing a doll. Reviewers label refusals and completions. | Robocurve | Embodied reasoning | Everyday | System cards only | 1 | Sep 2026 | Oct 2026 | ||
| Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. | OpenAI | Refusing harmful requests | Expert | System cards only | 1 | Sep 2026 | Sep 2026 | ||
| Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. | Cohere | Refusing harmful requests | Everyday | System cards only | 1 | Apr 2025 | Apr 2025 | ||
| SciTS Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric. | Shanghai AI Laboratory, Tsinghua University and others | Research science | Expert | System cards only | 1 | Sep 2025 | Mar 2026 | ||
| SeedKernelBench Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code. | ByteDance Seed | GPU kernels | Expert | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. | OpenAI | Refusing harmful requests | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Structured expert probing campaign: chem-bio novel design Biosecurity and chemistry experts test whether a model helps design novel chem-bio threats. Scenarios are rated against Preparedness risk levels. | OpenAI with Signature Science | Hazardous knowledge | Research frontier | System cards only | 1 | Dec 2024 | Dec 2024 | ||
| SWE agentic fabrication eval A grounding autorater checks whether claims made by coding agents on long SWE tasks are supported by tool outputs. | Google DeepMind | Hallucination rates | Expert | System cards only | 1 | Oct 2026 | |||
| Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. | Anthropic | Deception and alignment | Everyday | System cards only | 1 | Sep 2025 | Sep 2025 | ||
| Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. | OpenAI | Dangerous capabilities | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Trae Agent Bench Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup. | ByteDance Seed | Fixing real repositories | Expert | System cards only | 1 | Jun 2026 | Jun 2026 | ||
| Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. | Trajectory Labs for Google DeepMind | Jailbreaks and prompt injection | Expert | System cards only | 1 | Oct 2026 | Oct 2026 | ||
| Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. | Mistral AI | Knowledge in many languages | School | System cards only | 1 | Jan 2024 | Jan 2024 | ||
| Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Masakhane with OpenAI and collaborators (Bayes et al.) | Reasoning in many languages | School | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Masakhane with OpenAI and collaborators (Bayes et al.) | Knowledge in many languages | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Uhura-Eval Models answer new reading comprehension questions written in Amharic, Hausa and Yoruba. Scoring is accuracy. | Uhura collaborators (Adelani, Kgomo, Bayes and others) | Reading in many languages | Everyday | System cards only | 1 | Aug 2024 | Aug 2024 | ||
| Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. | Google DeepMind | Dangerous capabilities | Everyday | System cards only | 1 | Mar 2024 | Jul 2024 | ||
| OTIS Mock AIME 2024-2025 Models solve 45 problems from three OTIS Mock AIME exams written by students of the OTIS program. Integer answers are scored by exact match. | Epoch AI (problems by OTIS students) | Competition problems with answers | Competition | Not in cards | GPT-5.5 100% | Dec 2024 | Oct 2026 | ||
| SnakeBench Two language models each steer a snake on a shared board in a turn-based game, and the one that survives longest wins. Ratings come from many head-to-head games. | ARC Prize Foundation (Greg Kamradt) | Multi-agent games | Everyday | Not in cards | GPT-5.1-Codex-Max 38 | Feb 2025 | Sep 2026 | ||
| OpenVLM Leaderboard OpenCompass runs vision-language models through VLMEvalKit on a fixed set of image benchmarks, such as MMBench, MMStar and MathVista. The headline is the average score across that set. | OpenCompass (Shanghai AI Laboratory) | Composite indices | University | Not in cards | Gemini 2.5 Pro 80.1 | Jan 2024 | Aug 2025 | ||
| ACCoRD Models state probabilities and preferences on conceptual questions, and the benchmark checks 567 expert-approved logical consistency constraints such as conjunction rules and order invariance. | Redwood Research (Cooper, Kastner, Oesterheld et al.) | Other | Expert | Not in cards | Claude Opus 5.5 86.5 | Aug 2026 | Oct 2026 | ||
| Conceptual Reasoning Index An index that ranks models on reasoning about hard-to-verify topics such as AI alignment and decision theory. It weights LMCA 60 percent and ACCoRD and DTBench 20 percent each. | Redwood Research (conceptual reasoning team) | Composite indices | Expert | Not in cards | Claude Opus 5.5 77.9 | Aug 2026 | Oct 2026 | ||
| LMCA Models rate arguments against position texts on philosophy, decision theory, and AI risk. Scored by agreement with expert ratings, with a ceiling near 85. | Emery Cooper, Caspar Oesterheld, and collaborators | Other | Expert | Not in cards | Claude Sonnet 5.5 68.3% | Jul 2026 | Oct 2026 | ||
| LegalBench Models do 162 legal reasoning tasks written by legal professionals, covering six types of legal reasoning such as issue spotting and rule application. | Stanford University (Hazy Research) and collaborators | Law | Expert | Not in cards | Claude Fable 5 88.6% | Aug 2023 | Oct 2026 | ||
| Vision Arena People send an image with a prompt, compare answers from two anonymous vision-language models, and vote for the better one. Votes become Bradley-Terry ratings. | Arena (formerly LMArena) | General visual questions | Everyday | Not in cards | Claude Fable 5 1,309 | Jun 2024 | Oct 2026 | ||
| LisanBench Models build the longest chain of distinct English words from a given start word, each differing from the last by one letter. The score sums valid chain lengths over all start words. | Lisan al Gaib (scaling01, independent) | Puzzles and chess | Everyday | Not in cards | Muse Wonder Full 37152.2 | May 2025 | Oct 2026 | ||
| TaxEval v2 Models answer expert-written tax questions. A judge grades answer correctness and stepwise reasoning against expert solutions. | Vals AI | Finance | Expert | Not in cards | Muse Spark 1.2 80.4% | Sep 2026 | |||
| LLM Chess Models play chess in multi-turn dialogs against a random player or the Komodo Dragon engine. The board reports wins, Elo and game duration, which tracks instruction following. | Maxim Saplin | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 1,614 | Sep 2024 | Oct 2026 | ||
| BullshitBench V2 Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback. | Peter Gostev | Hallucination rates | Everyday | Not in cards | Claude Opus 4.8 95% | Mar 2026 | Oct 2026 | ||
| BullshitBench V1 Replaced in Mar 2026 | Not in cards | Claude Opus 4.8 96.4% | Feb 2026 | Sep 2026 | |||||
| ALE-Bench Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants. | Sakana AI and AtCoder | Competitive programming | Competition | Not in cards | GPT-6 Astra 2,951 | Jun 2025 | Oct 2026 | ||
| Longform Creative Writing (EQ-Bench) Models plan, revise and then write a short story or novella over eight 1,000-word turns from a minimal prompt. A judge model scores quality, and slop, repetition and degradation are tracked. | EQ-Bench (Sam Paech) | Creative writing | University | Not in cards | Claude Opus 5 86.3 | May 2025 | Oct 2026 | ||
| A-Bench Models answer multiple-choice questions about AI-generated images, covering both meaning and low-level visual quality, to test them as judges of image generators. | Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) | General visual questions | Everyday | Not in cards | Qwen2.5-VL-72B 81% | Jun 2024 | Jun 2025 | ||
| SimpleBench Models answer over 200 multiple-choice trick questions on spatial, temporal, and social reasoning that ordinary people solve. Scored by average accuracy over five runs. | SimpleBench Team | Abstract puzzles and logic | Everyday | Not in cards | Claude Opus 5.5 88.4% | Oct 2024 | Sep 2026 | ||
| Vectara Hallucination Leaderboard (2025 dataset) Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries. | Vectara | Faithful to sources | Everyday | Not in cards | GPT-5.4 Nano 3.10% | Nov 2025 | Sep 2026 | ||
| Vectara Hallucination Leaderboard (HHEM) v1 Replaced in Nov 2025 | System cards only | Gemma 1.1 2B IT 27.8% | 1 | Nov 2023 | Oct 2025 | ||||
| MedScribe Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure. | Vals AI (data partner Protege) | Medicine | Expert | Not in cards | Claude Opus 5.5 91.4% | Feb 2026 | Oct 2026 | ||
| Extended NYT Connections Models solve New York Times Connections puzzles with up to four extra trick words added. Each puzzle scores by the square of the share of groups found. | Lech Mazur | Language reasoning | School | Not in cards | Gemini 4 Argon 98.6% | Oct 2024 | Oct 2026 | ||
| MedCode Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders. | Vals AI | Medicine | Expert | Not in cards | Claude Opus 5 63.6% | Feb 2026 | Oct 2026 | ||
| MortgageTax Vision models read mortgage tax certificates given as images. They extract the year, parcel number and county, and compute the annualized amount due. | Vals AI (data from Vontive) | Finance | Everyday | Not in cards | Claude Opus 5 72.1% | Sep 2026 | |||
| Chess Puzzles Models find the single best next move in 100 chess puzzles that Epoch generated with a chess engine. Scored by the share solved. | Epoch AI | Puzzles and chess | Competition | Not in cards | GPT-6 Astra 76% | Dec 2025 | Oct 2026 | ||
| Mystery Game Puzzles 100 mid-game positions from an undisclosed puzzle variant of a well-known game. Model picks the single best move, checked against an answer key. | Epoch AI | Puzzles and chess | Competition | Not in cards | GPT-6 Astra 84% | Jul 2026 | Oct 2026 | ||
| Tax Agent Bench Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real. | Vals AI | Finance | Expert | Not in cards | Gemini 4 Carbon 51.8% | Sep 2026 | Oct 2026 | ||
| LMArena Text-to-Image Arena Image generation models answer the same user prompt side by side and anonymously. Community votes between the two images give Elo-style ratings. | Arena (formerly LMArena) | Image generation and editing | Everyday | Not in cards | GPT-Image-2.5 Sunburst 1,427 | Dec 2024 | Oct 2026 | ||
| Wer wird Millionär? LLM Benchmark Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms. | ikiruneo (GitHub user) | Knowledge in many languages | Everyday | Not in cards | GPT-6 Astra 644444.0 | Sep 2025 | Sep 2026 | ||
| HELM Safety Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard. | Stanford CRFM | Refusing harmful requests | Everyday | Not in cards | GPT-5 nano 98.6 | Nov 2024 | Nov 2025 | ||
| HELM Lite A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014. | Stanford CRFM | Composite indices | University | Not in cards | GPT-4o 93.8 | Dec 2023 | Mar 2025 | ||
| Human Pathogen Capabilities Test v2 (HPCT v2) Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT. | SecureBio | Hazardous knowledge | Research frontier | Not in cards | Claude Opus 5 80% | 2026 | Oct 2026 | ||
| Human Pathogen Capabilities Test v1 Replaced in Jan 2026 | Expert | System cards only | GPT-5.6 Sol 68.4% | 9 | Aug 2025 | Oct 2026 | |||
| CAIS Text Capabilities Index Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests. | Center for AI Safety | Composite indices | Research frontier | Not in cards | Gemini 4 Argon 66.5 | Dec 2025 | Oct 2026 | ||
| CAIS Risk Index Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games. | Center for AI Safety | Composite indices | Research frontier | Not in cards | Claude Opus 5.5 26.2 | Dec 2025 | Oct 2026 | ||
| Phare Models are probed in English, French and Spanish for hallucination, harmful misguidance, stereotypes and jailbreaks. Hallucination tasks cover factuality, misinformation, debunking and tool use. | Giskard | Hallucination rates | Everyday | Not in cards | Claude Haiku 4.5 83.2% | Feb 2025 | Jun 2026 | ||
| SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. | AI Singapore + Stanford CRFM | Reasoning in many languages | School | Not in cards | Gemini 3.1 Pro 77.4 | Feb 2025 | Sep 2026 | ||
| HELM Capabilities Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public. | Stanford CRFM | Composite indices | University | Not in cards | GPT-5 mini 81.9 | Mar 2025 | Nov 2025 | ||
| Step Race Several models talk in public, then secretly pick moves in a race where colliding moves cancel. The game tests cooperation and deception. | Lech Mazur | Multi-agent games | Everyday | Not in cards | GPT-5 5 | Jan 2025 | Dec 2025 | ||
| LLM Confabulation Leaderboard Models answer misleading questions about recent articles whose answers are not in the text. The board ranks confabulation rate against non-response rate. | Lech Mazur | Faithful to sources | Everyday | Not in cards | GPT-5 10.3% | Oct 2024 | Aug 2025 | ||
| Fiction.LiveBench Models answer 36 questions about 30 Fiction.live stories that need inference about plot and characters. Each story is tested at several context lengths. | Fiction.live | Reasoning over long documents | Everyday | Not in cards | o3-pro 97.2% | Feb 2025 | Jan 2026 | ||
| PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. | Kilo AI | General assistants | Everyday | Not in cards | Claude Opus 4.8 93.5% | May 2026 | Oct 2026 | ||
| PinchBench 1.x Replaced in May 2026 | In use | KAT-Coder-Pro V2.5 94.9% | 12 | 6 | Feb 2026 | Oct 2026 | |||
| Elimination Game Models play a multi-player game of public and private talk, alliances and votes to eliminate each other. A jury of eliminated players picks the winner. | Lech Mazur | Multi-agent games | Everyday | Not in cards | GPT-5.2 8 | Feb 2025 | Dec 2025 | ||
| LMArena Image Edit Arena Image editing models apply the same user edit request to the same image, anonymously and side by side. Community votes give Elo-style ratings. | Arena (formerly LMArena) | Image generation and editing | Everyday | Not in cards | GPT-Image-2.5 Sunburst 1,517 | Jun 2025 | Oct 2026 | ||
| VoxelBench Language models turn text or image prompts into voxel structures. Voters compare anonymous builds from the same prompt, and Glicko-2 ratings rank the models. | VoxelBench | Spatial reasoning and counting | Everyday | Not in cards | GPT-6 Astra 2,757 | Sep 2025 | Oct 2026 | ||
| Image-to-WebDev Arena Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals. | Arena (formerly LMArena) | Web and app front ends | Everyday | Not in cards | Claude Opus 5.5 1,753 | Apr 2026 | Oct 2026 | ||
| VISTA Models answer 758 single-turn prompts about one image each, covering perception and reasoning. Three LLM judges grade answers with yes/no rubric questions. The main score is average task accuracy. | Scale AI | General visual questions | Expert | Not in cards | Gemini 2.5 Pro 54.6% | Dec 2024 | Mar 2026 | ||
| CAIS Vision Capabilities Index Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval. | Center for AI Safety | Composite indices | Research frontier | Not in cards | Gemini 4 Argon 83.3 | Dec 2025 | Oct 2026 | ||
| CaseLaw v2 Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents. | Vals AI (with Jurisage, Together AI and Stanford CodeX) | Law | Expert | Not in cards | Grok 4.3 79.3% | Aug 2025 | Apr 2026 | ||
| Pencil Puzzle Bench Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules. | Approximate Labs | Puzzles and chess | Expert | Not in cards | Claude Fable 5 97.6% | Mar 2026 | Jun 2026 | ||
| ProofBench v1.1 Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts. | Vals AI | Formal proofs | Expert | Not in cards | Claude Fable 5.1 100% | Aug 2026 | Oct 2026 | ||
| ProofBench v1 Replaced in Aug 2026 | Not in cards | Claude Opus 5 78% | Jan 2026 | Oct 2026 | |||||
| PolicyBench No-tools estimate of US taxes and benefits for 100 households across 18 programs. Headline metric is answers within $1 of PolicyEngine-US. | PolicyEngine | Finance | Expert | Not in cards | GPT-6 Sol 95.2% | Jul 2026 | Oct 2026 | ||
| Public Benefits Bench Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts. | Vals AI | Other professions | Expert | Not in cards | Claude Opus 5 76.9% | Jun 2026 | Oct 2026 | ||
| PCT (Polarized Contrastive Pairs) Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair. | Center for AI Safety | Bias, fairness and politics | Everyday | Not in cards | Muse Spark 1.1 27.9% | May 2026 | Oct 2026 | ||
| IOI v2 (Vals) Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run. | Vals AI | Competitive programming | Competition | Not in cards | GPT-6 Astra 100% | Oct 2026 | Oct 2026 | ||
| IOI (Vals) v1 Replaced in Oct 2026 | Not in cards | Claude Opus 5 91.7% | Aug 2025 | Aug 2026 | |||||
| LMArena Text-to-Video Arena Video generation models make clips from the same text prompt, anonymously and side by side. Votes from logged-in users give Elo-style ratings. | Arena (formerly LMArena) | Video generation | Everyday | Not in cards | Gemini Omni 1.1 Flash 1,516 | Jan 2026 | Sep 2026 | ||
| Misguided Attention Models get slight variations of famous riddles, paradoxes and thought experiments. A model fails when it answers the well-known original instead of the problem as stated. | cpldcpu (GitHub user) | Abstract puzzles and logic | Everyday | Not in cards | Grok-3 mini 73.7% | May 2024 | Jun 2025 | ||
| Lech Mazur Creative Writing Models write short stories that must use 10 assigned elements such as characters, objects, and motivations. Judge models compare matched stories pairwise. | Lech Mazur | Creative writing | University | Not in cards | GPT-5 8.60 | Jan 2025 | Aug 2025 | ||
| LMArena Image-to-Video Arena Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings. | Arena (formerly LMArena) | Video generation | Everyday | Not in cards | MiniMax H3 1,495 | Jan 2026 | Sep 2026 | ||
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Vals AI | Finding and exploiting real bugs | Expert | Not in cards | Muse Wonder 80.4% | Oct 2026 | Oct 2026 | ||
| CyberBench (Vals) v1 Replaced in Oct 2026 | Not in cards | GPT-5.6 Sol 88.1% | Jun 2026 | Sep 2026 | |||||
| BRUMO 2025 Models answer the 30 final-answer problems of the 2025 Brown University Math Olympiad. MathArena scores the average accuracy over four attempts. | Brown University (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | Gemini 3 Flash 100% | Apr 2025 | Feb 2026 | ||
| DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. | Scale AI | Wellbeing and sycophancy | Expert | Not in cards | Muse Spark 1.2 88.4% | Oct 2026 | Oct 2026 | ||
| KORA Simulated child users hold 3-turn conversations probing 26 child-specific risks. Judge model rates replies failing, adequate or exemplary. | KORA (non-profit) | Wellbeing and sycophancy | Not in cards | GPT-6 Astra 73.2% | Feb 2026 | Oct 2026 | |||
| ParseBench Document parsers process about 2,000 human-verified enterprise pages. Over 169,000 test rules score tables, charts, content faithfulness, formatting and visual grounding. | LlamaIndex | Documents, charts and text in images | Expert | Not in cards | Claude Opus 5.5 78.9% | Apr 2026 | Oct 2026 | ||
| RewardBench 2 Reward models must score one correct answer above three wrong ones on 1,865 new human prompts. Subsets cover factuality, precise instruction following, math, safety, focus and ties, and the score averages them. | Allen Institute for AI (Ai2) | Overall chat quality | University | Not in cards |  Skywork-Reward-V2-Llama-3.1-8B 84.1% | Jun 2025 | Jun 2025 | ||
| RewardBench v1 Replaced in Jun 2025 | Allen Institute for AI (Lambert et al.) | Everyday | Dropped |  INF-ORM-Llama3.1-70B 95.1% | 1 | 2 | Mar 2024 | Aug 2025 | |
| EnigmaEval 1,184 multimodal puzzle-hunt puzzles (v2 dataset has 1,161). Model gives one exact answer per puzzle. Normal and Hard splits. | Scale AI + Center for AI Safety + MIT | General visual questions | Competition | Not in cards | Claude Fable 5 39.3% | Feb 2025 | Jul 2026 | ||
| VHELM Vision-language models are scored across nine aspects, including perception, reasoning, knowledge, bias, fairness, multilinguality, robustness, toxicity and safety, from many existing datasets. | Stanford CRFM | General visual questions | University | Not in cards | Gemini 2.5 Pro 87.1 | Oct 2024 | Jun 2025 | ||
| Judgemark v4 The model under test judges fixed creative-writing samples against low and high anchor responses. The score is how well its ratings separate writer models. | EQ-Bench (Sam Paech) | Taste and style | Expert | Not in cards | Claude Opus 4.6 90.7 | May 2026 | Aug 2026 | ||
| Judgemark v2.1 Replaced in May 2026 | Not in cards | Claude Sonnet 4.6 94.1 | Jan 2025 | Mar 2026 | |||||
| SMT 2025 Models answer 53 final-answer problems from the 2025 Stanford Math Tournament. MathArena scores the average accuracy over four attempts. | Stanford Math Tournament (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | Gemini 3 Pro 93.4% | May 2025 | Feb 2026 | ||
| ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. | Inception and MBZUAI (Jais project) | Knowledge in many languages | University | Not in cards | Qwen2.5-72B 72.2% | Sep 2024 | Aug 2025 | ||
| FACTS Multimodal Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | Gemini 3.8 Flash 50.7% | Dec 2025 | Sep 2026 | ||
| Game Arena Chess Language models play full chess games against each other with the board given as FEN and PGN text. Ratings come from many games between each pair. | Kaggle and Google DeepMind | Puzzles and chess | Expert | Not in cards | Gemini 3.8 Flash 1,453 | Aug 2025 | Oct 2026 | ||
| Kaggle Game Arena Language models play each other in board, card and word games through a text harness. One leaderboard ranks models across all Game Arena games. | Kaggle and Google DeepMind | Multi-agent games | Expert | Not in cards | Gemini 4 Argon 535 | Aug 2025 | Oct 2026 | ||
| BALROG Agentic reasoning on games (NetHack, Crafter, BabyAI, MiniHack...) for LLMs and VLMs. | UCL DARK et al. | Video games | Everyday | Not in cards | GPT-6 Astra 68.3% | Nov 2024 | Sep 2026 | ||
| Game Arena Checkers Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 1,338 | Jun 2026 | Oct 2026 | ||
| Game Arena Dark Hex Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Gemini 3.8 Flash 612 | Jun 2026 | Oct 2026 | ||
| Game Arena Four in a Row Language models play a Connect Four style game and try to line up four of their tokens before the opponent does. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Gemini 4 Argon 923 | Feb 2026 | Oct 2026 | ||
| MultiNRC Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer. | Scale AI | Reasoning in many languages | Expert | Not in cards | Muse Spark 1.1 65.6% | Jul 2025 | Jul 2026 | ||
| ClockBench Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage. | Alek Safar | Spatial reasoning and counting | Everyday | Not in cards | Gemini 4 Argon 83.3% | Sep 2025 | Sep 2026 | ||
| Document Arena People upload their own PDFs with a question, compare answers from two anonymous models, and vote for the better one. Votes become Bradley-Terry ratings. | Arena (formerly LMArena) | Reasoning over long documents | Everyday | Not in cards | Claude Fable 5 1,499 | Mar 2026 | Sep 2026 | ||
| Excel Modeling Benchmark v1.1 Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna. | Vals AI | Finance | Expert | Not in cards | Claude Fable 5.1 76.7% | Oct 2026 | Oct 2026 | ||
| Excel Modeling Benchmark 1.0 Replaced in Oct 2026 | Not in cards | Claude Fable 5.1 76.7% | Jul 2026 | Oct 2026 | |||||
| Game Arena Dots and Boxes Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 1,219 | Jun 2026 | Oct 2026 | ||
| MMLU-CF Models answer new MMLU-style multiple-choice questions written to avoid training data contamination. A 10,000 question test set stays private. | Microsoft Research | Exam questions | University | Not in cards | GPT-4o 73.4% | Dec 2024 | Dec 2024 | ||
| Game Arena Bargaining Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values. | Kaggle and Google DeepMind | Multi-agent games | Everyday | Not in cards | GPT-5.5 277 | Jul 2026 | Oct 2026 | ||
| Game Arena Clobber Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 847 | Jun 2026 | Oct 2026 | ||
| Game Arena Go Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games. | Kaggle and Google DeepMind | Puzzles and chess | Expert | Not in cards | GPT-6 Astra 1,762 | Sep 2026 | Oct 2026 | ||
| Game Arena Lines of Action Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Gemini 4 Argon 1,185 | Jul 2026 | Oct 2026 | ||
| Game Arena Nine Men's Morris Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Claude Opus 5 1,022 | Aug 2026 | Oct 2026 | ||
| Game Arena Reversi Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Gemini 4 Argon 1,102 | Aug 2026 | Oct 2026 | ||
| Game Arena Ultimate Tic-Tac-Toe Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | GPT-6 Astra 608 | Jul 2026 | Oct 2026 | ||
| Game Arena Word Association Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team. | Kaggle and Google DeepMind | Multi-agent games | Everyday | Not in cards | Gemini 4 Argon 596 | May 2026 | Oct 2026 | ||
| Furniture Assembly Benchmark Model gets an IKEA manual and a photo of a partly built item, then names the faulty step or next step. Accuracy on 60 photos. | Epoch AI | Spatial reasoning and counting | Everyday | Not in cards | Claude Opus 5.5 83.3% | Sep 2026 | Oct 2026 | ||
| Game Arena Coin Game Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest. | Kaggle and Google DeepMind | Multi-agent games | Everyday | Not in cards | GPT-5.5 514 | Jul 2026 | Oct 2026 | ||
| Game Arena Word Art Language models take turns drawing a word as ASCII art and guessing what the other model drew. | Kaggle and Google DeepMind | Multi-agent games | Everyday | Not in cards | Gemini 4 Argon 2,473 | Aug 2026 | Oct 2026 | ||
| Vals Finance Agent v2.1 Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones. | Vals AI | Finance | Expert | Not in cards | Claude Opus 5.5 67.8% | Oct 2026 | Oct 2026 | ||
| Vals Finance Agent v2 Replaced in Oct 2026 | In use | Gemini 4 Argon 65.4% | 10 | 9 | May 2026 | Oct 2026 | |||
| Vals Finance Agent v1.1 Replaced in May 2026 | Fading | Claude Opus 4.7 64.4% | 6 | 6 | Feb 2026 | Jun 2026 | |||
| Vals Finance Agent v1 Replaced in Feb 2026 | Vals AI (with Stanford researchers and a global systemically important bank) | Not in cards | Claude Opus 4.6 60.6% | Apr 2025 | Feb 2026 | ||||
| Vals Finance Index Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP. | Vals AI | Composite work indices | Expert | Not in cards | Gemini 4 Carbon 67% | Oct 2026 | Oct 2026 | ||
| AutoCodeBench Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution. | Tencent Hunyuan | Function-level problems | University | Not in cards | Claude Opus 4 52.4% | Aug 2025 | Aug 2025 | ||
| BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. | ETH Zurich SRI Lab, LogicStar.ai, UC Berkeley and INSAIT | Secure code and review | Expert | Not in cards | Claude Opus 4.5 68.6% | Feb 2025 | Nov 2025 | ||
| CMIMC 2025 Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts. | Carnegie Mellon University (contest), MathArena (evaluation) | Competition problems with answers | Competition | Not in cards | DeepSeek-V3.2-Speciale 95.6% | May 2025 | Feb 2026 | ||
| Harvey LAB v1.1 (HLAB-Vals-1.1) Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes. | Harvey (board run by Vals AI) | Law | Expert | Not in cards | Gemini 4 Argon 27.1% | Oct 2026 | Oct 2026 | ||
| Harvey LAB (Legal Agent Benchmark) 1.0 Replaced in Oct 2026 | Harvey | In use | Muse Spark 1.2 25.4% | 13 | 15 | May 2026 | Oct 2026 | ||
| OpenHands Index Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported. | All Hands AI (OpenHands) | Fixing real repositories | Expert | Not in cards | Claude Fable 5 81% | Jan 2026 | Jun 2026 | ||
| Vals Legal Index Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue. | Vals AI | Composite work indices | Expert | Not in cards | Muse Wonder Full 77.9% | Oct 2026 | Oct 2026 | ||
| Vals Multimodal Index Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work. | Vals AI | Composite indices | Expert | Not in cards | Claude Fable 5 74.2% | Oct 2025 | Aug 2026 | ||
| EQ-Bench 4 Models hold 16-message roleplay chats with 120 simulated user personas whose preferences compete and are hidden. A three-judge panel compares transcripts pairwise, and ratings are fitted as Elo scores. | EQ-Bench (Sam Paech) | Multi-turn conversation | Everyday | Not in cards | Claude Opus 5 1,365 | Jul 2026 | Oct 2026 | ||
| EQ-Bench 3 Replaced in Jul 2026 | EQ-Bench | Fading | Claude Fable 5 2,096 | 4 | 2 | Apr 2025 | Oct 2026 | ||
| EQ-Bench v2 Replaced in Apr 2025 | Overall chat quality | Not in cards | Claude 3.5 Sonnet 86.4 | Jan 2024 | Mar 2025 | ||||
| ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. | Scale AI | Function calling | Expert | Not in cards | o1 70.1% | Jan 2025 | Mar 2025 | ||
| Vending-Bench Arena Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance. | Andon Labs | Simulated companies | Expert | Not in cards | Gemini 4 Carbon $15,962 | Nov 2025 | Oct 2026 | ||
| Game Arena Poker Language models play heads-up no-limit Texas Hold'em against each other. Models are ranked by mean big blinds won per 100 hands. | Kaggle and Google DeepMind | Multi-agent games | Expert | Not in cards | GPT-5.6 Sol 30.3 | Feb 2026 | Sep 2026 | ||
| GSO-Bench Agents speed up code in 102 tasks across 10 real codebases, given a performance test. Scored by the share matching the expert commit's speedup. | GSO team (UC Berkeley et al.) | Speed and optimization | Expert | Not in cards | Claude Fable 5.1 88.2% | May 2025 | Sep 2026 | ||
| LEXam Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match. | University of Zurich and collaborators | Law | Expert | Not in cards | GPT-5 70.2% | May 2025 | Nov 2025 | ||
| OncologyBench Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 50.7% | Oct 2026 | Oct 2026 | ||
| U-MATH Models solve 1,100 unpublished open-ended university math problems from teaching materials, balanced across six subjects. About 20% include images, and an LLM judge checks each solution. | Toloka AI and collaborators | Word problems | University | Not in cards | o1 86.8% | Dec 2024 | Jan 2025 | ||
| Code Migration v1.1 Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check. | Vals AI | Fixing real repositories | Expert | Not in cards | Claude Sonnet 5.5 86.9% | Oct 2026 | Oct 2026 | ||
| Code Migration 1.0 Replaced in Oct 2026 | Not in cards | Claude Sonnet 5.5 69.8% | Jun 2026 | Oct 2026 | |||||
| Humanity's Sixth Sense Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric. | Scale AI and Elorian | General visual questions | Everyday | Not in cards | Gemini 4 Argon 55.4% | Oct 2026 | Oct 2026 | ||
| MysteryMechanism Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold. | Vals AI | Abstract puzzles and logic | Expert | Not in cards | GPT-6 Astra 53.1% | Sep 2026 | Oct 2026 | ||
| Political Science LLM Benchmark 33 political-science text-coding tasks, 100 texts each (~3,300 items). Models scored by mean F1 against human-coded labels. | Hanno Hilbig (UC Davis) | Other | University | Not in cards | Claude Opus 5 71.4% | May 2026 | Sep 2026 | ||
| xbench-ScienceQA Models answer postgraduate-level science questions written by experts, in Chinese, as part of the xbench AGI Tracking series. | xbench (HongShan) | Science questions | Expert | Not in cards | Claude Opus 4.8 76.4% | May 2025 | Sep 2026 | ||
| DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. | Vercel | Finding and exploiting real bugs | Expert | Not in cards | GPT-6 Sol 40.8 | Jul 2026 | Oct 2026 | ||
| EBR-bench Agent plays the board game Earthborne Rangers 10-30 times in a row. Scored on objectives completed (of 21) to test learning from experience. | Epoch AI | Puzzles and chess | Everyday | Not in cards | GPT-6.1 Terra 78.1% | Jul 2026 | Oct 2026 | ||
| HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. | Stanford CRFM with Arabic.AI | Knowledge in many languages | School | Not in cards | Claude Opus 4.7 84.1 | Dec 2025 | Apr 2026 | ||
| MirrorCode Agent rebuilds a whole CLI program from scratch using only the original binary as an oracle. Solved only if 100% of end-to-end tests pass. | Epoch AI + METR | Building whole projects | Research frontier | Not in cards | Claude Sonnet 5.5 97.8% | Jun 2026 | Oct 2026 | ||
| SAGE v1.1 (Vals) Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading. | Vals AI | Other | University | Not in cards | Claude Opus 5.5 91.3% | Oct 2026 | Oct 2026 | ||
| SAGE (Vals) v1 Replaced in Oct 2026 | Not in cards | Claude Opus 4.7 56.1% | Oct 2025 | Oct 2026 | |||||
| TxBench-Antibody-Discovery Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 67.7% | Aug 2026 | Oct 2026 | ||
| TxBench-Oligo-Discovery Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 62.2% | Sep 2026 | Oct 2026 | ||
| TxBench-Preclinical-Pharmacology Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 65.3% | Jun 2026 | Oct 2026 | ||
| EpiBench Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 39.6% | Jun 2026 | Oct 2026 | ||
| scBench-Long Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 36.9% | Jun 2026 | Oct 2026 | ||
| SpatialBench-Long Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 40.5% | May 2026 | Oct 2026 | ||
| VariantBench Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5 52.8% | Jul 2026 | Oct 2026 | ||
| MetagenomicsBench Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result. | LatchBio | Genomics and life sciences | Expert | Not in cards | Claude Opus 5.5 62.7% | Sep 2026 | Oct 2026 | ||
| METR Time Horizon 1.1 Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time. | METR | Task length | Research frontier | Not in cards | Claude Mythos Preview 17.4 h | Jan 2026 | Apr 2026 | ||
| METR Time Horizon 1.0 Replaced in Jan 2026 | System cards only | Claude Opus 4.5 4.1 h | 7 | Mar 2025 | Jul 2026 | ||||
| PACT Two models bargain over 20 rounds as buyer and seller, each with a hidden private value. Matches across many pairings rank the models. | Lech Mazur | Multi-agent games | Everyday | Not in cards | GPT-5.5 1,607 | Jul 2025 | Jun 2026 | ||
| prinzbench Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99. | prinz-ai | Law | Expert | Not in cards | GPT-5.6 Sol Pro 91.0 | Jan 2026 | Jul 2026 | ||
| SEAL Coding Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores. | Scale AI | Function-level problems | University | Not in cards | o1-mini 1,237 | May 2024 | Jan 2025 | ||
| Surface Evolver Bench Agents write Surface Evolver datafiles that model liquid surfaces under physical constraints across 16 tasks. Deterministic checks give partial credit for matching hidden reference quantities. | Yann Henon | Research science | Expert | Not in cards | Claude Fable 5 95% | Jun 2026 | Oct 2026 | ||
| TaxCalcBench v2 2 (TY 2025) Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact. | Column Tax | Finance | Expert | Not in cards | Muse Wonder 77.5% | Jun 2026 | Oct 2026 | ||
| TaxCalcBench 1 (TY 2024) Replaced in Jun 2026 | Not in cards | GPT-5.4 62.8% | Jul 2025 | Mar 2026 | |||||
| Vibe Code Bench 1-100 Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it. | Vals AI | Building whole projects | Expert | Not in cards | Muse Wonder Full 34% | Sep 2026 | Oct 2026 | ||
| Vals RSI Index Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness. | Vals AI | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 0.37 | Aug 2026 | Oct 2026 | ||
| WeirdML v3 Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results. | Håvard Tveit Ihle (Norwegian Defence Research Establishment) | Machine learning engineering | Expert | Not in cards | GPT-6 Astra 42.2% | Sep 2026 | Oct 2026 | ||
| WeirdML v2 Replaced in Sep 2026 | Håvard Tveit Ihle | Not in cards | GPT-6 Astra Pro 93.6% | Jun 2025 | Sep 2026 | ||||
| AgentIF (2026-10 setting) Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model. | Tsinghua University KEG (THU-KEG) | Following precise instructions | Everyday | Not in cards | DeepSeek-V4-Pro 67.4% | Oct 2026 | Oct 2026 | ||
| AgentIF Original Replaced in Oct 2026 | In use | o1-mini 59.8% | 1 | 1 | May 2025 | Oct 2026 | |||
| FailureSensorIQ Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation. | IBM Research | Other professions | Expert | Not in cards | GPT-6 Astra 69.5% | Jun 2025 | Sep 2026 | ||
| Immigration Bench Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric. | Vals AI + Talent Mobility Fund (Renaissance Philanthropy) | Law | Expert | Not in cards | Claude Fable 5.1 75.4% | Oct 2026 | Oct 2026 | ||
| ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. | SCB 10X + Stanford CRFM | Knowledge in many languages | School | Not in cards | Claude 3.5 Sonnet 75.1% | Sep 2024 | Aug 2024 | ||
| BuzzBench Models analyze humor, such as why a joke or comic piece works. The leaderboard reports a score and response length per model. | EQ-Bench (Sam Paech) | Taste and style | Everyday | Not in cards | Gemini 2.5 Pro 71.1 | Jun 2025 | |||
| GBAEval Agents get 24 hours to write a Game Boy Advance emulator in Rust for WebAssembly. Replay, hardware, and audio tests compare it with Mesen2. | Mechanize | Building whole projects | Research frontier | Not in cards | Claude Opus 5 79.6% | May 2026 | Jul 2026 | ||
| General365 Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks. | Meituan LongCat Team | Abstract puzzles and logic | Everyday | Not in cards | Gemini 3 Pro 62.8% | Apr 2026 | Feb 2026 | ||
| HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. | Stanford CRFM with Arabic.AI | Knowledge in many languages | Expert | Not in cards | Claude Opus 4.7 77.9 | May 2026 | Apr 2026 | ||
| ResearchClawBench Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs. | Intern Science (Shanghai AI Laboratory) | Research science | Research frontier | Not in cards | DeepSeek-V4-Pro 38.6 | May 2026 | Aug 2026 | ||
| Senior SWE-Bench Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate. | Snorkel AI | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 39.3% | Jun 2026 | Sep 2026 | ||
| Vals Science Index Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science. | Vals AI | Composite indices | Expert | Not in cards | GPT-6 Astra 65.1% | Oct 2026 | Oct 2026 | ||
| Verifying the Verifiers (Project APE CRED benchmark) Verifier LLMs must find planted errors (prose-table, paper-code, execution, data) in AI-generated economics papers. Credit needs a verbatim quote of the edit. | Project APE, University of Zurich (Willner, Yanagizawa-Drott) | Research science | Expert | Not in cards | GPT-6 Astra 99.5% | Jul 2026 | Sep 2026 | ||
| SWE Atlas Test Writing Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests. | Scale AI | Fixing real repositories | Expert | Not in cards | Claude Fable 5.1 67% | Apr 2026 | Sep 2026 | ||
| ArtifactArena Models design robot bodies, controllers and strategy for a simulated sumo-style arena. Artifacts fight head-to-head and models get Elo ratings. | MIT CSAIL + MIT Media Lab + NYU + UCSF | Simulated robots | Expert | Not in cards | GPT-6 Astra 1,539 | Sep 2026 | Sep 2026 | ||
| FACTS Search V2 Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | GPT-6 Astra 89.9% | Jul 2026 | Oct 2026 | ||
| FACTS Search v1 Replaced in Jul 2026 | Not in cards | Gemini 3.1 Pro 85.6% | Dec 2025 | May 2026 | |||||
| Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. | Galileo | Customer service | Everyday | Not in cards | GPT-4.1 62% | Jul 2025 | Aug 2025 | ||
| Galileo Agent Leaderboard v1 Replaced in Jul 2025 | Function calling | Not in cards | Claude 3.7 Sonnet 0.95 | Feb 2025 | Apr 2025 | ||||
| MathBench Models answer 3,709 math questions in five stages from basic arithmetic to college, in Chinese and English. Each stage also tests theory knowledge, and Circular Evaluation reshuffles answer options to check consistency. | OpenCompass (Shanghai AI Laboratory) | Word problems | University | Not in cards | GPT-4o 70.9% | May 2024 | Jun 2024 | ||
| SkateBench v2 Models name skateboard tricks from technical trick definitions. The score is the share named correctly. | Theo Browne (t3.gg) | Trivia and open questions | Everyday | Not in cards | Gemini 3.1 Pro 96.9% | Mar 2026 | Mar 2026 | ||
| Spiral-Bench v1.2 Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score. | EQ-Bench (Sam Paech) | Wellbeing and sycophancy | Everyday | Not in cards | GPT-5 Chat 70.8 | Oct 2025 | Dec 2025 | ||
| Spiral-Bench v1.0 Replaced in Oct 2025 | Not in cards | GPT-5 87.0 | Aug 2025 | Sep 2025 | |||||
| BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. | Quesma | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 4.6 49.1% | Dec 2025 | Mar 2026 | ||
| BixBench3 Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts. | Edison Scientific | Research science | Research frontier | Not in cards | Gemini 4 Argon 52.3 | Aug 2026 | Sep 2026 | ||
| BixBench v1 Replaced in Aug 2026 | FutureHouse | Lab and data work | Expert | Fading | Grok 4.5 93.8% | 1 | 3 | Feb 2025 | Sep 2026 |
| CompileBench Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. | Quesma | Terminal tasks | Expert | Not in cards | Claude Opus 4.1 100% | Aug 2025 | Oct 2025 | ||
| Dictatorship Eval Scenarios at six difficulty levels testing whether models help with authoritarian requests, from blunt asks to requests hidden in code edits. LLM judge, no self-grading. | Free Systems (Andy Hall, Stanford Hoover) | Refusing harmful requests | Not in cards | Claude Fable 5 91.3% | Apr 2026 | Jul 2026 | |||
| FACTS Parametric V2 Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer. | Google DeepMind + Google Research + Kaggle | Short factual questions | University | Not in cards | GPT-6 Astra 83.8% | Oct 2026 | Oct 2026 | ||
| FACTS Parametric v1 Replaced in Oct 2026 | Fading | GPT-6 Astra 83.1% | 1 | Dec 2025 | Oct 2026 | ||||
| Game Arena Five in a Row Language models play a gomoku-style game and try to line up five of their tokens before the opponent does. | Kaggle and Google DeepMind | Puzzles and chess | Everyday | Not in cards | Gemini 3.1 Pro 1,131 | Apr 2026 | Jun 2026 | ||
| RCM-Bench-1 Agents handle healthcare revenue cycle work, such as claim submission, billing and eligibility checks, with tools. Checks cover tool calls, arguments, replies and final system state. | Commure | Medicine | Expert | Not in cards | Claude Opus 5 92.7% | Oct 2026 | |||
| RISEBench++ Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types. | VisionXLab (Shanghai Jiao Tong University) and collaborators | General visual questions | University | Not in cards | GPT-Image-2.5 Sunburst 56.6% | Oct 2026 | |||
| RISEBench v1 Replaced in Oct 2026 | System cards only | GPT-Image-1.5 51.4% | 1 | Apr 2025 | May 2025 | ||||
| TasteVal 8 AI R&D tasks. Models iteratively design experiments. Taste measured as the compute needed to reach expert-level scores. | P-Zero Research | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 230.4× | Oct 2026 | Sep 2026 | ||
| AlgoTune Agents rewrite code for 154 math, physics, and computer science functions to run faster and stay correct. Scoring is harmonic mean speedup over reference code. | University of Tübingen and Princeton University | Speed and optimization | Expert | Not in cards | GPT-5.2 2.05× | Jul 2025 | Mar 2026 | ||
| Brokk Power Ranking Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score. | Brokk (SlopCop) | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 37.9 | Aug 2025 | Oct 2026 | ||
| CEO-Bench Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score. | Princeton University (Narasimhan and Liu labs) | Multi-agent games | Expert | Not in cards | Kimi K3 $22,148,357 | Jun 2026 | Aug 2026 | ||
| Clipping Bench Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets. | Naïve | Simulated companies | Expert | Not in cards | GPT-6 Astra $9,514 | Oct 2026 | Sep 2026 | ||
| Code2Video Benchmark 168 product-video briefs. Agent writes HTML motion-graphics code rendered by HyperFrames. A trained pairwise judge on five axes produces Elo. | HeyGen Research + Kaggle | Video generation | Expert | Not in cards | Claude Opus 5.5 1,585 | Sep 2026 | Sep 2026 | ||
| Creation-MMBench Models write creative pieces grounded in up to nine images plus a role, background and instructions, across 765 cases in 51 tasks. A judge scores quality and visual fact alignment against case-specific criteria. | OpenCompass (Shanghai AI Laboratory) | Creative writing | Everyday | Not in cards | Gemini 2.0 Pro 4.48 | Mar 2025 | Feb 2025 | ||
| Drone-Bench Agent writes code for a small real drone across five tasks (reconstruct, localize, navigate, detect, follow). Each scored against a human-written baseline. | Andon Labs | Real robots | Expert | Not in cards | Claude Opus 5.5 97.3% | Jul 2026 | Sep 2026 | ||
| DrugDiscoveryBench Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics. | Scale AI | Genomics and life sciences | Expert | Not in cards | GPT-6 Astra 68.7% | Jun 2026 | Oct 2026 | ||
| E-Commerce Bench Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance. | Qwen Team, Alibaba Group, with Taobao and Tmall Group | Running a business | University | Not in cards | GPT-5.6 Sol 14.3 | Aug 2026 | Aug 2026 | ||
| FrontierPhysics 56 specialist physics-research tasks in 9 domains. Graded by verifier pass plus a weighted rubric score over 3 trials. | BenchFlow | Research science | Expert | Not in cards | Claude Opus 5.5 22.6% | Aug 2026 | Oct 2026 | ||
| Leipzig Benchmark Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026. | Benchmarks in Leipzig workshop at MPI MiS (49 mathematicians), board on ScienceBench (Christian Stump) | Research mathematics | Research frontier | Not in cards | GPT-5.6 Sol Pro 91% | Jun 2026 | Sep 2026 | ||
| OpenDesign Arena Models create design artifacts in the OpenDesign tool, and the board reports an evaluation score out of 100 with cost and speed per artifact. | OpenDesign | Web and app front ends | Everyday | Not in cards | Kimi K3.1 Pro 91.0 | Oct 2026 | |||
| RoboRewardBench Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses. | Stanford (CRFM HELM, Percy Liang) + collaborators | Real robots | Everyday | Not in cards | GPT-5.5 0.66 | Jan 2026 | Apr 2026 | ||
| SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. | Scale AI | Chat and instructions in many languages | Everyday | Not in cards | o1 1,134 | May 2024 | Jan 2025 | ||
| ApprenticeBench Agent joins a simulated company and processes 100 accounts-payable bills in Odoo via GUI or API, learning from records and feedback. Cumulative success rate. | NeoCognition | Web browsing | University | Not in cards | Claude Fable 5.1 72% | Sep 2026 | Sep 2026 | ||
| CivBench Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy. | Vox Deorum | Video games | Expert | Not in cards | GLM-5.3 1,634 | Oct 2026 | Sep 2026 | ||
| Image2Struct Vision-language models turn images of webpages, LaTeX and musical scores into code that renders them. The render is compared with the input image. | Stanford CRFM | Documents, charts and text in images | University | Not in cards | GPT-4o 94.7 | Oct 2024 | Oct 2024 | ||
| NatureBench Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results. | Frontis AI | Research science | Research frontier | Not in cards | Claude Opus 5 26.7% | Jun 2026 | Sep 2026 | ||
| Physical-Engineering's Last Exam Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective. | Google Cloud AI Research | CAD models | Expert | Not in cards | Claude Opus 5.5 0.49 | Oct 2026 | Sep 2026 | ||
| RSI-Exam Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research. | AIMING Lab (UNC Chapel Hill) | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 53.6% | Aug 2026 | Sep 2026 | ||
| SEAL Precise Instruction Following Models answer 1,054 private prompts with detailed instructions in nine categories. Human annotators check 12 yes or no criteria and compare responses side by side on a 7-point scale. | Scale AI | Following precise instructions | Everyday | Not in cards | o1 92.0 | May 2024 | Jan 2025 | ||
| SuperCLUE The CLUE team tests Chinese and foreign language models on periodic sets of Chinese questions covering language, professional knowledge, reasoning, agents and safety. Open answers are graded by a strong LLM judge, and results feed an overall ranking. | CLUE benchmark team (SuperCLUE) | Composite indices | University | Not in cards | Qwen3.8 Max (0902) 72.6 | May 2023 | Sep 2026 | ||
| Agent Poker Bench Models play 10-seat no-limit Texas Hold'em tournaments against each other. Chip counts and elimination order feed TrueSkill ratings. | Vals AI (with The Graphite Group) | Multi-agent games | Expert | Not in cards | GPT-5.2 1,132 | Jan 2026 | Dec 2025 | ||
| CORE-Bench Agents reproduce the results of published studies from their code and data. 270 tasks from 90 papers in computer science, social science and medicine come in three difficulty levels. | Princeton University | Research science | Expert | Not in cards | Claude Opus 4.5 77.8% | Sep 2024 | Nov 2025 | ||
| Echo General AI Prediction Leaderboard Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models. | UniPat AI and Peking University | Real-world events | Expert | Not in cards | DeepSeek-V4-Pro 1,018 | Mar 2026 | Apr 2026 | ||
| HELM Classic Models are evaluated with standardized prompts on the scenarios from the original HELM paper. Each scenario reports accuracy, calibration, robustness, fairness, bias, toxicity and efficiency. | Stanford CRFM | Composite indices | University | Not in cards | Llama 2-70B 94.3 | Nov 2022 | Jul 2023 | ||
| Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. | Nous Research | General assistants | Expert | Not in cards | Claude Opus 5.5 76.7% | Oct 2026 | Oct 2026 | ||
| Hermes Index Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent. | Nous Research | Composite indices | Expert | Not in cards | Claude Opus 5.5 63.3 | Oct 2026 | Oct 2026 | ||
| Legora BAR Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers. | Legora | Law | Expert | Not in cards | Claude Opus 5.5 106.0 | Sep 2026 | Oct 2026 | ||
| Remote Labor Index Agents complete real paid remote freelance projects from many sectors. Reviewers judge whether each deliverable would be accepted as commissioned work, giving an automation rate. | Center for AI Safety and Scale AI | Tasks from real jobs | Expert | Not in cards | Claude Opus 5.5 21.3% | Oct 2025 | Sep 2026 | ||
| Slop Score Model writing is scored by how often it uses words and patterns that are far more common in AI text than in human text. Lower means less slop. | EQ-Bench (Sam Paech) | Taste and style | Everyday | Not in cards | Kimi K2 18.3 | Oct 2025 | Sep 2025 | ||
| SWE Atlas Refactoring Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result. | Scale AI | Fixing real repositories | Expert | Not in cards | GPT-6 Astra 59% | May 2026 | Sep 2026 | ||
| SWE-Marathon v1.1 Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials. | Abundant AI | Building whole projects | Expert | Not in cards | Muse Wonder Full 53.1% | Jul 2026 | Oct 2026 | ||
| SWE-Marathon 1.0 Replaced in Jul 2026 | In use | Grok 4.5 29% | 9 | 6 | Jun 2026 | Oct 2026 | |||
| Agentic BAIM-LLM Evaluation (ABLE) Agents run a viral-protein redesign workflow. Nine tasks cover structure retrieval, design strategy, tool selection and biological AI model use. Outputs are auto-scored. | SecureBio | Hazardous knowledge | Research frontier | Not in cards | Gemini 3.1 Pro 84.6% | Oct 2026 | |||
| BalatroBench Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run. | Coder | Video games | Everyday | Not in cards | Gemini 3 Pro 19.3 | Aug 2025 | Jan 2026 | ||
| FrontierCS 2 Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference. | FrontierCS team (UC Berkeley and others) | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 58.3% | Oct 2026 | Oct 2026 | ||
| FrontierCS 1.0 Replaced in Oct 2026 | Competitive programming | System cards only | GPT-5.6 Sol 76.4 | 1 | Dec 2025 | Jul 2026 | |||
| SWE-Bench Pro V2 2.0 Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container. | Scale AI (co-developed with Reflection) | Fixing real repositories | Expert | Not in cards | Claude Opus 5 99.4% | Sep 2026 | Oct 2026 | ||
| SWE-Bench Pro 1.0 Replaced in Sep 2026 | Scale AI | In use | Muse Spark 1.1 61.5% | 73 | 35 | Sep 2025 | Oct 2026 | ||
| BrowseSafe++ v2 Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. | Perplexity | Jailbreaks and prompt injection | University | Not in cards | GPT-6 Astra 91.5% | Oct 2026 | Sep 2026 | ||
| BrowseSafe-Bench v1 Replaced in Oct 2026 | Not in cards | Nov 2025 | |||||||
| CadEval Models write OpenSCAD code for single mechanical parts from text descriptions. Rendered parts are checked against reference geometry, scored by pass rate. | Will Patrick | CAD models | University | Not in cards | o3 74% | Apr 2025 | Apr 2025 | ||
| FACTS Grounding v2 Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document. | Google DeepMind + Google Research + Kaggle | Faithful to sources | University | Not in cards | GPT-6 Astra 80.1% | Dec 2025 | Oct 2026 | ||
| FACTS Grounding v1 Replaced in Dec 2025 | Google DeepMind + Kaggle | Dropped | Kimi K2 88.5% | 10 | 11 | Jan 2025 | Oct 2026 | ||
| GSM1k Models solve new grade school math word problems written to mirror GSM8K in style and difficulty. The gap between GSM8K and GSM1k accuracy shows overfitting or contamination. | Scale AI | Word problems | School | Not in cards | Claude 3.5 Sonnet 96.6% | May 2024 | Aug 2024 | ||
| Long-Horizon-Terminal-Bench 2.0 Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. | Tencent HY + university collaborators (Zongxia Li et al.) | Long terminal sessions | Expert | Not in cards | GPT-6 Astra 69.2% | Oct 2026 | Sep 2026 | ||
| Long-Horizon-Terminal-Bench (LHTB) 1.0 Replaced in Oct 2026 | Not in cards | DeepSeek V4 Flash 0731 60.2% | Jul 2026 | Jul 2026 | |||||
| SWE-bench Lite Models resolve 300 self-contained GitHub issues chosen from SWE-bench to cut evaluation cost. Hidden repository tests decide each patch. | Princeton University (SWE-bench team) | Fixing real repositories | Expert | Not in cards | Claude Sonnet 4 60.3% | Mar 2024 | Jul 2025 | ||
| Thrive Holdings helpdesk ticket resolution benchmark Agent investigates IT support requests, takes action and decides when to escalate to a technician. Scored by pass rate. | Thrive Holdings | Tasks from real jobs | University | Not in cards | GPT-6 Astra 61.5% | Oct 2026 | Sep 2026 | ||
| FrontierMath Erdős Models try to prove or disprove 68 open Erdős problems in Lean 4. A result counts only if the Lean checker accepts the full proof. | Epoch AI | Research mathematics | Research frontier | Not in cards | GPT-6.1 Sol 4.41% | Sep 2026 | Sep 2026 | ||
| Next.js Agent Evals Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost. | Vercel | Web and app front ends | Expert | Not in cards | Claude Fable 5.1 97% | Oct 2025 | Oct 2026 | ||
| OTelBench Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation. | Quesma | Fixing real repositories | Expert | Not in cards | Claude Opus 4.5 29% | Jan 2026 | Dec 2025 | ||
| SlopCodeBench Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity. | University of Wisconsin-Madison (SprocketLab) | Building whole projects | Expert | Not in cards | GPT-5.5 28.1% | Mar 2026 | Apr 2026 | ||
| ToRR Models answer questions over tables given in several formats, and the board checks how much accuracy changes when the table format or prompt changes. | IBM Research (on HELM) | Language reasoning | University | Not in cards | Claude 3.5 Sonnet (October 2024) 49.9 | Feb 2025 | Mar 2025 | ||
| BlenderWorld Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project. | Alibaba Group (Qwen, Alibaba Token Hub) + Zhejiang University + CUHK | CAD models | Expert | Not in cards | GPT-6 Astra 79.9 | Oct 2026 | Sep 2026 | ||
| CommerceAgentBench Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds. | Alibaba.com (Accio) | Running a business | Expert | Not in cards | Claude Opus 5 61.7% | Aug 2026 | Aug 2026 | ||
| EvoCode-Bench v2 Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit. | UniPat AI with Peking University, Tsinghua University and HKU | Building whole projects | Expert | Not in cards | Claude Opus 4.8 59.1% | Jun 2026 | Jun 2026 | ||
| EvoCode-Bench v1 Replaced in Jun 2026 | Not in cards | Claude Opus 4.7 54% | May 2026 | Apr 2026 | |||||
| Holdings-IndividualTaxBench 88 real client document packages turned into Form 1040 fields. Fact-level accuracy against practitioner-reviewed returns, with an LLM judge matching values. | Thrive Holdings | Tasks from real jobs | Expert | Not in cards | Claude Opus 5.5 76.3% | Oct 2026 | Sep 2026 | ||
| ITBench Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. | IBM Research | Terminal tasks | Expert | Not in cards | Gemini 3 Pro 58.3% | Feb 2025 | Nov 2025 | ||
| MedHELM Models run a suite of medical tasks chosen by clinicians across five categories, from clinical decision support and note writing to patient messages and administration. | Stanford CRFM (HELM) + Stanford Medicine | Medicine | Expert | Not in cards | GPT-5 70.3 | Feb 2025 | Aug 2025 | ||
| RealmPathology Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria. | micro1 | Medicine | Expert | Not in cards | Claude Fable 5.1 78% | Oct 2026 | Oct 2026 | ||
| StockBench Agents get daily prices, fundamentals and news over months and decide to buy, sell or hold. Scoring uses cumulative return, maximum drawdown and related metrics. | StockBench authors | Finance | Expert | Not in cards | Qwen3-235B-A22B-Thinking 2.50% | Oct 2025 | Sep 2025 | ||
| Time Horizon Index: KSP Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions. | Vals AI | Video games | Expert | Not in cards | Claude Opus 5.5 91.3% | Jul 2026 | Sep 2026 | ||
| AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. | Tsinghua THUDM (Zhipu AI) | Function calling | Everyday | Not in cards | Claude Sonnet 4.5 58.9% | Oct 2025 | Sep 2025 | ||
| AgentBench v0.2 Replaced in Oct 2025 | Tsinghua THUDM | General assistants | Not in cards | Aug 2023 | |||||
| CAD Arena Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match. | Normal Research | CAD models | Expert | Not in cards | Gemini 4 Argon 0.75 | Sep 2026 | Sep 2026 | ||
| Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. | Claw-Eval-Live team (Li, Tang et al.) | General assistants | University | Not in cards | Claude Opus 4.6 66.7% | Apr 2026 | Mar 2026 | ||
| MathIF Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer. | Renmin University of China, Shanghai AI Laboratory and CUHK | Following precise instructions | School | Not in cards | Qwen3-14B 50.7% | May 2025 | Apr 2025 | ||
| Monthly-SWEBench Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1. | UniPat AI and xbench (HongShan) | Fixing real repositories | Expert | Not in cards | GPT-5.5 76% | Mar 2026 | Jun 2026 | ||
| PharmaBench (micro1) Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff. | micro1 | Medicine | Expert | Not in cards | Kimi K3.1 Pro 27.5% | Oct 2026 | Oct 2026 | ||
| RadLE 2.0 Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors. | CRASH Lab, Koita Centre for Digital Health, Ashoka University | Medicine | Expert | Not in cards | Claude Fable 5 758.0 | Jul 2026 | Jul 2026 | ||
| Radiology's Last Exam (RadLE) 1.0 Replaced in Jul 2026 | Not in cards | Sep 2025 | |||||||
| WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. | micro1 | Wellbeing and sycophancy | Everyday | Not in cards | GPT-6.1 Sol 66% | Oct 2026 | Oct 2026 | ||
| WitnessBench Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels. | Fleet AI | Puzzles and chess | Expert | Not in cards | Claude Opus 5.5 46.5% | Sep 2026 | Sep 2026 | ||
| Android Bench 2.0 Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run. | Google (Android Developers) | Building whole projects | Expert | Not in cards | Claude Sonnet 5.5 40% | Sep 2026 | Sep 2026 | ||
| AndroidBench 1.0 Replaced in Sep 2026 | Fixing real repositories | In use | Claude Opus 5 91.8% | 1 | Mar 2026 | Aug 2026 | |||
| CritPt v2 Models solve unpublished research-level physics challenges written by working physicists, with answers graded automatically. The v2 release revised the public test challenges. | CritPt collaboration | Research science | Research frontier | Not in cards | GPT-6 Astra 83.1% | Oct 2026 | Oct 2026 | ||
| CritPt v1 Replaced in Oct 2026 | In use | Gemini 4 Carbon 36% | 12 | 6 | Sep 2025 | Oct 2026 | |||
| Game Arena Chess Openings Language models play chess against each other from varied two-move openings sampled from 20 popular human openings. The board is given as text. | Kaggle and Google DeepMind | Puzzles and chess | Expert | Not in cards | Gemini 3 Flash 1,263 | Feb 2026 | Dec 2025 | ||
| Human Behavior Bench Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance. | Rehearsals with Google DeepMind and Kaggle | Real-world events | Expert | Not in cards | Gemini 4 Argon 80.6 | Oct 2026 | Sep 2026 | ||
| MME-CoT Models reason step by step on image questions in math, science, OCR, logic, space-time and general scenes. A judge scores reasoning quality, robustness and efficiency. | MME team (CUHK MMLab and others) | General visual questions | University | Not in cards | Kimi k1.5 64.2% | Feb 2025 | Jan 2025 | ||
| OCR Arena Users upload a document, compare two anonymous OCR or vision-language models side by side, and vote. Votes become Elo ratings. | Extend | Documents, charts and text in images | Everyday | Not in cards | Claude Opus 4.5 1,791 | Nov 2025 | Feb 2026 | ||
| RWE-bench Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement. | Xplore AI | CAD models | Expert | Not in cards | GPT-6 Astra 37% | Oct 2026 | Oct 2026 | ||
| ScienceVision Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field. | UniPat AI and Elorian | Spatial reasoning and counting | Expert | Not in cards | GPT-6 Astra 0.53 | Oct 2026 | Sep 2026 | ||
| AHELM Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning. | Stanford CRFM | Audio understanding and voice | Everyday | Not in cards | Gemini 2.5 Pro 80.3 | Aug 2025 | Sep 2025 | ||
| BTF-3 Agents forecast already-resolved binary and numeric questions while researching a frozen snapshot of the web. Scoring pools Brier and ranked probability scores. | FutureSearch | Real-world events | Expert | Not in cards | Claude Opus 5 0.12 | Jun 2026 | Sep 2026 | ||
| BTF-2 Replaced in Jun 2026 | Not in cards | Apr 2026 | |||||||
| Bench to the Future v1 Replaced in Apr 2026 | Not in cards | Jun 2025 | |||||||
| CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. | Center for AI Safety | Deception and alignment | Expert | Not in cards | Gemini 4 Argon 8.60% | Sep 2026 | Oct 2026 | ||
| Continual Learning Exam Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics. | University College London (Jun Wang) + Imperial College London + University of Liverpool | Learning from the context | Expert | Not in cards | Claude Opus 5 47.7% | Oct 2026 | Aug 2026 | ||
| Dual-Arm Sim Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit. | micro1 | Simulated robots | Expert | Not in cards | GPT-6 Astra 49.5 | Oct 2026 | Oct 2026 | ||
| ExplorationBench Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help. | Tencent Hunyuan | Research science | Research frontier | Not in cards | Claude Opus 5 89% | Sep 2026 | Sep 2026 | ||
| FoodTruck Bench Agents run a food truck in Austin, Texas for a 30-day simulation, choosing locations, menus, prices, inventory, staff and loans with 34 tools. Humans can play the same game. | FoodTruck Bench maintainers | Simulated companies | Everyday | Not in cards | Claude Opus 5 $75,264 | Oct 2026 | |||
| GameDevBench Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result. | Carnegie Mellon University | Building whole projects | University | Not in cards | Claude Opus 5.5 70.6% | Feb 2026 | Sep 2026 | ||
| IMProofBench Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically. | ETH Zurich (Johannes Schmitt et al.) with a contributor community | Research mathematics | Research frontier | Not in cards | GPT-5.4 49.3% | Sep 2025 | Mar 2026 | ||
| LMArena Video Edit Arena Video editing models apply the same edit request to the same clip, anonymously and side by side. Community votes give Elo-style ratings. | Arena (formerly LMArena) | Video generation | Everyday | Not in cards |  Wan 3.0 1,414 | Aug 2026 | |||
| OR-License Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem. | ALLE AI | Tasks from real jobs | Expert | Not in cards | Grok 4.6 88.4% | Oct 2026 | Sep 2026 | ||
| PaperBenchX Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims. | UniPat AI | Research science | Research frontier | Not in cards | GPT-6 Astra 14% | Oct 2026 | Sep 2026 | ||
| RBench 650 image+text prompts for robot video generation (tasks and embodiments). MLLM judge scores task completion, physical plausibility, stability. | Peking University DAGroup | Video world models | Not in cards | Wan 2.6 60.7% | Jan 2026 | ||||
| ScarfBench Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change. | IBM Research | Fixing real repositories | Expert | Not in cards | Gemini 4 Argon 75.2% | May 2026 | Sep 2026 | ||
| SuperGLUE Models solve eight English language understanding tasks, including BoolQ, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC. Scored by the average task score. | NYU, Facebook AI, University of Washington, and DeepMind | Language reasoning | Everyday | Not in cards | T5-11B 88.9% | May 2019 | Apr 2022 | ||
| Tencent WorkBuddy Bench Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search. | Tencent | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 81.6% | Jul 2026 | Oct 2026 | ||
| AutoResearchExam Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed. | Bespoke Labs | Machine learning engineering | Research frontier | Not in cards | Claude Fable 5.1 0.60 | Sep 2026 | Sep 2026 | ||
| BIRD-CRITIC Models fix real user SQL issues in MySQL, PostgreSQL, SQL Server and Oracle applications. Test cases check each fix. | BIRD team (HKU) | Data science and SQL | Expert | Not in cards | o1-preview 35.5% | Jan 2025 | Mar 2025 | ||
| CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. | Salesforce AI Research | Customer service | Everyday | Not in cards | o1 64.3% | Nov 2024 | Jan 2025 | ||
| CUA-bench Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out. | Vals AI | Video games | Everyday | Not in cards | GPT-6 Astra 19.2% | Sep 2026 | Oct 2026 | ||
| DataCenterBench Tier 1 Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos. | Robocurve | Real robots | Everyday | Not in cards | GPT-6 Astra 40% | Oct 2026 | Oct 2026 | ||
| DeepTerminalBench Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4. | UniPat AI | Terminal tasks | Expert | Not in cards | Claude Opus 4.7 34% | May 2026 | Apr 2026 | ||
| EnterpriseClawBench Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics. | Frontis AI | Documents and spreadsheets | Expert | Not in cards | GPT-5.5 66.3% | Jun 2026 | Jun 2026 | ||
| MDPBench Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage. | Huazhong University of Science and Technology (MultimodalOCR group) | Documents, charts and text in images | Everyday | Not in cards | Gemini 3 Pro 86.4 | Mar 2026 | Jul 2026 | ||
| MissionReport Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision. | micro1 | Documents and spreadsheets | Expert | Not in cards | Gemini 4 Argon 64.1% | Oct 2026 | Oct 2026 | ||
| MMAU v05.15.25 Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set. | University of Maryland (GAMMA lab) | Audio understanding and voice | University | Not in cards | Nova 2 Omni Preview 77.8% | May 2025 | Dec 2025 | ||
| MMAU Original Replaced in May 2025 | University of Maryland | In use | Qwen3.5-Omni-Plus 82.2% | 11 | 14 | Oct 2024 | Sep 2026 | ||
| MyPCBench Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks. | Carnegie Mellon University | Desktop apps | Everyday | Not in cards | Claude Opus 4.8 62% | Jun 2026 | Jul 2026 | ||
| Quant-Arena Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window. | Einsia AI (Navers Lab, LaPlace's Eye) | Finance | Expert | Not in cards | GPT-6 Astra 51.3 | Oct 2026 | Sep 2026 | ||
| RoboColiseum AgiBot Genie robot in Genie Sim 3.0 (Isaac Sim): instruction, robustness, manipulation and spatial boards after post-training on provided sim data. | AgiBot | Simulated robots | Everyday | Not in cards | Claude Opus 5.5 76.3% | Jan 2026 | Oct 2026 | ||
| RoboDojo-RC Tier 1 Six real RoboDojo tasks on I2RT YAM arms, 20 attempts each. Robot policies and tool-calling LLMs scored by rubric progress. | Robocurve PBC | Real robots | Everyday | Not in cards | Gemini 4 Argon 48.2% | Sep 2026 | Sep 2026 | ||
| Short Form Bench Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder. | Naïve | Simulated companies | Expert | Not in cards | GPT-6.1 Sol $2,949 | Oct 2026 | Sep 2026 | ||
| StarSkirmish Bench Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots. | Kai McPheeters | Video games | Expert | Not in cards | GPT-6 Astra 57.3% | Oct 2026 | Oct 2026 | ||
| SWE-bench Models resolve 2,294 real GitHub issues from 12 Python repositories. A patch counts as resolved when the hidden repository tests pass. | Princeton University (SWE-bench team) | Fixing real repositories | Expert | Not in cards | Claude Opus 4.5 52.6% | Oct 2023 | Nov 2025 | ||
| ViBench 2.0 Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans. | ViBench team (Replit and collaborators) | Building whole projects | Expert | Not in cards | GPT-6.1 Sol 64.7% | May 2026 | Sep 2026 | ||
| VibeVideoBench Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result. | BUPT + Zhongguancun Academy + Peking University + Jilin University + Kling AI | Video generation | Expert | Not in cards | GPT-6 Astra 4.11 | Oct 2026 | Sep 2026 | ||
| Web Clone Bench Agent gets a running reference app (Gmail, Jira, Figma, etc.) and 8 hours to build a clone. Expert-written workflows are replayed as end-to-end tests. | Plato | Building whole projects | Research frontier | Not in cards | Claude Opus 5.5 34.2% | Oct 2026 | Oct 2026 | ||
| BiomniBench Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer. | Phylo, Humanlaya Data Lab and xbench (HongShan) | Genomics and life sciences | Expert | Not in cards | Claude Opus 4.7 63.9 | May 2026 | Apr 2026 | ||
| CodeClash Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings. | SWE-bench team (Stanford University and Princeton University) | Building whole projects | Expert | Not in cards | Claude Sonnet 4.5 1,385 | Nov 2025 | Sep 2025 | ||
| CriticEval Models critique responses across four dimensions in nine task scenarios, giving both scalar scores and written critiques. GPT-4 grades written critiques against a large set of annotated reference critiques. | OpenCompass (Shanghai AI Laboratory) and collaborators | Overall chat quality | University | Not in cards | GPT-4 Turbo 7.86 | Feb 2024 | Jan 2024 | ||
| Game Arena Werewolf Teams of language models play the social deduction game Werewolf in natural language. Villagers must find the hidden werewolves, who try to deceive them. | Kaggle and Google DeepMind | Multi-agent games | Expert | Not in cards | GPT-5.4 0 | Feb 2026 | Mar 2026 | ||
| LegalReasonBench Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings. | micro1 | Law | Expert | Not in cards | Gemini 4 Argon 50.8% | Oct 2026 | Sep 2026 | ||
| micro1 Medical Index Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench. | micro1 | Medicine | Expert | Not in cards | GPT-6 Astra 71% | Oct 2026 | Oct 2026 | ||
| Physics-IQ Verified Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally. | Anates Labs and Google DeepMind (Physics-IQ authors) | Physical prediction | Everyday | Not in cards | Flux 3 Video 10.9 | Jun 2026 | |||
| Physics-IQ Original Replaced in Jun 2026 | Google DeepMind | Not in cards | Sora 2 46.4% | Jan 2025 | Apr 2026 | ||||
| RSI Bench Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks. | Scale AI | AI research tasks | Research frontier | Not in cards | GPT-6 Astra 38.9 | Aug 2026 | Sep 2026 | ||
| SEAL Adversarial Robustness Models receive 1,000 single-turn adversarial prompts written by 10 full-time red teamers across nine harm categories. Safety specialists grade responses, and models are ranked by the number of high-harm violations. | Scale AI | Jailbreaks and prompt injection | Expert | Not in cards | Gemini 1.5 Pro 8.00 | Jul 2024 | |||
| VLA-Arena VLAs fine-tuned on level-0 tabletop tasks, tested on 170 tasks in 11 suites at harder unseen levels L1/L2. Success rate. | PKU-Alignment | Simulated robots | Everyday | Not in cards | MolmoAct 2 72.1% | Sep 2025 | May 2026 | ||
| xbench-Recruitment Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5. | xbench (HongShan) | Running a business | Expert | Not in cards | o3 78.5 | Jun 2025 | May 2025 | ||
| ClawBench V2 Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run. | TIGER-AI-Lab (University of Waterloo) and NAIL Group | Web browsing | Everyday | Not in cards | Claude Opus 4.7 44.6% | May 2026 | Apr 2026 | ||
| ClawBench V1 Replaced in May 2026 | Not in cards | Claude Opus 4.6 61.4% | Apr 2026 | Mar 2026 | |||||
| DeepRSI 15 open-ended ML tasks with a 48-hour budget each. Private verifiers score gains over baselines. Models ranked by Bradley-Terry Elo. | Hillclimb | AI research tasks | Research frontier | Not in cards | Claude Opus 5.5 1,629 | Oct 2026 | Sep 2026 | ||
| Embodied First Exam Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor. | Northwestern University and collaborators (CMU, UC Berkeley, Stanford and others) | Simulated robots | Everyday | Not in cards | GPT-6 Astra 32.7% | Oct 2026 | Sep 2026 | ||
| Ivo-micro1 ContractBench Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty. | micro1 with Ivo | Law | Expert | Not in cards | Grok 4.7 58.3% | Oct 2026 | Sep 2026 | ||
| LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. | Salesforce AI Research | Deep research | Expert | Not in cards | GPT-5 73.7 | Oct 2025 | Aug 2025 | ||
| OmniDocBench v1.6 Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula. | Shanghai AI Lab (OpenDataLab) | Documents, charts and text in images | Everyday | Not in cards | GLM-OCR 95.2 | Apr 2026 | Jan 2026 | ||
| OmniDocBench v1.5 Replaced in Apr 2026 | In use | GLM-OCR 94.6 | 20 | 10 | Sep 2025 | Oct 2026 | |||
| OmniDocBench 1.0 Replaced in Sep 2025 | In use | MiMo-V2.5 87.2% | 6 | 5 | Dec 2024 | Jul 2026 | |||
| PaintBench Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks. | VISIONx, New York University | General visual questions | Everyday | Not in cards | GPT-Image-2 16.3% | May 2026 | Jun 2026 | ||
| Robo-bench General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks. | Vals AI | Simulated robots | Expert | Not in cards | GPT-6.1 Sol 28.3% | Oct 2026 | Oct 2026 | ||
| SMDD-Bench 502 small-molecule drug-design tasks in 5 types. Agent uses a fixed toolkit to produce a molecule that must pass all task criteria. | CMU | Genomics and life sciences | Expert | Not in cards | GPT-5.4 40.2% | May 2026 | Mar 2026 | ||
| SpireClimb Agents play Slay the Spire 1 and 2 at max difficulty through a computer-use harness on fixed seeds. Score is the mean win rate. | Epoch AI | Video games | Expert | Not in cards | GPT-6 Astra 25.7% | Oct 2026 | Sep 2026 | ||
| SpireClimb StS1 A20 Heart Agents play Slay the Spire 1 through its graphical interface at Ascension 20 and must defeat the Heart. Scored by win rate over fixed seeds. | Epoch AI | Video games | Expert | Not in cards | GPT-6.1 Sol 20% | Oct 2026 | Sep 2026 | ||
| SpireClimb StS2 A10 Agents play Slay the Spire 2 at Ascension 10 through screenshots and mouse and keyboard, 15 runs per character. Scoring is the win rate. | Epoch AI | Video games | Everyday | Not in cards | GPT-6 Astra 34.7% | Oct 2026 | Sep 2026 | ||
| SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. | CLUE benchmark team (SuperCLUE) | Function calling | Everyday | Not in cards | GPT-4o 78.4 | Oct 2023 | Jun 2024 | ||
| SuperCLUE-Safety Chinese language models answer 4,912 open-ended safety questions over more than 20 sub-dimensions in multi-round adversarial dialogues. Follow-up turns push harder than the first prompt. | CLUE benchmark team (SuperCLUE) | Jailbreaks and prompt injection | Everyday | Not in cards |  Yi-34B 89.3 | Sep 2023 | Nov 2023 | ||
| VoiceCodeBench Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly. | besimple.ai (board run by Vals AI) | Speech recognition and translation | Everyday | Not in cards | gpt-4o-transcribe 59.7% | Aug 2026 | Mar 2026 | ||
| xbench-Marketing Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers. | xbench (HongShan) | Running a business | Expert | Not in cards | o3 50.8 | Jun 2025 | May 2025 | ||
| Brood War Bench Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game. | Ben Swerdlow | Video games | Expert | Not in cards | GPT-6 Astra 100% | Sep 2026 | Sep 2026 | ||
| CommonsenseQA 2.0 Models answer 14,343 yes or no commonsense questions that players wrote to fool an AI in a game. Scored by accuracy. | Allen Institute for AI and Tel Aviv University | Commonsense | Everyday | Not in cards | T5-11B 67.8% | Jan 2022 | Jul 2023 | ||
| CommonSenseQA 1.0 Replaced in Jan 2022 | Tel Aviv University and Allen Institute for AI | Dropped | PaLM 2-L 90.4% | 5 | 19 | Nov 2018 | Apr 2025 | ||
| CooperBench Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages. | Stanford NLP | Fixing real repositories | Expert | Not in cards | GPT-5 27.9% | Jan 2026 | Dec 2025 | ||
| Epoch Automation Reports Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees. | Epoch AI | Simulated companies | Expert | Not in cards | Claude Fable 5.1 65.3% | Oct 2026 | Sep 2026 | ||
| Factorio Learning Environment Agents write Python programs that control a Factorio factory, in lab tasks with fixed resources and in open play. Scored by production throughput, milestones and lab task success. | Jack Hopkins et al. (FLE team) | Video games | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 293206.0 | Mar 2025 | Feb 2025 | ||
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Alibaba Qwen Team, Alibaba Cloud security and National University of Singapore | Secure code and review | Expert | Not in cards | Claude Fable 5 25.1% | Oct 2026 | Aug 2026 | ||
| LLM-AggreFact Models judge whether claims are supported by a grounding document across 11 pooled datasets, such as RAGTruth and ExpertQA. Scored by balanced accuracy. | UT Austin + Salesforce AI Research (Tang, Laban and Durrett) | Faithful to sources | Everyday | Not in cards | Claude 3.5 Sonnet 77.2% | Apr 2024 | Dec 2024 | ||
| o11y-bench Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment. | Grafana Labs | Terminal tasks | Expert | Not in cards | Claude Opus 5.5 93.8% | Apr 2026 | Oct 2026 | ||
| OmniParseBench Document parsers convert single pages to text, and 16,288 binary unit tests on 2,937 pages check the output. The headline joins text, table and layout subscores. | Datalab | Documents, charts and text in images | Everyday | Not in cards | Gemini 3.8 Flash 92.9 | Oct 2026 | Sep 2026 | ||
| RoboDojo Generalist manipulation policies run 42 Isaac Sim tasks and 18 remote real-robot tasks. Scored by task score and success rate. | MMLab@HKU + UC Berkeley + Tsinghua + others (RoboDojo team) | Simulated robots | Everyday | Not in cards | GPT-6 Astra 29% | Jul 2026 | Sep 2026 | ||
| SciUniverse 92 tasks in 17 chemistry, biology and materials families run in C5R's lab or its digital twin. Pass@1 averaged equally across families. | C5R | Lab and data work | Expert | Not in cards | Claude Fable 5.1 45.3% | Sep 2026 | Sep 2026 | ||
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | University of Illinois Urbana-Champaign (Lingming Zhang) and University of Melbourne | Finding and exploiting real bugs | Expert | Not in cards | GPT-5.5 58.4% | Jun 2026 | Apr 2026 | ||
| SEC-bench Pro v260505 Replaced in Jun 2026 | University of Illinois Urbana-Champaign (Lingming Zhang) | In use | Gemini 4 Carbon 88.2% | 5 | 16 | May 2026 | Oct 2026 | ||
| SWE-PolyBench Coding agents resolve 2,110 tasks from 21 repositories in Java, JavaScript, TypeScript and Python. Tasks cover bug fixes, features and refactors, checked by execution. | Amazon | Fixing real repositories | Expert | Not in cards | Claude 3.5 Sonnet (October 2024) 14.1% | Apr 2025 | Jan 2025 | ||
| VGI-Bench Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame. | University of Illinois Urbana-Champaign | Video world models | Everyday | Not in cards | Dreamina Seedance 2.0 51% | Aug 2026 | |||
| ATLAS (OpenCompass) Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge. | OpenCompass (Shanghai AI Laboratory) and collaborators | Science questions | Research frontier | Not in cards | GPT-5 42.9% | Nov 2025 | Aug 2025 | ||
| Build-A-Chip-Bench Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates. | ChipAgents (with NVIDIA contributions) | Electronics | Research frontier | Not in cards | Claude Fable 5.1 16.6% | Oct 2026 | Sep 2026 | ||
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | UC Berkeley RDI (Sunblaze) and collaborators | Finding and exploiting real bugs | Expert | Not in cards | Claude Opus 5 89% | Jun 2026 | Jul 2026 | ||
| DecodingTrust Models are tested on eight trustworthiness areas, including toxicity, stereotype bias, robustness, privacy, machine ethics and fairness. | UIUC Secure Learning Lab (Bo Li) and coauthors | Bias, fairness and politics | Everyday | Not in cards | Gemini 1.0 Pro 80.6 | Jun 2023 | Dec 2023 | ||
| EdgeBench Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment. | ByteDance Seed | Long terminal sessions | Expert | Not in cards | Claude Opus 4.8 51.3 | Jul 2026 | May 2026 | ||
| K-Sort Arena Users rank outputs from K image or video generation models at once instead of a pair. Bayesian updates turn the rankings into ratings. | Li et al. (UC Berkeley and collaborators) | Image generation and editing | Everyday | Not in cards | GPT-Image-1 31 | Aug 2024 | |||
| VideoGameBench Vision-language models play 10 popular 1990s Game Boy and MS-DOS games in real time from raw frames. Scored by the share of each game completed. | Alex Zhang, Ofir Press and collaborators | Video games | Expert | Not in cards | Gemini 2.5 Pro 0.48% | May 2025 | Apr 2025 | ||
| CUA-World-Long Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task. | Carnegie Mellon University (Welleck and Neubig labs, CMU L3) | Desktop apps | Expert | Not in cards | GPT-5.4 27.5% | Apr 2026 | Mar 2026 | ||
| HarnessEval-W Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning. | MirroS Lab with xbench (HongShan) and universities | Interactive world models | Expert | Not in cards | Dreamina Seedance 2.0 75.5 | Aug 2026 | |||
| LADDER Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place. | INSAIT | Simulated robots | Everyday | Not in cards | GPT-6 Astra 63.3% | Oct 2026 | Sep 2026 | ||
| MobileGym Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state. | MobileGym authors | Phones | Everyday | Not in cards | Gemini 3.1 Pro 58.8% | May 2026 | Mar 2026 | ||
| Project Euler (MathArena) Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943. | MathArena (ETH Zurich SRI Lab and INSAIT) | Competition problems with answers | Expert | Not in cards | Gemini 3.1 Pro 89% | Sep 2025 | May 2026 | ||
| RedlineBench Agent acts as counsel at one turn of a SaaS contract negotiation and returns tracked-change redlines. Three LLM judges grade attorney-written rubrics. | Crosby Intelligence + micro1 | Law | Expert | Not in cards | GPT-5.5 50.5% | Jun 2026 | Jun 2026 | ||
| ReviewBench AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1. | GitHub and Microsoft | Fixing real repositories | Expert | Not in cards | Claude Opus 5.5 73.3% | Oct 2026 | Sep 2026 | ||
| RoboCasa365 Mobile-manipulator kitchen sim: train on 300-task Human300 data, test on 50 tasks incl. 16 unseen composite tasks. Success rate. | UT Austin + NVIDIA (RoboCasa team) | Simulated robots | Everyday | Not in cards | GR00T N1.5 23.9% | Mar 2026 | Dec 2025 | ||
| RoboCasa Original Replaced in Mar 2026 | Fading | GR00T N1.7 70.8% | 4 | Jun 2024 | Jul 2026 | ||||
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Vals AI | Deep research | Expert | Not in cards | GPT-5.6 Sol 49.1% | Jul 2026 | Jul 2026 | ||
| WBench Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics. | Meituan LongCat | Interactive world models | Everyday | Not in cards | Kling v3 Pro 79.0 | May 2026 | |||
| AI Detection Benchmark Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy. | Vals AI | Taste and style | Expert | Not in cards | Claude Opus 5.5 98.4% | Sep 2026 | Sep 2026 | ||
| Decision Index Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models. | Hugging Face community (apolinario, multimodalart) | Composite indices | Everyday | Not in cards | Gemma 4 31B IT 67.6 | Sep 2026 | Aug 2026 | ||
| iOSWorld Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history. | Carnegie Mellon University (Salakhutdinov lab) | Phones | Everyday | Not in cards | Claude Opus 4.6 51.9% | Jun 2026 | Mar 2026 | ||
| LIBERO-Plus LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate. | Fudan University + Tongji University + Shanghai Innovation Institute + NUS | Simulated robots | Everyday | Not in cards |  OpenVLA-OFT 69.6% | Oct 2025 | Feb 2025 | ||
| LIBERO-PRO LIBERO policies tested under object, position, instruction, task and environment perturbations. 50 episodes per task. Success rate. | Huazhong University of Science and Technology + Harvard + MIT + others | Simulated robots | Everyday | Not in cards | π0.5 53% | Oct 2025 | Apr 2025 | ||
| LongExtractionBench Extraction systems pull structured values from 225 long table-heavy documents averaging 358 pages and about 88,700 ground-truth fields each. | micro1 | Documents, charts and text in images | Expert | Not in cards | Claude Opus 4.8 70.7% | Jun 2026 | May 2026 | ||
| PhAIL DROID: Everyday tasks Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds. | Positronic Robotics | Real robots | Everyday | Not in cards | MolmoAct 2 46.6% | Oct 2026 | May 2026 | ||
| RoboArena Double-blind pairwise A/B trials of generalist DROID policies on real Franka arms, free-form tasks by volunteer evaluators. Elo-style rating. | UC Berkeley + Stanford + UPenn + others | Real robots | Everyday | Not in cards | π0.5 1,608 | Jun 2025 | Apr 2025 | ||
| AGI Score (A Definition of AGI) Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%. | Center for AI Safety and collaborators (Hendrycks et al.) | Composite indices | Expert | Not in cards | GPT-5 57% | Oct 2025 | Aug 2025 | ||
| AndroidLab Mobile agents complete 138 tasks across nine apps on predefined Android virtual devices, in text-only and multimodal modes. Checks on the final device state score each task. | Tsinghua University and Zhipu AI (THUDM) | Phones | University | Not in cards | GPT-4o 31.2% | Oct 2024 | Nov 2024 | ||
| PhAIL Real Franka bin-to-bin picking of four object types. VLAs scored on throughput and completion against human and teleop references. | Positronic Robotics | Real robots | Everyday | Not in cards | π0.5 75.1% | Mar 2026 | Dec 2025 | ||
| RoboTwin 2.0 Bimanual Aloha-AgileX sim: 50 tasks, policies trained on 50 clean demos per task, 100 trials each in clean and domain-randomized scenes. | RoboTwin Platform (HKU MMLab + others) | Simulated robots | Everyday | Not in cards | π0.5 58.3% | Jun 2025 | Apr 2025 | ||
| RoboTwin 1.0 Replaced in Jun 2025 | Not in cards | Sep 2024 | |||||||
| STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. | Microsoft | Customer service | Expert | Not in cards | GPT-5.5 58.9% | May 2026 | Apr 2026 | ||
| VideoPhy-2 200 action-centric prompts. Generated videos human-rated for semantic adherence, physical commonsense and physical-rule grounding. | UCLA + Google | Physical prediction | Not in cards | Sora 23.3% | Mar 2025 | ||||
| VideoPhy v1 Replaced in Mar 2025 | UCLA and Google Research (Hritik Bansal et al.) | Everyday | Not in cards | Jun 2024 | |||||
| AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. | xbench (HongShan) | General assistants | Expert | Not in cards | ChatGPT agent 62.6% | Dec 2025 | Jul 2025 | ||
| DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. | University of Science and Technology of China and Metastone Technology | Deep research | Expert | Not in cards | OpenAI deep research 43.1% | Jan 2026 | Feb 2025 | ||
| IMC 2025 Models write proofs for the 10 problems of the 2025 International Mathematics Competition for University Students. Human judges grade each proof. | IMC (contest), MathArena (evaluation) | Written proofs | University | Not in cards | Gemini 2.5 Pro 94.5% | Aug 2025 | Jun 2025 | ||
| Miklós Schweitzer 2025 Models write proofs for the 2025 Miklós Schweitzer competition, a hard Hungarian university-level contest. MathArena runs and grades it. | MathArena (ETH Zurich SRI Lab) | Written proofs | Research frontier | Not in cards | GPT-5 Pro 90% | Dec 2025 | Oct 2025 | ||
| MU-Bench Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts. | Sierra | Speech recognition and translation | Everyday | Not in cards | gpt-4o-mini-transcribe 27.3% | Apr 2026 | Mar 2025 | ||
| PAI-Bench Physical-AI video generation from 1,044 prompts in six domains, VLM-QA plus quality metrics. Companion control and understanding tracks. | SHI Labs + NVIDIA | Video world models | Not in cards | Veo 3 82.1% | Dec 2025 | ||||
| SuperCLUE-RAG Chinese models answer questions twice, first without documents and then with supplied documents. A judge model compares the answers to rate how well the system uses retrieved text. | CLUE benchmark team (SuperCLUE) | Faithful to sources | Everyday | Not in cards | GPT-4o 66.7 | Mar 2024 | May 2024 | ||
| 1X World Model Challenge Predict future egocentric frames of 1X's EVE humanoid from video and state. Compression scored by token cross-entropy, Sampling by PSNR on private tests. | 1X Technologies | Video world models | Not in cards | Jun 2024 | |||||
| 3D Arena Users compare two 3D models generated from the same input and vote for the better one. A live leaderboard ranks the generators. | Hugging Face (Dylan Ebert) | Spatial reasoning and counting | Everyday | Not in cards | |||||
| 3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. | Technology Innovation Institute (Boussaha et al.) | Reasoning in many languages | School | Not in cards | Jul 2025 | ||||
| 4DCodeBench Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion. | 4DCodeBench team | Physical prediction | Expert | Not in cards | Oct 2026 | ||||
| A-OKVQA Models answer about 25,000 visual questions that need world knowledge beyond the image, in multiple-choice and direct-answer forms. | Allen Institute for AI | General visual questions | Everyday | Not in cards | Jun 2022 | ||||
| A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. | Tencent | Secure code and review | Expert | Not in cards | Nov 2025 | ||||
| A.S.E (AICGSecEval) 1.0 Replaced in Nov 2025 | Not in cards | Jul 2025 | |||||||
| A^3-Bench Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates. | Jian Zhang, Yu He, Jun Liu et al. | Science questions | University | Not in cards | Jan 2026 | ||||
| AA-3D Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings. | Artificial Analysis | Web and app front ends | Expert | Not in cards | Oct 2026 | ||||
| AA-AnalystAgent Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right. | Artificial Analysis | Running a business | Expert | Not in cards | Aug 2026 | ||||
| AA-Image-Editing Validated private evaluators compare two image edits made from the same instruction and vote blind. Votes become Elo ratings. | Artificial Analysis | Image generation and editing | Everyday | Not in cards | |||||
| AA-LCR Extended Models answer questions that need reasoning across about 100k tokens of documents. Mercor adds 89 new expert-written tasks to the 100 public ones and reports both scores. | Mercor | Reasoning over long documents | Expert | Not in cards | Aug 2026 | ||||
| AA-Omniscience v1.1 Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones. | Artificial Analysis | Short factual questions | Expert | Not in cards | Oct 2026 | ||||
| AA-Omniscience 1.0 Replaced in Oct 2026 | In use | Claude Mythos Preview 70.8% | 15 | 18 | Nov 2025 | Oct 2026 | |||
| AA-Robotics Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks. | Artificial Analysis | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| AA-Video-Editing Voters compare two edits of the same video made from one instruction and pick the better one. Votes become Elo ratings. | Artificial Analysis | Video generation | Everyday | Not in cards | |||||
| AACR-Bench Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues. | Alibaba | Fixing real repositories | Expert | Not in cards | Jan 2026 | ||||
| AdaExploreBench Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice. | HKUST (KnowComp) | Puzzles and chess | Everyday | Not in cards | Oct 2026 | ||||
| AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. | University of Illinois Urbana-Champaign | General assistants | Everyday | Not in cards | Jun 2026 | ||||
| AfriMed-QA Models answer about 15,000 open and multiple-choice medical questions from over 60 medical schools in 16 African countries. Questions span 32 specialties. | AfriMed-QA team (Olatunji et al., Intron Health and partners) | Medicine | University | Not in cards | Nov 2024 | ||||
| Agent Arena (Gorilla) Users give a task, two agents built from different models, tools and frameworks run it, and users vote. A Bradley-Terry extension rates models, tools and frameworks. | UC Berkeley Gorilla team with LMSYS | General assistants | Everyday | Not in cards | Oct 2024 | ||||
| Agent Memory Leaderboard Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework. | AML-memory | Learning from the context | Everyday | Not in cards | Jul 2026 | ||||
| Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. | Tsinghua University (CoAI) | Refusing harmful requests | Everyday | Not in cards | Dec 2024 | ||||
| Agent4Sci Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking. | Anthropic | AI research tasks | Research frontier | Not in cards | Oct 2026 | ||||
| AgentClinic Doctor agents interview simulated patients, order tests and reach a diagnosis in multimodal clinical scenarios built from MedQA and NEJM cases. | Stanford and Johns Hopkins (Schmidgall et al.) | Medicine | Expert | Not in cards | May 2024 | ||||
| Agentic Abstention Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on. | University of Washington (Luo, Wen and Wang) | Hallucination rates | Everyday | Not in cards | Jun 2026 | ||||
| Agentic IF Index Composite of Meta internal evals on following tool-use restrictions, rubrics, compound constraints and long policies in agentic workflows. Scored 0-1. | Meta | Following precise instructions | Not in cards | Sep 2026 | |||||
| AgenticDataBench Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios. | Tsinghua University | Data science and SQL | Expert | Not in cards | Jul 2026 | ||||
| AgentKernelArena Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes. | AMD | GPU kernels | Expert | Not in cards | May 2026 | ||||
| AGORA (Agentic Global Open Research Arena) Autonomous AI systems get one secret open finance question, 72 hours and a fixed budget to write a paper. Five finance scholars judge it. | Penn State Smeal College of Business | Finance | Research frontier | Not in cards | Oct 2026 | ||||
| AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. | Mercor | General assistants | Everyday | Not in cards | Dec 2025 | ||||
| AI Diplomacy Models play the board game Diplomacy against each other, one power each. They negotiate, form alliances and betray each other in written messages. | Good Start Labs (Alex Duffy) | Multi-agent games | Everyday | Not in cards | Feb 2025 | ||||
| AidanBench Models keep giving new answers to open-ended questions until answers become incoherent or repeat earlier ones. The count of fresh, coherent answers is the score, with no ceiling. | Aidan McLaughlin (community) | Creative writing | Everyday | Not in cards | Aug 2024 | ||||
| AILuminate Safety 1.2 Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score. | MLCommons | Refusing harmful requests | Everyday | Not in cards | Sep 2026 | ||||
| MLCommons AILuminate 1.0 Replaced in Sep 2026 | Not in cards | Dec 2024 | |||||||
| AInsteinBench Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages. | ByteDance Seed and collaborators | Scientific code | Expert | Not in cards | Dec 2025 | ||||
| ALFWorld Agents complete household goals such as putting a washed apple in the fridge in a text world aligned with the ALFRED embodied simulator. Success is whether the goal state is reached. | ALFWorld team (Shridhar, Yuan, Côté, Bisk, Trischler, Hausknecht) | Simulated robots | Everyday | Not in cards | Oct 2020 | ||||
| Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. | Arena (formerly LMArena) | Deception and alignment | Everyday | Not in cards | Oct 2026 | ||||
| Alpha Arena Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns. | Nof1 | Finance | Expert | Not in cards | Oct 2025 | ||||
| AlphaEval Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics. | AlphaEval authors (Lu et al.) | Running a business | Expert | Not in cards | Apr 2026 | ||||
| AMBER Multimodal models are checked for object, attribute and relation hallucinations in generative and discriminative tasks, scored without an LLM judge. | Wang et al. (AMBER authors) | Hallucination rates | Everyday | Not in cards | Nov 2023 | ||||
| AMemGym Assistants hold long simulated conversations with users whose details and preferences change over time. Simulated users react on policy, and the test checks whether the assistant remembers and applies the right facts. | AGI-Eval (Meituan) | Learning from the context | Everyday | Not in cards | Jan 2026 | ||||
| Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic internal life-sciences evals Internal suite: morphology-to-molecule matching, medicinal chemistry, protein design, binder design, biomedical image analysis, organic chemistry. | Anthropic (with Axiom Bio and others) | Research science | Expert | Not in cards | 2026 | ||||
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Anthropic Frontier Red Team | Finding and exploiting real bugs | Expert | Not in cards | Oct 2026 | ||||
| Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. | Anthropic | Refusing harmful requests | Not in cards | ||||||
| Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. | Anthropic | Deception and alignment | Not in cards | Dec 2022 | |||||
| Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. | Anthropic | Deception and alignment | Not in cards | ||||||
| Antidote Experts such as doctors, lawyers and engineers compare model answers to prompts from their own history. They judge which answer would still help them a month later. | Surge AI | Overall chat quality | Expert | Not in cards | May 2026 | ||||
| APE (Attempt to Persuade Eval) A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion. | FAR.AI | Refusing harmful requests | Everyday | Not in cards | Jun 2025 | ||||
| APEX-1 Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics. | Mercor | Tasks from real jobs | Expert | Not in cards | Sep 2025 | ||||
| APEX-Accounting Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria. | Mercor with Ramp | Simulated companies | Expert | Not in cards | Jul 2026 | ||||
| APEX-Agents 1.1 Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero. | Mercor | Simulated companies | Expert | Not in cards | Sep 2026 | ||||
| APEX-Agents 1.0 Replaced in Sep 2026 | In use | Gemini 4 Argon 82.4% | 11 | 5 | Jan 2026 | Oct 2026 | |||
| APM-Bench Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions. | Shanghai Jiao Tong University | Long videos | Everyday | Not in cards | Sep 2026 | ||||
| Apodex Discovery Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors. | Apodex | Research science | Research frontier | Not in cards | Aug 2026 | ||||
| AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. | Stony Brook University and Allen Institute for AI | Apps and MCP servers | Everyday | Not in cards | Jul 2024 | ||||
| AQuA Two LLM research loops (factor discovery, model development) reuse evidence from their own past experiments inside sealed sandboxes. Scored by IC and held-out Sharpe. | Princeton University (Mengdi Wang) et al. | AI research tasks | Research frontier | Not in cards | Aug 2026 | ||||
| ArabCulture-Dialogue Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks. | MBZUAI (Al Kautsar, Almheiri et al.) | Chat and instructions in many languages | Everyday | Not in cards | Apr 2026 | ||||
| ArmnetBench 12 tabletop tasks on low-cost SO-101 arms. Each policy trained on the same 50 demos per task. 30 human-scored rollouts. | Armnet | Real robots | Everyday | Not in cards | Jul 2026 | ||||
| Artificial Analysis Decision Index Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning. | Artificial Analysis | Composite indices | Everyday | Not in cards | Oct 2026 | ||||
| Artificial Analysis Image Arena Voters compare images from two text-to-image models for the same prompt and pick the better one. Votes become Elo ratings. | Artificial Analysis | Image generation and editing | Everyday | Not in cards | |||||
| Artificial Analysis Progress Index Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5. | Artificial Analysis | Composite indices | Not in cards | Oct 2026 | |||||
| Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. | Artificial Analysis | Deep research | Expert | Not in cards | Aug 2026 | ||||
| Artificial Analysis Speech Arena Voters listen to two speech clips made from the same text and pick the more natural one. Votes become Elo ratings. | Artificial Analysis | Audio understanding and voice | Everyday | Not in cards | |||||
| Artificial Analysis Video Arena Voters compare videos from two text-to-video models for the same prompt and pick the better one. Votes become Elo ratings. | Artificial Analysis | Video generation | Everyday | Not in cards | |||||
| ArXivCS Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026. | MathArena (ETH Zurich SRI Lab) | Scientific code | Research frontier | Not in cards | Sep 2026 | ||||
| ArXivPhys Models answer research-level physics questions derived from recent arXiv papers, each with a reference final answer. | MathArena (ETH Zurich SRI Lab) | Research science | Research frontier | Not in cards | Sep 2026 | ||||
| ASI-Bench AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge. | Apexin AI | Research science | Research frontier | Not in cards | Aug 2026 | ||||
| Aspire Agent gets only a vague capability goal and must choose data, training and self-checks to evolve weights or harness. Scored on 520 hidden items. | ByteDance Seed et al. | AI research tasks | Research frontier | Not in cards | Aug 2026 | ||||
| AssetOpsBench Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders. | IBM Research | Other professions | Expert | Not in cards | Jun 2025 | ||||
| AssistantBench Web agents do 214 realistic, time-consuming tasks on the open web, such as monitoring real-estate markets. Answers are checked automatically against gold answers. | Tel Aviv University, Allen Institute for AI and others | Web browsing | Everyday | Not in cards | Jul 2024 | ||||
| ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. | Exa | Deep research | Expert | Not in cards | Oct 2026 | ||||
| ATLAS-Finance v1.1.0 Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level. | Handshake AI Research | Finance | Expert | Not in cards | Oct 2026 | ||||
| ATLAS-Finance 1.0.0 Replaced in Oct 2026 | Not in cards | Sep 2026 | |||||||
| AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. | ETH Zurich SRI Lab (von Arx et al.) | Secure code and review | Expert | Not in cards | Dec 2025 | ||||
| AutoEval Autonomous real-robot evaluation: WidowX arms run five Bridge tasks with learned success detectors and resets. 50 rollouts per task. | UC Berkeley | Real robots | Everyday | Not in cards | Mar 2025 | ||||
| Automated LLM Speedrunning Benchmark Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover. | Meta | AI research tasks | Research frontier | Not in cards | Jun 2025 | ||||
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Meta (PurpleLlama) | Secure code and review | Expert | Not in cards | Apr 2025 | ||||
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Politecnico di Torino and collaborators (Gioacchini et al.) | Finding and exploiting real bugs | Expert | Not in cards | Oct 2024 | ||||
| AutoResearch An agent edits a small nanochat training script and runs 5 minute training experiments overnight. Scored by final validation bits per byte, where lower is better. | Andrej Karpathy | AI research tasks | Research frontier | Not in cards | Mar 2026 | ||||
| AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. | Lei Xiong, Kun Luo, Zheng Liu et al. | Deep research | Expert | Not in cards | Apr 2026 | ||||
| aws-bench Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts. | aws-bench maintainers | Terminal tasks | Expert | Not in cards | Jul 2026 | ||||
| BabaIsBench Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost. | Quesma | Puzzles and chess | Expert | Not in cards | Jun 2026 | ||||
| BABILong Models answer bAbI reasoning questions whose supporting facts are scattered through very long distractor text, at lengths up to millions of tokens. | AIRI and collaborators (Kuratov et al.) | Reasoning over long documents | Everyday | Not in cards | Jun 2024 | ||||
| BackendBench LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed. | Meta (PyTorch) | GPU kernels | Expert | Not in cards | Jun 2025 | ||||
| BackSearch Paid API that searches a frozen web archive as of a given date, for forecasting backtests and reproducible web-enabled evals. A tool, not a benchmark. | General Reasoning | Real-world events | Not in cards | Jul 2026 | |||||
| BallotBench Nine frontier agents forecast 863 contests in the 2026 US midterms each day until polls close. Proper scoring rules grade forecasts against certified results, beside Kalshi, Polymarket and expert ratings. | Thoughtful Lab | Real-world events | Expert | Not in cards | Oct 2026 | ||||
| BankerToolBench v1.1 Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes. | Handshake AI Research | Finance | Expert | Not in cards | Oct 2026 | ||||
| BankerToolBench 1.0 Replaced in Oct 2026 | In use | Tencent Hy4 preview 78.6% | 3 | Apr 2026 | Oct 2026 | ||||
| BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. | Redwood Research | Deception and alignment | Expert | Not in cards | Dec 2025 | ||||
| Bedside Bench Models answer 500 physician-written clinical cases on diagnosis, workup, escalation and medication safety. An LLM judge grades each answer against the authors' criteria. | Doximity | Medicine | Expert | Not in cards | |||||
| BEHAVIOR Challenge 2026 Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit. | Stanford Vision and Learning Lab | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| BEHAVIOR Challenge 2025 Replaced in Jul 2026 | Stanford (BEHAVIOR team) | Fading | 1 | Sep 2025 | Dec 2025 | ||||
| BEHAVIOR-1K Embodied agents do 1,000 everyday household activities, such as cleaning and cooking, in realistic simulation built on OmniGibson. Success is checked against task goal definitions. | Stanford Vision and Learning Lab | Simulated robots | Everyday | Not in cards | Mar 2024 | ||||
| BIG-bench Language models do over 200 tasks contributed by hundreds of authors, from linguistics and math to social bias. Most tasks are scored by exact match or multiple choice. | Google and 400+ contributing authors | Language reasoning | University | Not in cards | Jun 2022 | ||||
| BigCodeArena Users ask for code, both models' programs run in a sandbox, and users try the results before voting for the better answer. Votes give Elo ratings. | BigCode project (Monash University and collaborators) | Function-level problems | Everyday | Not in cards | Oct 2025 | ||||
| BioRiskEval Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir. | Scale AI | Hazardous knowledge | Expert | Not in cards | Oct 2025 | ||||
| BIRD-Interact Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests. | BIRD team (HKU) with Google Cloud | Data science and SQL | Expert | Not in cards | Jun 2025 | ||||
| BizBench Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success. | Ant International | Running a business | Expert | Not in cards | Oct 2026 | ||||
| BLiMP Language models must assign higher probability to the grammatical sentence in each of 67,000 minimal pairs covering English syntax, morphology and semantics. | New York University (Warstadt et al.) | Commonsense | Everyday | Not in cards | Dec 2019 | ||||
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Stanford CRFM | Finding and exploiting real bugs | Expert | Not in cards | May 2025 | ||||
| BRIGHT Retrieval systems find documents for queries from StackExchange, LeetCode and math competitions where relevance needs reasoning. Scored by nDCG@10. | Su et al. (HKU and collaborators) | Questions that need search | Expert | Not in cards | Jul 2024 | ||||
| BrokenMath Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement. | INSAIT and ETH Zurich SRI Lab | Written proofs | Competition | Not in cards | Oct 2025 | ||||
| BrowseComp Extended Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores. | Mercor | Web browsing | Expert | Not in cards | Aug 2026 | ||||
| BrowseComp Long Context 256k Models answer BrowseComp questions from a context of 128K to 256K tokens built from the needed web pages and related distractor pages. Scored by accuracy. | OpenAI | Reasoning over long documents | Expert | Not in cards | Aug 2025 | ||||
| BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. | Scale AI and collaborators | Refusing harmful requests | Expert | Not in cards | Oct 2024 | ||||
| BU Bench V2.1 Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well. | Browser Use | Web browsing | Everyday | Not in cards | Sep 2026 | ||||
| BU Bench V2 V2 (60 tasks) Replaced in Sep 2026 | Not in cards | Jan 2026 | |||||||
| Butter-Bench A language model controls a real robot to find and deliver a pack of butter in an office. It is scored on completing each step of the task. | Andon Labs | Real robots | Everyday | Not in cards | Oct 2025 | ||||
| BVB Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit. | BVB authors (Tang et al.) | Video understanding | Expert | Not in cards | Sep 2026 | ||||
| CALVIN Simulated Franka follows chains of five language instructions from 34 tasks. ABC-to-D tests an unseen environment. Average chain length. | University of Freiburg | Simulated robots | Everyday | Not in cards | Dec 2021 | ||||
| CaP-X (CaP-Bench) Coding agents write perception and control code for 100+ sim manipulation tasks across 8 abstraction and interaction tiers. Scored by task success. | NVIDIA + UC Berkeley + CMU + Stanford | Simulated robots | University | Not in cards | Mar 2026 | ||||
| CAREBench Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not. | Handshake AI Research | Wellbeing and sycophancy | Everyday | Not in cards | Jun 2026 | ||||
| CASTLE Models read 250 hand-written compilable C programs and say whether each has a vulnerability and which CWE. Static analyzers run on the same set. | CASTLE authors (Richard et al.) | Finding and exploiting real bugs | Expert | Not in cards | Mar 2025 | ||||
| CATArena Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code, and scores separate first-round skill from improvement. | AGI-Eval (Meituan) | Building whole projects | Expert | Not in cards | Oct 2025 | ||||
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Apart Research and collaborators | Capture-the-flag challenges | Expert | Not in cards | Oct 2024 | ||||
| CausalVQA Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types. | Meta FAIR | Physical prediction | Everyday | Not in cards | Jun 2025 | ||||
| ChartX Vision-language models do 7 chart tasks, such as question answering, summarization and redrawing, over 18 chart types and 22 topics. | Shanghai AI Laboratory (Xia, Zhang et al.) | Documents, charts and text in images | Everyday | Not in cards | Feb 2024 | ||||
| CharXiv Extended Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores. | Mercor | Documents, charts and text in images | University | Not in cards | Aug 2026 | ||||
| CheckerBench Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback. | Humanlaya Data Lab | Fixing real repositories | Expert | Not in cards | Oct 2026 | ||||
| Chest ImaGenome Models draw bounding boxes around named anatomical regions in MIMIC-CXR chest X-rays. Scored by mean intersection over union. | IBM Research and MIT (Wu et al.) | Medicine | Expert | Not in cards | Jul 2021 | ||||
| ChestX-ray14 Models read frontal chest X-rays from the NIH Clinical Center and say whether thoracic conditions are present. | NIH Clinical Center (Wang et al.) | Medicine | Expert | Not in cards | May 2017 | ||||
| CheXpert Models read chest radiographs and label up to 14 observations. The test labels come from board-certified radiologists. | Stanford ML Group (Irvin, Rajpurkar et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| CHI-Bench Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs. | Actava AI | Medicine | Expert | Not in cards | May 2026 | ||||
| CineRecon Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips. | The University of Hong Kong | Video understanding | Expert | Not in cards | Oct 2026 | ||||
| ClaimBench Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency. | Vals AI | Finance | University | Not in cards | Oct 2026 | ||||
| CLASHarness Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment. | Google Research | Desktop apps | Everyday | Not in cards | Oct 2026 | ||||
| ClassEval Models generate whole Python classes for 100 hand-built tasks. Tests check each method and the class as a whole. | Fudan University | Function-level problems | University | Not in cards | Aug 2023 | ||||
| Claw-SWE-Bench General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches. | TokenRhythm | Fixing real repositories | Expert | Not in cards | Jun 2026 | ||||
| ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. | ClawProBench authors | General assistants | Expert | Not in cards | Aug 2026 | ||||
| CLBench-V Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition. | CLBench-V authors | General visual questions | Everyday | Not in cards | Jul 2026 | ||||
| CliniCARE-Bench Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases. | Scale AI | Medicine | Expert | Not in cards | Aug 2026 | ||||
| CLUE Models are fine-tuned or prompted on nine Chinese language understanding tasks, including sentence classification, sentence-pair inference and machine reading comprehension. The score averages the task results. | CLUE benchmark team | Reading in many languages | School | Not in cards | Apr 2020 | ||||
| CMMMU Multimodal models answer college-level questions in Chinese across many disciplines, following MMMU's annotation pattern. | M-A-P and collaborators | General visual questions | University | Not in cards | Jan 2024 | ||||
| CocoaBench Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output. | UC San Diego | Web browsing | Expert | Not in cards | Apr 2026 | ||||
| CoCoNot Models get 1,000 requests they should not fully comply with, such as incomplete, unsupported or humanizing requests. A judge model rates whether each response complies. | Allen Institute for AI | Deception and alignment | Everyday | Not in cards | Jul 2024 | ||||
| Code Review Bench (Martian) AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments. | Martian | Fixing real repositories | Expert | Not in cards | Feb 2026 | ||||
| Code2Video-Compass Multimodal models write programs that render complete videos from text, an image or a video. | Fudan University | Video generation | Everyday | Not in cards | Oct 2026 | ||||
| CodeCap-Bench Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail. | Kling AI (Kuaishou) + Nanjing University | Video captioning | Everyday | Not in cards | Oct 2026 | ||||
| cognition-golden Cognition tests coding agents on tasks modeled on real Devin use cases, in authentic development environments with fully autonomous grading. A train split serves self-improvement and a test split measures capability. | Cognition | Building whole projects | Expert | Not in cards | Sep 2024 | ||||
| Cohere North internal evals Cohere internal evals of its North agent platform: agentic question answering, spreadsheet data analysis and memory use. LLM-as-judge scoring. | Cohere | Documents and spreadsheets | University | Not in cards | May 2026 | ||||
| CompassArena Users chat with two anonymous models side by side, mostly in Chinese, and vote for the better answer. A Bradley-Terry model turns the votes into Elo-style ratings. | OpenCompass (Shanghai AI Laboratory) with ModelScope | Overall chat quality | Everyday | Not in cards | May 2024 | ||||
| CompassRank OpenCompass ranks open and API language models on its own evaluation sets and publishes monthly rankings. Model makers submit a repository link or API endpoint to be tested. | OpenCompass (Shanghai AI Laboratory) | Composite indices | Expert | Not in cards | Sep 2023 | ||||
| ComplexBench Models answer 1,150 hand-written Chinese instructions that compose several constraints by chaining, selection and nesting. LLM evaluators with rules check each constraint, and scores follow the dependency structure. | Tsinghua University (CoAI) and Zhipu AI | Following precise instructions | Everyday | Not in cards | Jul 2024 | ||||
| ComplexConstraints Models answer 75 expert-written professional prompts whose constraints depend on each other, fire conditionally or must be inferred. An LLM judge grades 1,559 rubric criteria one at a time, and a task passes only if all criteria pass. | Surge AI | Following precise instructions | University | Not in cards | Jun 2026 | ||||
| ComputeEval Models write CUDA code for handcrafted problems on kernels, runtime APIs and GPU libraries. Held-out test harnesses check correctness, and optional benchmarks time the code. | NVIDIA | Speed and optimization | Expert | Not in cards | Apr 2025 | ||||
| Continual Learning Bench AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning. | UC Berkeley, Snorkel AI and collaborators (Asawa, Orlanski et al.) | Learning from the context | Expert | Not in cards | Jun 2026 | ||||
| ContinualSkillBench Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks. | Peking University | Learning from the context | Expert | Not in cards | Aug 2026 | ||||
| Cooperate to Compete (C2C) Multi-player conquest game with private non-binding negotiation and secret asymmetric objectives. LM agents scored by win rate against humans and other agents. | UC Berkeley | Multi-agent games | Everyday | Not in cards | Apr 2026 | ||||
| Copilot Arena Developers inside their IDE see paired code completions from two models and pick one. Pairwise votes give an Elo-style ranking of code models. | Arena (formerly LMArena) with Carnegie Mellon University | Function-level problems | Everyday | Not in cards | Nov 2024 | ||||
| CoQA Models answer a series of linked conversational questions about a passage, where later questions depend on earlier turns. Scored by word-level F1. | Stanford University (Reddy, Chen and Manning) | Reading comprehension | Everyday | Not in cards | Aug 2018 | ||||
| CoreCodeBench Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels. | AGI-Eval (Meituan) | Fixing real repositories | Expert | Not in cards | Jul 2025 | ||||
| CorporateBench Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph. | Epiq AI Labs | Reasoning over long documents | Expert | Not in cards | Aug 2026 | ||||
| Coval Voice AI Benchmarks Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo. | Coval | Speech recognition and translation | Everyday | Not in cards | Apr 2026 | ||||
| CoVEBench Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged. | Nanjing University (NJU-LINK) | Video generation | Everyday | Not in cards | Jun 2026 | ||||
| CRAB Multimodal agents do tasks that span an Ubuntu desktop and an Android phone at once. Graph-based evaluators give credit for intermediate steps. | CAMEL-AI and collaborators | Desktop apps | Everyday | Not in cards | Jul 2024 | ||||
| CRAG Retrieval-augmented systems answer 4,409 factual questions across finance, sports, music, movies and open domain, using mock web and knowledge graph search APIs. Hallucinated answers are penalized. | Meta | Faithful to sources | Everyday | Not in cards | Jun 2024 | ||||
| Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. | Cresta | Customer service | Everyday | Not in cards | Oct 2026 | ||||
| CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. | Salesforce AI Research | Customer service | Everyday | Not in cards | May 2025 | ||||
| CrossPoint-Bench Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction. | Institute of Automation, Chinese Academy of Sciences (Wang, Ji et al.) | Spatial reasoning and counting | Everyday | Not in cards | Dec 2025 | ||||
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | ETH Zurich (SPY Lab) + Anthropic + University of Haifa + Tel Aviv University | Finding and exploiting real bugs | Research frontier | Not in cards | Jul 2026 | ||||
| CT-RATE Models read 3D non-contrast chest CT volumes and say whether each of 18 conditions is present. Scored by macro accuracy or F1 against report-derived labels. | University of Zurich and collaborators (Hamamci et al.) | Medicine | Expert | Not in cards | Mar 2024 | ||||
| CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. | CTI-REALM authors, implemented in UK AISI inspect_evals | Finding and exploiting real bugs | Expert | Not in cards | Mar 2026 | ||||
| CTIBench Models answer cyber threat intelligence questions. Tasks cover knowledge questions, CVE to CWE mapping, CVSS severity scoring, threat actor attribution and technique extraction. | Rochester Institute of Technology (Alam, Bhusal, Nguyen, Rastogi) | Finding and exploiting real bugs | Expert | Not in cards | Jun 2024 | ||||
| CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. | Cohere Labs Community | Reasoning in many languages | Everyday | Not in cards | Oct 2026 | ||||
| CVDP Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs. | NVIDIA | Electronics | Expert | Not in cards | Jun 2025 | ||||
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | UIUC (Kang Lab) | Finding and exploiting real bugs | Expert | Not in cards | Oct 2025 | ||||
| CVE-Bench v1 Replaced in Oct 2025 | System cards only | GPT-5.6 Sol 94.1% | 20 | Mar 2025 | Sep 2026 | ||||
| CyberMetric Models answer multiple-choice cybersecurity knowledge questions drawn from standards, papers and RFCs. Sets of 80, 500, 2,000 and 10,000 questions exist. | Technology Innovation Institute (Abu Dhabi) | Capture-the-flag challenges | Expert | Not in cards | Feb 2024 | ||||
| CyberOffense Agents attack two simulated company networks, Nova and Lighthouse, on their own with no human help. The score counts the attack objectives completed on each range. | Vals AI | Capture-the-flag challenges | Expert | Not in cards | Oct 2026 | ||||
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Meta (PurpleLlama) | Finding and exploiting real bugs | Expert | Not in cards | Jul 2024 | ||||
| CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | Dec 2023 | ||||
| CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | Apr 2024 | ||||
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Meta (PurpleLlama) | Secure code and review | Expert | Not in cards | Jul 2024 | ||||
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Meta (PurpleLlama) | Capture-the-flag challenges | Expert | Not in cards | Apr 2024 | ||||
| DABstep Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically. | Adyen and Hugging Face | Data science and SQL | Expert | Not in cards | Jun 2025 | ||||
| DAComp Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights. | ByteDance Seed | Data science and SQL | Expert | Not in cards | Dec 2025 | ||||
| Data Science Tasks (OpenAI) Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | OpenAI | Documents and spreadsheets | Expert | Not in cards | Sep 2026 | ||||
| Database Migration Tasks (OpenAI) Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading. | OpenAI | Fixing real repositories | Expert | Not in cards | Sep 2026 | ||||
| DataPrep-Bench Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score. | Peking University | Machine learning engineering | Expert | Not in cards | May 2026 | ||||
| DAYJOB: Accounting Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| DAYJOB: Data Science Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Data science and SQL | Expert | Not in cards | Oct 2026 | ||||
| DAYJOB: Finance Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Tasks from real jobs | Expert | Not in cards | Sep 2026 | ||||
| DAYJOB: Healthcare Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Medicine | Expert | Not in cards | Sep 2026 | ||||
| DAYJOB: Law Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria. | Surge AI | Law | Expert | Not in cards | Oct 2026 | ||||
| DAYJOB: Tax Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria. | Surge AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. | Xi'an Jiaotong University | Deception and alignment | Everyday | Not in cards | Oct 2026 | ||||
| Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. | FutureSearch | Deep research | Expert | Not in cards | May 2025 | ||||
| DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. | Google DeepMind | Dangerous capabilities | Not in cards | Mar 2024 | |||||
| DeepPHY Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles. | DeepPHY authors | Abstract puzzles and logic | Everyday | Not in cards | Aug 2025 | ||||
| DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. | DeepResearch Arena authors | Deep research | Expert | Not in cards | Sep 2025 | ||||
| Design Tasks (OpenAI) Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading. | OpenAI | Documents and spreadsheets | Expert | Not in cards | Sep 2026 | ||||
| DesignArena 3D Design Models build an interactive 3D scene that runs in the browser as a single HTML file. Users explore the scenes and vote for the better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Agentic Game Dev Coding agents build multi-file games from a user prompt. Users play the final games and vote, and the logs keep tool calls, retries and failures. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena Agentic Image to HTML Coding agents turn a user-uploaded image into a self-contained HTML website. Users compare how well each page reproduces the design and vote. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Android (Kotlin) Models build native Android apps in Kotlin from real user prompts. The apps run in an Android emulator, and users compare them and vote. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena ASCII Art Models draw a requested picture using only text characters. Users compare the ASCII art and vote for the better one. | DesignArena (Intelligence) | Creative writing | Everyday | Not in cards | |||||
| DesignArena Audio Realism Bench Voice models produce conversational speech, and users vote on which sounds more real. DesignArena built it because 1 to 5 opinion scores for TTS had saturated. | DesignArena (Intelligence) | Audio understanding and voice | Everyday | Not in cards | |||||
| DesignArena Builder AI app builders make full-stack web and mobile apps from the same user prompt. Users compare the results and vote for the better one. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena Code Users give a prompt, and anonymous models each build a website, UI component, browser game, data visualization or 3D scene as one HTML file. Users vote, and votes become Bradley-Terry ratings. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | Jul 2025 | ||||
| DesignArena Data Visualization Models turn a user prompt and its data into an interactive visualization as a single HTML file. Users compare the rendered charts and vote. | DesignArena (Intelligence) | Data science and SQL | Everyday | Not in cards | |||||
| DesignArena Frontend Web App Coding agents build multi-file React frontends from real user prompts. Users compare the running apps and vote, and the run logs record agent behavior. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Full Stack Coding agents build complete React apps with sign-in, databases and backend code from real user prompts. Users try the apps and vote for the better one. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena Game Dev Models build a playable browser game as a single HTML file from a real user prompt. Users play the games and vote for the better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Godot Game Dev Coding agents build complete Godot projects with GDScript, scenes and generated assets, exported to run in the browser. Users play the games and vote. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena HTML Slides Agents write a self-contained HTML slide deck that is converted to PPTX. Users compare the decks and vote for the better one. | DesignArena (Intelligence) | Documents and spreadsheets | Everyday | Not in cards | |||||
| DesignArena Image Image models generate a picture from a user prompt. Users compare two anonymous images and vote for the better one. | DesignArena (Intelligence) | Image generation and editing | Everyday | Not in cards | |||||
| DesignArena Image Editing Image models edit a user image following a text instruction. Users compare the two edits and vote for the better one. | DesignArena (Intelligence) | Image generation and editing | Everyday | Not in cards | |||||
| DesignArena Image to Frontend Coding agents turn a reference image into a multi-file frontend web app. Users compare the running apps and vote for the better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Image to HTML Models recreate a reference image or mockup as a working single-file HTML website. Users compare the rendered pages and vote for the closer, better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Image to Video Video models animate a user image into a short video. Users compare two anonymous videos and vote for the better one. | DesignArena (Intelligence) | Video generation | Everyday | Not in cards | |||||
| DesignArena Multi-Image to Video Video models compose one video from several reference images. Users compare two anonymous videos and vote for the better one. | DesignArena (Intelligence) | Video generation | Everyday | Not in cards | |||||
| DesignArena Multimodal to Video Video models make a video from a mix of text, images and other inputs. Users compare two anonymous videos and vote for the better one. | DesignArena (Intelligence) | Video generation | Everyday | Not in cards | |||||
| DesignArena Proprietary Harness Slides Slide-making products build a presentation from a user prompt in their own harness. Users compare the decks and vote for the better one. | DesignArena (Intelligence) | Documents and spreadsheets | Everyday | Not in cards | |||||
| DesignArena Python-PPTX Slides Agents build PowerPoint decks with python-pptx from a user prompt. Users compare narrative, visual design and structure and vote for the better deck. | DesignArena (Intelligence) | Documents and spreadsheets | Everyday | Not in cards | |||||
| DesignArena React Native Models build cross-platform mobile apps in React Native from real user prompts. Users compare the running apps and vote for the better one. | DesignArena (Intelligence) | Building whole projects | Everyday | Not in cards | |||||
| DesignArena Speech-to-Speech Voice models answer spoken input with speech. Users listen to two anonymous replies and vote for the better one. | DesignArena (Intelligence) | Audio understanding and voice | Everyday | Not in cards | |||||
| DesignArena Spreadsheet Agents build spreadsheets such as financial models, budgets and trackers as Excel or CSV workbooks. Users compare accuracy, structure and formatting and vote. | DesignArena (Intelligence) | Documents and spreadsheets | Everyday | Not in cards | |||||
| DesignArena SVG Models write SVG code for a design request such as an icon, logo, badge or illustration. Users compare the rendered graphics and vote. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Text-to-Speech Speech models read the same text aloud. Users listen to two anonymous clips and vote for the more natural one. | DesignArena (Intelligence) | Audio understanding and voice | Everyday | Not in cards | |||||
| DesignArena UI Component Models build one interface component as a single HTML file from a real user prompt. Users compare the rendered components and vote for the better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Video Arena Users compare videos from anonymous video generation models for the same prompt and vote for the better one. Votes become Elo ratings. | DesignArena (Intelligence) | Video generation | Everyday | Not in cards | |||||
| DesignArena Video Editing Video models edit a user video following a text instruction. Users compare the two results and vote for the better one. | DesignArena (Intelligence) | Video generation | Everyday | Not in cards | |||||
| DesignArena Video to Website Models watch a reference video and recreate its content as a working single-file HTML website. Users compare the rendered pages and vote. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DesignArena Website Models build a complete website as one HTML file from a real user prompt. Users compare the rendered pages and vote for the better one. | DesignArena (Intelligence) | Web and app front ends | Everyday | Not in cards | |||||
| DevBench Models work through the software lifecycle on 22 curated repositories in Python, C/C++, Java and JavaScript: design, environment setup, implementation, acceptance tests and unit tests. Tests score code and an LLM judge scores design. | OpenCompass (Shanghai AI Laboratory) and collaborators | Building whole projects | Expert | Not in cards | Mar 2024 | ||||
| dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. | depthfirst | Finding and exploiting real bugs | Expert | Not in cards | Aug 2026 | ||||
| DiG-bench Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt. | Princeton + MIT + collaborators (Battleday, Whittington, Griffiths, Tenenbaum and others) | Puzzles and chess | Everyday | Not in cards | Aug 2026 | ||||
| Digital Work Index Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to. | Snorkel AI | Composite work indices | Expert | Not in cards | Oct 2026 | ||||
| DISBench Multimodal agents search raw visual histories over several steps to find target images from implicit context cues. | NLPIR Lab, Renmin University of China | General visual questions | Everyday | Not in cards | Feb 2026 | ||||
| Doc2Harness Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers. | Institute of Computing Technology, Chinese Academy of Sciences | AI research tasks | Research frontier | Not in cards | Oct 2026 | ||||
| Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Cohere Labs | Chat and instructions in many languages | Everyday | Not in cards | Apr 2026 | ||||
| mDolly v1 Replaced in Apr 2026 | Not in cards | Feb 2024 | |||||||
| DP-Bench Document parsers and vision models convert PDFs and scanned pages into structured HTML or Markdown. Output is compared with reference structure. | Upstage | Documents, charts and text in images | Everyday | Not in cards | Oct 2024 | ||||
| DreamGen Bench Video generation benchmark measuring how well video models adapt to a target robot embodiment. Part of NVIDIA's DreamGen pipeline. | NVIDIA GEAR | Video world models | Not in cards | May 2025 | |||||
| DS-1000 Models write Python for 1,000 data science problems from StackOverflow across seven libraries such as NumPy and Pandas. Test cases and API constraints check each answer. | HKU, Stanford, UC Berkeley, UW, Meta and CMU | Data science and SQL | University | Not in cards | Nov 2022 | ||||
| Dual Exam Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases. | Dual Exam authors | Interactive world models | Everyday | Not in cards | Oct 2026 | ||||
| DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. | Decagon | Customer service | Expert | Not in cards | Jun 2026 | ||||
| DuoBench Eleven bimanual Franka FR3 Duo tasks in four coordination categories, in simulation and partly real. Stage-based scoring beyond binary success. | UTN Nuremberg + KIT + Franka Robotics + TU Munich | Simulated robots | Everyday | Not in cards | Jun 2026 | ||||
| Duplex-MPE Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios. | Peking University | Speech recognition and translation | Everyday | Not in cards | Sep 2026 | ||||
| EBench 26 Isaac Sim household mobile-manipulation task types annotated by 5 capability and 4 generalization dimensions. Platform-scored success rate. | Shanghai AI Lab (InternRobotics) | Simulated robots | Everyday | Not in cards | Jun 2026 | ||||
| EHRNoteQA Models read a patient's accumulated MIMIC-IV discharge summaries and answer clinical questions about them. The set has 962 question and answer pairs, each for a different patient. | KAIST and collaborators (Kweon et al.) | Medicine | Expert | Not in cards | Feb 2024 | ||||
| EHRQA (Google) Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| EmbodiedBench Multimodal LLMs act as embodied agents on 1,128 tasks in four simulators, from household planning (EB-ALFRED, EB-Habitat) to navigation and manipulation. Success rate. | University of Illinois Urbana-Champaign + others | Simulated robots | Everyday | Not in cards | Feb 2025 | ||||
| EmbodiedCLBench Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks. | Peking University (VaLuE Lab) | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| EmoBench-M Multimodal models are tested on emotional intelligence across 13 scenarios in three dimensions: emotion recognition, conversational emotion understanding and socially complex emotion analysis. | EmoBench-M authors | General visual questions | Everyday | Not in cards | Feb 2025 | ||||
| EngiWorld Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software. | Zhiman AI | CAD models | Expert | Not in cards | Sep 2026 | ||||
| EnterpriseBench: CoreCraft Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy. | Surge AI | Simulated companies | Expert | Not in cards | Feb 2026 | ||||
| EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. | Onyx | Deep research | Expert | Not in cards | Feb 2026 | ||||
| Epoch Capabilities Index An index that fits model capability and benchmark difficulty from scores on over 50 benchmarks in Epoch AI's hub, using item response theory. Scores read like Elo ratings. | Epoch AI | Composite indices | Research frontier | Not in cards | Oct 2025 | ||||
| Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. | Eon | Apps and MCP servers | University | Not in cards | Sep 2026 | ||||
| ESI-Bench Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action. | ESI-Bench team | Embodied reasoning | Everyday | Not in cards | May 2026 | ||||
| ETHICS Models predict common moral judgments about text scenarios covering justice, well-being, duties, virtues and commonsense morality. | UC Berkeley (Hendrycks et al.) | Deception and alignment | Everyday | Not in cards | Aug 2020 | ||||
| EurekaBench Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support. | Carnegie Mellon University + Stanford + collaborators | Research science | Research frontier | Not in cards | Sep 2026 | ||||
| EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. | UK AI Security Institute + LASR Labs | Deception and alignment | Expert | Not in cards | Jun 2026 | ||||
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | OpenAI and Paradigm | Finding and exploiting real bugs | Expert | Not in cards | Feb 2026 | ||||
| EvoArena Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes. | MIT | Learning from the context | Expert | Not in cards | Jun 2026 | ||||
| EvoPolicyGym Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves. | EvoPolicyGym authors | AI research tasks | Expert | Not in cards | Jul 2026 | ||||
| EWMBench Embodied world models generate robot manipulation videos from language. Scored on scene consistency, motion correctness, semantic alignment. | AgiBot | Video world models | Not in cards | May 2025 | |||||
| EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. | Sofia University and collaborators (Hardalov et al.) | Knowledge in many languages | School | Not in cards | Nov 2020 | ||||
| ExeWorldBench Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code. | University of Science and Technology of China | Physical prediction | Expert | Not in cards | Oct 2026 | ||||
| EXP-Bench Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results. | Curie team (Just-Curieous) | AI research tasks | Research frontier | Not in cards | May 2025 | ||||
| Expert-SWE (OpenAI) Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours. | OpenAI | Building whole projects | Expert | Not in cards | Apr 2026 | ||||
| ExtractBench Systems read a full business document and return a schema-valid JSON object with every requested value. Unified value F1 compares the output with the expected values. | LlamaIndex | Documents, charts and text in images | Everyday | Not in cards | Jul 2026 | ||||
| ExtractBench Pro Systems extract every schema value from long-tail synthetic documents with charts, drawn relations, house rules and jargon. Scored by value F1 and by grounded F1, which also needs the source location. | LlamaIndex | Documents, charts and text in images | Expert | Not in cards | Oct 2026 | ||||
| EyePACS (diabetic retinopathy grading) Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient. | Google (EyePACS images) | Medicine | Expert | Not in cards | Jul 2025 | ||||
| FailSafeQA Models answer finance questions over long documents while the query or the document is degraded, irrelevant or missing. An LLM judge scores robustness, context grounding and compliance. | Writer | Finance | Expert | Not in cards | Feb 2025 | ||||
| FDE-Gym Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1. | DeepLethe (with 20+ contributing institutions) | Building whole projects | Expert | Not in cards | Oct 2026 | ||||
| FeatureBench Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically. | LiberCoders | Fixing real repositories | Expert | Not in cards | Feb 2026 | ||||
| Finance Reasoning (Snorkel) Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness. | Snorkel AI | Finance | Expert | Not in cards | Jun 2025 | ||||
| FinQA (Snorkel OpenEnv environment) Agent answers 290 multi-step questions on SEC 10-K data via MCP tools and constrained SQL. Binary reward with 1% numeric tolerance. | Snorkel AI | Finance | University | Not in cards | Mar 2026 | ||||
| Fireworks Specialized Intelligence Index An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number. | Fireworks AI | Composite indices | Expert | Not in cards | Sep 2026 | ||||
| First Proof Third Batch Batch 3 The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution, with negative marks for wrong answers. | First Proof Foundation | Research mathematics | Research frontier | Not in cards | Oct 2026 | ||||
| First Proof Second Batch Batch 2 Replaced in Oct 2026 | Not in cards | Jun 2026 | |||||||
| First Proof Batch 1 Replaced in Jun 2026 | First Proof (Abouzaid, Blumberg, Hairer, Spielman, Williams and others) | Not in cards | Feb 2026 | ||||||
| FlashDexRetarget 50-motion benchmark Retarget 50 human hand-object motions (HOT3D, TACO, OakInk2) to XHand and Sharpa hands in Isaac Sim. Scored by object pose-error success thresholds. | KAIST AI + Holiday Robotics | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| FlashInfer-Bench Agents generate GPU kernels for LLM inference that are benchmarked on workloads from real serving traces and can be deployed into the FlashInfer stack. | FlashInfer team | GPU kernels | Expert | Not in cards | Jun 2025 | ||||
| FollowBench Models answer 820 English and Chinese instructions where each level adds one more constraint of content, situation, style, format or example type. Rules and an LLM judge check each constraint. | HKUST and Huawei Noah's Ark Lab (Jiang et al.) | Following precise instructions | Everyday | Not in cards | Oct 2023 | ||||
| FrogsGame post-training task Agent has 20 hours to post-train Qwen3-8B to solve FrogsGame puzzle boards. A hidden verifier scores the best checkpoint on unseen boards. | Thoughtful Lab (task in FrontierSWE v1) | Machine learning engineering | Research frontier | Not in cards | Apr 2026 | ||||
| FrontierCAD Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry. | Normal Research | CAD models | Expert | Not in cards | Oct 2026 | ||||
| FrontierChallenge Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable. | Apodex AI (FrontierAgent team) | Research science | Expert | Not in cards | Aug 2026 | ||||
| FrontierMath: Open Problems Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved. | Epoch AI | Research mathematics | Research frontier | Not in cards | Jan 2026 | ||||
| FrontierMemory A model reads 750 past agent trajectories and writes memory for a task agent. The score is how many new coding and knowledge-work tasks that agent then solves. | Letta | AI research tasks | Research frontier | Not in cards | |||||
| Full-Duplex-Bench v3 3.0 Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking. | Guan-Ting Lin, Hung-yi Lee and collaborators (National Taiwan University and others) | Speech recognition and translation | Everyday | Not in cards | Apr 2026 | ||||
| Full-Duplex-Bench v2 2.0 Replaced in Apr 2026 | Not in cards | Oct 2025 | |||||||
| Full-Duplex-Bench v1.5 Replaced in Oct 2025 | Not in cards | Jul 2025 | |||||||
| Full-Duplex-Bench 1.0 Replaced in Jul 2025 | Not in cards | Mar 2025 | |||||||
| FutureEval (Metaculus) Models forecast open Metaculus questions with a fixed prompt and are scored as the questions resolve. Seasonal bot tournaments and Pro Forecaster baselines run alongside. | Metaculus | Real-world events | Expert | Not in cards | Feb 2026 | ||||
| FutureX Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more. | ByteDance Seed with Fudan, Stanford and Princeton | Real-world events | Expert | Not in cards | Aug 2025 | ||||
| Galahad Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior. | phi-monster (Galahad authors) | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| GameCraft-Bench Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together. | CUHK Shenzhen | Building whole projects | Expert | Not in cards | Jun 2026 | ||||
| GameHorizon Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts. | Tencent ARC | Video games | Everyday | Not in cards | Sep 2026 | ||||
| GaP: Graph-as-Policy Multi-agent harness that compiles language tasks into typed robot skill graphs. Tested on 8 variational-automation tasks (4 sim, 4 real) by success rate. | Bosch + NVIDIA + UC Berkeley | Simulated robots | Everyday | Not in cards | Jul 2026 | ||||
| GDP.xlsx Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria. | Surge AI | Documents and spreadsheets | Expert | Not in cards | Sep 2026 | ||||
| GDPevo Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution. | Prism Shadow | Running a business | Expert | Not in cards | Aug 2026 | ||||
| GDPval Extended Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores. | Mercor | Tasks from real jobs | Expert | Not in cards | Aug 2026 | ||||
| Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. | Google DeepMind | Deception and alignment | Not in cards | ||||||
| Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. | Google DeepMind | Refusing harmful requests | Not in cards | ||||||
| Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. | Google DeepMind | Jailbreaks and prompt injection | Not in cards | ||||||
| Gemini Robotics generalization benchmark Robot policies run 85 real bi-arm manipulation tasks. About 80% test generalization to new scenes, new wording of instructions or new actions. Scored by average progress and success rate. | Google DeepMind | Real robots | Everyday | Not in cards | Mar 2025 | ||||
| GenAI-Arena Users compare outputs from two anonymous models for image generation, image editing and video generation, and vote. Votes give Elo ratings. | TIGER-Lab (University of Waterloo) | Image generation and editing | Everyday | Not in cards | Jun 2024 | ||||
| Genspark Slides Benchmark Models build finished slide decks from 200 real production tasks in the Genspark Slides agent harness. An internal grader scores quality, completion, content and design. | Genspark | Documents and spreadsheets | Everyday | Not in cards | |||||
| GeoGuesser environment (FineEnvs) Agent placed at unknown Mapillary street-level location looks, walks, pins candidates and guesses coordinates. Distance-based GeoGuessr-style reward. | FineEnvs (Hugging Face community) | General visual questions | Everyday | Not in cards | Sep 2026 | ||||
| GitTaskBench Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it. | QuantaAlpha | Fixing real repositories | Expert | Not in cards | Aug 2025 | ||||
| GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. | Cohere Labs | Reasoning in many languages | School | Not in cards | Mar 2026 | ||||
| GlobeBench Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue. | Patronus AI | Interactive world models | Expert | Not in cards | Oct 2026 | ||||
| GMAI-MMBench Vision-language models answer medical multiple-choice questions over many imaging modalities, clinical tasks and departments, at several perceptual granularities. | Shanghai AI Laboratory (uni-medical) | Medicine | Expert | Not in cards | Aug 2024 | ||||
| Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. | Google DeepMind | Dangerous capabilities | Not in cards | ||||||
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Google DeepMind (with Wiz) | Finding and exploiting real bugs | Expert | Not in cards | |||||
| Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. | Gray Swan | Jailbreaks and prompt injection | Not in cards | ||||||
| GSM8K-V Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures. | Zhejiang University | Visual math and diagrams | School | Not in cards | Sep 2025 | ||||
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | OpenCompass (Shanghai AI Laboratory) and collaborators | General assistants | Expert | Not in cards | Apr 2026 | ||||
| GTA v1 Replaced in Apr 2026 | Function calling | Everyday | Not in cards | GPT-4 Turbo 46.6% | Jul 2024 | May 2024 | |||
| Gym-Anything-RT Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time. | Carnegie Mellon University (Welleck lab, CMU L3) | Video games | Everyday | Not in cards | Oct 2026 | ||||
| HABIT-Bench Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later. | Shanghai Jiao Tong University (LUMIA) | Learning from the context | Everyday | Not in cards | Oct 2026 | ||||
| HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. | Meta (FAIR RAM group) | Deception and alignment | Expert | Not in cards | Oct 2026 | ||||
| HaluBench Models judge whether an answer is faithful to its context across about 15,000 labeled context, question and answer triplets. It tests hallucination detectors and LLM judges. | Patronus AI | Hallucination rates | University | Not in cards | Jul 2024 | ||||
| HaluMem Agent memory systems are tested for hallucinations at each operation, memory extraction, updating and question answering, rather than only end to end. | MemTensor | Hallucination rates | Everyday | Not in cards | Nov 2025 | ||||
| HappyWorld-Bench World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them. | Alibaba | Interactive world models | Everyday | Not in cards | Sep 2026 | ||||
| HarnessDev Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score. | ByteDance Seed | AI research tasks | Research frontier | Not in cards | Sep 2026 | ||||
| HCAST Agents attempt 189 machine learning, cybersecurity, software engineering and reasoning tasks. 563 human baselines time each task, from one minute to over 8 hours, so success can be read against human task length. | METR | Task length | Expert | Not in cards | Mar 2025 | ||||
| Hebbia Financial Services Benchmark Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion. | Hebbia | Finance | Expert | Not in cards | Oct 2026 | ||||
| HEIM Text-to-image models are scored on 12 aspects, such as alignment, quality, aesthetics, originality, reasoning, knowledge, bias and toxicity, with human raters and automatic metrics. | Stanford CRFM | Image generation and editing | Everyday | Not in cards | Nov 2023 | ||||
| Hemingway-bench Professional writers compare two models' answers blind on real writing prompts, from bedtime stories to business documents. The comparisons become ratings. | Surge AI | Creative writing | Expert | Not in cards | Feb 2026 | ||||
| HiL-Bench Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary. | Scale AI | Fixing real repositories | Expert | Not in cards | Apr 2026 | ||||
| HLE Extended Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores. | Mercor | Expert-written questions | Expert | Not in cards | Aug 2026 | ||||
| HumanoidBench Policies control a simulated Unitree H1 humanoid with dexterous hands on 15 whole-body manipulation and 12 locomotion tasks. Scored by task return or success. | UC Berkeley + Yonsei University (Sferrazza et al.) | Simulated robots | Everyday | Not in cards | Mar 2024 | ||||
| HumanoidToolBench Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1. | Seoul National University (SNU PI) | Real robots | Everyday | Not in cards | Oct 2026 | ||||
| IMO-GradingBench Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers. | Google DeepMind (Superhuman Reasoning team) | Written proofs | Competition | Not in cards | Nov 2025 | ||||
| IMO-LeanProofBench Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof. | Google DeepMind (Superhuman Reasoning team) | Formal proofs | Competition | Not in cards | Jun 2026 | ||||
| IndustryBench Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses. | Alibaba (1688 multimodal and industrial AI) | Other professions | Expert | Not in cards | May 2026 | ||||
| InnovationEval Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches. | Epoch AI | AI research tasks | Research frontier | Not in cards | Oct 2026 | ||||
| InteractiveBench Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome. | Meta + Duke University | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| Into the Unknown Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check. | Prometheus-Eval team with US DOE national laboratories (Seungone Kim et al.) | Research science | Research frontier | Not in cards | Oct 2026 | ||||
| Investor's Last Exam Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria. | Prophet Lab | Finance | Expert | Not in cards | Oct 2026 | ||||
| IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. | Masakhane and collaborators (Adelani et al.) | Knowledge in many languages | School | Not in cards | Jun 2024 | ||||
| ITBench-AA Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON. | Artificial Analysis + IBM Research | Terminal tasks | Expert | Not in cards | May 2026 | ||||
| JevBench Decision models, such as TypeSafe AI's Jev and compatible systems, answer typed decision questions. The board scores accuracy, probability calibration, latency and cost. | Benchmark Heaven | Other | Everyday | Not in cards | Sep 2026 | ||||
| K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. | Carnegie Mellon University and collaborators | Knowledge in many languages | Expert | Not in cards | Jun 2026 | ||||
| KellyBench Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth. | General Reasoning | Social simulation | Expert | Not in cards | Apr 2026 | ||||
| KeyFrame-Compass Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities. | KlingAI Research (Kuaishou) | Video generation | Everyday | Not in cards | Jul 2026 | ||||
| KilometerVision Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures. | Google DeepMind (Perception Test team) | Spatial reasoning and counting | Everyday | Not in cards | Jun 2026 | ||||
| Kimi Code Bench Moonshot in-house coding-agent tasks from internal engineering cases, production incidents and open-source projects across 10+ languages. 80 tasks. | Moonshot AI | Fixing real repositories | Expert | Not in cards | |||||
| KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. | HAE-RAE | Knowledge in many languages | University | Not in cards | Jan 2024 | ||||
| KnowMe-Bench Models read long autobiographical narratives and must infer a person's stable motivations and decision principles, not just retrieve facts. | QuantaAlpha | Learning from the context | Everyday | Not in cards | Jan 2026 | ||||
| KoBEST Models do five Korean tasks designed by professional linguists: BoolQ, COPA, WiC, HellaSwag and SentiNeg. All data is human-annotated. | SK Telecom and collaborators (Kim, Jang et al.) | Reasoning in many languages | Everyday | Not in cards | Apr 2022 | ||||
| KoMT-Bench Models follow multi-turn instructions in Korean on a translated and culturally adapted version of MT-Bench, reviewed by expert linguists. | LG AI Research | Chat and instructions in many languages | Everyday | Not in cards | Aug 2024 | ||||
| Kontext Bench Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency. | Black Forest Labs | Video generation | Everyday | Not in cards | Jun 2025 | ||||
| KORA Apps Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks. | KORA (non-profit) | Wellbeing and sycophancy | Everyday | Not in cards | Jul 2026 | ||||
| LAB: Contracts 500 in-house contracting tasks (drafting, redlines, issues lists, escalation memos) across 10 contract categories. All-pass expert-rubric grading. | Harvey | Law | Expert | Not in cards | Jun 2026 | ||||
| LAB: Diligence Agent researches a synthetic M&A virtual data room (thousands of documents, up to ~80M tokens) and drafts a diligence memo scored against planted-issue rubric criteria. | Harvey | Law | Expert | Not in cards | Jul 2026 | ||||
| LAB: US Case Law Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings. | Harvey | Law | Expert | Not in cards | Oct 2026 | ||||
| LABBench2 FigQA2 Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | Feb 2026 | ||||
| LAB-Bench FigQA v1 Replaced in Feb 2026 | FutureHouse | System cards only | Claude Opus 4.8 80.4% | 14 | Jul 2024 | Jun 2026 | |||
| LABBench2 LitQA3 Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | Feb 2026 | ||||
| LAB-Bench LitQA2 v1 Replaced in Feb 2026 | FutureHouse | Not in cards | Jul 2024 | ||||||
| LABBench2 ProtocolQA2 Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | Feb 2026 | ||||
| LAB-Bench ProtocolQA v1 Replaced in Feb 2026 | FutureHouse | Lab protocols | System cards only | Claude Opus 4.5 90.7% | 18 | Jul 2024 | Oct 2026 | ||
| LABBench2 SeqQA2 Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2. | Edison Scientific (formerly FutureHouse) | Genomics and life sciences | Expert | Not in cards | Feb 2026 | ||||
| LAB-Bench SeqQA v1 Replaced in Feb 2026 | FutureHouse | System cards only | Muse Spark 1.1 98.2% | 15 | Jul 2024 | Oct 2026 | |||
| Last Translation Benchmark Strong models translate hard texts chosen to stay unsaturated, and experts assess the outputs instead of automatic metrics that can be reward-hacked. | Zouhar et al. | Translation | Research frontier | Not in cards | Sep 2026 | ||||
| Learn2Play Bench Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure. | National University of Singapore | Puzzles and chess | Everyday | Not in cards | Oct 2026 | ||||
| Legacy-Bench Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass. | Factory | Fixing real repositories | Expert | Not in cards | Sep 2026 | ||||
| LEGO-Puzzles Vision-language models answer LEGO assembly questions, from basic spatial understanding to planning several assembly steps. The elementary set has 1,100 questions over 11 tasks. | Shanghai AI Laboratory and collaborators (Tang, Duan et al.) | Spatial reasoning and counting | Everyday | Not in cards | Mar 2025 | ||||
| LEXam-hard Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed. | Joel Niklaus (LEXam co-author) | Law | Research frontier | Not in cards | Sep 2026 | ||||
| LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. | Scale AI | General assistants | Expert | Not in cards | Feb 2026 | ||||
| LIBERO-MAX Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing. | LIBERO-MAX authors | Simulated robots | Everyday | Not in cards | Aug 2026 | ||||
| LIBERO-Para VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric. | Soongsil University + Chung-Ang University | Simulated robots | Everyday | Not in cards | Mar 2026 | ||||
| LibraryDesignBench An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity. | SprocketLab (University of Wisconsin-Madison) with Snorkel AI | Building whole projects | Expert | Not in cards | Sep 2026 | ||||
| LiveMACE Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns. | Fudan University | Finance | Expert | Not in cards | Oct 2026 | ||||
| LiveMath Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures. | Google DeepMind | Competition problems with answers | Competition | Not in cards | Sep 2025 | ||||
| LiveMathBench Models solve recent competition math problems, sampled many times. The G-Pass@k metric rewards both peak accuracy and stable answers across samples. | OpenCompass (Shanghai AI Laboratory) | Competition problems with answers | Competition | Not in cards | Dec 2024 | ||||
| LLaVA-Bench (In-the-Wild) Multimodal assistants answer open questions about a small set of diverse real images. A GPT-4 judge scores each answer relative to a reference answer. | UW-Madison and Microsoft Research (LLaVA team) | General visual questions | Everyday | Not in cards | Apr 2023 | ||||
| LLM Colosseum Two models each control a fighter in Street Fighter III and choose moves from text descriptions of the game. Wins give Elo ratings. | phospho (OpenGenerativeAI) | Video games | Everyday | Not in cards | Mar 2024 | ||||
| LMBuild Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended. | University of Illinois Urbana-Champaign | CAD models | Expert | Not in cards | Oct 2026 | ||||
| lmgame-Bench Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds. | Hu et al. (lmgame-org) | Video games | Everyday | Not in cards | May 2025 | ||||
| LoCoMo Models answer questions, summarize events and continue dialogue over very long multi-session chats of about 300 turns and 9K tokens, up to 35 sessions. Answers are scored against references. | Snap Research, UNC Chapel Hill and USC (Maharana et al.) | Finding facts in long inputs | Everyday | Not in cards | Feb 2024 | ||||
| LogiQA Models answer reading comprehension questions that test logical reasoning, taken from Chinese civil service exams and translated to English. | Liu et al. (LogiQA authors) | Language reasoning | University | Not in cards | Jul 2020 | ||||
| LongAV-Compass Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time. | KlingAI Research (Kuaishou) and Peking University | Video generation | Everyday | Not in cards | May 2026 | ||||
| LongMemEval-V2 Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized. | UCLA NLP | Learning from the context | Expert | Not in cards | May 2026 | ||||
| LongMemEval v1 Replaced in May 2026 | LongMemEval team (Wu et al., ICLR 2025) | Finding facts in long inputs | Everyday | In use | Nemotron 3.5 Bolt E4B 69.9% | 1 | 1 | Oct 2024 | Oct 2026 |
| LoopArena Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit. | AMAP-ML (Alibaba) | Building whole projects | Expert | Not in cards | Aug 2026 | ||||
| m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Cohere Labs | Chat and instructions in many languages | University | Not in cards | Apr 2026 | ||||
| m-ArenaHard v2.0 Replaced in Apr 2026 | Not in cards | Jun 2025 | |||||||
| m-ArenaHard v1 Replaced in Jun 2025 | System cards only | 2 | Oct 2024 | May 2025 | |||||
| MA-Bench Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics. | Scale AI (Lafuente-Mercado et al.) | Other | Expert | Not in cards | Oct 2026 | ||||
| MaCBench Vision language models answer chemistry and materials questions from images. Tasks cover reading data, understanding experiments and interpreting results. | MaCBench authors (Alampara et al., ChemBench group) | Lab and data work | Expert | Not in cards | Nov 2024 | ||||
| MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. Annotations score power-seeking, disutility and ethical violations next to game reward. | UC Berkeley and Center for AI Safety | Deception and alignment | Everyday | Not in cards | Apr 2023 | ||||
| MacroscopeBench Models review commits from public repositories and must flag the known bugs a commit introduced without false reports. LLM judges match findings to the bugs. | Macroscope | Fixing real repositories | Expert | Not in cards | |||||
| MAEB Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages. | MTEB community | Audio understanding and voice | Everyday | Not in cards | Feb 2026 | ||||
| Management Consulting Tasks (OpenAI) Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading. | OpenAI | Running a business | Expert | Not in cards | Jul 2026 | ||||
| ManipArena 20 real bimanual tasks on X2Robot platforms: 10 execution, 5 semantic reasoning and 5 mobile manipulation, with in-domain and shifted trials. Partial-credit scoring. | X Square Robot | Real robots | Everyday | Not in cards | Mar 2026 | ||||
| ManiSkill3 GPU-parallel robot simulator and task suite across 12 domains for manipulation learning. Success rate per task. | UC San Diego (Hao Su Lab) + Hillbot | Simulated robots | Everyday | Not in cards | Oct 2024 | ||||
| Mantic vs Polymarket backtest Point-in-time backtest of Mantic's AI forecasts against prices on 4,139 Polymarket markets at volume snapshots. Scored by Brier score and calibration. | Mantic | Real-world events | Expert | Not in cards | Sep 2026 | ||||
| MBABench 2.0 Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar. | Columbia University (Namkoong Lab), hosted by Snorkel AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| MBABench v1 Replaced in Oct 2026 | Columbia University (Namkoong Lab) | Not in cards | May 2026 | ||||||
| MC-Bench Models write code that builds a Minecraft structure from a prompt. Users vote between two builds without knowing the models. | MC-Bench (Aditya Singh and volunteers) | Spatial reasoning and counting | Everyday | Not in cards | Mar 2025 | ||||
| MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. | Accenture | Apps and MCP servers | Expert | Not in cards | Aug 2025 | ||||
| MedAgentBench Agents use FHIR APIs in a virtual electronic health record to do 300 patient tasks in 10 categories written by physicians. | Stanford University | Medicine | Expert | Not in cards | Jan 2025 | ||||
| MedChemBench (OpenAI) Models answer chemistry questions from an OpenAI internal set, grouped with OpenAI's life science research evaluations. OpenAI has not published the task format. | OpenAI | Genomics and life sciences | Expert | Not in cards | Jul 2026 | ||||
| MedXpertQA MM Extended Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores. | Mercor | Medicine | Expert | Not in cards | Aug 2026 | ||||
| MemEye Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply. | MemEye authors (Kwok et al.) | Finding facts in long inputs | Everyday | Not in cards | May 2026 | ||||
| MemLens Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths. | NVIDIA and collaborators | Finding facts in long inputs | Everyday | Not in cards | May 2026 | ||||
| MemoryAgentBench Memory agents receive long inputs in chunks over many turns and are tested on accurate retrieval, test-time learning, long-range understanding and selective forgetting. It adds two new sets, EventQA and FactConsolidation. | UC San Diego (Hu, Wang, McAuley) | Learning from the context | Everyday | Not in cards | Jul 2025 | ||||
| MemTrapBench Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance. | Zhejiang University (ZJUNLP) | Learning from the context | Everyday | Not in cards | Aug 2026 | ||||
| MerchantBench Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence. | Alibaba | Simulated companies | Expert | Not in cards | Jul 2026 | ||||
| Meta-World Robot policies do simulated tabletop manipulation tasks, scored by success rate. Splits test multi-task learning, such as MT10 and MT50, and meta-learning, such as ML45. | Farama Foundation (originally Stanford, Berkeley and others) | Simulated robots | Everyday | Not in cards | Oct 2019 | ||||
| metal-arm-harness Safety-enveloped CLI that lets any coding agent drive a real MakerMods Metal or Maker arm one command at a time from camera JPEGs. | MakerMods Robotics | Real robots | Everyday | Not in cards | Sep 2026 | ||||
| METR pre-deployment AI R&D tasks METR's long-horizon tasks for Anthropic pre-deployment tests: Train a Program, Gaming Bot, Sunlight open-ended research, plus Budget NanoGPT and LMCA. | METR | AI research tasks | Research frontier | Not in cards | |||||
| METR Time Horizon suite Software tasks with human completion times. Reports the task length at which a model succeeds 50% of the time. | METR | Task length | Research frontier | Not in cards | Mar 2025 | ||||
| MeViS Models segment the objects in a video that a sentence picks out by their motion, not by static looks. Scored by J&F. | Nanyang Technological University and Fudan University (Ding, Liu, He, Jiang, Loy) | Video understanding | Everyday | Not in cards | Aug 2023 | ||||
| MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. | Reasoning in many languages | School | Not in cards | Nov 2025 | |||||
| MGSM v1 Replaced in Nov 2025 | Google Research | Fading | Cogito v2 Preview 671B MoE 93% | 33 | 36 | Oct 2022 | Sep 2026 | ||
| MIEB Image and image-text embedding models are scored on 130 tasks in 38 languages, grouped into 8 task types such as retrieval, clustering and classification. | MTEB community | General visual questions | Everyday | Not in cards | Apr 2025 | ||||
| MIMIC-CXR (classification) Models read chest X-rays from Beth Israel Deaconess and say whether five common conditions are present. Scored by average F1 over the conditions. | MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| MIMIC-CXR report generation Models write the findings of a radiology report for a MIMIC-CXR chest X-ray. Scored against the radiologist's report, most often by RadGraph F1. | MIT Laboratory for Computational Physiology and Beth Israel Deaconess Medical Center (Johnson et al.) | Medicine | Expert | Not in cards | Jan 2019 | ||||
| Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. | Ohio State University (OSU NLP Group) | Deep research | Expert | Not in cards | Jun 2025 | ||||
| Mind2Web v1 Replaced in Jun 2025 | Web browsing | Everyday | Dropped | Qwen3-Coder-480B-A35B 55.8% | 1 | Jun 2023 | Jul 2025 | ||
| MindEval2 Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring. | Sword Health | Medicine | Expert | Not in cards | Oct 2026 | ||||
| MindEval v1 Replaced in Oct 2026 | Not in cards | Nov 2025 | |||||||
| MineBench Models turn a prompt into a Minecraft-style voxel build as raw 3D coordinates or tool calls. Users compare builds blind, and votes become rankings. | Ammaar Alam | Spatial reasoning and counting | Everyday | Not in cards | Jan 2026 | ||||
| MINERVA-Cultural Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2. | Google DeepMind | Long videos | Expert | Not in cards | Jan 2026 | ||||
| Minerva-Ego Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects. | Google DeepMind | Video understanding | Expert | Not in cards | May 2026 | ||||
| MiniWoB++ Agents operate over 100 small synthetic web pages, such as forms, menus and date pickers, from a natural-language instruction. Each task returns a success reward. | Stanford University, now maintained by the Farama Foundation | Web browsing | Everyday | Not in cards | Feb 2018 | ||||
| MINTEval Memory-augmented agents answer questions after long, frequently updated histories where memories interfere. Instances average 138,800 tokens and reach 1.8 million. | MINTEval authors (Lee et al.) | Learning from the context | Expert | Not in cards | May 2026 | ||||
| MLAgentBench Agents run machine learning experiments on 13 tasks, from improving CIFAR-10 accuracy to research problems like BabyLM. They read and write files, run code and inspect outputs. | Stanford University (SNAP) | Machine learning engineering | Expert | Not in cards | Oct 2023 | ||||
| MLCommons AILuminate Jailbreak Models answer hazard prompts with and without taxonomy-based jailbreak attacks. The Resilience Gap is the drop in safety score under attack. | MLCommons | Jailbreaks and prompt injection | Everyday | Not in cards | Dec 2025 | ||||
| MLCR-AA Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy. | Wisedocs + Artificial Analysis | Medicine | Expert | Not in cards | Aug 2026 | ||||
| MLGym-Bench Agents work on 13 open-ended AI research tasks in vision, language, reinforcement learning and game theory. Scores compare the agent's result with baselines on each task's metric. | Meta | Machine learning engineering | Expert | Not in cards | Feb 2025 | ||||
| MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. | Facebook AI Research | Reading in many languages | Everyday | Not in cards | Oct 2019 | ||||
| MM-HELIX Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking. | MM-HELIX team (Zhao et al.) | Visual math and diagrams | University | Not in cards | Oct 2025 | ||||
| MM-NIAH Vision-language models answer questions about long documents of interleaved images and text. Tasks are retrieval, counting and reasoning over needles hidden in the document. | Shanghai AI Laboratory (OpenGVLab) | Finding facts in long inputs | Everyday | Not in cards | Jun 2024 | ||||
| MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. | East China Normal University and Shanghai AI Laboratory | Jailbreaks and prompt injection | Everyday | Not in cards | Nov 2023 | ||||
| MMAE Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types. | MMAE authors | Audio understanding and voice | Everyday | Not in cards | Jun 2026 | ||||
| MMDU Vision-language models hold long conversations about many images, up to 20 images and 27 turns per dialog. A GPT-4o judge scores the answers. | Shanghai AI Laboratory and collaborators (Liu, Zang et al.) | General visual questions | Everyday | Not in cards | Jun 2024 | ||||
| MME-Reasoning Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning. | MME team (Alpha-Innovator) | Visual math and diagrams | University | Not in cards | May 2025 | ||||
| MMEB-V3 Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval. | University of Waterloo (TIGER-Lab) and collaborators | General visual questions | Everyday | Not in cards | Apr 2026 | ||||
| MMEB-V2 Replaced in Apr 2026 | University of Waterloo (TIGER-Lab) + Salesforce Research | Not in cards | May 2025 | ||||||
| MMEB v1 Replaced in May 2025 | University of Waterloo (TIGER-Lab) + Salesforce Research | Not in cards | Oct 2024 | ||||||
| MMEnterprise-Bench Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough. | MLV Lab (mlvlab) | Simulated companies | Expert | Not in cards | Oct 2026 | ||||
| MMLU Medical Models answer the multiple-choice questions from the medical subjects of MMLU, such as anatomy, clinical knowledge and professional medicine. | UC Berkeley (Hendrycks et al.), subset chosen by medical model reports | Medicine | University | Not in cards | Sep 2020 | ||||
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University of Tokyo (Li Lab) and collaborators | Knowledge in many languages | University | Not in cards | May 2025 | ||||
| MMLU-Redux 2.0 Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them. | University of Edinburgh | Exam questions | University | Not in cards | Aug 2024 | ||||
| MMLU-Redux 1.0 Replaced in Aug 2024 | Fading | Qwen3.7-Max 95% | 26 | 25 | Jun 2024 | Jun 2026 | |||
| MMMU-Pro Extended Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores. | Mercor | General visual questions | University | Not in cards | Aug 2026 | ||||
| MolmoAct2 sim_eval Zero-shot ManiSkill evaluation of MolmoAct2 policies: put-everything-in-box tasks for Franka DROID and bimanual YAM, 10 episodes per task. | Ai2 | Simulated robots | Everyday | Not in cards | Jun 2026 | ||||
| Movie Gen Video Bench Video generation models make clips from a fixed prompt set, and people compare them. Meta released non-cherry-picked Movie Gen videos for comparison. | Meta | Video generation | Everyday | Not in cards | Oct 2024 | ||||
| MP-DocVQA Models answer questions about scanned documents of up to 20 pages, where the answer sits on one page. Scored by ANLS against the reference answer. | Computer Vision Center, Universitat Autonoma de Barcelona (Tito, Karatzas, Valveny) | Documents, charts and text in images | Everyday | Not in cards | Dec 2022 | ||||
| MPIE-Bench Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other. | muset.ai | Image generation and editing | Everyday | Not in cards | Jul 2026 | ||||
| MS-CXR-T Models compare a current and a prior chest X-ray of one patient and say whether a finding improved, stayed stable or got worse. | Microsoft Health Futures (Bannur et al.) | Medicine | Expert | Not in cards | Jan 2023 | ||||
| MSEB Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction. | Google Research | Audio understanding and voice | Everyday | Not in cards | Feb 2026 | ||||
| MTEB Massive Text Embedding Benchmark: retrieval, clustering, classification across languages and modalities. | MTEB community | Other | Not in cards | Oct 2022 | |||||
| MTEB Arena Users compare retrieval, clustering and similarity results from two anonymous embedding models and vote for the better one. | MTEB community | Other | Everyday | Not in cards | Jul 2024 | ||||
| Multi-LCB Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation. | Multi-LCB team | Competitive programming | Competition | Not in cards | Jun 2026 | ||||
| Music Arena Users write a music prompt, listen to tracks from two text-to-music systems, and vote. Preferences, listening data and feedback feed a leaderboard. | Carnegie Mellon University (G-CLef) with Arena and Sony researchers | Audio understanding and voice | Everyday | Not in cards | Jul 2025 | ||||
| MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. | MyClaw.ai | General assistants | Everyday | Not in cards | Mar 2026 | ||||
| NanoGPT-Bench Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered. | Intology | AI research tasks | Research frontier | Not in cards | May 2026 | ||||
| NaturalBench Models answer 10,000 human-checked questions on natural images. Each question pairs with two images that need different answers, so a model cannot pass without looking. | Carnegie Mellon University and others (Baiqi Li, Zhiqiu Lin et al.) | General visual questions | Everyday | Not in cards | Oct 2024 | ||||
| NeedleBench V2 Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks. | OpenCompass (Shanghai AI Laboratory) | Finding facts in long inputs | Everyday | Not in cards | May 2025 | ||||
| NeedleBench v1 Replaced in May 2025 | System cards only | 1 | Jul 2024 | Jul 2024 | |||||
| NoLiMa Models find a needle in a long haystack when the question and the needle share almost no words, so they must infer a latent link. Accuracy is compared across context lengths. | Adobe Research and LMU Munich (Modarressi et al.) | Finding facts in long inputs | Everyday | Not in cards | Feb 2025 | ||||
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | New York University (NYU Tandon) | Capture-the-flag challenges | University | Not in cards | Jun 2024 | ||||
| OBLIQ-Bench Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems. | MIT (Tchuindjo, Shah and Khattab) | Questions that need search | Expert | Not in cards | May 2026 | ||||
| OCBench 28 simulated manipulation tasks with human-like scripted demonstrators for controlled BC and RL studies. Policies scored by task success rate. | UC Berkeley (Seohong Park, Sergey Levine) | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| OccuBench Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable. | Qwen (Alibaba) | Tasks from real jobs | Expert | Not in cards | Apr 2026 | ||||
| OdysseyArena Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks. | OdysseyArena authors | Puzzles and chess | Expert | Not in cards | Feb 2026 | ||||
| olmOCR-Bench OCR systems convert 1,403 PDFs to markdown. 7,010 unit tests check text presence and absence, reading order, tables and math. | Allen Institute for AI | Documents, charts and text in images | Everyday | Not in cards | Feb 2025 | ||||
| Omni-InteractScope Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing. | Alibaba Group (Qwen, Alibaba Token Hub) + Hong Kong Polytechnic University | Video understanding | Everyday | Not in cards | Oct 2026 | ||||
| OmniVBench Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly. | Tencent | Video generation | Everyday | Not in cards | Sep 2026 | ||||
| Open LLM Leaderboard v2 Hugging Face ran open models on IFEval, BBH, MATH Level 5, GPQA, MuSR and MMLU-Pro and averaged normalized scores. | Hugging Face | Composite indices | University | Not in cards | Jun 2024 | ||||
| OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. | OpenAI | Deception and alignment | Not in cards | ||||||
| OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. | OpenAI | Jailbreaks and prompt injection | Not in cards | ||||||
| OpenAI internal AI self-improvement evals Internal Research Debugging (41 real research bugs plus alignment-auditing tasks) and KernelGen 1P (kernels for OpenAI first-party hardware). | OpenAI | Labs' internal R&D tests | Expert | Not in cards | |||||
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | OpenAI | Finding and exploiting real bugs | Expert | Not in cards | |||||
| OpenFrontiers-Bench Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings. | University of Illinois Urbana-Champaign (Tong Zhang) | Formal proofs | Research frontier | Not in cards | Oct 2026 | ||||
| OpenRSI Index Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from. | OpenRSI Foundation (UW-led, academic and industry partners) | AI research tasks | Research frontier | Not in cards | Sep 2026 | ||||
| OpenThoughts-TBLite Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates. | OpenThoughts Agent team, Snorkel AI and Bespoke Labs | Terminal tasks | Expert | Not in cards | Feb 2026 | ||||
| OSGym Infrastructure running 1,000+ parallel OS sandboxes for computer-use agent training and evaluation. Tasks span browsing, documents, SWE and multi-app work. | Zengyi Qin et al. (agiopen-org) | Desktop apps | Everyday | Not in cards | Nov 2025 | ||||
| OST-Bench Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes. | Shanghai AI Laboratory (OpenRobotLab) and collaborators | Embodied reasoning | Everyday | Not in cards | Jul 2025 | ||||
| OSWorld-Science Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task. | DiscoAILab | Desktop apps | Expert | Not in cards | Sep 2026 | ||||
| OVO-S-Bench Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos. | Shanghai AI Laboratory (InternLM) | Video understanding | Everyday | Not in cards | Jun 2026 | ||||
| Paloma Language models are measured by perplexity on held-out text from 585 domains, from news sites to individual subreddits. | Allen Institute for AI | Commonsense | Everyday | Not in cards | Dec 2023 | ||||
| Pantograph frontier-LLM robot benchmark (Pandroid) GPT-6 Astra and Claude Fable 5.1 drive Pandroid robots through a fixed tool harness on eight simple manipulation tasks, 10 blind-graded attempts each. | Pantograph PBC | Real robots | Everyday | Not in cards | Sep 2026 | ||||
| ParallelKernelBench Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline. | Together AI and Stanford | GPU kernels | Expert | Not in cards | Jun 2026 | ||||
| PatchEval Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. | ByteDance | Finding and exploiting real bugs | Expert | Not in cards | Nov 2025 | ||||
| PathMCQA (Google) Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| PatientAgentBench Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria. | Amazon | Medicine | Expert | Not in cards | Jul 2026 | ||||
| PAWBench Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior. | PAWBench team (with Hugging Face's Sayak Paul) | Physical prediction | Everyday | Not in cards | Aug 2026 | ||||
| Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. | Robocurve | Jailbreaks and prompt injection | Everyday | Not in cards | Oct 2026 | ||||
| Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. | Anthropic | Deception and alignment | Expert | Not in cards | Jan 2026 | ||||
| Petri 1.0 Replaced in Jan 2026 | System cards only | 8 | Oct 2025 | Jul 2026 | |||||
| PhysClosureBench Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them. | Yijia Mao et al. | Research science | Research frontier | Not in cards | Oct 2026 | ||||
| PhysicalRealismBench-U VLM watches 1,300 synthetic Isaac Sim videos, flags physics-law violations, names object and frames. Scored by joint realism F1. | Reka | Physical prediction | Everyday | Not in cards | Jun 2026 | ||||
| PhysVista Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos. | University of Science and Technology of China | Physical prediction | Everyday | Not in cards | Sep 2026 | ||||
| PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. | University of Illinois Urbana-Champaign | Function calling | Expert | Not in cards | Jun 2026 | ||||
| PlanningBench Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers. | Tencent Hunyuan | Language reasoning | Expert | Not in cards | May 2026 | ||||
| PokeAgent Challenge Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning. | Princeton University and collaborators | Video games | Expert | Not in cards | Mar 2026 | ||||
| PRDBench Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria. | AGI-Eval (Meituan) | Building whole projects | Expert | Not in cards | Oct 2025 | ||||
| ProcessBench Models read step-by-step solutions to GSM8K, MATH, OlympiadBench and Omni-MATH problems and name the earliest wrong step, or say none is wrong. Scored by F1. | Qwen Team, Alibaba Group | Competition problems with answers | Competition | Not in cards | Dec 2024 | ||||
| ProofRank Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings. | MathArena (ETH Zurich SRI Lab and INSAIT) | Written proofs | Competition | Not in cards | May 2026 | ||||
| PROWBench Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules. | Alaya Lab | Interactive world models | Everyday | Not in cards | Oct 2026 | ||||
| Putnam-like Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading. | Google DeepMind | Written proofs | University | Not in cards | Sep 2025 | ||||
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Novee | Finding and exploiting real bugs | Expert | Not in cards | Aug 2026 | ||||
| Q-Bench Multimodal models are tested on low-level vision: perceiving image distortions, describing them and predicting quality scores. | Nanyang Technological University and collaborators | General visual questions | Everyday | Not in cards | Sep 2023 | ||||
| Q-Bench-Video Models judge the quality of natural, AI-generated and computer-graphics videos through yes-or-no, what-how, open-ended and pairwise comparison questions. | Q-Future (Nanyang Technological University and Shanghai Jiao Tong University) | Video understanding | Everyday | Not in cards | Sep 2024 | ||||
| QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. | Technology Innovation Institute (AlQadi, Alzubaidi et al.) | Knowledge in many languages | School | Not in cards | Jan 2026 | ||||
| Qodo AI Code Review Benchmark Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1. | Qodo | Fixing real repositories | Expert | Not in cards | Feb 2026 | ||||
| Qwen-Image-Bench Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images. | Qwen (Alibaba) | Video generation | Everyday | Not in cards | May 2026 | ||||
| ReactHuman Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate. | ReactHuman authors | Simulated robots | Everyday | Not in cards | Sep 2026 | ||||
| Real-SWE Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors. | Specific Labs | Fixing real repositories | Expert | Not in cards | Sep 2026 | ||||
| RealCompanion Models read months of real chats between ten people and an AI companion, then answer questions about who each person is. The release has 27,218 messages over up to 120 days. | Quis Lab | Multi-turn conversation | Everyday | Not in cards | Oct 2026 | ||||
| REALM High-fidelity simulation benchmark for VLA generalization: 15 perturbation factors, 7 skills, 3,500+ objects. Success rate validated against real-robot results. | Czech Technical University (CIIRC) + University of Amsterdam | Simulated robots | Everyday | Not in cards | Dec 2025 | ||||
| ReasonVOS Models segment the object in a video that an implicit query describes, where finding it needs world knowledge and reasoning. Scored by J&F. | University of Technology Sydney and Alibaba (Yan et al., VISA) | Video understanding | Everyday | Not in cards | Jul 2024 | ||||
| RedTeam Arena Players get 60 seconds to make a model say a target word. An extended Elo ranks players, models and target prompts, with models ranked by resistance. | Arena (formerly LMArena) | Jailbreaks and prompt injection | Everyday | Not in cards | Sep 2024 | ||||
| Ref-DAVIS17 Models segment and track the object that a sentence describes through each frame of a DAVIS 2017 video. Scored by J&F, the mean of mask overlap and contour accuracy. | Max Planck Institute for Informatics (Khoreva, Rohrbach, Schiele) | Video understanding | Everyday | Not in cards | Mar 2018 | ||||
| Ref-YouTube-VOS Models segment and track the object that a sentence describes in YouTube-VOS videos. Scored by J&F on a server-graded validation split. | Seoul National University and Adobe Research (Seo, Lee, Han, URVOS) | Video understanding | Everyday | Not in cards | Aug 2020 | ||||
| RefSpatial-Expand-Bench Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit. | BAAI (RoboRefer) | Embodied reasoning | Everyday | Not in cards | Oct 2025 | ||||
| RefSpatial-Bench 1.0 Replaced in Oct 2025 | Fading | Qwen3.5 397B-A17B 73.6% | 6 | 4 | Jun 2025 | Apr 2026 | |||
| RepoChat Arena Users give a GitHub repository link and a question. A retriever model picks files, two answer models reply, and users vote for the better answer. | Arena (formerly LMArena) | Fixing real repositories | Everyday | Not in cards | Jan 2025 | ||||
| RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. | RiOSWorld authors (Yang et al.) | Refusing harmful requests | Everyday | Not in cards | May 2025 | ||||
| RMBench Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate. | HKU MMLab + Peking University + others (RoboTwin team) | Simulated robots | Everyday | Not in cards | Mar 2026 | ||||
| RoboChallenge Table30 v2 2.0 Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit. | Dexmal + Hugging Face | Real robots | Everyday | Not in cards | May 2026 | ||||
| RoboChallenge 1.0 Replaced in May 2026 | Not in cards | π0.5 42.7% | Oct 2025 | Apr 2025 | |||||
| Robocurve physical tool-use benchmark (unidentified) Unnamed Robocurve physical tool-use robot task. | Robocurve | Real robots | Everyday | Not in cards | |||||
| Robocurve simple tasks (LLM control of YAM arms) Frontier LLMs control bimanual I2RT YAM arms through Inspect Robots tools on block-into-bowl, puzzle, push-ball and stack-bowls. Staged scores, 20 trials. | Robocurve | Real robots | Everyday | Not in cards | Sep 2026 | ||||
| RoboGate 68 industrial pick-and-place scenarios (nominal, edge, adversarial, randomized) for a Franka arm in Isaac Sim. Organizers run submitted VLAs and report success. | AgentAI Co., Ltd. | Simulated robots | Everyday | Not in cards | Mar 2026 | ||||
| RoboMemArena Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots. | OpenHelix Team | Real robots | Everyday | Not in cards | May 2026 | ||||
| Robot Capabilities Index (Robocurve) An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3. | Robocurve | Real robots | Everyday | Not in cards | Oct 2026 | ||||
| Robotic Origami Challenge (IROS 2026) Fold a six-fold paper airplane on a supplied bimanual Sharpa-hand rig. Origami judges give pass/fail, and passes are ranked by folding time. | Sharpa + BitRobot/FrodoBots + Nippon Origami Association + partners | Real robots | Everyday | Not in cards | Jun 2026 | ||||
| RobotWorld Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks. | RobotWorld team | Simulated robots | Everyday | Not in cards | Oct 2026 | ||||
| RoboWorld Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1. | Carnegie Mellon University (Welleck lab, CMU L3) | Simulated robots | Expert | Not in cards | Oct 2026 | ||||
| robust-kbench Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel. | Sakana AI | GPU kernels | Expert | Not in cards | Sep 2025 | ||||
| Role-play Benchmark (MiniMax) Role-play agents are tested on situated reenactment in Chinese and English scenes. Grading targets the immersive experience rather than a verifiable answer. | MiniMax | Creative writing | Everyday | Not in cards | Jan 2026 | ||||
| RoTA-Bench Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette. | MAC-AutoML (Xiamen University) | Spatial reasoning and counting | Everyday | Not in cards | Oct 2026 | ||||
| RSI Arena Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback. | Bake AI | Machine learning engineering | Research frontier | Not in cards | Sep 2026 | ||||
| RSI Index (OpenAI) An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model. | OpenAI | Labs' internal R&D tests | Research frontier | Not in cards | Jul 2026 | ||||
| RSIBench-Data Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks. | Evolvent AI + National University of Singapore | Machine learning engineering | Research frontier | Not in cards | Jul 2026 | ||||
| RTEB Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10. | MTEB community and Hugging Face (with MongoDB) | Other | University | Not in cards | Oct 2025 | ||||
| RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. | Microsoft (de Wynter et al.) | Bias, fairness and politics | Everyday | Not in cards | Apr 2024 | ||||
| RULERv2 Models solve synthetic multi-key and multi-value needle tasks and QA tasks at basic, easy, medium and hard levels. Contexts run from 8K to 1M tokens, and accuracy is averaged across lengths. | NVIDIA | Finding facts in long inputs | Everyday | Not in cards | Oct 2025 | ||||
| RULER v1 Replaced in Oct 2025 | In use | Nemotron-Cascade 2 30B-A3B 99% | 16 | 33 | Apr 2024 | Oct 2026 | |||
| RuleTrek Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements. | RuleTrek authors | Language reasoning | Expert | Not in cards | Oct 2026 | ||||
| S3Gym Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict. | ByteDance Seed | AI research tasks | Expert | Not in cards | Aug 2026 | ||||
| SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. | Vals AI | Wellbeing and sycophancy | Everyday | Not in cards | Oct 2026 | ||||
| SafeLIBERO VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success. | Tsinghua University (THU-RCSCT, Hu et al.) | Simulated robots | Everyday | Not in cards | Dec 2025 | ||||
| Sakana kana-letter reading-order set Model recovers the reading order of 25 expert-annotated classical Japanese kana letters (chirashigaki). Scored by normalized edit distance. | Sakana AI | Documents, charts and text in images | Expert | Not in cards | Jun 2026 | ||||
| ScholarCatalyst Given a research question, retrieve the 'catalyst papers' that 184 lead authors labeled for 207 projects, searching only prior literature. Recall@20. | Stanford + collaborators | Research science | Expert | Not in cards | Oct 2026 | ||||
| SciArena Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings. | Yale NLP and Allen Institute for AI | Science questions | Expert | Not in cards | Jul 2025 | ||||
| ScienceAgentBench Agents write programs for 102 data-driven discovery tasks from 44 papers in four disciplines. Outputs are checked by execution and task-specific metrics. | Ohio State University (OSU NLP Group) | Research science | Expert | Not in cards | Oct 2024 | ||||
| ScienceBoard Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking. | OS-Copilot team (Shanghai AI Laboratory and HKU) | Desktop apps | Expert | Not in cards | May 2025 | ||||
| ScienceWorld Agents carry out elementary-school science tasks, such as measuring conductivity or growing plants, by acting in an interactive text environment. Each task scores progress toward its goal. | Allen Institute for AI (Wang, Jansen, Côté, Ammanabrolu) | Lab and data work | School | Not in cards | Mar 2022 | ||||
| SciPredict Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge. | Scale AI | Research science | Expert | Not in cards | Jan 2026 | ||||
| SciQ Models answer crowdsourced multiple-choice science exam questions in physics, chemistry and biology, each with a supporting passage. | Allen Institute for AI (Welbl et al.) | Science questions | School | Not in cards | Jul 2017 | ||||
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Anthropic (with MATS and Anthropic Fellows) | Finding and exploiting real bugs | Expert | Not in cards | Dec 2025 | ||||
| SCROLLS Models do seven long-text tasks such as summarization, question answering and natural language inference over naturally long documents. | Tel Aviv University and collaborators | Reasoning over long documents | University | Not in cards | Jan 2022 | ||||
| SEAL Showdown Users of Scale's Playground app chat with a model and sometimes compare its reply with a random opponent's reply. Scale ranks models with a Bradley-Terry model that controls for length and formatting. Rankings can be split by region, profession and age. | Scale AI | Overall chat quality | Everyday | Not in cards | Sep 2025 | ||||
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | University of Illinois Urbana-Champaign (Lingming Zhang) + Purdue | Finding and exploiting real bugs | Expert | Not in cards | Jun 2025 | ||||
| SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. | Cribl AI Research Lab | Finding and exploiting real bugs | Expert | Not in cards | Aug 2026 | ||||
| SecQA Models answer multiple-choice computer security questions at two difficulty levels. Questions come from a security textbook and were checked by people. | SecQA author (Zefang Liu) | Finding and exploiting real bugs | University | Not in cards | Dec 2023 | ||||
| SEED-Bench-2 Multimodal models answer multiple-choice questions that cover comprehension of interleaved image and text inputs and image generation, in hierarchical capability levels. | Tencent AI Lab (ARC Lab) | General visual questions | Everyday | Not in cards | Nov 2023 | ||||
| SEED-Bench v1 Replaced in Nov 2023 | Tencent AI Lab and ARC Lab (Li et al.) | Fading | LFM2.5-VL-3B 77.7% | 3 | 2 | Jul 2023 | Aug 2026 | ||
| SemComp-Bench Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference. | FrameX AI | Video generation | Everyday | Not in cards | Aug 2026 | ||||
| SEvenLLM Models analyze cyber security incidents and write responses. Tasks split into understanding and generation over 28 subcategories, in Chinese and English. | SEvenLLM authors (Ji et al.) | Finding and exploiting real bugs | Expert | Not in cards | May 2024 | ||||
| SheetBench Agents solve finance and accounting spreadsheet tasks through a browser in a fresh Google Sheet. The score is the share of expected answer cells that match exactly. | HUD | Finance | Expert | Not in cards | |||||
| Simreal ML Benchmark ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them. | Simreal | Machine learning engineering | Expert | Not in cards | Sep 2026 | ||||
| SimRepairBench Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers. | HCLTech | CAD models | Expert | Not in cards | Oct 2026 | ||||
| SimuVerity Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements. | Xi'an Jiaotong University | Electronics | Expert | Not in cards | Oct 2026 | ||||
| SlideVQA Models answer questions about whole slide decks, with single-hop, multi-hop and numerical reasoning. The set has 14.5k questions over 2.6k decks. | NTT (Tanaka, Nishida et al.) | Documents, charts and text in images | Everyday | Not in cards | Jan 2023 | ||||
| Slop Arena Site collecting writing-preference data to tune slop detection to a user's taste, paired with the slop-guard tool. | Taste and style | Not in cards | Mar 2026 | ||||||
| Snorkel Agentic Coding 2.0 Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior. | Snorkel AI | Building whole projects | Research frontier | Not in cards | Oct 2026 | ||||
| Snorkel Agentic Coding 1.0 Replaced in Oct 2026 | Expert | Not in cards | Jan 2026 | ||||||
| SnorkelFinance 2.0 Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes. | Snorkel AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| SnorkelFinance 1.0 Replaced in Oct 2026 | University | Not in cards | 2025 | ||||||
| SnorkelGraph Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth. | Snorkel AI | Word problems | University | Not in cards | Sep 2025 | ||||
| SnorkelLegal Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority. | Snorkel AI | Law | Expert | Not in cards | Oct 2026 | ||||
| SnorkelManufacturing Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks. | Snorkel AI | Other professions | Expert | Not in cards | Oct 2026 | ||||
| SnorkelRevOps Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records. | Snorkel AI | Running a business | Expert | Not in cards | Oct 2026 | ||||
| SnorkelSequences Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math. | Snorkel AI | Word problems | School | Not in cards | Jun 2025 | ||||
| SnorkelSpatial Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning. | Snorkel AI | Language reasoning | School | Not in cards | Oct 2025 | ||||
| SnorkelUnderwrite 2.0 Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows. | Snorkel AI | Finance | Expert | Not in cards | Oct 2026 | ||||
| SnorkelUnderwrite 1.0 Replaced in Oct 2026 | Not in cards | May 2025 | |||||||
| SnorkelWordle Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate. | Snorkel AI | Puzzles and chess | Everyday | Not in cards | Jul 2025 | ||||
| SoccerNet-Caption Models write timestamped commentary for soccer broadcasts. The set has almost 37k commentaries over 715.9 hours of video. Streaming reports also score when the model chooses to speak. | University of Liege and KAUST (SoccerNet) | Video captioning | Everyday | Not in cards | Apr 2023 | ||||
| SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. | King's College London, Fudan University and The Alan Turing Institute | Deception and alignment | Everyday | Not in cards | Jun 2026 | ||||
| SOL-ExecBench Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes. | NVIDIA | GPU kernels | Expert | Not in cards | Mar 2026 | ||||
| Soohak Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems. | OneLineAI, EleutherAI and CMU with 64 mathematicians (Guijin Son et al.) | Research mathematics | Research frontier | Not in cards | May 2026 | ||||
| SOP-Bench Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points. | Amazon | Running a business | Expert | Not in cards | Feb 2026 | ||||
| SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. | Princeton University and collaborators | Refusing harmful requests | Everyday | Not in cards | Jun 2024 | ||||
| SoSBench Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse. | SoSBench authors (Jiang et al.) | Hazardous knowledge | Expert | Not in cards | May 2025 | ||||
| SpaceXAI internal R&D evals Internal xAI model-development tasks (MTS Eval) and production inference-stack patches (InferenceEval). | xAI (SpaceXAI) | Labs' internal R&D tests | Expert | Not in cards | |||||
| SpatialBench (Ropedia) Spatial foundation models are tested across many 3D tasks, viewpoints, scene domains, input densities and hardware limits, beyond the domains they were trained for. | Ropedia | Spatial reasoning and counting | Everyday | Not in cards | May 2026 | ||||
| SpatialWorld Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions. | Tsinghua University | Spatial reasoning and counting | Everyday | Not in cards | Jun 2026 | ||||
| SpeedrunBench Agents try to finish goals in 10 video games as fast as possible. Score is time to goal in frames or seconds, lower is better. | Patronus AI | Video games | Everyday | Not in cards | Sep 2026 | ||||
| Spider2-V Multimodal agents do data science and engineering workflows in real enterprise apps such as BigQuery, dbt and Airbyte, on a desktop computer. Execution-based checks score each task. | HKU XLANG Lab and collaborators | Desktop apps | Expert | Not in cards | Jul 2024 | ||||
| StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. | MIT, Intel AI and McGill University | Bias, fairness and politics | Everyday | Not in cards | Apr 2020 | ||||
| StreamEval Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing. | StreamEval authors | Video generation | Everyday | Not in cards | Oct 2026 | ||||
| StudentBench AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems. | Handshake AI Research | Other | University | Not in cards | Sep 2026 | ||||
| sudo L7 Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change. | Surge AI | Fixing real repositories | Expert | Not in cards | Oct 2026 | ||||
| Summary of a Haystack Models read a synthetic haystack of conversation or news documents where specific insights repeat, then write a query-focused summary that cites its sources. An automatic evaluation scores insight coverage and citation accuracy. | Salesforce AI Research | Reasoning over long documents | University | Not in cards | Jul 2024 | ||||
| Surge AI blind coding evaluation Surge AI raters blindly score model coding outputs on a 1-5 scale. Mistral reports five models. | Surge AI (commissioned by Mistral) | Building whole projects | University | Not in cards | Oct 2026 | ||||
| SWE Atlas Codebase QnA Extended Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores. | Mercor | Fixing real repositories | Expert | Not in cards | Aug 2026 | ||||
| SWE-bench CLI Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result. | Snorkel AI | Fixing real repositories | Research frontier | Not in cards | Oct 2026 | ||||
| SWE-Bench ProMax Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions. | ByteDance | Fixing real repositories | Expert | Not in cards | Aug 2026 | ||||
| SWE-bench Science Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix. | Fudan University OpenMOSS (Zhipeng Xu et al.) | Scientific code | Expert | Not in cards | Aug 2026 | ||||
| SWE-bench Verified Extended Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores. | Mercor | Fixing real repositories | Expert | Not in cards | Aug 2026 | ||||
| SWE-CI Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes. | SKYLENAGE AI | Fixing real repositories | Expert | Not in cards | Mar 2026 | ||||
| SWE-Clar Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build. | Upwork | Building whole projects | Expert | Not in cards | Oct 2026 | ||||
| SWE-Explore Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail. | Shanghai Jiao Tong University | Fixing real repositories | Expert | Not in cards | Jun 2026 | ||||
| SWE-Interact Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session. | Scale AI | Fixing real repositories | Expert | Not in cards | Jun 2026 | ||||
| SWM-Bench (Social World Models) Predict how Kalshi and Polymarket belief trajectories shift after news events. 12k+ points across politics, finance and crypto, chronologically split. | UIUC (ulab-uiuc) | Social simulation | Expert | Not in cards | Jun 2026 | ||||
| SynthDocBench Vision-language models answer questions about synthetic long documents where length, layout, modality and difficulty are varied one at a time to locate failure causes. | ServiceNow | Documents, charts and text in images | Everyday | Not in cards | Jul 2026 | ||||
| TASTE 92 pairwise comparisons of AI safety research proposals. Model must match experienced researchers' preference. Accuracy versus ~77% human agreement. | Anthropic Fellows Program + Anthropic | AI research tasks | Expert | Not in cards | Aug 2026 | ||||
| TCS-Bench Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof. | Google Research + collaborators (Cohen-Addad, Woodruff and others) | Research mathematics | Research frontier | Not in cards | Aug 2026 | ||||
| Terminal-Bench 2.0 Verified (Z.ai) Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks. | Z.ai (Zhipu AI) | Terminal tasks | Expert | Not in cards | Feb 2026 | ||||
| Terminal-Bench 2.1 Extended Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores. | Mercor | Terminal tasks | Expert | Not in cards | Aug 2026 | ||||
| Terminal-Bench Challenges Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric. | Terminal-Bench team (Stanford, Harbor and Laude Institute) | Long terminal sessions | Research frontier | Not in cards | Jun 2026 | ||||
| Terminal-X A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories. | UniPat AI | Terminal tasks | Expert | Not in cards | May 2026 | ||||
| TestPrism Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one. | Nanjing University (NJU-LINK Lab) | Function-level problems | Expert | Not in cards | Oct 2026 | ||||
| TextArena Collection of competitive text games for LLM eval and RL, with an online Elo leaderboard. | TextArena | Multi-agent games | Not in cards | Apr 2025 | |||||
| TextEdit Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity. | OpenCompass (Shanghai AI Laboratory) | Documents, charts and text in images | Everyday | Not in cards | Mar 2026 | ||||
| TheoPhysEval Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate. | ICTS Bengaluru (Suvrat Raju) with 35 researchers incl. Google DeepMind, Harvard and KU Leuven | Research science | Research frontier | Not in cards | Oct 2026 | ||||
| Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. | Toloka | Customer service | Expert | Not in cards | Apr 2026 | ||||
| ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. | OpenBMB and Tsinghua University | Function calling | Everyday | Not in cards | Jul 2023 | ||||
| TraceSpatial-Bench Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth. | Beihang University + Peking University + BAAI | Embodied reasoning | Everyday | Not in cards | Dec 2025 | ||||
| TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. | TraceLite AI | General assistants | Expert | Not in cards | Oct 2026 | ||||
| TRAIL Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks. | Patronus AI | General assistants | Expert | Not in cards | May 2025 | ||||
| TritonBench Models write Triton GPU operators in two channels, one from real GitHub kernels (G) and one matching PyTorch operators (T). Execution checks correctness and speedup. | Tsinghua University (THUNLP) | GPU kernels | Expert | Not in cards | Feb 2025 | ||||
| TTS Arena Users listen to two anonymous text-to-speech models read the same text and vote for the more natural one. Votes become Elo ratings. | TTS-AGI (Hugging Face community) | Audio understanding and voice | Everyday | Not in cards | Feb 2024 | ||||
| TUA-Bench Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work. | Meta | Terminal tasks | Everyday | Not in cards | Jun 2026 | ||||
| Tuesday Work Index A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday. | Surge AI | Composite work indices | Expert | Not in cards | Aug 2026 | ||||
| TutorBench Models act as tutors on three tasks: adaptive explanations for a confused student, feedback on student work, and hints. Graded against sample-specific rubrics. | Scale AI | Other | University | Not in cards | Oct 2025 | ||||
| UI-Vision Computer use agents are tested offline on human desktop demonstrations from 83 applications. Three tasks go from element grounding to layout grounding to next-action prediction for clicks, drags and key presses. | ServiceNow Research and collaborators | Finding things on screen | Everyday | Not in cards | Mar 2025 | ||||
| UltraText Bench Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility. | Westlake University | General visual questions | Everyday | Not in cards | Oct 2026 | ||||
| UMI-Bench 10 real tabletop tasks trained from standardized UMI hand-held gripper demos. 50 rollouts per task. Progress score and success rate. | Soochow University + Lumos Robotics + others | Real robots | Everyday | Not in cards | Jun 2026 | ||||
| US-DermMCQA (Google) Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions. | Medicine | Expert | Not in cards | Jul 2025 | |||||
| USACO Models solve 307 USA Computing Olympiad problems. Hidden unit tests check correctness and efficiency, scored as pass@1. | Princeton University | Competitive programming | Competition | Not in cards | Apr 2024 | ||||
| UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. | Salesforce AI Research | General assistants | Everyday | Not in cards | Jul 2025 | ||||
| Vals Router Index Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights. | Vals AI | Composite indices | Expert | Not in cards | Oct 2026 | ||||
| VBench-2.0 Video generation models are scored on intrinsic faithfulness, such as human fidelity, controllability, creativity, physics and commonsense, beyond pixel quality and basic prompt match. | NTU S-Lab and Shanghai AI Laboratory (Vchitect) | Video generation | Everyday | Not in cards | Mar 2025 | ||||
| VBench 1.0 Replaced in Mar 2025 | Fading | ERNIE 5.0 84.2% | 1 | 1 | Nov 2023 | Feb 2026 | |||
| vCX Benchmark Models run inside a Retell voice agent on pass or fail checks for business rules, grounding, tool calls and speech-ready replies. Each suite reports its own pass rate. | Retell AI | Speech recognition and translation | Everyday | Not in cards | |||||
| VDEBench Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved. | Yi et al. (VDE-Bench authors) | Documents, charts and text in images | Everyday | Not in cards | Jan 2026 | ||||
| Verifiable Browsing Task (Meta) Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools. | Meta (Meta Superintelligence Labs) | Web browsing | Everyday | Not in cards | Oct 2026 | ||||
| VIALS Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key. | Handshake AI Research | Genomics and life sciences | Expert | Not in cards | Aug 2026 | ||||
| Video Reality Test Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection. | Video Reality Test authors | Video understanding | Everyday | Not in cards | Dec 2025 | ||||
| Video-Oasis A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions. | NAVER and collaborators | Video understanding | Everyday | Not in cards | Mar 2026 | ||||
| Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. | BAAI FlagEval team | Jailbreaks and prompt injection | Everyday | Not in cards | May 2025 | ||||
| VideoFDB Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes. | NVIDIA | Speech recognition and translation | Everyday | Not in cards | May 2026 | ||||
| VisualAgentBench Large multimodal models act as agents in embodied, GUI and visual design environments, including mobile, WebArena-based web and Minecraft tasks. | Tsinghua University and Zhipu AI (THUDM) | Web browsing | Everyday | Not in cards | Aug 2024 | ||||
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | Scale AI | Function calling | University | Not in cards | Oct 2025 | ||||
| VisualWebArena Multimodal web agents complete 910 visually grounded tasks on self-hosted classifieds, shopping and Reddit-style sites. Execution-based checks score each task. | Carnegie Mellon University | Web browsing | University | Not in cards | Jan 2024 | ||||
| VizWiz Models answer over 31,000 spoken questions that blind people asked about photos they took with phones. Some questions are unanswerable from the image. | University of Texas at Austin and collaborators (Gurari et al.) | General visual questions | Everyday | Not in cards | Feb 2018 | ||||
| VL-RewardBench Vision-language reward models pick the better of two responses on general queries, hallucination detection and reasoning tasks. | HKU and collaborators | General visual questions | Expert | Not in cards | Nov 2024 | ||||
| VLABench 100 language-conditioned manipulation task categories needing world knowledge, implicit intent and long-horizon reasoning. Evaluates VLAs and VLMs. | Fudan University (OpenMOSS) | Simulated robots | Everyday | Not in cards | Dec 2024 | ||||
| VoxMem Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible. | University of Melbourne | Audio understanding and voice | Everyday | Not in cards | Sep 2026 | ||||
| VQA-RAD Vision-language models answer clinicians' questions about 315 radiology images, including head CTs, MRIs and chest X-rays. Questions are closed yes or no, or open free text. | NIH and collaborators (Lau et al.) | Medicine | Expert | Not in cards | Nov 2018 | ||||
| VSI-Bench-Debiased Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge. | New York University (VisionX) | Spatial reasoning and counting | Everyday | Not in cards | Nov 2025 | ||||
| VSI-SUPER Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours. | New York University (Cambrian-S team, Yang et al.) | Long videos | Everyday | Not in cards | Nov 2025 | ||||
| VSR Models judge whether captions with 65 kinds of spatial relation, such as under or facing, are true of an image. Over 10,000 text and image pairs. | University of Cambridge (Liu et al.) | Spatial reasoning and counting | Everyday | Not in cards | Apr 2022 | ||||
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Tencent | Finding and exploiting real bugs | Expert | Not in cards | Apr 2026 | ||||
| WARC-Bench Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state. | Uniphore | Web browsing | Everyday | Not in cards | Oct 2025 | ||||
| WeaveBench Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts. | Microsoft and collaborators | Desktop apps | Expert | Not in cards | Jun 2026 | ||||
| Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. | Nimble | Deep research | Expert | Not in cards | Oct 2026 | ||||
| Web UI Bench Many models build the same 12 app UIs from one shared prompt. Outputs are shown side by side for people to compare. No automatic score. | Eric Zakariasson (Cursor) | Taste and style | Not in cards | May 2026 | |||||
| WebCanvas Web agents do tasks on live websites, scored on key intermediate states rather than one fixed action path, so the test survives site changes. | iMean AI | Web browsing | Everyday | Not in cards | Jun 2024 | ||||
| WebLINX Agents follow a user's multi-turn chat instructions to navigate real websites. 100,000 interactions from 2,300 expert demonstrations give turn-level metrics. | McGill University and Mila | Web browsing | Everyday | Not in cards | Feb 2024 | ||||
| WebShop Agents search and click through a simulated shopping site with 1.18 million real products to buy an item that matches a text instruction. Reward reflects how well the purchase matches. | Princeton University | Web browsing | Everyday | Not in cards | Jul 2022 | ||||
| WebSRC Models answer questions about a web page from its screenshot, HTML or both. The set has 400K question and answer pairs from 6.4K web pages. Scored by exact match and F1. | Shanghai Jiao Tong University (X-LANCE lab) | Documents, charts and text in images | Everyday | Not in cards | Jan 2021 | ||||
| WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. | Alibaba Tongyi Lab | Deep research | Everyday | Not in cards | Jan 2025 | ||||
| Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. | USTC and Metastone Technology (DeepResearch Bench team) | Deep research | Expert | Not in cards | Feb 2026 | ||||
| WildArtifactBench Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo. | Meta (Meta Superintelligence Labs) | Tasks from real jobs | Expert | Not in cards | Aug 2026 | ||||
| WildGraphBench GraphRAG systems answer 1,100 questions over long, mixed reference documents behind Wikipedia articles in 12 topics, from single facts to section summaries. | USTC and Metastone Technology (DeepResearch Bench team) | Reasoning over long documents | University | Not in cards | Feb 2026 | ||||
| WildVision Arena Users upload an image with a question, compare answers from two anonymous vision-language models, and vote. Votes give Elo ratings. | Allen Institute for AI and UCSB | General visual questions | Everyday | Not in cards | Feb 2024 | ||||
| WorkArena Web agents do routine knowledge-work tasks in a live ServiceNow instance, such as filling forms and filtering lists. WorkArena++ adds 682 compositional tasks at levels 2 and 3. | ServiceNow Research | Web browsing | University | Not in cards | Mar 2024 | ||||
| WorkArena++ Web agents complete 682 compositional knowledge-work tasks in ServiceNow, which need planning, reasoning and memory across many steps. | ServiceNow Research | Web browsing | Expert | Not in cards | Jul 2024 | ||||
| WorkplaceAgents Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics. | Snorkel AI | Tasks from real jobs | Expert | Not in cards | Oct 2026 | ||||
| World-in-World Closed-loop platform: world models plan for embodied agents in four environments via a standard action API. Scored by task success. | Interactive world models | Everyday | Not in cards | Oct 2025 | |||||
| WorldArena 2.0 Extends WorldArena to visuotactile prediction, world models as RL environments, and simulated plus real multi-embodiment robot platforms. | WorldArena team (Tsinghua FIB Lab) | Interactive world models | Not in cards | May 2026 | |||||
| WorldArena 1.0 Replaced in May 2026 | Tsinghua FIB Lab | Video world models | Not in cards | Feb 2026 | |||||
| WorldAuditBench Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects. | UC Santa Barbara (UCSB NLP) | Embodied reasoning | Everyday | Not in cards | Sep 2026 | ||||
| WorldExam Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states. | WorldExam authors | Video world models | Everyday | Not in cards | Aug 2026 | ||||
| WorldGUI Desktop GUI agents do 611 tasks that start from varied initial states, such as the target app not being open. Tasks come with instructional videos and project files. | Show Lab, National University of Singapore | Desktop apps | Everyday | Not in cards | Feb 2025 | ||||
| WorldModelBench Video generators judged as world models on instruction following and physics adherence in application domains. Human labels and a fine-tuned judge. | Video world models | Not in cards | Feb 2025 | ||||||
| WorldModelGym World models predict returns for five candidate action sequences across 100+ Atari, Meta-World, DMC and classic-control tracks. Scored by normalized regret. | Reka Labs | Interactive world models | Not in cards | Jul 2026 | |||||
| WorldScore 3,000 next-scene generation tasks with camera-trajectory layouts. Scores 3D/4D and video world generators on controllability, quality, dynamics. | Stanford | Video world models | Not in cards | Apr 2025 | |||||
| WorldSimBench Dual evaluation of video world simulators: human-preference perceptual scoring plus action-level embodied tasks in games, driving, manipulation. | Video world models | Not in cards | Oct 2024 | ||||||
| xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. | xAI (SpaceXAI) | Refusing harmful requests | Not in cards | ||||||
| Xbench Measures firsthand X posts about frontier models and coding harnesses. An LLM labels each post's sentiment, aggregated per model per week. | Nico Christie (offpolicyinc) | Other | Not in cards | Sep 2026 | |||||
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | XBOW | Capture-the-flag challenges | Expert | Not in cards | Aug 2024 | ||||
| XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. | Meta AI (XGLM team) | Reasoning in many languages | Everyday | Not in cards | Dec 2021 | ||||
| XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. | Google DeepMind | Deception and alignment | Expert | Not in cards | Sep 2026 | ||||
| YoCausal Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels. | Alaya Lab | Physical prediction | Everyday | Not in cards | May 2026 | ||||
| Yupp VIBE Score People use Yupp for real tasks, see answers from two or more models, and pick the one they prefer. The preferences become VIBE scores for each model. | Yupp (Ber Sarai Labs) | Overall chat quality | Everyday | Not in cards | Jun 2025 | ||||
| Z.ai Code Bench Zhipu in-house coding benchmark used for GLM-5.3. Reports pass rate and output tokens per task. Method unpublished. | Zhipu AI (Z.ai) | Fixing real repositories | Expert | Not in cards | Aug 2026 | ||||
| ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. | Z.ai (Zhipu AI) | General assistants | Expert | Not in cards | Mar 2026 | ||||
| τ-bench 2.0 2.0.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. | Sierra | Customer service | University | Not in cards | Oct 2026 | ||||
| tau3-bench Replaced in Oct 2026 | Everyday | In use | Mercury 2.5 96% | 12 | 1 | Mar 2026 | Oct 2026 | ||
| tau2-bench Replaced in Mar 2026 | Sierra Research | Everyday | In use | GPT-5.4 98.9% | 72 | 40 | Jun 2025 | Oct 2026 | |
| tau-bench v1 Replaced in Jun 2025 | Everyday | Dropped | Claude Opus 4 70.5% | 23 | 10 | Jun 2024 | Sep 2025 | ||
| τ^τ-bench Coding agent builds a customer-service agent from business records, a client and an API. The built agent is scored against held-out simulated users. | Sierra + Princeton | Building whole projects | Expert | Not in cards | Sep 2026 |