New benchmarks

Agents reconstruct dynamic scenes from video as executable graphics programs, including physical simulations, so rendered output matches the observed motion.

Agents in an offline sandbox turn a written brief into an interactive three.js scene. A recruited human panel votes on blind pairs, and votes become Elo ratings.

Models answer 3,000 knowledge questions in six domains without tools and may abstain. The index adds correct answers and subtracts wrong ones.

Language models in coding harnesses control a 6-axis robot arm with a gripper over HTTP, using camera views, to finish eight everyday tabletop tasks.

Language model agents explore environments with hidden rules, infer the rules from evidence and adapt when the rules change without notice.

Coding agents try to improve the published best model on 20 scientific ML benchmarks across 11 domains, under long in-the-wild runs. The paper reports mean relative improvement rather than a ranking.

Models follow the same AgentIF instructions in thinking and non-thinking modes with vendor decoding settings. A gpt-5.6-luna judge scores 3 runs per model.

Agents complete 122 public long-horizon professional workflow tasks across 55 sub-industries. Deterministic graders check outcomes against hidden references.

Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion.

Decision models answer structured questions with a probability for each option. Four evaluations weighted equally cover classification, tool calling, knowledge with hallucination, and reasoning.

Artificial Analysis's index that tracks how model capability changes over time on one continuous scale, launched alongside Intelligence Index v5.

Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct.

Agents do the same 100 ATLAS-Finance banking tasks, graded with revised rubrics. Weights use a 1 to 10 scale and tolerances and gates sit at the section level.

Agents complete 100 junior investment banker tasks and produce Excel, PowerPoint and Word deliverables. A task passes only if every critical rubric criterion passes.

Long-horizon agents run business operations such as payments under long, partly outdated or conflicting policies. Intermediate steps are checked, not only final success.

Visual agents turn a text, image or video brief into an editable Blender scene with rendered media. Hidden binary rubrics and eight visual quality dimensions score each project.

NVIDIA evaluation of short, direct answers for voice and device assistants. The card gives no link or method.

Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1.

Agents take a chip from workload specification to a routed GDS layout with open-source tools. Each design must pass hidden verification, routability and timing gates.

Agents manage a simulated business over a long horizon and make operating decisions. Moonshot reports a percentage score.

Long-horizon agents read a defect specification and a repository, then write and refine a working static-analysis checker using compiler and analyzer feedback.

Agents use tools to rebuild reference videos from film, real footage and games as editable dynamic 3D scenes. The set has 50 videos and 360 evaluation clips.

Models play full games of Civilization V with the Vox Populi mod against each other on matched maps and seeds. Pairwise skill ratings rank long-term strategy.

Models label whether a 10-K excerpt supports, contradicts or says nothing about a claim, across 1,960 claims. Small decision models are scored on accuracy, cost and latency.

Computer-use agents work in multi-agent setups where peers report stale states, give contradictory instructions or compete for files. Agents must check claims against the environment.

Agents run a simulated clip farm for about a month. They join pay-per-view campaigns, cut clips from licensed streams, post them and compete with rival clippers for budgets.

Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.

Multimodal models write programs that render complete videos from text, an image or a video.

Models watch a structured 2D video and write code that re-renders it. The rendered video is compared with the original on visual similarity, object timing and placement, and retained detail.

Agents work through 120 professional tasks, each a ladder of five levels that must be passed in order. Later levels reuse files, tools and lessons from earlier ones. An evaluation agent grades against expert rubrics.

Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest.

Models solve unpublished research-level physics challenges written by working physicists, with answers graded automatically. The v2 release revised the public test challenges.

Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning.

Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass.

Agents attack two simulated company networks, Nova and Lighthouse, on their own with no human help. The score counts the attack objectives completed on each range.

Frontier models watch through cameras and drive two real robot arms on tabletop tasks rebuilt from data-center equipment manuals. LLM judges score each try from photos.

Agents do accounting assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents do data science assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents do legal assignments from a short request and a workspace of files. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents take tax assignments from client request to finished return, workpaper or advice, using messy records. A pass needs 95% of the expert rubric criteria.

LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance.

Models build presentations for real user queries with python-pptx, pptxjs or a Kimi slide language, with search and image generation. An automated grader rates them as Elo scores.

Models make graphic designs from direct prompts or few-shot reference images. An automated grader rates the results as Elo scores.

Models are scored on eight benchmarks in coding, reasoning, writing and enterprise work. Weights come from the wages of the 29 digital occupations the benchmarks map to.

Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics.

Models build a coding-agent harness from an interface skeleton and a development document. The frozen harness then runs 328 scripted test cases in separate containers.

Multimodal agents are tested under fixed environment responses, and world models under fixed actions. The agent track has 1,088 items and the world model track 400 cases.

Frontier models control a two-armed robot in a simulated operating room from cameras and sensors. Tasks include swapping instruments, replacing a drug cartridge and packing a surgical kit.

Coding agents answer frozen robot tasks from many benchmark suites with a program. Each program is verified in a separate sandbox, run on Harbor.

Self-evolving embodied agents learn tasks in sequence in simulation. The test measures whether they keep past experience and transfer it to new tasks.

Agents in coding harnesses do real Epoch AI work tasks such as graphic design, data insights and research design. Epoch staff grade outputs against the rubric they use for employees.

Agents build the same 103 Excel financial modeling tasks as EMB v1.0, with the Scratch rubric judge changed to GPT-6 Luna.

Coding agents build executable code-based 3D worlds that capture how a world works, testing world understanding and its realization in code.

Systems extract every schema value from long-tail synthetic documents with charts, drawn relations, house rules and jargon. Scored by value F1 and by grounded F1, which also needs the source location.

Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer.

Agents act as forward-deployed engineers in live customer systems, talking to scripted stakeholders to find unstated constraints. Score is the uplift on the customer's own success metric, from 0 to the reference solution's 1.

The editors run five AI systems once each on ten unpublished research problems with proofs of up to 12 pages. About 30 referees score correctness, exposition and attribution, with negative marks for wrong answers.

Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes.

Agents build 39 machined parts from engineering drawings and place parts into 16 assemblies, in their own coding harnesses. A verifier that tolerates rotation and checks hole positions scores the geometry.

Agents work inside the released repository of a recent systems or programming-languages paper and try to beat that paper's method. A hidden held-out suite grades the final patch against the human reference.

Models act as text world models. Given an agent's past actions and observations, they predict the environment's next observation across code, enterprise, games, math and science, and dialogue.

Vision-language agents play 100 real-time browser minigames by screenshots and mouse and keyboard input. Each game also has a paused, non-real-time counterpart, so the gap shows the cost of acting in real time.

Agent memory systems must infer user habits from weak evidence spread across many past conversations and apply them correctly later.

Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices.

Agents produce legal work product from matter files with file tools and docx, pptx and xlsx skills. A task passes only if every expert rubric criterion passes.

Agents in Hebbia's production harness answer questions over deal and research documents, build Excel models and make client decks. A run passes only if it meets every rubric criterion.

Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind.

Nous Research averages pass@1 and cost per task over Hermes Bench, Terminal-Bench 4.0, Terminal-Bench Science and SkillsBench, all run in Hermes Agent.

Models predict how specified groups of U.S. adults answer closed survey questions, as a distribution over options. Scored as 1 minus total variation distance.

Models answer open-ended questions about what an image or short video implies but does not show, such as hidden objects, causes and intentions. An LLM judge checks each answer against a human-written rubric.

Humanoid robot policies pick a suitable tool and use it, walking when needed, across 18 tasks in simulation and on a real Unitree G1.

Models answer expert-written US immigration law questions from seven personas, such as F-1 students, employers and DACA recipients. An AI grader checks each answer against an expert rubric.

Agents with thousands of GPU hours try to discover an ML technique that matches a recent human paper they have not seen. The score is the share of that paper's gain the agent reaches.

Multimodal models act as a wearable assistant that sees through a simulated person's eyes and guides that person through household tasks. State-based checks in a BEHAVIOR-1K home score the outcome.

Agents get a science paper and an unpublished follow-up direction from a DOE national lab scientist, then write the follow-up paper. Rubrics built from the scientist's conditions grade it, plus an integrity check.

Financial agents make equity investment judgments from a frozen set of decision-time evidence. Predictions are scored against later outcomes and professional criteria.

Agents solve 18 International Olympiad in Informatics problems from 2024, 2025 and 2026 in C++ inside OpenCode. Official tests and graders score the final solution by subtask after the run.

Models review contracts and return redlines across 350 attorney-written tasks in 35 scenarios and five agreement types. Tasks cover first review and replies to a counterparty.

Coding agents carry out long multi-step engineering tasks with tools across several engineering domains. Moonshot reports a percentage score.

Models answer hard prompts drawn from practical work tasks. Fact-checking agents score the share of claims that are hallucinated.

Agents draft a litigation brief for the same party and motion as a real filed federal brief, from the case record. Model judges compare briefs pairwise and wins become Elo ratings.

Policies, orchestrators and frontier-model agents do 145 long-horizon tabletop tasks of 1 to 10 subgoals in three simulators. A task counts only when every object ends in place.

Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit.

Agents play unfamiliar games whose rules are not given and must learn from experience across episodes. Improvement over repeated play is the measure.

Models work long-horizon legal matters built by practicing lawyers, writing memoranda and court filings across dependent assignments, in closed-record and open-research settings.

Agents trade in live financial markets, and the benchmark scores the process capabilities behind their decisions, not only returns.

Language model agents design 3D structures as parts, joints, materials and assembly steps. Scoring checks that the structures can be built and work as intended.

Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards.

Models do military mission analysis under the joint planning process on long-context scenarios, graded against expert rubrics.

Agents build spreadsheet financial models for 101 tasks written by finance experts. A task passes only if every number matches the reference and the workbook meets a quality bar.

Models recall key facts from multi-turn conversations of 64K to 100K tokens that include distractor chats and turns that conflict with or update the facts. An LLM judge scores answers on weighted rubrics.

Models are scored on the equal-weight average of micro1 medical benchmarks, currently RealmPathology, PharmaBench and WellnessBench.

Models support simulated patients from MindSim, a tool-using harness with detailed profiles. Psychologist-written rubrics (MHQR) replace Likert scales for scoring.

Models write government status reports in Word from working files and a template, across 20 expert-written Department of Defense style assignments. Expert rubrics score each decision.

Enterprise agents answer 1,613 questions over workplace documents, chats and figures that change over time. They must resolve conflicting versions and say when evidence is not enough.

Models answer chat prompts from an internal NVIDIA set. The Nemotron 3.5 Super VL card labels the score an LMArena v3 proxy.

Omnimodal models watch continuous video and audio streams and must respond at the right moment with answers, alerts, warnings, tool calls or step guidance. Scoring joins content and timing.

Document parsers convert single pages to text, and 16,288 binary unit tests on 2,937 pages check the output. The headline joins text, table and layout subscores.

Separate agents run analyses of cancer research data, and a final agent makes a decision on detection, diagnosis or treatment. Strict scoring needs full marks on both parts.

Agents try to rediscover proofs of research math problems, most resolved during 2026, stated in Lean 4. Two independent checkers grade the formal proofs, with closed-book, frozen-retrieval and live-web settings.

Agents turn original client materials into validated operations research solutions without an expert first reformulating the problem.

Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house.

Agents reproduce published results from 12 research areas outside machine learning, using each field's own scientific software. Verifiers check that the evidence regenerates and supports the paper's claims.

Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals.

Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.

Policies run 286 language-instructed tabletop tasks worded as DROID annotations on a real Franka FR3. Operators mark stages reached within 240 seconds.

Agents write clinical trial documentation inside 8 synthetic drug programs, across 120 tasks written by biostatisticians, medical writers, safety physicians and regulatory staff.

Agents run 50 expert-verified physics research tasks in strong-field and attosecond physics, deciding how to generate and validate evidence. A 20-criterion score and a closure level from RC0 to RC4 grade them.

Agents produce billable engineering deliverables such as calculation packages, drawings and design reviews in 85 tasks across electrical, civil, structural and mechanical engineering. Automatic graders check constraints, safety limits and the design objective.

Programmable world models must render video that follows the rules and interactions their program specifies. The test checks visual fidelity to fine-grained rules.

Models act as quant researchers on 2022 to 2023 China A-share data with bounded feedback, then commit one strategy. A cash-account evaluator with trading costs scores it once on a sealed 2024 window.

Models read months of real chats between ten people and an AI companion, then answer questions about who each person is. The release has 27,218 messages over up to 120 days.

Models interpret real anatomic pathology reports and must not claim a diagnosis, stage or treatment the report does not support. Rubrics have 772 positive and 299 penalty criteria.

AI code review systems review 219 real pull requests from 187 repositories in 19 languages. A judge scores found issues against a multi-source golden set for precision, recall and F1.

Image editing models follow 1,000 bilingual instructions over six reasoning types, adding counterfactual and hybrid reasoning. It covers single-image, multi-image and multi-turn editing across 65 task types.

General agents control a simulated 7-DoF Franka arm and a 29-DoF humanoid through motor commands, sensors and camera images only, across 24 tasks.

An index fitted like Epoch ECI over five RoboDojo-RC Tier 1 tasks run by frontier LLMs on real YAM arms. Anchored at GPT-5.6 Sol minus 3 and GPT-6 Astra plus 3.

Multimodal agents control robots through robot interfaces in simulation. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving and aerial control, with executable success checks.

Coding agents control simulated robots through an HTTP API to do physical jobs such as welding, sewing, pouring and forklift driving. Each simulation checks the task and returns a reward from 0 to 1.

Vision-language models rotate a 3D object step by step, using visual feedback, until its projection matches a target silhouette.

Agents must find and correctly apply domain-specific rules when the scenario's facts are spread across many files, unlike rule benchmarks built from a few statements.

Agents do mechanical, electrical and aerospace design tasks, such as a crankshaft, dense PCB routing or turbine-vane cooling, with CAD and CAE tools. A deterministic verifier runs physics solvers and passes a design only if it meets every requirement.

Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate.

Models grade handwritten undergraduate math and computer science work against rubrics. Scored by class-balanced accuracy of each rubric check against expert grading.

Agents inspect 3D scientific data such as proteins, crystals, CAD assemblies, buildings and CT volumes, then answer 250 expert questions. A rule-based scorer grades each answer field.

Agents act as growth lead of a small brand for a simulated week. They post short videos, run the store and answer customers, sponsors and the founder.

Agents find and repair modeling errors in 3D OpenUSD assets for robot simulation, from 25 damaged objects in NVIDIA's SimReady collection, with partial credit from verifiers.

Agents build executable Simulink models from text for 101 tasks in ten engineering domains. Simulations check that each model meets its engineering requirements.

Agents plan, run, check and recover on 200 terminal-native engineering tasks across 9 task types and 9 languages. Harbor checks task tests, required outputs, milestones and trace behavior.

Agents do 200 investment banking and private equity workflow tasks in a simulated firm with incomplete requests, scattered evidence, compliance rules and required state changes.

Agents work 200 tasks inside a simulated mid-market law firm. Each request ties to a matter, a procedural stage and a role with set authority.

Agents make engineering, production, quality, supplier and safety decisions in a simulated discrete-parts plant across 200 tasks, a third of them CAD tasks.

Agents work 200 tasks in a simulated B2B SaaS revenue team spanning CRM, CPQ, billing, compensation, marketing and finance records.

Agents do 200 commercial property and casualty underwriting tasks in a simulated insurer with partial information, role permissions and stateful workflows.

Each model gets one hour in its lab's own CLI to write a StarCraft: Brood War bot in C++. The score is its win rate against human-written bots.

Streaming video generation models handle 300 scripted cases of 15 to 300 seconds. Requests arrive over time across avatar dialogue, performance, world generation and editing.

Coding agents get staff-level engineering requests in mostly private production codebases. Graders score whether the agent found hidden risks and shipped the right change.

Models answer materials chemistry questions in a QA part and solve materials tasks with tools in an Agent part.

Agents make verified multi-file changes in unfamiliar open-source repositories across 200 tasks, 11 languages and 10 technical domains. Harbor runs fail-to-pass and pass-to-pass test suites on the result.

Agents question a simulated client about an incomplete request, then build greenfield software from real Upwork jobs. Acceptance tests score the build.

Agents get a real repository with no hint of bug type or location and must find and repair as many bugs as they can. Score is the share of bugs resolved.

Agents browse the web to retrieve information and do research tasks. iFlytek built it in house.

Coding agents write tests for 300 tasks. A test passes only if it fails on the starting code, accepts every valid implementation and rejects every invalid one.

Agents solve 50 unpublished research-level theoretical physics problems on their own, with literature search and scripts. Each final answer is checked against an expert answer, and the score is the pass rate.

Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause.

Image generators render dense bilingual text in many regions from 432 prompts. Scores check string accuracy, placement and legibility.

Agents answer analyst-grade questions on public company filings, scored with dealbreaker-gated partial credit. A three-model jury grades answers and fails hedged ones.

Agents are scored on Finance Agent v2.1, Excel Modeling Benchmark v1.1 and Tax Agent Bench, combined with weights from each sub-sector share of US GDP.

Agents are scored on Legal Research Bench, HLAB-Vals-1.1 and Tax Agent Bench, mapped to practice areas weighted by share of US law-office revenue.

Model routers run all eight Vals Index v2.2 benchmarks through the same agents as single models and are scored with the Index weights.

Agents are scored on the equal average of BioMysteryBench, MysteryMechanism and Terminal-Bench-Science.

Agents answer list-style information-seeking questions, written as realistic user requests, using browser search, open and find tools.

Agents write code to render a finished video with sound, with no image or video generation model, in their native harness within one hour. Program checks, task rubrics and a judge panel score the result.

Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages.

Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful.

Agents produce finished work products for 200 professional tasks across 96 occupations in 19 sectors. Harbor grades deliverables with code checks and rubrics.

Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string.

Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture.

Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT.

Models answer 100 questions that need reasoning across sets of real documents of about 100K tokens. An LLM equality checker with a fixed system prompt compares each answer with the key.

Systems label private pre-2022 human documents and AI rewrites of them as human or AI written. Both LLMs and commercial detectors are scored by balanced accuracy.

Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score.

Coding agents complete 30 long Android tasks such as building an app from mocks, library migrations and Flutter to native conversion. Tests and an LLM visual judge grade each run.

Agents do long banking, consulting and law tasks in worlds with docs, sheets, email and chat. A new judge scores hedged multiple answers as zero.

Streaming video assistants must remember experiences across separate sessions of a person's day. The set has 549 sessions in 104 life trajectories with objective and open questions.

Version 1.5 of Artificial Analysis's coding agent index, which moved to the harder Terminal-Bench 4.0 and DeepSWE 1.1.

Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate.

Version 4.2 of Artificial Analysis's index, which added multi-hour office work and document tasks and dropped GPQA Diamond.

Version 4.3 of Artificial Analysis's index, eleven weighted parts led by knowledge work, office tasks and Terminal-Bench 4.0.

Models answer research-level computer science questions derived from recent arXiv papers, each with a reference final answer. The first set has 46 questions from August 2026.

Models answer research-level physics questions derived from recent arXiv papers, each with a reference final answer.

Agents do 100 investment banking tasks inside 13 simulated firms. They gather facts from email, chat, calendar and files over MCP, then build an Excel model graded against expert rubrics.

Agents get 24 hours on a CPU or GPU machine to improve solutions to open-ended ML research tasks through experiments. The score is the area under the improvement curve on a hidden test set, rewarding both quality and speed.

Coding agents play StarCraft: Brood War against each other in real time. Systems are ranked by win rate over matches, with cost per game.

Browser agents do 200 real web tasks. A findings judge checks tool evidence, answers and files, and deterministic integrity checks run as well.

Multimodal agents rebuild real-world videos as animated Blender scenes by writing code. The renders are compared with the source video under a shared cost limit.

Agents rebuild licensed engineering drawings as editable CAD parts through the scripting APIs of NX, SolidWorks, Onshape, Fusion or Build123d. A verifier scores geometric match.

Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not.

Computer-use agents play six commercial video games, such as Minecraft, SUPERHOT and eFootball, using only a keyboard and mouse. Half of the games are held out.

Agents do data science and analysis tasks from an OpenAI internal set. OpenAI has not published the task count or grading.

Agents carry out database migration tasks from an OpenAI internal set. OpenAI has not published the task count or grading.

Agents get short requests from finance colleagues plus a workspace of filings, ledgers and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Agents handle short requests from healthcare staff using charts, claims exports, policies and emails. An agent judge applies an expert rubric, and a pass needs 95% of criteria.

Decision models read a state and a typed question and return a decision. The index weights private same-skill sets and new-domain sets, equated on 21 base models.

Models produce design work such as slides and documents that follow templates, from an OpenAI internal set. OpenAI has not published the grading.

Full-duplex speech models join conversations of three or four people and must decide when to answer, stay silent or stop speaking. The set has 2,000 scenarios.

Agents work through the full industrial engineering design loop on 1,301 expert-curated tasks that need reasoning over geometric and physical constraints across software.

Agents run long experiments and propose mechanisms that explain observations in six sciences. Mechanisms are scored on known constraints, predictive accuracy and the 306 expert-listed insights they should support.

Agents explore verifiable alien worlds with hidden laws, framing hypotheses and running experiments. The worlds are built so that recalled pretraining knowledge does not help.

Models do three kinds of finance work: retrieving and checking live financial facts, building company valuations, and writing full research reports. Graders check accuracy, sourcing and reproducible calculations.

An index that gathers 66 practitioner-built benchmarks in 14 domains, such as legal, finance, healthcare and software. It combines score, cost and duration per benchmark into one number.

Language models play Go against each other through the Game Arena text harness. Ratings come from head-to-head games.

Models are tested on gameplay over several time horizons. Tasks combine visual understanding, instruction decomposition, planning and precise action control, using offline data instead of noisy rollouts.

Models sit the five 2026 Chinese Gaokao mathematics papers (National I, National II, Beijing, Shanghai and Tianjin). Each paper is graded out of 150 points.

Agents open a professional Excel workbook in a sandbox and answer a short question about it, as a domain professional would. A Gemini autograder checks the final answer against criteria.

World models are tested on the quality of the worlds they generate and on staying consistent and responsive while agents explore, interact and modify them.

Agents build a complete agent harness from a minimal seed, then revise their own harness using downstream execution feedback. The harness's task results are the score.

Models answer 1,000 cleaned expert questions from Humanity's Last Exam, 500 on reasoning and 500 on knowledge, closed book. Graded on exact answers.

Decision models, such as TypeSafe AI's Jev and compatible systems, answer typed decision questions. The board scores accuracy, probability calibration, latency and cost.

Strong models translate hard texts chosen to stay unsaturated, and experts assess the outputs instead of automatic metrics that can be reward-hacked.

Agents debug, add features to and migrate enterprise code in COBOL, Java 7, BASIC, C89, Fortran and Assembly. A task passes only when hidden tests all pass.

Models work private legal cases end to end inside Legora's agent harness, with the matter's documents and templates. An LLM judge scores outputs against rubrics written by lawyers.

Agents with legal research tools answer the 518 LEXam open questions that strong open models score lowest on. Questions answerable from memory or unanswerable from their text are removed.

An author agent designs a software library from an open spec. Three fresh agents then solve 242 problems with it, and the library scores by their pass rate and code simplicity.

Agents analyze microbial communities from shotgun, amplicon, long-read and spatial metagenomics data. A grader checks the biological result.

Coding agents work through a varied set of software tasks built in house by Xiaomi. The report gives no task count or grading details.

Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details.

Models build websites from requests and turn reference images into matching front-end code. Tasks include WebDev and Image2Code.

Agents recover a hidden mathematical law from two noisy observations and a few experiments they choose. The submitted expression is run on private probes and passes if its error is under a noise-aware threshold.

Video generators make videos that follow many kinds of reference inputs at once. Scoring checks that each reference factor is kept, separated and placed correctly.

Agents run autoresearch on real open-source model development projects with their original code, data and checkpoints, for 60 or more hours. Fixed verifiers score the result against the human recipe they started from.

Computer-use agents do 146 scientific software tasks, such as molecular drawing, across several domains. Produced artifacts are checked to score each task.

Vision-language models perceive, reason about and judge whether videos follow real-world physics. The benchmark also tests if they can judge the physical realism of generated videos.

Multimodal models act as the decision core of a household robot. They must react at once to sudden physical hazards, such as catching a slipping plate.

Models resolve tasks in private production codebases licensed from real companies. The board reports resolution rate with standard errors.

Ten frontier agents each post-train the same Nemotron base model on a shared GPU cluster for six days. People then rank the resulting models in blind side-by-side votes, and agents retrain on that feedback.

ML research agents study a dataset, build a model and submit predictions on 60 tasks in tabular, forecasting, vision, language, audio and multimodal learning. Competition platforms score them.

Models do software engineering tasks in 553 repositories, across 9 task types, 20 application domains and 33 languages. Task types include feature changes, bug repair, refactoring and environment setup.

Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts.

AI tutors teach real students GRE material and are scored by the students measured learning gains. A second study has expert tutors compare AI lesson plans and practice problems.

Agents resolve 642 repaired issue tasks from the SWE-Bench Pro public split in Go, Python, JavaScript and TypeScript. A locked protocol re-grades each patch in a clean container.

Agents resolve the 51 SWE-Bench Pro V2 tasks that at least two of five model families failed. Graded with the V2 locked protocol.

Agents research 391 expert-written US corporate tax questions with tools and submit one cited written answer. A judge grades it against a rubric and checks that citations are real.

Agents make oligonucleotide discovery decisions, such as ASO and siRNA design, from real data. A grader checks each decision.

Version 2.1 of Vals AI's GDP-weighted index, which moved coding to Terminal-Bench 4.0 and added a tax sector.

Agents start from a working web app built for Vibe Code Bench v1.1. They extend it through up to ten dependent plain-English product requests without breaking it.

Audio language models answer questions that need memory of earlier spoken sessions. Questions cover what was said, who said it, how it was spoken and what was audible.

Agents play unseen line-puzzle games in the style of The Witness without being told the rules. They must discover rules by experiment and apply them to harder levels.

Multimodal agents explore interactive 3D worlds to find anomalies such as floating objects, walls that can be walked through or out-of-place objects.

Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice.

Agents answer 80 numeric analyst questions from real spreadsheets and documents in 14 domains, using Python in a sandbox. A question counts only if all five attempts are right.

Models answer questions that need reasoning across about 100k tokens of documents. Mercor adds 89 new expert-written tasks to the 100 public ones and reports both scores.

Models state probabilities and preferences on conceptual questions, and the benchmark checks 567 expert-approved logical consistency constraints such as conjunction rules and order invariance.

Solver systems of model, harness and tools run long, verifiable investigations on high-value problems scouted from 561 industries in 16 sectors.

Version 1.4 of Artificial Analysis's coding agent index, which moved to the full Terminal-Bench 2.1 and began scoring reward-hacked runs as zero.

One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores.

AI systems try to explore unknown problems, create new knowledge and turn new ideas into verifiable results, rather than apply learned knowledge.

Agents rebuild the full computational analysis of 20 published biology papers from raw data on a cloud VM. Output tables are graded against the papers' own artifacts.

Agents browse the web to find hard-to-locate facts with short answers. Mercor adds 100 new expert-written tasks to the 130 public ones and reports both scores.

Models answer descriptive and reasoning questions about scientific charts. Mercor adds 150 new expert-written tasks to the 5000 public ones and reports both scores.

Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups.

Agents answer a clinical question about one real MIMIC-IV patient through governed chart tools, then give a four-way verdict with cited evidence across 750 cases.

Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds.

An index that ranks models on reasoning about hard-to-verify topics such as AI alignment and decision theory. It weights LMCA 60 percent and ACCoRD and DTBench 20 percent each.

Agents work through five domains of 100 linked subtasks of rising difficulty, testing whether skill libraries they build in context actually help later tasks.

Agents answer questions over the email archives of four synthetic companies, up to 232,692 documents. Answers are checked against each company's ground-truth knowledge graph.

Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately.

Agents play text-only games whose transformation rules and win conditions are hidden, so they must discover the rules by experimenting. Every game was beaten by at least one human on a first attempt.

Agents complete Tencent in-house work tasks that need code. Task and scoring details are not published.

Agents start with 100,000 yuan and run up to four online stores for 365 simulated days on real market data. They source, negotiate with suppliers, price, promote and manage cash. Scored by end-of-year assets over the opening balance.

Agents complete end-to-end scientific workflows in chemistry, materials, molecular dynamics and life science from fixed inputs, producing a set of required deliverables. Task-specific graders check each deliverable.

Language models play Nine Men's Morris. Forming a row of three, called a mill, lets a player remove one of the opponent's pieces.

Language models play Reversi, also called Othello, flipping the opponent's disks that they flank in a line.

Language models take turns drawing a word as ASCII art and guessing what the other model drew.

Agents learn from training tasks drawn from enterprise workflows and must reuse that experience on related test tasks. Gains at test time measure self-evolution.

Models produce work products for occupations across nine GDP sectors. Mercor adds 132 new expert-written tasks to the 213 public ones and reports both scores.

Video and world models generate rollouts from image and action prompts. An evaluator agent plans checks, spawns sub-agents, gathers spatiotemporal evidence and scores observation, transition and persistence with written reasoning.

Models answer closed-ended frontier academic questions. Mercor adds 447 new expert-written tasks to the 2158 public ones and reports both scores.

Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published.

Coding agents complete a second version of Tencent in-house company engineering tasks. Task and scoring details are not published.

Agents complete Tencent in-house finance analyst tasks. Task and scoring details are not published.

Agents build financial models for a second version of Tencent in-house finance tasks. Task and scoring details are not published.

Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published.

Robot policies face 8,000 paired cases where the scene changes during execution, such as a target moving, the viewpoint shifting or an obstacle appearing.

Models answer 3,240 questions about 360 English and Chinese recordings from 10 minutes to over 2 hours. Each question is scored as multiple choice and as open-ended.

Models act as the controller loop around a coding agent. They monitor progress, assign work, run checks and decide when the task is safe to submit.

Models answer multimodal medical questions over clinical images and patient details. Mercor adds 51 new expert-written tasks to the 200 public ones and reports both scores.

Models with memory are tested on cases where correct, relevant memories still distort their reasoning or beliefs and lower task performance.

Models reason over synthetic medical claim files of about 25k to 64k tokens, as a claims reviewer would. Three judge models check completeness and accuracy.

Models answer multimodal questions that need both the image and the text. Mercor adds 140 new expert-written tasks to the 1730 public ones and reports both scores.

Agents answer 90 analytical questions using about 120,000 pages of U.S. Treasury Accounts of Receipts and Expenditures from 1793 to 2024. Answers are scored by exact match.

Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts.

Video generators are run many times on the same start and action, and the outcomes become a distribution over physical behaviors. The test checks whether it matches reference probabilities and covers every valid behavior.

Models write Lean 4 proofs of graduate-level theorems from paired natural-language and formal statements. Scored by the share the Lean checker accepts.

Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth.

Agents get a starting environment, a GPU budget and a verifier suite for an AI R&D task such as data curation or post-training. The score is mean reward per task, averaged across tasks.

Agents get a weak method on a long research task and must turn it into one that scores better on hidden data. It measures recursive self-improvement on executable research.

Models try to improve themselves through self-testing, self-judging and self-improvement in seven text-based games with executable verifiers. Held-out evaluation is kept strict.

Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions.

Video generation models must complete an intended task outcome from a reference image while keeping the high-level semantics of the reference.

Agents answer questions about large codebases, graded with rubrics. Mercor adds 50 new expert-written tasks to the 124 public ones and reports both scores.

Agents do 170 expert-curated, large refactoring tasks from real commits in Python, Java, TypeScript, Go, C, C++ and Rust, with rewritten issue descriptions.

Coding agents fix 119 repository-level tasks from 98 scientific software projects in 20 domains, split into issue-driven, expert-exploratory and integration tasks. Tests check each fix.

Agents resolve real GitHub issues by editing codebases, scored by repository tests. Mercor adds 50 new expert-written tasks to the 500 public ones and reports both scores.

Models write proofs of target results from STOC, FOCS and SODA papers, given the needed context. A verification agent, checked against expert judgments, grades each proof.

Agents complete long command-line tasks in containers, scored by verification tests. Mercor adds 99 new expert-written tasks to the 89 public ones and reports both scores.

A composite score built from the Surge AI benchmark suite, such as DAYJOB, GDP.pdf, Riemann-bench and ComplexConstraints. It asks whether models get through an ordinary workday.

Agents make antibody discovery and development decisions from real experimental data. A grader checks each decision.

Version 2 of Vals AI's GDP-weighted index, which brought back a legal sector and added financial modeling in Excel and code migration.

Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness.

Video generation models solve 27 visual reasoning tasks with 810 instances through generated frames. Scoring checks that the process is valid, not only the final frame.

Vision models answer 161 questions about lab images such as gel blots, plasmid maps, microscopy and flow cytometry plots. An LLM judge grades each answer against a key.

Speech-to-text models transcribe 300 English workplace recordings. They are scored on whether every target entity, such as an address or email, survives exactly.

Agents produce real deliverables in any format, from audio analysis and medical video reads to 3D meshes and kitchen layouts. Human and agent judges compare them with a baseline agent's work to give win rates and Elo.

Controllable video generators are tested as world models. Levels run from visual appearance to whether the depicted world reacts plausibly beyond what the prompt states.

Models write proofs for 245 advanced problems at undergraduate and doctoral qualifying exam levels. A verifier trained on expert annotations grades each proof.

Memory systems for agents are evaluated on storing, retrieving and using long-term memory across textual, coding and multimodal tracks in one reproducible framework.

Data agents carry out data science workflows over raw, mixed data and are scored at a fine-grained level across many scenarios.

Agents run month-end close tasks across 10 frozen business worlds with the apps an accountant uses. 100 tasks are graded against 1,475 rubric criteria.

Robot reasoning agents decide when to refuse, stop or ask a human. Parts cover safety constraints, people within one meter, safe VLA tool calls, task feasibility and unclear instructions or scenes.

Coding agents and model pairs diagnose misconfigurations, provision infrastructure and operate live cloud setups in disposable real AWS accounts.

Robot policies solve 100 full-length household tasks in seven BEHAVIOR-1K scenes from RGB, depth and proprioception. Ranking uses average task success with partial credit.

Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails.

Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate.

Models must learn from task-specific multimodal context, such as figures, tables, maps and web pages, instead of prior knowledge. Tasks run from grounding to knowledge acquisition.

Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break.

Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.2 added instruction following and advanced tool use problems. Agentic graders score correctness.

Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug.

Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings.

Agents complete Tencent in-house work tasks in the working agent group. Task and scoring details are not published.

Agents iterate for up to 12 hours on 134 executable real-world tasks with multi-level feedback. Scores are tracked at 2 hour steps to measure learning from the environment.

Models hold 16-message roleplay chats with 120 simulated user personas whose preferences compete and are hidden. A three-judge panel compares transcripts pairwise, and ratings are fitted as Elo scores.

Agents repeatedly edit an executable policy for compact interactive RL environments under a fixed interaction budget. The score tracks how much the policy improves.

Agents build investment banking and private equity financial models in Excel from a written brief and source files. Rubrics give partial credit for tabs, structure, formulas and formatting.

Systems read a full business document and return a schema-valid JSON object with every requested value. Unified value F1 compares the output with the expected values.

Models answer information-seeking questions with a search tool and must combine what they find correctly. Second version of FACTS Search on Kaggle.

Agents answer 220 investment research queries from finance experts across six use cases of the investor workflow. Three judge models check each answer against 11,543 source-attributed rubrics.

Vision-language-action policies are tested on whether they follow instructions. One word of the instruction changes and the policy must change its behavior.

Two language models negotiate how to split a pool of items, each with private item values. Each side tries to reach a good deal without giving away its values.

Two language models move on a grid and pick up coins. Each should collect the coins that score for it and avoid the rest.

Language models play Lines of Action, where a piece moves as far as the number of pieces on its line. The goal is to join all of one's pieces into one group.

Language models play nine linked tic-tac-toe boards. Each move decides which small board the opponent must play next.

Agents do routine company work with mock email, chat, calendar, issue and commerce tools while following a 20 to 124 page handbook. Programmatic criteria check required and forbidden actions, and a trial passes only if all pass.

Models judge from robot camera views whether a human is within 1 meter, a spatial awareness check for safe robot behavior. Scored by accuracy.

Coding agents complete Tencent in-house backend development tasks in the Claude Code scaffold. Task and scoring details are not published.

Coding agents complete Tencent in-house company engineering tasks. Task and scoring details are not published.

Models with tools solve Tencent in-house STEM problems. Task and scoring details are not published.

Agents build financial models for Tencent in-house finance tasks. Task and scoring details are not published.

Models solve Tencent in-house math problems without tools. Task and scoring details are not published.

Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published.

Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables.

Agents turn a natural-language requirement into a patch in a real repository drawn from Kuaishou internal development work. Each task pins its commit, environment and an executable verification entry point across 12 languages.

Models answer Korean finance questions in a multiple-choice format. Upstage built the set in house.

Video generators must reproduce a given sequence of keyframes while keeping overall video quality. The set has 386 samples over three domains and four keyframe densities.

Models solve AIME 2025 competition problems in Korean. Upstage built the set in house.

Agents complete 170 Korean office tasks from 58 occupations in 11 industries and produce deliverable files. Rubrics grade data, content, structure, quality and format.

Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks.

Models do management consulting work tasks from OpenAI's internal set. OpenAI has not published the task format or grading.

Models answer chemistry questions from an OpenAI internal set, grouped with OpenAI's life science research evaluations. OpenAI has not published the task format.

Agents run a seller's e-commerce store for 365 simulated days at order level. Actions constrain later choices, so the test measures long-term coherence.

Models find the moment in a robot video when an event happens. Scored by accuracy and mean absolute time distance in seconds.

Image editing models place several named people into shared contact actions such as an embrace or a carry. Scoring targets fused limbs, extra extremities and bodies passing through each other.

Patient-facing health agents talk with a simulated patient and use a sandbox of healthcare tools for scheduling, prescriptions, telehealth and escalation. An LLM jury scores over a hundred clinician-grounded criteria.

Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3.

Models assign each frame of a robot video to one of five task-progress bands from 0 to 100 percent. Scored by accuracy.

Models diagnose single-image radiology cases with a confidence score from 0 to 4, or say they do not know. A confidence-weighted index rewards right answers and penalizes confident errors.

An aggregate of OpenAI's internal evaluations of progress toward recursive self-improvement. The parts are real AI research tasks such as debugging research systems, optimizing kernels and training recipes, and improving another model.

Coding agents act as data-centric researchers. With a fixed Tinker post-training stack and budget, they refine training data from dev-benchmark feedback, and the final model is scored on held-out test benchmarks.

Agents complete 20 multi-hour software tasks such as library rebuilds, product clones and ML work. Hidden tests, parity checks and integrity checks grade each run, scored over 8 trials.

Vision-language models answer questions about synthetic long documents where length, layout, modality and difficulty are varied one at a time to locate failure causes.

Coding agents do tasks in four work domains: code, web, office and security. Each task is rebuilt from a real commit or business scenario as a short request that resists web search.

Agents get five days to build and run a space program in Kerbal Space Program. The score is progress through 30 fixed missions.

Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill.

Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable.

Agents discover and interpret genetic variants from real sequencing analyses. A deterministic grader checks the result.

Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear.

Agents complete workflows taken from market-validated AI-native startup products and their users. Domain experts wrote the tasks, attachments and rubrics after user interviews.

Agents work in web shopping, terminal and search question tasks where some goals cannot be met. They must abstain at the right turn instead of acting on.

Language world models predict the next environment observation after an agent action, in 7 domains from MCP and search to Android and desktop OS. A judge model scores each prediction against the real observation on five dimensions.

Version 1.1 of Artificial Analysis's coding agent index, which replaced SWE-Bench-Pro-Hard-AA with DeepSWE.

Version 4.1 of Artificial Analysis's index, which moved to GDPval-AA v2, Terminal-Bench 2.1 and τ³-Banking and put the most weight on agents.

Agents play levels of the puzzle game Baba Is You through a text interface and win by rewriting the rules. Models are compared on pass rate, speed and cost.

Agents answer workflow-grounded financial research questions with web search, SEC EDGAR, page fetch and Python. A judge grades each answer against an expert rubric and reference.

Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not.

Agents steer a simulated AI startup for 500 days, gathering noisy information and adapting plans. The outcome of the simulated company is the score.

General-purpose agent harnesses such as OpenClaw do multilingual SWE-bench-style coding tasks through an adapter that turns their output into scorable patches.

Models port a working program into another language offline and ship a Dockerfile. A hidden behavior-test suite the model never sees scores the port, after an anti-cheat check.

Models answer 75 expert-written professional prompts whose constraints depend on each other, fire conditionally or must be inferred. An LLM judge grades 1,559 rubric criteria one at a time, and a task passes only if all criteria pass.

AI systems work through sequences of related task instances in six domains and should get better as they go. A gain score subtracts each system's own stateless baseline to isolate learning.

Video editing models follow compositional instructions with several coupled edits to subjects, actions and camera, while unrelated content must stay unchanged.

Agents complete productivity tasks in Notion, Canva and Figma by mixing GUI actions with MCP tool calls.

Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass.

Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects.

Models answer the turn of a real Doubao conversation where the user was dissatisfied, with the earlier turns as context. Rubrics grade each answer.

Agents do early-stage drug discovery work, from target identification to lead optimization, in 82 tasks with one checkable answer each. A judge model grades answers against expert rubrics.

Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head.

Agents do 852 tasks rebuilt from real proprietary workplace agent sessions, each with fixtures, hard rules and semantic rubrics.

Agents analyze real epigenomics datasets in a sandbox with standard tools. A deterministic grader checks the biological result.

A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is.

Agents work in environments whose knowledge, tools and task conditions change over time. The suite tests whether agent memory keeps up with the changes.

Coding agents work through 26 multi-round tasks with 227 graded rounds on the official Harbor multi-step format. The score is the mean of binary per-round rewards. A case score gives partial credit.

The editors run AI harnesses once each, with 24 hours, on ten unpublished research problems with human proofs of up to eight pages. Referees grade each answer blind.

Agents complete 180 office tasks across 15 corporate professions inside a workspace of xlsx, docx, pptx, pdf and other files. An LLM judge checks the output files against expert rubrics, and a task passes only if every rubric item passes.

Language models play American checkers against each other, with forced jumps and crowned kings. Ratings come from head-to-head games.

Language models play Clobber, where every move captures an adjacent enemy stone. The player who makes the last legal move wins.

Language models play Hex without seeing the opponent's stones, so they must probe cells to learn the hidden board while building a path.

Language models take turns drawing lines between dots on a grid. Closing the fourth side of a box claims it.

Coding agents build playable games end to end inside a real game engine, where scripts, scenes, assets and runtime behavior must work together.

Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier.

Models generate JSON or YAML that must follow a requested schema and phrasing constraints, without constrained decoding. A response passes only if every structural constraint holds.

Models study several real photos of a home and draw its floor plan. Scored by an average quality score.

Models write Lean 4 proofs for the 60 IMO-ProofBench problems, formalized by hand and checked by Lean experts. The Lean compiler verifies each proof.

Phone agents work in a native iOS simulator across 26 purpose-built apps that share one persistent user's data. Tasks span single apps, multi-app flows and memory or personalization from the user's history.

Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers.

Models watch walking-tour videos geolocated over up to 1 km and answer questions on landmarks, routes and map layout, such as distances and loop closures.

Models answer 899 knowledge questions across 261 fine-grained disciplines, each with ten lettered options. Scored by accuracy against the key.

Models answer 100 research-level math questions with known answers, written by 49 mathematicians at a Max Planck Institute workshop in April and May 2026.

Extraction systems pull structured values from 225 long table-heavy documents averaging 358 pages and about 88,700 ground-truth fields each.

Audio models edit audio from instructions across a broad range of real-world editing tasks and sound types.

Models answer culture-specific knowledge questions in 11 major languages.

Models solve LiveCodeBench-style contest problems in twelve programming languages, with dated problems for contamination-aware evaluation.

Computer-use agents work in a personalized Linux desktop with 17 simulated logged-in web apps and a user's history, across 184 tasks.

Coding agents work on 90 tasks distilled from Nature-family papers, each in its own container, and try to match or beat the published results.

Models write captions for videos with audio while following detailed user constraints. Captions are scored on format and content compliance.

Multimodal models answer spatial questions about places and layouts during a continuous egocentric video stream. The set has 1,680 human-annotated questions over 348 videos.

Models replace PyTorch and NCCL reference code with multi-GPU CUDA kernels for 87 problems from codebases such as Megatron-LM and DeepSpeed. Scored on correctness and speed over the baseline.

Video models continue 198 real filmed physics experiments from their first frames. Generated videos are compared with the real outcome, with every sample and metric weighted equally.

Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures.

Models answer SNAP benefit questions across seven lifecycle phases and 25 topics. A judge grades each answer against rubrics validated by SNAP policy experts.

Agents run long single-cell RNA-seq analyses that need sustained empirical work and biological reasoning.

Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release.

Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables.

Models write or optimize CUDA style kernels for operator tasks mined from FlashInfer, SGLang, FlashAttention, DeepGEMM, CUTLASS and cuDNN. Scored as average speedup over reference code.

Agents build features and fix runtime bugs in real open-source repos from loose, natural instructions. Behavioral tests and codebase quality metrics give a tasteful solve rate.

RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose.

Multimodal agents act in real-world style environments to show interactive spatial understanding, rather than answer static spatial questions.

Coding agents are scored on separate repository skills, such as understanding, context retrieval, code localization and bug diagnosis, instead of one pass or fail.

Coding agents work 75 multi-turn software tasks driven by a simulated developer who adds requests over a long session.

Models read W-2, 1099 and other PDFs and compute full federal and state returns line by line. A return counts only when every evaluated line is exact.

Agents build a large project end to end with no time limit, such as a faster Rust compiler or a WebGL renderer in WebAssembly. Each challenge has its own test bed and metric.

Coding agents complete tasks drawn from real use of the Trae coding product, such as web development, bug fixing, code generation, artifacts and environment setup.

Agents use a terminal for general computer tasks beyond coding, such as file, system and everyday productivity work.

Agents analyze preclinical pharmacology data and make development decisions. A grader checks each decision.

Agents build front-end and full-stack Web, Android and iOS projects from scratch in Claude Code. An agent-as-verifier pipeline checks interaction logic and visual output.

Computer-use agents do 114 long tasks in 8 work domains that mix GUI actions with command-line and code operations in one trajectory. Results are checked against verifiable artifacts.

Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments.

Models answer hand-written reasoning questions about images drawn from a taxonomy of thousands of visual concepts. Questions were kept only if frontier models failed them.

Models answer open-ended requests grounded in real user needs across more than 30 daily-life, study and office scenarios. Rubrics grade completion, correctness, coverage and usefulness.

Agents complete 100 video post-production tasks in 4 task families built from real workflows by 20 industry experts. Programmatic verifiers and expert rubrics score the results.

Coding agents optimize GPU kernels through full workflows with compilers and profilers. Scores check speedups and whether gains generalize beyond tuned shapes.

Experts such as doctors, lawyers and engineers compare model answers to prompts from their own history. They judge which answer would still help them a month later.

Artificial Analysis's first index of coding agents, each a model in a harness, averaged over three equally weighted coding benchmarks.

Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response.

Agents redo data analysis tasks from 100 published biomedical studies in a Harbor environment. An LLM judge grades the whole trajectory against rubrics written with the paper authors, so method quality counts, not just the final answer.

Agents run long healthcare operations workflows such as prior authorization through 87 MCP tools over 20 simulated apps. Tasks need policy lookups, role handoffs and multi-turn dialogs.

Browser agents do 130 new everyday tasks on live websites under six supported harnesses. Request interception and a more lenient LLM judge score each run.

Agents do ambiguous multi-file coding tasks from real Cursor sessions in Cursor's harness. Version 3.1 added codebase understanding, bug finding, planning and code review problems. Agentic graders score correctness.

Models prepare training data across six domains, and a base model is fine-tuned on their output. Downstream results of that model give the score.

Agents get one attempt at each of 50 deep engineering tasks in Docker, in the Terminal-Bench 2.0 format. A task passes only if every hidden test passes. Tasks were calibrated so Claude Opus 4.6 still fails some at pass@4.

Embodied agents act to reveal what is unseen, such as occluded structure, containment and function, then reason about how observations change with action.

Voice agents hold simulated spoken conversations to complete enterprise tasks. Scores cover task success and voice-specific failure modes.

Coding agents keep one project working across 26 tasks of 5 to 15 rounds each, while requirements change, pile up and conflict. Cumulative hidden tests check every round in a persistent container.

Teams of language models give one-word clues and guess linked words, in a game like Codenames, against an opposing team.

Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.

Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute.

Models answer 2,049 Chinese industrial procurement questions grounded in national standards and product records. Answers must respect regulated thresholds and safety clauses.

Agents read an offline snapshot of a failing Kubernetes cluster and name the entities that caused the incident, in structured JSON.

The model under test judges fixed creative-writing samples against low and high anchor responses. The score is how well its ratings separate writer models.

Models generate minute-long videos with sound from text, an image or a video. Scoring covers identity consistency, narrative coherence and audio-visual alignment over time.

Memory systems help agents learn the affordances, dynamics and failure modes of customized web environments. Questions check whether that experience was internalized.

Agents build complete financial models in Excel from a case prompt and documents. An LLM judge grades accuracy, formulas and formatting on a 0 to 100 scale.

Multimodal agent memory systems answer questions that need fine visual evidence kept from earlier sessions, which captions or text traces alone cannot supply.

Vision-language models and memory agents answer 789 questions over multimodal multi-session conversations, covering extraction, temporal reasoning, knowledge updates and refusal at four context lengths.

Models answer 1,160 multiple-choice questions on egocentric kitchen videos from HD-EPIC. Each question has a hand-written reasoning trace grounded in time and objects.

Memory-augmented agents answer questions after long, frequently updated histories where memories interfere. Instances average 138,800 tokens and reach 1.8 million.

Mobile GUI agents use 28 simulated apps in a browser-hosted phone, across 416 parameterized task templates. Deterministic judges check the app state.

Models answer five kinds of music questions that need the correct time region of the audio, such as instrument entries. Music experts validated the items.

Coding agents with a fixed H100-hour budget and no internet try to improve on a strong NanoGPT speedrun starting point. The score is the share of later human progress recovered.

Retrievers search large real corpora for documents that match a latent pattern, such as an implicit stance. The suite has five such oblique search problems.

Image editing models do precise single-answer edits in 20 operations such as geometric transforms, color changes and symbolic edits. Procedural generation gives deterministic checks.

Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair.

Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge.

Models solve generated planning problems that coordinate goals, constraints and resources. Instances are scalable and come with verifiers.

Models write proofs for hard competition problems. An LLM judge compares correct proofs pairwise on conciseness, computational ease, simplicity, diversity and adaptivity, giving Elo ratings.

Text-to-image models are tested on professional creative work, from faithful real-world reconstruction to creative expression. A released judge model scores the images.

Models turn images, videos and design references into executable code. Qwen reports ratings near 1,500 to 1,900, not percentages.

Agents get the literature and raw data behind a published paper, with the paper hidden, and try to rediscover its results. Expert rubrics with weighted criteria grade the outputs.

Agents start from an older version of an open-source repository and implement a roadmap of features from its next real release. The set has 115 tasks from 17 repositories in 5 languages, scored by tests.

Policies run 30 new real tabletop tasks on ARX5, UR5, ALOHA and DOS-W1 robots. Each task has a 10-point stage rubric for partial credit.

Robot policies do long-horizon tasks in partially observable settings that need memory of past observations and actions, in simulation and on real robots.

Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal.

Agents migrate enterprise Java applications from one framework to another while keeping behavior the same. Build config, dependency injection, persistence and deployment all change.

Models solve 439 new research-level and graduate math problems written by mathematicians. A 340-item Challenge subset tests solving and a 99-item Refusal subset tests spotting ill-posed problems.

Spatial foundation models are tested across many 3D tasks, viewpoints, scene domains, input densities and hardware limits, beyond the domains they were trained for.

Agents run long, multi-step analyses of real spatial biology datasets that need sustained biological reasoning.

Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy.

Models fix software issues inside several coding agent harnesses, such as Claude Code, OpenCode and Hermes Agent. The score is the average over harnesses.

Coding agents refactor code in 70 tasks in real repositories with internet access cut off. Tests and software-quality rubrics grade the result.

A suite of three terminal datasets for coding agents. DeepTerminalBench tests single-shot depth, EvoCode-Bench tests multi-round iteration and RoadmapBench tests version upgrades on real repositories.

Agents build 17 real web apps from empty projects one feature at a time from short specs. A grading agent uses each app in a browser and follows test plans.

Multimodal agents are evaluated on two-person video conference recordings. They cover 11 conversational dynamics across verbal, nonverbal and mixed behavior, annotated by people in three passes.

Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics.

Interactive video world models run 289 multi-turn cases with navigation, subject actions, event edits and perspective switches. 22 metrics cover quality, adherence, consistency and physics.

Video generation models are tested for causal and arrow-of-time understanding by comparing real videos with their time-reversed versions, in two levels.

Agent products such as Claude Code and Codex do 94 production tasks from 7 companies across 6 O*NET domains. Grading mixes formal checks, UI tests and rubrics.

Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks.

Models prove or disprove theorems from recent arXiv papers, stated in Lean 4. A strict comparator checks each submitted proof against the fixed reference statement.

Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions.

Agents forecast 1,417 binary questions asked in October 2025 and resolved in December 2025, researching offline in a frozen corpus of 15 million documents. Scored by Brier score.

Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state.

Browser agents do 153 everyday tasks on live websites, such as booking travel, ordering food and applying for jobs. Request interception stops real side effects, and an LLM judge scores the five-layer traces.

Unified digital agents do long human-designed tasks that mix vision, search and coding. An automatic function checks each final output.

Speech-to-text, text-to-speech and speech-to-speech models from almost 40 providers are scored every 30 minutes on call-like audio with accents, noise and echo.

Computer-use agents do one long task in each of 200 real applications, such as 3D Slicer, GIMP and OpenEMR, often needing 500 or more steps. Checklist verifiers with privileged state score each task.

Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report.

Agents do long-horizon coding tasks with a median estimated human completion time of 20 hours.

Agents work up to 20 hours on 17 ultra-long engineering, performance and research tasks in their own harnesses. Models are ranked by average task rank and dominance.

Voice agents hear real human speech with five kinds of disfluency and must make chained API calls across four task domains. Scored on Pass@1, response quality, latency and turn-taking.

Language models play a gomoku-style game and try to line up five of their tokens before the opponent does.

Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state.

Models solve 365 hand-written reasoning problems and 1,095 variants in eight categories that need only K-12 knowledge. Scoring mixes rule-based and model-based checks.

Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts.

Coding agents face tasks with hidden blockers such as missing or contradictory requirements. The score rewards asking for help only when a blocker makes it necessary.

Coding agents complete Tencent in-house backend development tasks. Task and scoring details are not published.

Coding agents solve 300 Tencent in-house software tasks in Claude Code with a 200-turn budget and no web access. Hidden verifier tests grade each task.

Coding agents complete development workflows drawn from real Tencent users. Task and scoring details are not published.

Users upload a screenshot, wireframe or mockup, two anonymous models build the website in Code Arena, and users vote for the better result. Votes become ratings with bootstrapped confidence intervals.

Models read gauges, sight glasses, digital displays, linear scales, rulers and thermometers from robot camera images. ER 2 tests 10 instrument types.

Agents bet through a simulated 2023 to 24 English Premier League season with historical data. The score is long-term bankroll growth.

Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images.

Models answer multiple choice questions about theorems from arXiv papers published after their training cutoff. Wrong options follow plausible but invalid proof sketches.

Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report.

Models do varied coding tasks inside agent harnesses such as Claude Code. Tasks cover repo understanding, project building, code review, artifact generation, planning and SWE.

Omni-modality embedding models are scored on 190 tasks across text, image, video, audio, visual documents and agent-centric retrieval.

Speech-to-text providers transcribe 4,270 caller utterances from real 8 kHz customer service phone calls in five locales, scored against human word-level transcripts.

Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment.

Agents do 100 professional task scenarios across 10 industries and 65 domains. Language world models simulate each domain's tools so tasks stay solvable.

Models parse 1,651 PDF pages, including nested tables, dense formulas and unusual layouts, into text, tables and formulas. Scored by the v1.5 Overall formula.

Document parsers process about 2,000 human-verified enterprise pages. Over 169,000 test rules score tables, charts, content faithfulness, formatting and visual grounding.

Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark.

Models find the time span of short sound events hidden in long, unstructured YouTube recordings. It has 400 audio, query and timestamp triplets.

Coding agents write production-quality tests for 90 tasks in real repositories with internet access cut off. Programmatic checks and rubrics grade the tests.

Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed.

Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following.

White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace.

Models are asked to prove statements from recent arXiv papers that were changed to be plausible but false. Full credit goes to models that say the statement is false.

Models answer 100 nonsense questions across software, finance, legal, medical and physics, built with 13 techniques. A three-judge panel scores clear pushback.

Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry.

People upload their own PDFs with a question, compare answers from two anonymous models, and vote for the better one. Votes become Bradley-Terry ratings.

Models forecast unresolved events drawn from prediction markets, synthesized news questions and expert-written questions. Forecasts made at aligned times are compared in pairwise battles, and an Elo fit ranks the models.

Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task.

Agents complete the formal Lean proofs and define the new mathematical concepts in real pull requests to the Fermat's Last Theorem (FLT) formalization project. Lean and SafeVerify check each submitted diff.

Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number.

Models attack 113 mostly unsolved problems in eight areas of mathematics and the mathematical sciences. A program checks each proposed closed form, construction or bound.

Models answer 320 hand-written questions about music, curated and checked by people with musical training.

Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise.

VLA policies run LIBERO tasks with over 4,000 paraphrased instructions that vary object and action wording separately. Scored by success rate and the difficulty-aware PRIDE metric.

Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8.

Models answer questions about long videos with rich audio, often tens of minutes long. The questions need both the audio and the video track.

Document parsers read 3,400 digital and photographed document images in 17 languages. Public and private splits limit leakage.

Models answer 20,000 questions about 11,877 long web videos, using sound, images and text together. Questions cover 13 skill categories.

Coding agents resolve 100 fresh GitHub tasks each month, half bug fixes and half features, from newly merged pull requests. Harbor verifiers score pass@1.

Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge.

Models solve 300 pencil puzzles of 20 varieties with unique solutions, either in one shot or agentically with step checks. Every board state can be verified against the rules.

Agents compete in Pokemon battles under partial observability in a Battling Track and play the Pokemon RPG in a Speedrunning Track that needs long-horizon planning.

Policies run 9 simulated bimanual manipulation tasks on RoboTwin 2.0 that need memory of earlier observations, at several levels of memory complexity. Success rate.

Models name skateboard tricks from technical trick definitions. The score is the share named correctly.

Agents repeatedly extend their own code across 36 problems and 196 checkpoints as the specification changes. Tests check correctness, and metrics track structural erosion and verbosity.

Systems write CUDA kernels for 235 workloads taken from real model architectures, run on B200 GPUs. The SOL score measures how much of the gap between a baseline and the analytic hardware speed limit a kernel closes.

Agents maintain a repository from a base commit toward a later reference commit over 100 commit pairs, looping through tests, requirement writing and code changes.

Image generation models edit text inside 2,148 real and synthetic images across 18 subcategories. OCR, image-fidelity metrics and a multimodal LLM judge score target accuracy, text preservation and scene integrity.

A diagnostic suite that audits video understanding benchmarks and keeps only samples that need visual and temporal input. Models are tested on the remaining video-native questions.

Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use.

Full-duplex voice agents handle real-time support calls in retail, airline and telecom with tools and a simulated caller. Scoring covers task completion and natural interaction.

Models reason from experimental results in peer-reviewed biology papers to draw conclusions, with causal and cross-scale inference.

Models get prompts built on nonsensical premises. A three-judge model panel grades whether each answer pushes back clearly, challenges partly or accepts the nonsense.

Models solve 53 Codeforces Div. 2 problems from September 2024 to February 2025. An offline judge with rebuilt test cases scores C++, Python and Java submissions.

Models follow complex Chinese production prompts with constraints on format, conditions, content, phrasing, tone, emoji, few-shot style and length. It has 912 cases over 17 weighted dimensions.

AI code review tools review pull requests from five large open-source projects. An LLM judge matches their comments to human-verified golden comments.

Models identify several elements of a business text in one inference, such as user intent and slot values.

Models answer 500 advice queries taken from Reddit, Stack Exchange and forums. Model judges score usefulness, logic, instruction following and tone against an expert reference set at 88.

Agents in Claude Code solve 50 business data-analysis tasks with CSV data. A model grader scores outputs against expert rubrics averaging 26.9 items a task.

Multimodal agents search raw visual histories over several steps to find target images from implicit context cues.

Agents work inside a simulated hardware startup with more than 2,500 entities and 23 tools. They triage incidents, audit shipments and handle customers under company policy.

Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them.

Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests.

Agents implement whole features in real repositories, beyond single-PR bug fixes. Execution-based tests check each feature, and new tasks are generated automatically.

AI systems attempt ten unpublished research problems that arose in the authors' own work. Mathematicians judge whether each proof is correct.

Models forecast open Metaculus questions with a fixed prompt and are scored as the questions resolve. Seasonal bot tournaments and Pro Forecaster baselines run alongside.

Language models play chess against each other from varied two-move openings sampled from 20 popular human openings. The board is given as text.

Language models play a Connect Four style game and try to line up four of their tokens before the opponent does.

Language models play heads-up no-limit Texas Hold'em against each other. Models are ranked by mean big blinds won per 100 hands.

Teams of language models play the social deduction game Werewolf in natural language. Villagers must find the hidden werewolves, who try to deceive them.

Agents complete game development tasks in the Godot engine, derived from web and video tutorials, across 2D and 3D graphics, user interfaces and gameplay logic. Tests in the engine check each result.

Models complete a ByteDance Seed subset of GDPval occupational tasks. Rubrics grade each deliverable automatically.

Professional writers compare two models' answers blind on real writing prompts, from bedtime stories to business documents. The comparisons become ratings.

Models read scientific figures and answer in open response. Served as images or as PDFs. The FigQA2 subtask of LABBench2.

Models answer open-response questions whose answers appear once in recent biology papers. The LitQA3 subtask of LABBench2.

Models find and fix errors planted in published lab protocols, answering in open response. The ProtocolQA2 subtask of LABBench2.

Models reason about DNA and protein sequences and answer in open response. The SeqQA2 subtask of LABBench2.

Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success.

Audio embedding models are scored on 30 tasks across speech, music, environmental sound and audio-text retrieval in over 100 languages.

Models read de-identified patient records and output primary and secondary ICD-10-CM diagnosis codes. Answers are checked against codes assigned by professional coders.

Models write SOAP notes from clinical visit transcripts. Notes are graded against expert gold-standard rubrics for required content and section structure.

Audio representations from any multimodal system are scored on eight core tasks, including transcription, classification, retrieval, reasoning, segmentation, clustering, reranking and reconstruction.

Agents explore interactive environments built from four transition primitives and must work out the hidden dynamics over long horizons. OdysseyArena-Lite has 120 standard tasks.

Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data.

Agents solve 100 Terminal-Bench tasks picked to track Terminal-Bench 2 scores while running faster. Tasks are balanced across four difficulty bands using Claude Haiku 4.5 pass rates.

Models review real merged pull requests with planted functional bugs and best-practice violations. Findings are matched to the planted issues and scored by F1.

Agents follow expert-written standard operating procedures across 12 industrial domains, using executable tool interfaces over more than 2,000 tasks with 10 to 50 decision points.

Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents.

Agents do the Terminal-Bench 2.0 tasks after Z.ai fixed environments for 89 tasks and instructions for 11. A further 2026-08-18 round fixed tests and instructions in 6 more tasks.

Models analyze business text and return labels or structured outputs, for example sentiment or the main viewpoints in user content.

Agents build full-stack projects for Web, Android, iOS and Windows, including server APIs and databases. Claude Code based verifiers check interaction logic and visual output.

Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability.

GraphRAG systems answer 1,100 questions over long, mixed reference documents behind Wikipedia articles in 12 topics, from single facts to section summaries.

Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data.

Models solve 2,198 science reasoning problems annotated with the anchors and attractors (prior knowledge) they need. An AAUI metric measures how much of that knowledge the model activates.

Models review 200 real pull requests in many languages with full repository context. Their review comments are matched against expert-annotated issues.

Models play 10-seat no-limit Texas Hold'em tournaments against each other. Chip counts and elimination order feed TrueSkill ratings.

Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect.

Assistants hold long simulated conversations with users whose details and preferences change over time. Simulated users react on policy, and the test checks whether the assistant remembers and applies the right facts.

Version 4.0 of Artificial Analysis's index, ten evaluations in four equal categories that brought in real-world knowledge work and hallucination tests.

Models answer questions about over one thousand iconic internet sounds and videos, from surface content to context, emotion, usage and world knowledge.

Browser agents do real web tasks with the Browser Use library. A judge scores each run against a rubric.

Models handle 16 everyday coding tasks such as explaining, debugging, refactoring and testing code. A GPT-5 judge checks five rubrics per item.

Two coding agents each get a different feature for the same repository and must coordinate so both changes work together. Over 600 tasks span 12 libraries in 4 languages.

Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria.

Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors.

Models attack unsolved research math problems proposed by working mathematicians. Each problem has a program that checks a proposed solution. Epoch tracks which problems AI has solved.

Models watch and hear 919 videos and answer 1,034 multiple-choice questions that ask what will happen next. Questions need audio and visual cues together.

Models answer HealthBench-style medical prompts. A pipeline splits each answer into medical claims and checks them against medical evidence. The score is a weighted hallucination rate.

Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk.

Models read long autobiographical narratives and must infer a person's stable motivations and decision principles, not just retrieve facts.

Video generation models animate the same input image with the same prompt, anonymously and side by side. Votes give Elo-style ratings.

Video generation models make clips from the same text prompt, anonymously and side by side. Votes from logged-in users give Elo-style ratings.

Agents attempt 228 software and research tasks timed against skilled humans, run in Inspect. The score is the human task length at which the agent succeeds half the time.

Models turn a prompt into a Minecraft-style voxel build as raw 3D coordinates or tool calls. Users compare builds blind, and votes become rankings.

Models answer about 2,200 open-ended questions on 540 videos from 18 locales, written in native languages by local experts. An LLM judge scores each answer 0, 1 or 2.

Models answer questions about molecular graphs, such as counting, indexing and constrained generation, and a symbolic checker verifies each answer.

Models run in the same OpenHands agent setup on five areas: issue resolution, greenfield builds, frontend work, test writing and information gathering. Accuracy and cost are both reported.

Coding agents add OpenTelemetry tracing, metrics and logging to codebases in several languages. Tests check the instrumentation.

Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors.

Models research fact patterns under U.S. statutes and regulations and hunt for obscure public information online. 33 private questions run three times give a score out of 99.

Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review.

Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs.

Vision-language models act as reward models on real robot videos from Open X-Embodiment and RoboArena, judging task success and partial progress. Counterfactual relabeling adds calibrated failures and near misses.

Role-play agents are tested on situated reenactment in Chinese and English scenes. Grading targets the immersive experience rather than a verifiable answer.

Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer.

Models act as doctors in 303 simulated clinical cases from 12 departments. They take a history, order lab tests and give a final diagnosis, scored per stage.

Models predict the outcomes of 405 real physics, biology and chemistry experiments published after March 31, 2025, with and without expert background knowledge.

Agents complete 100 multi-step coding tasks in sandboxed containers, spread evenly over four difficulty tiers. Unit tests and rubrics grade both outputs and the agent trajectory.

Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies.

Image editing models make local text and table edits to 3,506 Chinese and English document images. Scores check the edit and that the rest of the page is preserved.

Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies.

Models answer multiple-response questions on a small set of weaponizable pathogens that biosecurity experts rank highest priority. A revised version of HPCT.

Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements.

Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer.

Agents fix issues and add features in production scientific code repositories, such as quantum chemistry and astrophysics codes. Fail-to-pass tests in containers check each change.

Coding agents do development tasks taken from maintainer pull requests in six widely used scientific codebases, such as quantum chemistry and molecular simulation packages.

Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench.

Models answer 3,212 multiple-choice questions about who speaks, what is said and when in real-world videos. Questions need audio and video together.

Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates.

Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software.

Models answer 150 olympiad-level biology questions, 80% of them with images. Domain experts annotated and reviewed each question.

Averages seven safety measures that correlate weakly with general capability. They cover agent prompt injection, bioweapons help, political bias, overconfidence, lying under pressure and harmful or deceptive play in text games.

Averages model scores on the text tests that the Center for AI Safety runs for its AI Dashboard. On 2026-11-01 the index listed Humanity's Last Exam and TextQuests.

Averages model scores on five vision tests run by the Center for AI Safety: ERQA, MindCube, SpatialViz-Bench, IntPhys 2 and EnigmaEval.

Models answer short bilingual fact questions about programming concepts and implementations across languages and computer science domains.

Models carry out step-by-step tasks that follow a given standard operating procedure, such as multi-turn after-sales service in support, sales and healthcare.

Models see two views of a scene and must find the point in the second view that matches a point in the first. Tasks go from perception and reasoning to exact coordinate prediction.

Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom.

Data agents do 210 enterprise data tasks: building multi-stage SQL pipelines on industrial schemas and analyzing tables into decision-oriented insights.

Models solve, grade, explain and write K to 12 problems across core subjects in teaching scenarios.

Models judge questions composed at test time from knowledge statements extracted from authoritative textbooks. Random composition is meant to resist contamination and test several facts at once.

Models are scored on four factuality tests: Parametric, Search, Multimodal and Grounding v2. The FACTS Score averages accuracy over public and private sets of all four.

Models write long answers from a supplied document of up to 32k tokens. Judge models check that each answer stays faithful to the document.

Models answer questions about input images and must combine what they see with world knowledge. A judge checks coverage and contradictions.

Models answer closed-book factoid questions from their own knowledge, without tools. A judge model grades each answer.

Models answer information-seeking questions with a web search tool (the Brave Search API) and must combine what they find correctly.

Models write programs for 156 open-ended computer science problems whose optimum is unknown. An automatic evaluator gives partial scores against an expert reference solution.

Models solve olympiad-style physics, chemistry and biology problems at the level of IPhO, IChO and IBO, written by medalists and coaches. Graded on short final answers.

Models work on open-ended PhD-level research subtasks in physics, chemistry and biology, written by working scientists. A grader scores each answer against a 10-point rubric.

Models answer 400 human-translated MMLU questions per language in 18 languages, adding Albanian, Burmese and Welsh to the first 15. Accuracy.

Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU.

Models extract words, sentences or fields from meeting records, legal texts, contracts, product descriptions and knowledge bases.

Models analyze large collections of user-generated content, for example summarizing viewpoints and judging overall sentiment.

Models classify intent from dialogues, meeting transcripts, social posts and articles, such as purchase intent in customer service chats.

Models answer questions about videos that cannot be answered from vision or audio alone. Questions cover speech and sound events over several scene spans.

Models solve AIME 2024 competition problems translated into Korean. Upstage built the set in house.

Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house.

Models write proofs for the 2025 Miklós Schweitzer competition, a hard Hungarian university-level contest. MathArena runs and grades it.

Models answer hazard prompts with and without taxonomy-based jailbreak attacks. The Resilience Gap is the drop in safety score under attack.

Models read a long synthetic chat that hides 2, 4 or 8 identical writing requests and must reproduce the response to the i-th one. Scored by mean string match ratio per context bin.

VLA policies run 16 LIBERO tasks with an added obstacle at two levels, near the target or in the path. 50 episodes per scenario. Scored by collision avoidance and task success.

Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail.

Models work through more than 1,000 expert samples in four scientist tasks: deep research, idea generation, dry and wet experiments, and experimental reasoning.

Models solve 400 real spreadsheet manipulation questions from Excel forums, an expert annotated subset of SpreadsheetBench. Each answer runs against several test spreadsheets.

Models solve 500 expert-written chemistry reasoning problems, offered in multimodal and text-only forms. Expert solution paths also allow a reasoning path score.

Coding agents implement release-note changes in seven mature Python projects, editing about 21 files per task. Large test suites check each of the 48 tasks.

Models find the start and end time of a described event in a video. The benchmark re-annotates Charades-STA, ActivityNet Captions and QVHighlights with strict quality rules, and scores by mean IoU.

Models plan an object-centric 3D trace, an ordered list of metric 3D waypoints, from language instructions in real cluttered indoor scenes with depth.

Models build interactive artifacts such as small apps and pages for 199 cases across 13 topics. A GUI agent simulates a non-expert user to grade interactivity, looks and fit to the need.

Vision-language models and humans judge whether ASMR videos with tight audio and visual coupling are real or AI-generated. It tests perceptual realism detection.

Models complete expert tasks from more than 140 task types in law, finance, education, humanities and science. Over 200 domain experts wrote the prompts and weighted rubrics.

Models answer real-world video reasoning questions that need high frame rate motion perception or multi-hop reasoning over long videos. Videos average 1,672 seconds.

Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security.

Agents complete 350 daily tasks in 94 real closed-source Android apps. A process-aware reviewer model checks each trajectory against written guidelines.

Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench.

Models answer about 800 original PhD-level problems across mathematics, physics, chemistry, biology, computer science, earth science and materials science. Answers are open-form and checked by an LLM judge.

Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples.

Models prove 91 theorems in algebra, geometry, number theory, combinatorics and inequalities at Chinese National High School Mathematics League level. A verifier model scores the proofs.

Models build and improve their own codebases over many rounds, then compete in arenas such as Halite, poker and BattleSnake. Results give Elo ratings.

Models reason across several videos at once in 10 tasks, with 9,015 questions on 5,331 videos in single-choice, multiple-choice and open-ended formats.

Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness.

Provers write Lean proofs for 100 abstract and commutative algebra problems at honors course or graduate exam level. The Lean compiler checks each proof.

Provers write Lean proofs for 100 algebra problems at or beyond PhD qualifying exam level, some beyond what Mathlib covers. The Lean compiler checks each proof.

Agent memory systems are tested for hallucinations at each operation, memory extraction, updating and question answering, rather than only end to end.

Models grade 1,000 olympiad proof attempts, and their grades are compared with human expert grades. Tests automatic evaluation of long-form math answers.

Models write proofs for 60 olympiad problems, split into a basic set and an advanced set up to IMO-hard. Each proof is graded from 0 to 7 by experts or a Gemini autograder.

Models answer 430 long-tail professional questions drawn from real forum discussions in 7 academic and industrial domains.

Models solve MathVision style problems shown in photos taken in natural settings, judging from the name. The Qwen3-VL report gives no citation or further description.

Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy.

Models hold multi-turn therapy conversations with simulated patients designed with licensed clinical psychologists. An LLM judge scores each session.

Models answer 1,173 expert-annotated vision-grounded questions in 11 task types grouped as spatial, geometric and visual knowledge reasoning.

Users upload a document, compare two anonymous OCR or vision-language models side by side, and vote. Votes become Elo ratings.

Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch.

Models summarize over 7,700 private articles of up to 32K tokens using only facts in the text. Vectara's commercial HHEM model flags hallucinated summaries.

Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance.

Agents build web applications from scratch from a natural-language spec in OpenHands. The built apps are tested against functional requirements.

Models answer the 2,362 VSI-Bench questions left after removing 2,768 that text-only models could solve from answer priors or world knowledge.

Video models watch arbitrarily long streams and must recall where objects appeared (VSR) or keep counting objects across scenes (VSC). Video lengths run from 10 minutes to 4 hours.

Computer-use agents complete 609 hand-verified tasks on live websites across 11 categories, such as bookings, real estate and job search. A companion set of 111 harmful tasks tests refusals.

Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused.

Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy.

Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format.

Experts test models on ten cognitive domains from Cattell-Horn-Carroll theory, such as reasoning, memory and perception, with adapted human psychometric tests. Each domain counts 10% toward a score out of 100%.

Frontier models trade live markets with real money over a season, and the leaderboard ranks them by returns.

Models answer probing questions about coherent multi-session user conversations of 128K, 500K, 1M and 10M tokens. Questions test ten memory abilities such as knowledge updates, temporal reasoning and contradiction resolution.

Users ask for code, both models' programs run in a sandbox, and users try the results before voting for the better answer. Votes give Elo ratings.

Researchers measure biorisk-relevant capability kept by open-weight DNA and protein foundation models, with three task sets named Gen, Mut and Vir.

Models are asked to prove false versions of 2025 competition problems. An LLM judge counts how often a model writes a proof instead of rejecting the statement.

Code agents write programs that play board and card games in repeated tournaments. Between rounds they refine their code through self-reflection and by studying rivals' code, and scores separate first-round skill from improvement.

Models solve 50 research-level condensed matter theory problems written and checked by a panel of expert physicists. Answers are graded automatically.

Agents answer real agent requests from Cursor engineers in Cursor's harness. Graders compare against hand-curated solutions and check use of the codebase's existing abstractions.

Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution.

An index that fits model capability and benchmark difficulty from scores on over 50 benchmarks in Epoch AI's hub, using item response theory. Scores read like Elo ratings.

An automated examiner holds real-time multi-turn audio conversations with the model over WebRTC or WebSocket. A model judge scores task completion under conversational constraints.

Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts.

A dynamic fork of Humanity's Last Exam where the makers remove, fix and re-add questions over time and publish an update log.

LIBERO policies run 10,030 perturbed tasks across seven dimensions: layout, camera, robot start state, language, light, background and sensor noise. Success rate.

Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports.

Vision language models answer 168 multiple-choice problems from the 2025 Math Kangaroo, given as screenshots across six grade groups. Many problems need reading a diagram.

Models answer 5,000 English and Chinese math questions whose solutions need visual reasoning, such as drawing auxiliary lines or plotting functions. Answers are checked against ground truth.

Models solve 3,000 math problems that call for interleaved visual and textual solutions, such as drawing auxiliary lines in geometry.

Vision-language models read values from images of measuring instruments, such as gauges and rulers, in real and synthesized photos. Scored by matching the reading to the key.

Models answer tacit knowledge and troubleshooting questions on wet-lab workflows for proxy biological agents. Has 200 single-answer and 100 multiple-answer questions.

Models answer 400 questions on sequence prediction, correction and missing steps in 60 expert protocols for 15 low-risk proxy agents.

Multimodal models solve 1,260 items from 42 synthetic tasks that need long reflective reasoning with backtracking.

Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10.

Coding agents with their models do real Next.js tasks. The board reports success rate with and without an AGENTS.md file, plus duration and cost.

Models write detailed captions of audio, visual and audio-visual clips. A cloze test on the caption content scores the detail.

Models answer 1,000 questions about videos that need audio and visual evidence together. Each question has annotated reasoning steps.

Code agents build 50 real-world Python projects across 20 domains from product requirement documents. A trained judge agent checks each project against structured criteria.

Models and agents forecast open Kalshi prediction-market events and are scored once the events resolve. The live board ranks a difficulty-adjusted Arena score and also reports calibration error and market return.

Trained conversationalists hold three minute live video calls with a model. Real users rate each session for naturalness and fluency on a 0 to 3 scale.

Models point to an object or free space described by multi-step spatial relations, in more indoor scenes plus new outdoor streets and parking lots. Scored by mask hit.

Embedding models retrieve documents on open and private datasets across 20 languages and domains such as law, healthcare, code and finance. Scored by NDCG@10.

Models solve synthetic multi-key and multi-value needle tasks and QA tasks at basic, easy, medium and hard levels. Contexts run from 8K to 1M tokens, and accuracy is averaged across lengths.

Models grade handwritten undergraduate work in PDEs, linear algebra and computer science against rubrics. Scores compare their deductions with expert graders.

Models understand, edit and generate SVG graphics in domains such as icons, illustrations and chemistry diagrams. Generation is scored by comparing rendered output with references.

Model writing is scored by how often it uses words and patterns that are far more common in AI text than in human text. Lower means less slop.

Models track particles on a 2D grid through 10 to 200 text-described moves, then report final positions and orientations. The 330 samples test allocentric and egocentric spatial reasoning.

Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score.

Agents get daily prices, fundamentals and news over months and decide to buy, sell or hold. Scoring uses cumulative return, maximum drawdown and related metrics.

Models act as tutors on three tasks: adaptive explanations for a confused student, feedback on student work, and hints. Graded against sample-specific rubrics.

Models answer 1,250 human-written omni-modal questions and 2,480 uni-modal questions across 44 task types and 5 modality mixes. Multiple choice and multi-step open questions are scored, the open ones by a scoring model.

Vals AI's first index of agentic work in finance, law and coding, with each sector weighted by its share of U.S. GDP.

Models are scored on a GDP-weighted mix of seven Vals finance, coding and education benchmarks, including MortgageTax and SAGE, which grades handwritten math work.

Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics.

Models judge whether visual outcomes in multimodal reasoning are correct, across 16 task categories such as boxes, points, counts, mazes, GUIs, charts and LaTeX.

Screenshot-only agents do short web subtasks, such as date pickers and scroll containers, on real sites replayed from Web Archive snapshots. A program checks the end state.

Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key.

Models complete expert-written work cases with source documents for four jobs: investment banking, consulting, big law and primary care. A judge model grades each response against rubrics.

Version 3 of Artificial Analysis's index, ten evaluations that added agentic terminal and tool-use tasks to the knowledge, math and coding tests.

Models turn about 20 interior photos of each of 50 apartments into a 2D floor plan. Scored by similarity of room connectivity and size ranking to the true plan.

Models read analog clock faces and answer questions about the time shown. Most clocks stay private to avoid training leakage.

Agents perform refactor-style tasks taken from large, established repositories in Python, Go and OCaml. One example threads a ctx variable through 232 files and 3,541 lines of Gitea.

Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall.

Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies.

Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage.

Models answer spatial reasoning questions about ego-centric multi-view scenes, such as those from self-driving cars. Some items ask for distances, scored by error.

Vision-language models solve GSM8K word problems that are redrawn as multi-image visual scenes, so the facts must be read from pictures.

Models judge whether 230 Python programs translated from SV-COMP and TPDB terminate. A divergence claim must give an input that times out after 5 seconds.

Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points.

Agents with web search and SageMath write proofs for private PhD-level problems written by mathematicians. Experts grade the proofs, and final-answer subquestions are graded automatically.

Models follow 1,012 Chinese and English instructions that conflict with habits learned in fine-tuning, such as writing code without comments. A model judge scores compliance.

Models solve 250 math problems written with six professional mathematicians. The sets are 150 regular high school problems, 50 at about AIME level and 50 multimodal problems with figures.

Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category.

Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation.

Models answer the final turn of MultiChallenge conversations converted to speech, with speech-to-text and speech-to-speech versions. Xiaomi built the conversion in house.

Streaming video models answer real-time questions about autonomous driving footage.

Models parse 1,355 PDF pages into text, tables and formulas. The Overall score averages text accuracy from edit distance, table TEDS and formula CDM.

Models solve recent Project Euler problems that need both mathematical insight and code. MathArena adds problems as Project Euler releases them, starting from problem 943.

Models write proofs for 96 university problems in 8 sets of 12, in the style of the easier and middle Putnam tiers. Each problem has a Lean statement and a point rubric for grading.

Multimodal models make a single spot diagnosis from images alone on 50 hard radiology cases. Board-certified radiologists scored about 83%.

Agents write CUDA kernels that are tested across several weight initializations and input shapes, so shape and magnitude exploits fail. Correctness checks and profiling score each kernel.

Models perform scientific sequence and text tasks such as format translation, extraction, property prediction and molecule design, drawn from up to 103 tasks.

Models understand and generate scientific time series in 43 tasks across 12 domains. Signals range from one to ten million points. Each task uses its own metric.

Users of Scale's Playground app chat with a model and sometimes compare its reply with a random opponent's reply. Scale ranks models with a Bradley-Terry model that controls for length and formatting. Rankings can be split by region, profession and age.

Models solve 200 procedurally generated graph problems, such as minimum vertex cover, given as node and edge lists in text. Answers are checked against verifiable ground truth.

Embodied reasoning models judge from robot camera images or video whether a manipulation task succeeded. Later versions use raw video to catch mid-execution failures.

Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages.

GUI agents play 15 browser games such as 2048, Infinity-Loop and Snake-solver from screenshots. Each game score is divided by a human score, and the mean is reported.

Models search high-resolution images for small targets to answer questions, often over many crop and zoom turns. Answers are checked against a key.

Language models turn text or image prompts into voxel structures. Voters compare anonymous builds from the same prompt, and Glicko-2 ratings rank the models.

Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms.

Models write solutions to problems from premier programming contests such as IOI and ICPC. Expert-validated test suites judge each solution.

Audio-language models are scored across 10 aspects with standard prompts. The suite combines existing datasets with two new ones, PARADE for stereotypes and CoRe-Bench for conversational reasoning.

Version 2.2 of Artificial Analysis's index, which swapped the older math tests for AIME 2025 and added instruction following and long-context reasoning.

Models solve automatically generated hard coding problems across many programming languages. A sandbox runs generated tests on each solution.

Models play the deck-building roguelike Balatro through an API bot. The leaderboard compares how far each model and strategy gets in a run.

Models solve 181 real tickets from a 1.1 million line Rust codebase. Model judges compare solutions in pairs, and a Bradley-Terry fit gives each model a strength score.

Models answer BrowseComp questions from a context of 128K to 256K tokens built from the needed web pages and related distractor pages. Scored by accuracy.

Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key.

Models answer private questions over recent Canadian court decisions, such as finding key cases and answering across several documents.

Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code.

Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs.

Agentic vision-language models act in simulated physics environments of varied difficulty that test understanding of basic physical principles.

Agents predict upcoming events collected daily from 195 websites in 11 domains, within a one-week window. Answers are scored once real results are crawled, with harder tiers weighted more.

Language models play full chess games against each other with the board given as FEN and PGN text. Ratings come from many games between each pair.

Code agents solve 54 real tasks across 7 modalities and 7 domains by using a given GitHub repository, from understanding and setup to running it.

Models write proofs for the 10 problems of the 2025 International Mathematics Competition for University Students. Human judges grade each proof.

Agents solve International Olympiad in Informatics problems from 2024 and 2025 in C++. A submission tool modeled on the contest grades up to 50 tries per problem by subtask.

Language models play each other in board, card and word games through a text harness. One leaderboard ranks models across all Game Arena games.

Models read and reason over Korean text in images across diverse visual contexts. It is a text-rich visual question answering test for Korean.

Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal.

Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs.

Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning.

Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path.

Models answer 5,305 expert-written questions about speech, sounds, music and their mixes, covering 49 listening skills. Some clips run up to 10 minutes.

Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key.

Speech-to-speech models answer an audio-friendly subset of MultiChallenge multi-turn questions that OpenAI converted from text to speech. Scored on instruction following accuracy.

Models get requests in scenarios that closed and open models commonly refuse. The score is the percent of questions the model answers, averaged over 5 trials.

Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements.

Speech role-playing agents talk in character across 98 roles. A model judge scores interaction ability, speech expressiveness and role fidelity from the generated speech.

Models chat for 20 turns with a simulated suggestible user played by Kimi-K2. A judge logs protective and risky behaviors such as pushback, sycophancy and delusion reinforcement, which combine into a safety score.

Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation.

Models generate code inside real GitHub projects at spots taken from CVE patches. Static analysis then checks whether the generated code is secure.

Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them.

Models answer hard questions about 40 long-form audio recordings. Mistral built it in-house to test audio understanding beyond transcription.

Models complete over 1,500 repository-level Python tasks covering development, bug fixing and test-driven work, with single-function and multi-function variants. The CorePipe pipeline builds tasks from repositories at set difficulty levels.

Users give a prompt, and anonymous models each build a website, UI component, browser game, data visualization or 3D scene as one HTML file. Users vote, and votes become Bradley-Terry ratings.

Models answer questions about long synthetic outpatient health records in FHIR format. Questions cover 42 types in 10 categories and need multi-hop, temporal and arithmetic reasoning.

Models grade diabetic retinopathy in a fundus photo on a 5-level scale, posed as a 5-option question. Google's test set has 3,614 images, one per patient.

Full-duplex speech models are tested on overlap scenarios: listener backchannel, side conversation and ambient speech. Metrics cover how the model yields, continues or responds.

Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality.

Models solve the written theoretical problems of the 57th International Chemistry Olympiad, held in Dubai in July 2025. Answers are graded against the official marking scheme.

Models write proofs for the 31 problems of the 2024 IMO Shortlist. DeepSeek scores the proofs with its own verifier model.

Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them.

Models build first- to third-year investment banking analyst models, such as three-statement and leveraged buyout models, with formatting and citations. OpenAI grades the outputs internally.

Models solve the theory problems of the 2025 International Physics Olympiad. Answers are graded against the official marking scheme.

Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them.

Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them.

Models answer multiple-choice questions taken from Korean National Professional Licensure exams. It tests professional knowledge used in Korean industry.

Models answer multiple-choice questions from Korean National Technical Qualification exams. The set is rebuilt from KMMLU with erroneous questions removed.

Models answer questions over long Korean contexts. Tasks cover document QA, story and dialogue history understanding, in-context learning, structured QA and RAG.

Memory agents receive long inputs in chunks over many turns and are tested on accurate retrieval, test-time learning, long-range understanding and selective forgetting. It adds two new sets, EventQA and FactConsolidation.

GUI agents are tested at four levels on Windows, macOS, Linux, iOS, Android and the web: content understanding, element grounding, task automation and task collaboration.

Agents fix real GitHub issues in several languages, a 300-instance subset of Multi-SWE-bench chosen for fast evaluation and agent rollouts. Scored by tests passing.

Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language.

Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer.

Users write a music prompt, listen to tracks from two text-to-music systems, and vote. Preferences, listening data and feedback feed a leaderboard.

Vision-language models act as an agent exploring a 3D scene and answer spatial questions from the views seen so far. The set has 10k questions over 1.4k scenes.

Two models bargain over 20 rounds as buyer and seller, each with a hidden private value. Matches across many pairings rank the models.

Provers write Lean proofs for 155 past IMO problems, mostly adapted from Compfiles and MiniF2F.

Models answer 4 to 9 option questions on histopathology patches about cancer identification, grading and subtype. The test split has 450 patches from 354 slides.

Robot policies perform eight bimanual manipulation tasks with controlled variations in simulation. Scores add behavior and progress metrics, such as coordination and efficiency, to task success.

Researchers ask scientific literature questions and vote between two models' long cited answers. Votes give Elo ratings.

Models play Wordle on 100 rare five-letter English words, with six guesses and letter feedback after each guess. The score is the win rate.

Models compute complete US federal tax returns line by line from taxpayer data, with no tax engine. A return counts only when every evaluated line is exact.

Models answer 405 visual questions about small targets in dense SA-1B images and give bounding boxes as evidence. Scored by answer accuracy, with box overlap as a second measure.

Models pick the skin condition in a teledermatology image from 4 options. The set has one image per patient from 1,996 patients and 136 conditions.

Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user.

A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion.

Apple's in-house graders compare two models' responses to locale-specific text and image prompts and pick the preferred one. Results are win, tie and loss fractions per comparison.

Agents automate industrial asset work, such as condition monitoring, maintenance planning and intervention scheduling, using sensor data, failure models and work orders.

Agents try to reproduce 19 past records of the NanoGPT speedrun, given the earlier code and hints of varying detail. Scored by how much of each record's speedup they recover.

LLMs write kernels that override PyTorch core operators. PyTorch's own OpInfo and FACTO tests check correctness, and real model tensor shapes measure speed.

Agents forecast hundreds of already-resolved questions while researching an offline corpus of web pages saved before the resolution. Scored by Brier score.

Models solve ambiguous database tasks by talking with a simulated user, in a conversational mode and an agentic mode. 600 BI and CRUD tasks are checked by executable tests.

Models answer questions about real egocentric videos that probe physical cause and effect, in counterfactual, hypothetical, anticipation, planning and descriptive types.

Models answer Tencent in-house natural language understanding tasks with complex inputs. Task and scoring details are not published.

Models and agents solve hardware design problems in Verilog, such as spec to RTL, code completion, debugging and testbench writing. Simulation tests check the outputs.

Agents answer over 450 data analysis questions from a payments analytics platform. They combine code over data files with reading documentation, and factoid answers are checked automatically.

Web agents answer complex shopping queries on live sites with product attributes, search filters and sorting preferences. An LLM judge checks whether results meet each requirement.

Models follow extremely complex instructions that bundle several tasks and many constraints in one request, as in real product workflows.

Models answer multiple-choice questions on how industrial sensors relate to failure modes of assets. The Kaggle board shows it under IBM's AssetOpsBench citation.

Agents answer expert-written financial reasoning questions grounded in 10-K tables loaded into a database, using SQL tools over many steps. A judge scores correctness and completeness.

Models solve financial numerical reasoning problems that need one or more finance formulas. Answers are numbers checked against reference Python solutions.

Agents generate GPU kernels for LLM inference that are benchmarked on workloads from real serving traces and can be deployed into the FlashInfer stack.

Models write long-form RAG answers to 2,366 questions with human-annotated grounding passages. Scoring checks whether answers use the relevant passages.

Models answer challenging long or repetitive prompts from Mistral's internal set. The score is the share of responses that never stop or loop, so lower is better.

Models write text under detailed instructions in a Tencent in-house text creation test. Task and scoring details are not published.

Text-to-speech models follow natural-language style instructions in English and Chinese across acoustic parameter, descriptive style and role-play tasks. Gemini judges each clip.

Models watch simulated videos and judge whether each event is physically possible, testing object permanence, immutability, continuity and solidity. Scored by accuracy.

Image editing models get 1,026 image and instruction pairs across local edits, global edits, text editing, style reference and character consistency.

Models write text that must meet a requested length, in a Tencent in-house test. Task and scoring details are not published.

Image editing models apply the same user edit request to the same image, anonymously and side by side. Community votes give Elo-style ratings.

Models answer 498 English prompts from Arena-Hard v2.0, translated into 22 more languages for 11,454 prompts in all. An LLM judge compares answers pairwise and reports win rates.

Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution.

Models answer questions about spoken language that need its content, its paralinguistic cues such as emotion and pitch, and its phonology.

Models answer questions embedded in 500 scripted video clips across six reasoning categories, including abstract, physical, planning, spatial and temporal reasoning.

Models answer video questions on spatial, temporal and intuitive physics. Each question comes in a pair of near-identical videos with opposite answers, so text or single-frame shortcuts fail.

Models write programs for 250 original olympiad in informatics problems. Hidden tests check correctness, and time and space completion curves compare efficiency with reference solutions.

Models solve generated competition-style math problems that test exploratory, compositional and transformative generalization beyond training patterns. Answers are checked automatically.

Vision language models answer over 8,400 spatial reasoning questions in four groups: dynamic reasoning, complex spatial logic, spatial interaction and perspective taking.

LLM agents play 12 popular video games across most genres through an MCP interface. Scores come from in-game progress or outcomes per title.

Reward models must score one correct answer above three wrong ones on 1,865 new human prompts. Subsets cover factuality, precise instruction following, math, safety, focus and ties, and the score averages them.

Models answer 830 expert-verified visual questions on scientific data across signal perception, attribute understanding and comparative reasoning.

Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops.

Models compute values over integer sequences built from chains of filters, transforms and operators. The 250 procedurally generated questions test compositional multi-step math.

Models answer more than 5,000 questions about indoor 3D scenes that test 10 atomic and 8 compositional spatial abilities.

Models solve 4K spatial tasks such as 2D and 3D transformations, cube net folding, tangrams, perspective and temporal reasoning that people solve by visual simulation.

Models hold spoken conversations built from real chat data, with prosody, homophones and stuttering. A judge model scores the replies.

Systems translate the WMT25 general task test sets in 16 directions, mostly out of English. New pairs include English to Bhojpuri, Korean and Maasai. Humans evaluate the systems.

Models translate sentence, paragraph and document level text with a supplied source to target glossary. Scoring checks quality and whether the prescribed target terms appear.

Models solve Tencent in-house word level language tasks. Task and scoring details are not published.

Agents search for influencers who fit 50 real advertiser briefs, scored against a curated pool of 836 influencers.

Agents do 50 headhunting tasks from live business cases: company mapping, finding facts about people and finding people who fit a profile. A judge scores answers 1 to 5.

People use Yupp for real tasks, see answers from two or more models, and pick the one they prefer. The preferences become VIBE scores for each model.

Models follow long system prompts from 50 real agentic applications, each with many constraints. Scored by constraint and instruction success rates.

Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars.

Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types.

Models answer 40 final-answer problems from the 2025 Carnegie Mellon Informatics and Mathematics Competition. MathArena scores the average accuracy over four attempts.

Provers solve 100 combinatorics problems formalized in Lean 4, from middle school to IMO and university level. Lean checks each proof.

Agents resume an existing Cascade session with a half-finished task and answer the next user query. A 0 to 10 score blends judge scores for helpfulness, efficiency and correctness with target-file edit accuracy.

Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private.

Models answer 1,378 Chinese STEM questions in physics, chemistry, biology and math that need domain knowledge and evidence from 14 kinds of scientific images.

Text-to-speech models clone voices from in-the-wild reference audio across languages, dialects and emotions. Scored by ASR error rate, speaker similarity and DNSMOS audio quality.

Models answer 1,197 multiple choice questions about everyday videos that need audio and visual events aligned in time. Scored by accuracy.

Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people.

Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer.

Models answer 2,612 visual perception questions on 1,758 images across seven 2D and 3D subtasks drawn from human psychology. Questions are multiple choice or numeric.

Agents start a fresh Cascade conversation from a user intent and must pass a selected set of unit tests. A 0 to 10 score blends test pass rate and a judge score.

Agents get a research question and incomplete starter code from an influential AI paper. They must form hypotheses, design and run experiments, and analyze results.

Provers write Lean 4 proofs for 5,560 formal statements from high-school olympiad problems to undergraduate theorems. The Lean checker scores each proof.

Image editing models carry out edit instructions such as add, remove, replace, style transfer and background change. A GPT-4.1 judge scores each result from 1 to 5.

Image editing models complete 350 free-form edits that need reasoning and world knowledge. GPT-4o scores each output from 0 to 2 against a reference image, normalized to 100.

Models answer 700 expert-written multiple-choice questions on 24 Korean linguistic phenomena in syntax, semantics, pragmatics, phonology and morphology.

Image editing models follow 1,267 instructions that need factual, conceptual or procedural knowledge across 22 tasks. A GPT-4o judge scores the edits.

Bimanual robot policies run 49 manipulation tasks in the Drake simulator from random starting conditions. Scored by success rate per task.

Models answer 7,537 questions from 340 law exams at Swiss, EU and international law courses, in English and German. Open answers are judged against reference answers, and multiple-choice items are exact match.

Models build the longest chain of distinct English words from a given start word, each differing from the last by one letter. The score sums valid chain lengths over all start words.

Models play platformer, puzzle and narrative games such as Sokoban, Tetris, 2048 and Candy Crush through a Gym-style API with perception and memory scaffolds.

Models plan, revise and then write a short story or novella over eight 1,000-word turns from a minimal prompt. A judge model scores quality, and slop, repetition and degradation are tracked.

Models ground many objects at once by outputting boxes for named categories in LVIS images.

Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate.

Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate.

Models handle long multi-turn dialogues built from play-by-play sports commentary. Tasks test ultra multi-turn, interactive multi-turn and cross-turn reasoning.

Reasoning models solve 420 math problems from GSM8K, MATH-500, Minerva, Olympiad and AIME while obeying added verifiable instructions. Code checks the instructions and the answer.

Text-to-speech models clone two Common Voice speakers per language and read 100 test sentences in each of 24 languages. Scored by word error rate and speaker similarity.

Models answer 1,000 questions about real-world audio of speech, sound, music and their mixes. Questions span four reasoning layers from signal to cultural knowledge.

Models answer multiple-choice questions about speech, sound and music clips that need expert knowledge and reasoning. Scored by accuracy on test-mini or the hidden-answer test set.

Multimodal models answer logic questions grouped into inductive, deductive and abductive reasoning.

Models answer 2,000 questions about text that appears in 1,464 videos, across 25 tasks from recognition to reasoning.

Multimodal embedding models are scored on 78 tasks over images, videos and visual documents. V2 adds visual document retrieval, video retrieval, temporal grounding, video classification and video QA.

Vision-language models solve 13,331 long-context examples in five task types, such as visual RAG, needle search and many-shot in-context learning. Each example comes at five lengths from 8K to 128K tokens.

Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy.

Models answer 1,000 human-written questions that need spatial reasoning across several images. Images come from indoor scans, driving, egocentric video and robot data.

Models answer questions about figures from Earth science papers across the atmosphere, cryosphere, hydrosphere, lithosphere and biosphere. Labs report the multiple choice split.

Models retrieve one or several needles and reason over fictional multi-needle facts in English and Chinese haystacks from 4K to 1M tokens. The overall score is the plain average of three tasks.

Models answer 1,069 human-annotated questions that need reasoning over text-rich images. The questions cover 6 reasoning abilities and 18 practical tasks.

Models map 564 desktop instructions to the right screen element or point, covering text matching, element recognition, layout understanding and fine manipulation.

Models point to the image location a language request describes, in about 1,000 tasks across five reasoning categories. A point inside the target mask counts as correct.

Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk.

Computer-use agents do scientific workflows in desktop software such as ChimeraX, Celestia, GRASS GIS and Lean 4. The apps are modified to expose state for checking.

Models answer more than 12,000 internal image questions over 100 tasks, from perception, OCR and STEM to agents, code and out-of-distribution puzzles. LLM judges grade answers, and the overall is a weighted average.

Models answer multiple-choice spatial questions over single images, multiple images and video, drawn from about 30 vision datasets. Tasks span figural to environmental scale.

Models answer 53 final-answer problems from the 2025 Stanford Math Tournament. MathArena scores the average accuracy over four attempts.

Agents help simulated small business insurance underwriters across six task types in multi-turn chats, using tools and asking the underwriter for missing facts.

Models answer hazardous requests drawn from real regulations in six scientific fields. Scored by how often the model complies with a request it should refuse.

Models generate or convert structured outputs in 18 formats, such as JSON, TOML and HTML. StructEval-T covers text-only formats and StructEval-V renderable ones.

Models solve Sudoku variants with unusual rule sets that need creative, multi-step deduction. Scored by the share of puzzles solved.

Agents resolve recent GitHub issues gathered by an automated pipeline, with 50 new verified Python tasks added each month. Tests in a docker sandbox check each patch.

Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks.

Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed.

Models answer 1,837 questions about 270 suspense short films across seven reasoning tasks. Each question needs several visual clues from different parts of the film.