Tool use
We track 125 benchmarks in tool use, or 136 if you count each version. They run from everyday level to expert level.
Anthropic held the top tool use score 47% of the contested time
That's about 22 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
OpenAI Anthropic ByteDance Seed Z.ai Meta Moonshot AI Qwen MiniMax Inception Labs Google DeepMind Arcee AI Kuaishou Xiaomi Corp xAI DeepSeek Mistral AI
Everyday Most adults, without training
Berkeley Function Calling Leaderboard (BFCL) Passed 85% in Jul 2024, by Llama 3.1-405B
Llama 3.1-405B 88.5% University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Share of contested time at number one
Anthropic 47% OpenAI 34% 
Google DeepMind 6% Moonshot AI 4% 
Qwen 3% Meta 2% Z.ai 1% 
DeepSeek 1%
Google DeepMind 6% Moonshot AI 4% Qwen 3% Meta 2% Z.ai 1% DeepSeek 1% Closed out: 9 benchmarks passed 85%
Meta 2 Moonshot AI 1 Xiaomi Corp 1 Qwen 1 Anthropic 1 Google DeepMind 1 OpenAI 1 
MiniMax 1 - Jul 2026 MCPMark Kimi K3
- Jul 2026 MCP Atlas Muse Spark 1.1
- May 2026 PinchBench 2.0 MiMo-V2.5
- Mar 2026 tau3-bench Qwen3.5 397B-A17B
- Feb 2026 DeepSearchQA Claude Opus 4.6
- Feb 2026 PinchBench Gemini 3 Flash
- Dec 2025 GAIA GPT-5.2
- Oct 2025 tau2-bench MiniMax-M2
- Jul 2024 Berkeley Function Calling Leaderboard (BFCL) Llama 3.1-405B
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
OpenAI Anthropic Salesforce Moonshot AI Qwen Google DeepMind Xiaomi Corp xAI Z.ai Meta
Everyday Most adults, without training
Berkeley Function Calling Leaderboard (BFCL) Replaced by BFCL V4, Jul 2025
Llama-xLAM-2-70b-fc-r 78.5% University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Share of contested time at number one, independent runs only
OpenAI 49% Anthropic 33% Google DeepMind 10% Salesforce 4% Moonshot AI 2% Meta 2% xAI 1%
Google DeepMind 10% Salesforce 4% Moonshot AI 2% Meta 2% xAI 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From everyday to expert level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- tau2-bench
- Berkeley Function Calling Leaderboard (BFCL)
- BFCL V4
- PinchBench
- GAIA
- τ³-bench Banking
- PinchBench 2.0
- Agent Arena
- Claw-Eval
- tau-bench
- VITA-Bench
- LMArena Search Arena
- Galileo Agent Leaderboard
- tau3-bench
- VitaBench 2.0
- AceBench
- Galileo Agent Leaderboard v2
- Meta Agents Research Environments (ARE) / Gaia2
- GTA
- AgentBench FC
- DeepPlanning
- QwenClawBench
- UniClawBench
- ToolSandbox
- CRMArena
- SuperCLUE-Agent
- Chinese Tool-Use Benchmark
- tau2-bench Verified
- FocusBench (Meta)
- T-Eval
- API-Bank
- Customer Support Q&A (ByteDance Seed in-house)
- Gorilla APIBench
- MM-ClawBench
- WANDR
- ComplexFuncBench Audio (OpenAI)
- m-tau-bench (Cohere)
- Mobile Actions
- PAgentBench
- RelyToolBench
- SeedClawBench
- tau2-Noise (LongCat)
- ToolcallFormatIFBench
- VitaBench-Noise (LongCat)
- When2Call
- WorldTravel
- Mistral internal function calling benchmark
- Search triggering
- Template Following generalization (Qwen)
- ToolCall-Badcase
- ToolUse (Qwen)
- AdaPlanBench
- Agent Arena (Gorilla)
- AgentBench
- AI Consumer Index
- AppWorld
- Cresta-Bench
- CRMArena-Pro
- MyClaw Bench
- ToolBench
- UserBench
- WebWalkerQA
School
Students, from grade school to high school
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
- DeepSearchQA
- BrowseComp-ZH
- DeepResearch Bench
- ToolComp
- LiveClawBench
- Hermes Bench
- DRACO
- MM-BrowseComp
- ResearchRubrics
- RWSearch
- BrowseComp-Plus
- LiveResearchBench
- xbench-DeepSearch
- BrowseComp-VL
- DeepConsult
- EnterpriseOps-Gym
- Research-Eval
- Vals Web Search Index
- X Browse (xAI)
- xbench-DeepSearch-2510
- RISE
- DeR2 Bench
- Hy-BrowseComp-Pro2
- Hy-LifeSearch
- Hy-SkillsWorld
- MMSearch-Plus
- ProdBench (Tencent)
- STATE-Bench
- SWebBench
- Agent Behavior Bench
- AgentIF-OneDay
- Deep Research Bench (Moonshot)
- DeepResearch Bench II
- KAET (Kimi Autonomous Execution Tasks)
- Multi-agent search (Anthropic internal benchmark)
- Artificial Analysis Search Index
- ATLAS (Exa)
- AutoResearchBench
- ClawProBench
- Deep Research Bench (FutureSearch)
- DeepResearch Arena
- DuetBench
- EnterpriseRAG-Bench
- GTA-2
- LHAW
- MCP-Bench
- Mind2Web 2
- PlanBench-XL
- Toloka Arena
- TraceTrial-Bench
- TRAIL
- Web Context (Nimble)
- Wiki Live Challenge
- ZClawBench
By kind
Function calling
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Berkeley Function Calling Leaderboard (BFCL) Model emits function calls for single, parallel, multi-turn and agentic scenarios. Scored by AST and execution checks. | Everyday | Checked automatically | 88.5% | In use |
| BFCL V4 Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. | Everyday | Checked automatically | 75% | In use |
| ToolComp Models chain several tool calls to answer 485 human-verified prompts. Most final answers are numbers. ToolComp-Enterprise gives 11 tools and ToolComp-Chat targets general chat tool use. Step labels also support process reward model tests. | Expert | Checked automatically | 70.1% | Not in cards |
| Galileo Agent Leaderboard Models pick and call tools on samples drawn from existing tool-use benchmarks. Galileo scores Tool Selection Quality with an LLM judge and averages the datasets equally. | Everyday | Graded by a model | 0.95 | Not in cards |
| AceBench Models call tools from 4,538 Chinese and English APIs in normal, special, and multi-turn agent cases. Answers are checked without LLM judges or live APIs. | Everyday | Checked automatically | 85.4% | Fading |
| GTA Agents answer 229 human-written queries that need 1 to 4 real deployed tools for perception, operation, logic and creativity, often with image inputs. Queries leave the tools and steps implicit. | Everyday | Checked automatically | 46.6% | Not in cards |
| AgentBench FC Agents work five containerized AgentBench environments, ALFWorld, DB, knowledge graph, OS and WebShop, through function-calling prompts instead of the original text format. | Everyday | Checked automatically | 58.9% | Not in cards |
| SuperCLUE-Agent Chinese language models are tested on agent skills in Chinese tasks. These include tool use with several API calls, task planning and memory over long dialogues. | Everyday | Graded by a model | 78.4 | Not in cards |
| Chinese Tool-Use Benchmark Models pick and call plugins in ReAct format for Chinese prompts. Scores are tool selection accuracy, Rouge-L on tool inputs, and false positive rate. | Everyday | Checked automatically | 98% | Dropped |
| Code Interpreter Benchmark Models use a Python code interpreter for math, data visualization, and general tasks. Scoring covers result accuracy and the share of code that runs. | School | Checked automatically | 81.7% | Dropped |
| ComplexFuncBench Models make function calls for 1,000 travel booking style samples that need several steps, user constraints, implicit parameter reasoning, long parameter values or 128K context. | University | Checked automatically |  61.2% | Dropped |
| Nexus Models write function calls for queries over real-world APIs, including nested and parallel calls. Scoring is the function calling success rate. | University | Checked automatically | 58.7% | Dropped |
| HuggingFace Agent Benchmark Agents write Python code that calls Hugging Face Transformers tools to solve short tasks. Scores cover tool selection, tools used, and code correctness. | School | Checked automatically | 93.5% | Dropped |
| T-Eval Models use tools step by step, and each sub-process is scored separately: instruction following, planning, reasoning, retrieval, understanding, and review. | Everyday | Checked automatically | 76.7% | Dropped |
| API-Bank Models make API calls in dialogues that need tools such as calendars and search. Scored on call accuracy. | Everyday | Checked automatically | 92.3% | System cards only |
| Gorilla APIBench Models write calls to machine learning APIs from Hugging Face, TorchHub and TensorHub. Scored by AST match. | Everyday | Checked automatically | 35.3% | System cards only |
| C3-Bench Agents complete multi-task tool use dialogues built to stress tool dependencies, long context dependencies and frequent switches between policy types. | University | Checked automatically | 63.5% | Dropped |
| ComplexFuncBench Audio (OpenAI) Speech-to-speech models solve ComplexFuncBench function-calling tasks whose text prompts OpenAI converted into speech. Covers multi-step calls, constraints, implicit parameters and long inputs. | Everyday | Checked automatically | 66.5% | Dropped |
| Mobile Actions Model converts natural-language phone requests into on-device function calls (calendar, email, maps, flashlight, Wi-Fi). Call accuracy. | Everyday | Checked automatically | 88.4% | Fading |
| RelyToolBench Agents answer queries with simulated APIs built on the StableToolBench server. Scoring targets tool hallucinations, such as calling wrong or missing tools. | Everyday | Graded by a model |  69.3% | In use |
| ToolcallFormatIFBench Models make tool calls that must follow format rules across many agent scaffolds and tool sets. Qwen uses it to measure real-world tool call following. | Everyday | Checked automatically | 86.1% | Fading |
| When2Call Models decide when to call a tool, when to ask a follow-up question and when to say the tools cannot answer. Scored on multiple-choice and LLM-judged test sets. | Everyday | Checked automatically | 60.5% | In use |
| Mistral internal function calling benchmark Models call functions on Mistral's in-house test set. The Magistral paper does not describe the tasks or the metric. | Everyday | Checked automatically |  87.4% | System cards only |
| Search triggering Models decide whether to call search for queries that need fresh facts or answer directly when search is unneeded. Reports F1. | Everyday | Checked automatically | 88% | System cards only |
| Template Following generalization (Qwen) Models follow system prompts and emit exactly formatted XML or JSON tool calls for five scaffolds derived from popular IDE and CLI coding agents. | Everyday | Checked automatically | 92.7% | System cards only |
| ToolCall-Badcase Models choose a tool and its arguments for 200 cases drawn from real user failure reports. Each case has a ground-truth call. | Everyday | Checked automatically | 95.8% | System cards only |
| ToolUse (Qwen) Models make single-turn, multi-turn and multi-step tool calls. The score combines intent recognition, format and parameter accuracy. | Everyday | Checked automatically | 85.5% | System cards only |
| PlanBench-XL Tool-use agents solve 327 retail tasks over 1,665 tools that they must first retrieve, chaining calls to uncover evidence. An optional blocking mode simulates tool failures. | Expert | Checked automatically | Not in cards | |
| ToolBench Models plan and call real REST APIs from RapidAPI, over 16,000 of them, to answer instructions that need one or many tools. ToolEval judges pass rate and win rate. | Everyday | Graded by a model | Not in cards | |
| VisualToolBench Vision-language models solve 1,204 image tasks with tools, half single-turn and half multi-turn, graded per turn against weighted rubrics. | University | Graded by a model | Not in cards |
Customer service
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| tau2-bench Dual-control customer-service simulations (retail, airline, telecom) where agent and simulated user both act. Graded on final state. | Everyday | Checked automatically | 98.9% | In use |
| τ³-bench Banking Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. | Everyday | Checked automatically | 55.1% | In use |
| tau-bench Agents chat with a simulated user and call retail or airline APIs under policy rules. A task passes if the final database matches the goal. | Everyday | Checked automatically | 70.5% | Dropped |
| VITA-Bench Agents handle multi-turn requests in simulated food delivery, in-store, and travel services with 66 tools. A rubric-based judge scores task success. | Everyday | Graded by a model | 32.5% | Fading |
| tau3-bench Agents use tools and follow policies while talking with simulated users in airline, retail, telecom, and banking domains. Scoring is task success rate. | Everyday | Checked automatically | 96% | In use |
| VitaBench 2.0 Agents serve one simulated user across a sequence of life-service subtasks over days or months. They must infer, use and update the user's preferences, judged by rubrics. | Everyday | Graded by a model | 50.3% | In use |
| Galileo Agent Leaderboard v2 Agents handle simulated multi-turn customer support conversations in banking, healthcare, insurance, investment and telecom, with synthetic personas and domain tools. Scores are Action Completion and Tool Selection Quality. | Everyday | Graded by a model | 62% | Not in cards |
| CRMArena Agents do nine customer-service CRM tasks for service agent, analyst and manager personas inside a real Salesforce org. The org holds 16 linked object types with seeded hidden patterns. | Everyday | Checked automatically | 64.3% | Not in cards |
| tau2-bench Verified Agent serves simulated customers in airline, retail and telecom under policies, with Amazon-fixed faulty tasks from τ²-bench. Pass rate. | Everyday | Checked automatically |  73.5% | Fading |
| Customer Support Q&A (ByteDance Seed in-house) Models answer user questions from noisy enterprise knowledge base results, such as policy questions in e-commerce, government services and telecom. | Everyday | Graded by a model |  72.4% | Fading |
| m-tau-bench (Cohere) Agents handle tau-bench retail and airline customer service tasks translated into Japanese, Korean, Arabic, Spanish and French, plus English. Scored by average pass^1 over 10 runs. | Everyday | Checked automatically |  46.5% | Dropped |
| STATE-Bench Agents work multi-step enterprise tasks in travel, customer support and shopping with a sandbox database, domain tools and a simulated user. They must apply the right policy. | Expert | Graded by a model | 58.9% | Not in cards |
| tau2-Noise (LongCat) Agents run tau2-bench tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Checked automatically |  67.1% | Fading |
| Cresta-Bench Contact-center agents take simulated calls for lending, health and airline lines. Tool calls get deterministic checks and LLM judges grade the rest. | Everyday | Graded by a model | Not in cards | |
| CRMArena-Pro Agents do 19 expert-checked sales, service and quoting tasks in B2B and B2C Salesforce orgs, in single-turn and persona-driven multi-turn settings. It also tests whether agents keep confidential data private. | Everyday | Checked automatically | Not in cards | |
| DuetBench Agents diagnose and improve customer service agents on simulated deployments with procedures, tools and synthetic traffic. A blinded judge compares responses head to head. | Expert | Graded by a model | Not in cards | |
| Toloka Arena Agents handle private simulated enterprise workflows with live databases, API tools and strict business rules across nine domains. The main metric pass^5 needs all five trials to succeed. | Expert | Checked automatically | Not in cards | |
| τ-bench 2.0 Agents serve simulated customers in airline, retail, telecom and banking domains in six languages, by text or voice. Release 2.0 adds frozen task manifests and an intake domain for entity capture. | University | Checked automatically | Not in cards |
Apps and MCP servers
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| Toolathlon 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | University | Checked automatically | 56.5% | In use |
| MCP Atlas Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. | University | Graded by a model | 88.1% | In use |
| AutomationBench Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. | University | Checked automatically | 53.9% | In use |
| MCPMark Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. | University | Checked automatically | 94.5% | In use |
| Toolathlon-Verified 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. | University | Checked automatically | 78.4% | In use |
| MCP-Universe Agents complete tasks through 11 real MCP servers across six domains, such as navigation, repositories, finance, and browsing. Scored by execution-based success rate. | University | Checked automatically | 44.6% | Fading |
| EnterpriseOps-Gym Agents do 1,150 expert-written tasks in eight enterprise domains, such as IT, HR and customer service, using 512 tools over live MCP servers. SQL checks on the final database state score each task. | Expert | Checked automatically |  33.6% | In use |
| Random Complex Tasks (LongCat) Agents solve 100 tasks sampled from a generator that builds random executable tool environments across more than four domains. Scored by Avg@4, averaged over three runs. | University | Checked automatically | 35.8% | Fading |
| VitaBench-Noise (LongCat) Agents run VitaBench life-service tasks after a pipeline injects instruction noise and tool failures into the environment. The score is the average over several noisy copies. | Everyday | Graded by a model | 20.5% | Fading |
| AppWorld Agents write code against 457 APIs of 9 simulated everyday apps to finish 750 tasks for about 100 fictional users. State-based unit tests check the outcome and collateral damage. | Everyday | Checked automatically | Not in cards | |
| Era by Eon Generates a simulated enterprise (Salesforce, Slack, Jira, Zendesk, etc.) with exact ground truth. Agents answer questions through MCP/REST. | University | Checked automatically | Not in cards | |
| MCP-Bench Agents solve multi-step tasks using 28 live MCP servers with 250 tools in areas such as finance, travel, scientific computing and academic search. Scoring covers tool use, coordination and planning. | Expert | Graded by a model | Not in cards |
Deep research
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| DeepSearchQA Questions that need multi-step search and synthesis. Final answers graded against references. | Expert | Checked automatically | 95% | In use |
| LMArena Search Arena People compare answers from two anonymous search-augmented models on their own questions and vote for the better one. Votes become Bradley-Terry ratings. | Everyday | Ranked by votes | 1,258 | Fading |
| BrowseComp-ZH Agents browse the Chinese web to answer 289 multi-hop questions with short, verifiable answers. Scored by accuracy. | Expert | Checked automatically | 82.4% | Fading |
| DeepResearch Bench Agents write cited research reports for 100 PhD-level tasks across 22 fields. Judges score report quality against references and check citation accuracy. | Expert | Graded by a model | 55.3% | Fading |
| DRACO 100 deep-research tasks from real Perplexity queries across 10 domains, graded on about 40 expert rubric criteria each. | Expert | Graded by a model | 88.6% | In use |
| MM-BrowseComp Browsing agents answer 400 hand-written questions whose evidence sits in web images and videos. Each question has a checklist for analysis of the reasoning path. | Expert | Checked automatically | 52% | Fading |
| ResearchRubrics 101 deep-research prompts with 2,593 expert-written weighted rubric criteria (including penalties). Reports graded for grounding, reasoning and clarity. | Expert | Graded by a model | 76.2% | Fading |
| RWSearch Agents search the live web to answer 200 expert-written Chinese questions that need many retrieval steps, filtering and arithmetic. Each question has one objective answer. | Expert | Checked automatically | 82% | Fading |
| BrowseComp-Plus Deep research agents answer 830 BrowseComp questions by searching a fixed corpus of about 100,000 human-verified documents. | Expert | Graded by a model | 67.6% | In use |
| LiveResearchBench Deep research agents write cited reports for user-centric questions that need current information from many live web sources. Rubric-based judges score the reports. | Expert | Graded by a model | 73.7 | Not in cards |
| OmniGAIA Agents answer multi-hop questions that need reasoning and multi-turn tool use across video, audio and image inputs. Questions come from an omni-modal event graph built on real data. | University | Checked automatically | 74% | Fading |
| xbench-DeepSearch Agents search the web to answer 100 Chinese questions that need multi-step retrieval. Scored by accuracy against reference answers. | Expert | Checked automatically |  83.7% | Fading |
| BrowseComp-VL Agents answer BrowseComp-style questions that need web search across both images and text. Answers are short and checked against a key. | Expert | Checked automatically | 59% | Fading |
| DeepConsult Deep research agents answer business and consulting queries. A model judge compares each report pairwise with an OpenAI Deep Research baseline answer. | Expert | Graded by a model | 61.1% | Fading |
| Research-Eval Search-augmented models answer 374 grounded multi-hop questions that need sources across many kinds of websites. An LLM judge checks each answer against a checklist of requirements. | Expert | Graded by a model | 66% | Dropped |
| Vals Web Search Index Agents work Finance Agent v2 and Legal Research Bench tasks with one web search tool as their only outside source. It compares native provider search with Exa and Keenable. | Expert | Graded by a model | 49.1% | Not in cards |
| X Browse (xAI) Agents answer questions that need multi-hop search and browsing on the X platform. Scored by answer accuracy. | Expert | Checked automatically | 58% | Dropped |
| xbench-DeepSearch-2510 Agents answer 100 new Chinese deep search questions that need many web searches and cross-checks. Answers are matched to a key. | Expert | Checked automatically | 56.3% | Fading |
| RISE Agents answer real questions from human experts that need multi-step search and interaction with information-dense web pages. MiniMax has not published the scoring method. | Expert | Checked automatically | 64.3% | Fading |
| WANDR 500 wide-search tasks: find tens to hundreds of entities and facts, then return structured records with citations. | Everyday | Graded by a model | 72.3% | System cards only |
| DeR2 Bench Models reason over supplied scientific documents in a sandbox that separates evidence access from reasoning. Four regimes range from instructions only to relevant documents plus distractors. | Expert | Checked automatically | 58.2% | Fading |
| Hy-BrowseComp-Pro2 Search agents answer hard browsing questions in a Tencent in-house set named after BrowseComp. Task and scoring details are not published. | Expert | Checked automatically | 56.1% | In use |
| Hy-LifeSearch Search agents answer Tencent in-house everyday life questions with web search. Task and scoring details are not published. | Expert | Checked automatically | 49.2% | In use |
| MMSearch-Plus Multimodal browsing agents solve 311 search tasks that need fine-grained visual reasoning, provenance checks and long tool use. Answers are short and checked against a key. | Expert | Checked automatically | 41.4% | In use |
| SWebBench Agents browse the web to retrieve information and do research tasks. iFlytek built it in house. | Expert | Checked automatically | 71.7% | In use |
| Deep Research Bench (Moonshot) Agents answer deep-research queries curated by domain experts. Expert-aligned rubrics grade the reports. | Expert | Graded by a model | 90% | System cards only |
| DeepResearch Bench II Deep research agents write reports for 132 research briefs in 22 domains, each derived from a review article they may not consult. A judge model checks 9,287 expert criteria. | Expert | Graded by a model | 43.1% | Not in cards |
| Multi-agent search (Anthropic internal benchmark) Agents solve difficult information retrieval problems with web search, alone or as orchestrators that delegate to subagents. | Expert | Graded by a model | 74.8% | System cards only |
| Artificial Analysis Search Index One fixed agent answers DeepSearchQA, BrowseComp and AA-Omniscience questions while the search API provider changes. The index is the equal-weighted mean of the three scores. | Expert | Checked automatically | Not in cards | |
| ATLAS (Exa) Agents search the web to build a full table of every entity that meets a set of conditions, with 2 to 10 attributes each. A row counts only if every cell is correct. | Expert | Graded by a model | Not in cards | |
| AutoResearchBench Agents search the scientific literature, either tracking down one target paper through multi-step probing or collecting every paper that meets given conditions. | Expert | Checked automatically | Not in cards | |
| Deep Research Bench (FutureSearch) Web research agents work on multi-step research tasks against RetroSearch, a frozen copy of 10,000 to 100,000 web pages per task. Answers are compared with answers worked out by people. | Expert | Checked automatically | Not in cards | |
| DeepResearch Arena Deep research agents get research tasks generated from academic seminar transcripts, which reflect expert discussion and reduce data leakage. | Expert | Graded by a model | Not in cards | |
| EnterpriseRAG-Bench Retrieval systems and agents answer 500 questions over a synthetic company's internal documents, slightly over 500,000 of them. | Expert | Graded by a model | Not in cards | |
| Mind2Web 2 Agents browse the live web for 130 long tasks and return answers with citations. Task-specific judge agents built from rubric trees grade correctness and attribution. | Expert | Graded by a model | Not in cards | |
| Web Context (Nimble) Agents with only web search and page reading enrich company records, find matching entities or write research reports. LLM judges check each fact against source pages. | Expert | Graded by a model | Not in cards | |
| WebWalkerQA Agents start from a website's home page and traverse its subpages to find the facts that answer a question. Answers are judged against references. | Everyday | Graded by a model | Not in cards | |
| Wiki Live Challenge Deep research agents write articles on topics of 100 recent Wikipedia Good Articles, which serve as expert references. Scored on 39 writing criteria and factual verifiability. | Expert | Graded by a model | Not in cards |
General assistants
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| PinchBench Real-world tasks (scheduling, email triage, coding, research, files) for an LLM driving an OpenClaw agent. Graded by automated checks and LLM judge. | Everyday | Graded by a model |  94.9% | In use |
| GAIA Agents answer real-world questions that need web browsing, file handling, multimodal input, and tools, at three difficulty levels. Scored by exact answer match. | Everyday | Checked automatically | 91.7% | Fading |
| PinchBench 2.0 Models drive an OpenClaw agent through 148 real-world tasks, including data and log analysis, meeting notes, DevOps, PDFs, research and writing. Graded by checks and a Haiku judge. | Everyday | Graded by a model | 93.5% | Not in cards |
| Agent Arena Arena ranks models as the orchestrator in real Agent Mode sessions on arena.ai. Scores use behavioral signals such as file downloads, disapproval events, retries and steerability. | Everyday | Ranked by votes | 0.21 | System cards only |
| Claw-Eval 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. | Everyday | Graded by a model | 81.4% | In use |
| Claw-Eval Pass^3 Agents complete 300 human-verified tasks in service orchestration, multimodal interaction, and professional dialogue. Pass^3 credits a task only if all three trials pass. | University | Graded by a model | 70.4% | In use |
| LiveClawBench Agents act as OpenClaw personal assistants on 134 multi-step tasks across 10 app domains backed by stateful mock services. A run passes when its reward exceeds 0.8. | Expert | Checked automatically | 76% | In use |
| Meta Agents Research Environments (ARE) / Gaia2 Simulated app environments for agents with time and events. Hosts the Gaia2 benchmark. | Everyday | Checked automatically | 42.1% | Fading |
| Hermes Bench Agents in Hermes Agent do 150 tasks across 25 categories, such as research, diagrams, memory, tool use and safety. Graders check the files and state left behind. | Expert | Checked automatically | 76.7% | Not in cards |
| Claw-Eval-Live Agents complete 105 workflow tasks in 17 families drawn from live ClawHub workflow signals, scored on both the workflow and the final workspace state. | University | Graded by a model | 66.7% | Not in cards |
| DeepPlanning Agents use tools to plan multi-day trips and multi-item shopping orders under time and budget limits. Plans are checked against verifiable constraints. | Everyday | Checked automatically | 62.3% | Fading |
| QwenClawBench Agents complete 100 tasks across 8 domains in isolated OpenClaw workspaces drawn from real user patterns. Automated checks and an LLM judge score each task. | Everyday | Checked automatically | 64.3% | Fading |
| UniClawBench Agents act as proactive assistants on real-world tasks with live tools across many turns. Tasks are grouped by the capability they test, so failures can be traced to one skill. | Everyday | Checked automatically | 47.5% | In use |
| ToolSandbox Models use stateful tools in multi-turn conversations with a simulated user. Tools have hidden state dependencies, and runs are scored against milestones. | Everyday | Checked automatically | 80.7% | In use |
| WildClawBench 60 original in-the-wild tasks run inside a live OpenClaw assistant environment, chaining 10-60+ tool calls. Scored by task-level graders. | University | Checked automatically | 71% | In use |
| FocusBench (Meta) Agents do personal tasks in domains such as housing, travel, IT and finance. The test checks whether they keep working through real-world blockers across several interactions as the environment changes. | Everyday | Checked automatically | 54.2% | In use |
| MM-ClawBench Agents complete common OpenClaw tasks such as study planning, office documents, scheduled research, and code maintenance. Scoring is accuracy. | Everyday | Graded by a model | 62.7% | Fading |
| Hy-SkillsWorld Agents use skills to complete Tencent in-house working tasks. Task and scoring details are not published. | Expert | Checked automatically | 45.8% | In use |
| KAT Claw Bench Agents complete business tool-use tasks from Kuaishou internal agent needs in seven categories, such as office work, content operations, data analysis and monitoring. Automatic checks grade objective outputs, and model and human review grade open deliverables. | University | Graded by a model | 85.5% | In use |
| PAgentBench Agents act as personal assistants for task management, file operations and system interactions. iFlytek built it in house. | Everyday | Checked automatically | 74.1% | In use |
| ProdBench (Tencent) Agents complete Tencent in-house productivity tasks in the OpenClaw harness. Scored by pass^3. | Expert | Checked automatically | 23% | In use |
| SeedClawBench Agents complete 100 OpenClaw-style user tasks built from Ark usage logs and crowdsourced tasks, such as data analysis and office work. An agent judge checks rubric items against the trace and deliverables. | Everyday | Graded by a model | 66.6% | In use |
| WorldTravel Agents plan trips for 150 real-world scenarios across 5 cities with more than 15 tightly coupled time and logic constraints each. A web environment supplies multimodal travel data. | Everyday | Checked automatically | 47.2% | Fading |
| Agent Behavior Bench Agents are scored on process quality, including tool-use behavior, efficiency and discipline, together with task completion. | Expert | Graded by a model | 65% | System cards only |
| AgentIF-OneDay Agents complete 104 tasks that stand for a full day of human work, life and study. Tasks include tedious workflows, inferring rules from examples and changing requirements. | Expert | Graded by a model | 62.6% | Not in cards |
| KAET (Kimi Autonomous Execution Tasks) Agents execute long-horizon tasks that simulate real user requests and enterprise system operations. | Expert | Checked automatically | 83.5% | System cards only |
| AdaPlanBench Agents plan under world and user constraints that are revealed step by step during interaction, and must adapt plans as new constraints appear. | Everyday | Checked automatically | Not in cards | |
| Agent Arena (Gorilla) Users give a task, two agents built from different models, tools and frameworks run it, and users vote. A Bradley-Terry extension rates models, tools and frameworks. | Everyday | Ranked by votes | Not in cards | |
| AgentBench 8 environments (OS, DB, KG, games, web) testing LLMs as agents. | Everyday | Checked automatically | Not in cards | |
| AI Consumer Index Models with web search handle everyday consumer requests in shopping, food, gaming and DIY. Prompt-specific rubrics with hurdle and grounding criteria grade each answer. | Everyday | Graded by a model | Not in cards | |
| ClawProBench Agents on stateful runtimes are scored with their traces, covering evidence gathering, runtime routing, safety boundaries and repeated runs in frozen workplace setups. | Expert | Checked automatically | Not in cards | |
| GTA-2 Agents face two tiers: the original atomic tool-use queries and new long-horizon workflows with open-ended deliverables. Workflows are graded with recursive checkpoints that split each goal into checkable parts. | Expert | Graded by a model | Not in cards | |
| LHAW Agents face 285 underspecified variants of long-horizon tasks with goals, constraints, inputs or context removed. Scored on strategic clarification and task success. | Expert | Checked automatically | Not in cards | |
| MyClaw Bench Models run inside the OpenClaw agent on 45 tasks in four difficulty tiers built from real usage data. Some tasks use an LLM judge. | Everyday | Graded by a model | Not in cards | |
| TraceTrial-Bench Agent harnesses, not only models, are tested on reliability across seven components, with traces used to attribute each failure to its cause. | Expert | Checked automatically | Not in cards | |
| TRAIL Models read long agent workflow traces and must find and classify the errors in them, using a formal taxonomy of agent error types. The set has 148 large human-annotated traces from established agent benchmarks. | Expert | Checked automatically | Not in cards | |
| UserBench Agents help simulated users who start with vague goals and reveal preferences bit by bit. Agents must ask clarifying questions and use tools to reach choices that fit the user. | Everyday | Checked automatically | Not in cards | |
| ZClawBench Agents do end-to-end OpenClaw workflows such as environment setup, development, information retrieval, data analysis and content creation, drawn from real OpenClaw use. | Expert | Checked automatically | Not in cards |