StableToolBench
Models solve user requests by calling tools from ToolBench's large set of real-world APIs. A virtual API server with a cache and API simulators stands in for live APIs, so results stay stable over time. GPT-4 judges whether each request was solved and which of two answers is better, giving a solvable pass rate and a win rate.
It evolves from ToolBench, which the almanac carries, and changes the scoring, so its scores are not comparable with ToolBench's. The judge model and the simulated API answers both shape the scores. About 240 GitHub stars.
- Made by
- Tsinghua University and collaborators
- Released
- March 2024
- Kind
- Function calling
- Level
- Everyday
- Scored
- Graded by a model
- Answers
- Agent task
- Tasks
- All public
- Strongest model tested
- GPT-4o mini #177 on the capability index
- Last model added
- July 2024
- Scores refreshed
- November 12th, 2026
In the authors’ words
Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.
GPT 4o (snapshot unspecified) tops the leaderboard with 32%
Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is GPT-4o mini, at 26.4%.
- 1 GPT 4o (snapshot unspecified) 32%
- 2 GPT-4o mini 26.4%
- 3 ToolLLaMA v2 22.9%
Source: Official StableToolBench leaderboard
| # | Entry | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT 4o (snapshot unspecified) | 32% | CoT | Official leaderboard | |
| 2 | GPT-4o mini | 26.4% | DFS | Jul 2024 | Official leaderboard |
| 3 | ToolLLaMA v2 | 22.9% | DFS | Official leaderboard |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Source: Official StableToolBench leaderboard
No lab model card reports it yet
We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.
More in Tool use
- AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state.
Gemini 4 Carbon 53.9% - MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Gemini 4 Argon 90.7%
- BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes.
Qwen3.7-Max 75% - τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
- Toolathlon-Verified In use 108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state. GLM-5.3-Flash 78.4%
- DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%