Benchmark Almanac

Toolathlon-Verified

GLM-5.3-Flash 78.4% leads since Aug 2026 In use

108 long-horizon tasks across 32 apps through about 600 MCP tools. Execution-based checkers inspect final environment state.

A checked subset of Toolathlon tasks.

Made by HKUST NLP Released Jun 2026 Apps and MCP servers All tasks public Last in a launch post Oct 2026 Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Leaderboard
25
Models scored on its own leaderboard
7
Launch posts list it, from 6 labs
9
System cards report it, from 5 labs

GLM-5.3-Flash leads with 78.4% among models released by Aug 2026

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best.

The newest model with a score here came out in Aug 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

25 models. Scroll sideways for the rest.

  1. 78.4 GLM-5.3-Flash
  2. 76.5 Kimi K3
  3. 76.2 Claude Opus 4.8
  4. 75.9 Muse Spark 1.2
  5. 75.6 Muse Spark 1.1
  6. 74.4 DeepSeek V4 Pro 0813
  7. 73.5 GPT-5.5
  8. 71.6 Claude Sonnet 5
  9. 70.7 DeepSeek V4 Flash 0731
  10. 67.3 Gemini 3.5 Flash
  11. 61.1 Gemini 3.1 Pro
  12. 59.9 GLM-5.2
  13. 58.0 Kimi K2.6
  14. 58.0 Kimi K2.7 Code
  15. 57.1 Gemini 3.5 Flash-Lite
  16. 56.8 Tencent Hy3
  17. 55.9 DeepSeek-V4-Pro
  18. 54.4 Inkling-Small
  19. 50.9 DeepSeek-V4-Flash
  20. 49.1 MiMo-V2.5
  21. 47.5 MiniMax-M2.7
  22. 45.5 Inkling
  23. 40.7 Qwen3.5 397B-A17B
  24. 34.3 Nemotron 3 Ultra
  25. 33.0 Kimi K2.5
  1. 1 GLM-5.3-Flash max 78.4%
  2. 2 Kimi K3 max 76.5%
  3. 3 Claude Opus 4.8 max 76.2%
  4. 4 Muse Spark 1.2 xhigh 75.9%
  5. 5 Muse Spark 1.1 xhigh 75.6%
  6. 6 DeepSeek V4 Pro 0813 max 74.4%
  7. 7 GPT-5.5 xhigh 73.5%
  8. 8 Claude Sonnet 5 max 71.6%
  9. 9 DeepSeek V4 Flash 0731 max 70.7%
  10. 10 Gemini 3.5 Flash high 67.3%
  11. 11 Gemini 3.1 Pro high 61.1%
  12. 12 GLM-5.2 max 59.9%
  13. 13 Kimi K2.6 58%
  14. 14 Kimi K2.7 Code 58%
  15. 15 Gemini 3.5 Flash-Lite high 57.1%
  16. 16 Tencent Hy3 high 56.8%
  17. 17 DeepSeek-V4-Pro max 55.9%
  18. 18 Inkling-Small xhigh 54.4%
  19. 19 DeepSeek-V4-Flash max 50.9%
  20. 20 MiMo-V2.5 49.1%
  21. 21 MiniMax-M2.7 47.5%
  22. 22 Inkling xhigh 45.5%
  23. 23 Qwen3.5 397B-A17B 40.7%
  24. 24 Nemotron 3 Ultra 34.3%
  25. 25 Kimi K2.5 33%
#ModelScoreSettingDateSource
1 GLM-5.3-Flash 78.4% max Aug 2026 Official leaderboard
2 Kimi K3 76.5% max Jul 2026 Official leaderboard
3 Claude Opus 4.8 76.2% max May 2026 Official leaderboard
4 Muse Spark 1.2 75.9% xhigh Aug 2026 Official leaderboard
5 Muse Spark 1.1 75.6% xhigh Jul 2026 Official leaderboard
6 DeepSeek V4 Pro 0813 74.4% max Aug 2026 Official leaderboard
7 GPT-5.5 73.5% xhigh Apr 2026 Official leaderboard
8 Claude Sonnet 5 71.6% max Jun 2026 Official leaderboard
9 DeepSeek V4 Flash 0731 70.7% max Jul 2026 Official leaderboard
10 Gemini 3.5 Flash 67.3% high May 2026 Official leaderboard
11 Gemini 3.1 Pro 61.1% high Feb 2026 Official leaderboard
12 GLM-5.2 59.9% max Jun 2026 Official leaderboard
13 Kimi K2.6 58% Apr 2026 Official leaderboard
14 Kimi K2.7 Code 58% Jun 2026 Official leaderboard
15 Gemini 3.5 Flash-Lite 57.1% high Jul 2026 Official leaderboard
16 Tencent Hy3 56.8% high Jul 2026 Official leaderboard
17 DeepSeek-V4-Pro 55.9% max Apr 2026 Official leaderboard
18 Inkling-Small 54.4% xhigh Jul 2026 Official leaderboard
19 DeepSeek-V4-Flash 50.9% max Apr 2026 Official leaderboard
20 MiMo-V2.5 49.1% Apr 2026 Official leaderboard
21 MiniMax-M2.7 47.5% Mar 2026 Official leaderboard
22 Inkling 45.5% xhigh Jul 2026 Official leaderboard
23 Qwen3.5 397B-A17B 40.7% Feb 2026 Official leaderboard
24 Nemotron 3 Ultra 34.3% Jun 2026 Official leaderboard
25 Kimi K2.5 33% Jan 2026 Official leaderboard

Labs report scores for 13 models in their cards. We plot every source in the charts below.

3 models have held number one since its release in Jun 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

Anthropic Moonshot AI Z.ai Meta
Own leaderboard Not tested since Aug 2026
GLM-5.3-Flash 78.4%
Lab claims 2 leaders
Claude Opus 5 80.6%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 Claude Opus 4.8 Gemini 3.1 Pro Qwen3.5 397B-A17B Kimi K2.5 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Claude Opus 5 GPT-5.5

6 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Thinking Machines
Poolside
Tencent
StepFun
Xiaomi Corp
Meta
DateLaunch postScoreAs the post writes it
15 Oct 2026 Muse Spark Open 72.8 Toolathlon Verified
22 Sep 2026 MiMo-V2.6 Pro 76.9 Toolathlon-Verified
20 Sep 2026 Step 5 Preview 74.1 Toolathlon-Verified
28 Aug 2026 Tencent Hy4 preview 74.1 Toolathlon-Verified
30 Jul 2026 Inkling-Small 54.4 Toolathlon Verified
21 Jul 2026 Laguna S 2.1 49.7 Toolathlon Verified
15 Jul 2026 Inkling 45.5 Toolathlon Verified

9 system cards report it, from 5 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
28 Sep 2026 Claude Sonnet 5.5 system card 77.8 Toolathlon-Verified Pass@1 over three trials (Pass@3 85.2, Pass^3 68.5) (Table 8.14.5.A) Capabilities
22 Sep 2026 MiMo-V2.6 Pro technical report 76.9 Toolathlon-Verified (Table 3) Capabilities
22 Sep 2026 Claude Opus 5.5 system card 77.8 Toolathlon-Verified Pass@1 over three trials, max effort, production classifiers on (Pass@3 82.4, Pass^3 72.2) (Table 8.14.5.A) Capabilities
1 Sep 2026 Claude Fable 5.1 system card 77.8 Toolathlon Verified Pass@1 over 3 trials, max effort, production classifiers with Opus 4.8 fallback; Pass@3 81.5, Pass^3 73.1 (Table 8.15.5.A) Capabilities
30 Jul 2026 Inkling-Small model card 54.4 Toolathlon Verified (5. Evaluations table) Capabilities
27 Jul 2026 Kimi K3 technical report 76.5 Toolathlon-Verified, cited from the official leaderboard as of 2026-07-24 (Table 2) Capabilities
24 Jul 2026 Claude Opus 5 system card 80.6 Toolathlon-Verified Pass@1 over three trials, max effort, internal harness, no safety classifiers (Pass@3 87.0, Pass^3 73.1) (Table 8.13.6.A) Capabilities
15 Jul 2026 Inkling model card 45.5 Toolathlon Verified (5. Evaluations table, effort 0.99) Capabilities
9 Jul 2026 Muse Spark 1.1 system card 75.6 Toolathlon-Verified (Figure 44) Capabilities

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

Anthropic OpenAI Google DeepMind Meta Moonshot AI Z.ai
Toolathlon-Verified 16 leaders
Claude Opus 5 80.6%
Toolathlon Replaced by Toolathlon-Verified, Aug 2026
GLM-5.3-Flash 78.4%
Toolathlon-Verified 3 leaders
Claude Opus 5 80.6%
Replaces Toolathlon

More in Tool use

All 125 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Muse Spark 1.1 88.1%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%