Benchmark Almanac

τ³-bench Banking

Qwen 3.8 Max 55.1% leads since Aug 2026 In use

Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state.

A separate domain of τ³-bench. Its overall score covers retail, airline and telecom only.

Made by Sierra Released Mar 2026 Customer service All tasks public Last in a launch post Oct 2026 Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Leaderboard
26
Entries on its own leaderboard, 25 models we track
18
Launch posts list it, from 11 labs
11
System cards report it, from 7 labs

Qwen 3.8 Max leads with 55.1% among models released by Aug 2026

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Aug 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

26 entries. Scroll sideways for the rest.

  1. 55.1 Qwen 3.8 Max
  2. 48.7 Claude Opus 5
  3. 47.9 Grok 4.5
  4. 46.9 GPT-5.6 Sol
  5. 44.6 GPT-5.5
  6. 40.5 Muse Spark 1.1
  7. 40.2 Claude Opus 4.7
  8. 39.7 Claude Opus 4.8
  9. 39.7 Claude Fable 5
  10. 39.4 GPT-5.4
  11. 37.1 GLM-5.2
  12. 37.1 Kimi K3
  13. 32.2 GPT-5.2
  14. 27.3 Gemini 3 Flash
  15. 27.3 Claude Opus 4.6
  16. 26.0 Gemini 3.1 Pro
  17. 25.3 Claude Sonnet 4.5
  18. 25.0 Inkling
  19. 24.7 Claude Opus 4.5
  20. 18.0 Gemini 3 Pro
  21. 18.0 Grok 4.2
  22. 15.7 Grok 4 Fast
  23. 13.7 Gemini 2.5 Pro
  24. 13.1 Grok 4.1 Fast
  25. 9.79 GLM-5
  26. 9.79 Qwen3.5 397B-A17B
  1. 1 Qwen 3.8 Max xhigh 55.1%
  2. 2 Claude Opus 5 max 48.7%
  3. 3 Grok 4.5 high 47.9%
  4. 4 GPT-5.6 Sol xhigh 46.9%
  5. 5 GPT-5.5 xhigh 44.6%
  6. 6 Muse Spark 1.1 xhigh 40.5%
  7. 7 Claude Opus 4.7 max 40.2%
  8. 8 Claude Opus 4.8 max 39.7%
  9. 9 Claude Fable 5 max 39.7%
  10. 10 GPT-5.4 xhigh 39.4%
  11. 11 GLM-5.2 xhigh 37.1%
  12. 12 Kimi K3 max 37.1%
  13. 13 GPT-5.2 high 32.2%
  14. 14 Gemini 3 Flash high 27.3%
  15. 15 Claude Opus 4.6 max 27.3%
  16. 16 Gemini 3.1 Pro high 26%
  17. 17 Claude Sonnet 4.5 enabled 25.3%
  18. 18 Inkling max 25%
  19. 19 Claude Opus 4.5 high 24.7%
  20. 20 Gemini 3 Pro high 18%
  21. 21 Grok 4.2 high 18%
  22. 22 Grok 4 Fast high 15.7%
  23. 23 Gemini 2.5 Pro Jun 2025, high 13.7%
  24. 24 Grok 4.1 Fast high 13.1%
  25. 25 GLM-5 enabled 9.79%
  26. 26 Qwen3.5 397B-A17B enabled 9.79%
#EntryScoreSettingDateSource
1 Qwen 3.8 Max 55.1% xhigh Aug 2026 Official leaderboard
2 Claude Opus 5 48.7% max Jul 2026 Official leaderboard
3 Grok 4.5 47.9% high Jul 2026 Official leaderboard
4 GPT-5.6 Sol 46.9% xhigh Jul 2026 Official leaderboard
5 GPT-5.5 44.6% xhigh Apr 2026 Official leaderboard
6 Muse Spark 1.1 40.5% xhigh Jul 2026 Official leaderboard
7 Claude Opus 4.7 40.2% max Apr 2026 Official leaderboard
8 Claude Opus 4.8 39.7% max May 2026 Official leaderboard
9 Claude Fable 5 39.7% max Jun 2026 Official leaderboard
10 GPT-5.4 39.4% xhigh Mar 2026 Official leaderboard
11 GLM-5.2 37.1% xhigh Jun 2026 Official leaderboard
12 Kimi K3 37.1% max Jul 2026 Official leaderboard
13 GPT-5.2 32.2% high Dec 2025 Official leaderboard
14 Gemini 3 Flash 27.3% high Dec 2025 Official leaderboard
15 Claude Opus 4.6 27.3% max Feb 2026 Official leaderboard
16 Gemini 3.1 Pro 26% high Feb 2026 Official leaderboard
17 Claude Sonnet 4.5 25.3% enabled Sep 2025 Official leaderboard
18 Inkling 25% max Jul 2026 Official leaderboard
19 Claude Opus 4.5 24.7% high Nov 2025 Official leaderboard
20 Gemini 3 Pro 18% high Nov 2025 Official leaderboard
21 Grok 4.2 18% high May 2026 Official leaderboard
22 Grok 4 Fast 15.7% high Sep 2025 Official leaderboard
23 Gemini 2.5 Pro Jun 2025 13.7% high Jun 2025 Official leaderboard
24 Grok 4.1 Fast 13.1% high Nov 2025 Official leaderboard
25 GLM-5 9.79% enabled Feb 2026 Official leaderboard
26 Qwen3.5 397B-A17B 9.79% enabled Feb 2026 Official leaderboard

Labs report scores for 20 models in their cards. We plot every source in the charts below.

7 models have held number one since its release in Mar 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Anthropic xAI Qwen NVIDIA Moonshot AI Google DeepMind StepFun Upstage Ant Group
Own leaderboard Not tested since Aug 2026
Qwen 3.8 Max 55.1%
Lab claims 7 leaders
Ling-3.1-flash 47.8%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Claude Opus 5 Claude Opus 4.7 GPT-5.2 Claude Sonnet 4.5 Grok 4 Fast 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Qwen 3.8 Max Claude Opus 5

11 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

NVIDIA
Thinking Machines
Upstage
LG AI Research
Liquid AI
Meta
Google DeepMind
StepFun
Reflection
Ant Group
Inception Labs
DateLaunch postScoreAs the post writes it
19 Oct 2026 Mercury 3 30.2 τ³-bench Banking
15 Oct 2026 Nemotron 3.5 Bolt 74B-A9B 17 Tau3 Banking
15 Oct 2026 Nemotron 3.5 Super VL 18.9 tau3-bench banking
13 Oct 2026 LFM2.5-24B-A2B 8.04 τ³-Bench Banking
13 Oct 2026 Ling-3.1-flash 47.8 TAU3-Banking
5 Oct 2026 Beam 1.0 38 tau3 banking
1 Oct 2026 Solar Mini 4 47.2 τ³-Banking (a separate evaluation, printed as 47.2)
20 Sep 2026 Step 5 Preview 42.5 τ³-Banking
15 Sep 2026 Gemini 3.8 Live Extended Thinking 35.1 τ³-Banking Leaderboard (Sierra), voice setting, High
11 Aug 2026 Nemotron 3.5 Lightning 30B-A3B 9.28 τ³-bench (Banking)
11 Aug 2026 Solar Pro 4 23 τ³-Banking (Artificial Analysis score quoted by Upstage)
10 Aug 2026 Muse Glimmer 23.5 τ3-Banking
4 Aug 2026 LFM2.5-2.6B 5.67 τ³-Bench Banking
31 Jul 2026 K-EXAONE 2.0 14.2 τ3-Banking
30 Jul 2026 Inkling-Small 15.5 Tau 3 Banking
22 Jul 2026 Solar Open 2 19.6 τ³ (banking)
15 Jul 2026 Inkling 23.7 Tau 3 Banking (Artificial Analysis)
4 Jun 2026 Nemotron 3 Ultra 22.6 TauBench V3 Banking

11 system cards report it, from 7 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
19 Oct 2026 Muse Glimmer technical report 23.5 τ3-Banking, high, 26.0 xhigh (Tables 2 and 5) Capabilities
15 Oct 2026 Nemotron 3.5 Super VL technical report 18.9 Tau3 Banking (Table 10) Capabilities
15 Oct 2026 Nemotron 3.5 Lightning 30B-A3B technical report 9.9 Tau3 Banking, BF16 acc. (Table 22) Capabilities
15 Oct 2026 Nemotron 3.5 Bolt 74B-A9B technical report 17 Tau3 Banking, released 24GB checkpoint (Table 7). BF16 17.01 Capabilities
5 Aug 2026 K-EXAONE 2.0 technical report 14.2 tau3-Banking (Table 6, Reasoning mode) Capabilities
30 Jul 2026 Inkling-Small model card 15.5 Tau 3 Banking (5. Evaluations table) Capabilities
27 Jul 2026 Kimi K3 technical report 33.4 τ3-Banking, cited from Artificial Analysis (Table 2) Capabilities
22 Jul 2026 Solar Open 2 technical report 19.6 τ3 (banking) (Table 3) Capabilities
15 Jul 2026 Inkling model card 23.7 Tau 3 Banking (5. Evaluations table, effort 0.99) Capabilities
14 Jul 2026 Grok 4.5 model card 32.6 τ³-banking (AA), accuracy, high (4.2) Capabilities
9 Jun 2026 Nemotron 3 Ultra technical report 22.6 TauBench V3 Banking (Table 10) Capabilities

More in Tool use

All 125 in Tool use
  • AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state. Gemini 4 Carbon 53.9%
  • MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Muse Spark 1.1 88.1%
  • BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
  • Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%
  • DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
  • MCPMark In use Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. Kimi K3 94.5%