AutomationBench
Gemini 4 Carbon 53.9% leads since Oct 2026 In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state.
Gemini cards report the private set from Zapier's leaderboard. Zapier ships 1.0.x patch releases, such as 1.0.6 on 2026-07-31, that fix task fairness and make private tasks harder. It then reruns all models, so the board stays one record.
In the authors’ words
Existing AI benchmarks for software automation rarely combine cross-application coordination, autonomous API discovery, and policy adherence. Real business workflows demand all three: a single task may span a CRM, inbox, calendar, and messaging platform - requiring the agent to find the right endpoints, follow a policy document, and write correct data to each system. To address this gap, we introduce AutomationBench, a benchmark for evaluating AI agents on cross-application workflow orchestration via REST APIs. Drawing on real workflow patterns from Zapier's platform, tasks span Sales, Marketing, Operations, Support, Finance, and HR domains. Agents must discover relevant endpoints themselves, follow layered business rules, and navigate environments with irrelevant and sometimes misleading records. Grading is programmatic and end-state only: whether the correct data ended up in the right systems. Even the best frontier models currently score below 10%. AutomationBench provides a challenging, realistic measure of where current models stand relative to the agentic capabilities businesses actually need.
- Level
- University Undergraduates and entry-level professionals
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Gemini 4 Carbon leads with 53.9%
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 1 Gemini 4 Carbon high 53.9%
- 2 Gemini 4 Argon high 51.3%
- 3 Claude Sonnet 5.5 max 44.8%
- 4 Claude Opus 5.5 max 42.5%
- 5 GPT-6 Astra max 41.4%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon | 53.9% | high | Oct 2026 | Official leaderboard |
| 2 | Gemini 4 Argon | 51.3% | high | Sep 2026 | Official leaderboard |
| 3 | Claude Sonnet 5.5 | 44.8% | max, default fallbacks | Sep 2026 | Official leaderboard |
| 4 | Claude Opus 5.5 | 42.5% | max, default fallbacks | Sep 2026 | Official leaderboard |
| 5 | GPT-6 Astra | 41.4% | max | Sep 2026 | Official leaderboard |
Labs report scores for 41 models in their cards. We plot every source in the charts below.
More effort adds a median 4 points
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Gemini 4 Carbon Gemini 4 Argon | Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Gemini 4 Carbon | high | 53.9% | |||
| Gemini 4 Carbon | medium | 53% | |||
| Gemini 4 Argon | high | 51.3% | |||
| Gemini 4 Argon | medium | 50.1% | |||
| Claude Sonnet 5.5 | max | default fallbacks | 44.8% | ||
| Claude Opus 5.5 | max | default fallbacks | 42.5% | ||
| GPT-6 Astra | max | 41.4% | |||
| GPT-6 Astra | xhigh | 39% | |||
| GPT-6 Astra | high | 37.1% | |||
| Claude Sonnet 5.5 | xhigh | default fallbacks | 36.8% |
5 models have held number one since its release in Apr 2026
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
18 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 28 Oct 2026 | Gemini 4 Carbon | 53.9 | AutomationBench (Zapier) |
| 27 Oct 2026 | LFM3-1.7B | 41.3 | AutomationBench (public) |
| 24 Oct 2026 | Kimi K3.1 Pro | 73.3 | Automation Bench-AA |
| 19 Oct 2026 |  MiniMax-M3.1-Flash | 62.8 | Automation-Bench v1.0.6 |
| 17 Oct 2026 |  DeepSeek V4.1 Flash 1017 | 60.5 | Automation-Bench v1.0.6 |
| 15 Oct 2026 |  Nemotron 3.5 Super VL | 32.4 | AutomationBench-AA [offline] |
| 13 Oct 2026 | LFM2.5-24B-A2B | 27.7 | AutomationBench |
| 13 Oct 2026 |  Ling-3.1-flash | 52.5 | Automationbench (Public) |
| 13 Oct 2026 | Gemini 4 Argon | 51.3 | AutomationBench (Score) |
| 12 Oct 2026 |  Muse Wonder | 39.1 | AutomationBench-AA |
| 12 Oct 2026 | Muse Wonder Full | 45.4 | AutomationBench-AA |
| 6 Oct 2026 |  Mistral Large 4 | 59.9 | Agentic behaviour: Artificial Analysis - AutomationBench |
| 5 Oct 2026 | Beam 1.0 | 37 | AutomationBench public |
| 1 Oct 2026 |  Solar Mini 4 | 22.3 | AutomationBench-AA (Artificial Analysis score quoted by Upstage) |
| 29 Sep 2026 | GPT-6.1 Sol | 36.1 | AutomationBench 1.0.6 (GPT-6.1 Sol Max) |
| 28 Sep 2026 | Claude Sonnet 5.5 | 44.7 | AutomationBench |
| 22 Sep 2026 | GPT-6 Luna | 20.7 | AutomationBench 1.0.6 (GPT-6 Luna max) |
| 22 Sep 2026 | MiMo-V2.6 Pro | 53.1 | AutomationBench v1.0.6 |
| 22 Sep 2026 | Claude Opus 5.5 | 40 | Business workflows AutomationBench |
| 22 Sep 2026 | GPT-6 Sol | 33.2 | AutomationBench 1.0.6 (GPT-6 Sol xhigh) |
| 20 Sep 2026 |  Step 5 Preview | 44 | AutomationBench (public); AutomationBench-AA 51.0 |
| 10 Sep 2026 | DeepSeek V4.1 Flash | 54.8 | AutomationBench (Pass@1) |
| 3 Sep 2026 | GPT-6 Astra | 41.4 | AutomationBench |
| 2 Sep 2026 | Muse Spark 1.3 | 49.6 | AutomationBench (E2E business workflows) |
| 1 Sep 2026 | Claude Fable 5.1 | 31.4 | Business workflows AutomationBench |
| 28 Aug 2026 |  Tencent Hy4 preview | 32.1 | AutomationBench (v1.0.6) |
| 26 Aug 2026 | GLM-5.3-Flash | 48.8 | AutomationBench (v1.0.6) |
| 21 Aug 2026 | DeepSeek-V4-Flash-Vision-Exp | 25.7 | AutomationBench (Public) |
| 14 Aug 2026 | GLM-5.3 | 48.2 | AutomationBench (v1.0.6) |
| 13 Aug 2026 | DeepSeek V4 Pro 0813 | 31.8 | AutomationBench (Public) |
| 13 Aug 2026 | Gemini 3.7 Flash | 30.4 | AutomationBench (Private set) |
| 3 Aug 2026 |  Qwen 3.8 Max | 27.3 | Automation-Bench (Pass@1) |
| 31 Jul 2026 | DeepSeek V4 Flash 0731 | 25.1 | AutomationBench Public |
| 24 Jul 2026 | Claude Opus 5 | 26 | Business workflows AutomationBench |
| 16 Jul 2026 | Kimi K3 | 30.8 | AutomationBench |
| 9 Jul 2026 | GPT-5.6 Luna | 14.9 | AutomationBench |
| 9 Jul 2026 | GPT-5.6 Terra | 15.2 | AutomationBench |
| 9 Jul 2026 | GPT-5.6 Sol | 18.1 | AutomationBench (Sol, max) |
| 30 Jun 2026 | Claude Sonnet 5 | 13.5 | AutomationBench |
| 9 Jun 2026 | Claude Fable 5 | 17.4 | Tool use AutomationBench |
| 28 May 2026 | Claude Opus 4.8 | 15.5 | Automation Bench |
14 system cards report it, from 6 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 13 Oct 2026 | Gemini 4 Argon model card | 51.3 | AutomationBench, Score (results table, p.5) | Capabilities |
| 12 Oct 2026 | Muse Wonder model card | 39.1 | AutomationBench-AA strict pass@1 from Artificial Analysis, Muse Wonder (max) (capabilities table image, page 3) | Capabilities |
| 12 Oct 2026 | Muse Wonder Full model card | 45.4 | AutomationBench-AA strict pass@1 from Artificial Analysis, Muse Wonder Full (max) (capabilities table image, page 3) | Capabilities |
| 28 Sep 2026 | Claude Sonnet 5.5 system card | 44.7 | AutomationBench at max effort with default fallbacks, run by Zapier (Table 8.1.A, 8.14.6) | Capabilities |
| 22 Sep 2026 | MiMo-V2.6 Pro technical report | 53.1 | AutomationBench v1.0.6 (Table 3) | Capabilities |
| 22 Sep 2026 | Claude Opus 5.5 system card | 40 | AutomationBench, private held-out leaderboard set, max effort, run by Zapier (Table 8.1.A, 8.14.6) | Capabilities |
| 10 Sep 2026 | DeepSeek V4.1 Flash technical report | 54.8 | Automation-Bench (Pass@1), public set of AutomationBench v1.0.6 (Table 3) | Capabilities |
| 1 Sep 2026 | Claude Fable 5.1 system card | 31.4 | AutomationBench, private held-out leaderboard set, max effort (Table 8.1.A, 8.15.6) | Capabilities |
| 13 Aug 2026 | Gemini 3.7 Flash model card | 30.4 | AutomationBench, private set (results table, p.5) | Capabilities |
| 27 Jul 2026 | Kimi K3 technical report | 30.8 | AutomationBench, 600-task public subset (Table 2) | Capabilities |
| 24 Jul 2026 | Claude Opus 5 system card | 26 | AutomationBench, private held-out leaderboard set, max effort (24% at medium) (Table 8.1.A, 8.13.7) | Capabilities |
| 30 Jun 2026 | Claude Sonnet 5 system card | 13.5 | AutomationBench, private held-out leaderboard set, max effort, run by Zapier (Table 8.1.A, 8.11.6) | Capabilities |
| 9 Jun 2026 | Claude Fable 5 system card | 17.4 | AutomationBench, private held-out leaderboard set, max effort, measured by Zapier (Table 8.1.A, 8.17.9) | Capabilities |
| 28 May 2026 | Claude Opus 4.8 system card | 15.5 | AutomationBench, Zapier private held-out leaderboard set, max effort (Table 8.1.A, 8.13.8) | Capabilities |
More in Tool use
- MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Muse Spark 1.1 88.1%
- BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes. Qwen3.7-Max 75%
- τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
- Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%
- DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%
- MCPMark In use Agent stress-tests realistic MCP tool use (GitHub, Notion, Postgres, browser). Graded by programmatic verification scripts. Kimi K3 94.5%