CommerceAgentBench
Agents finish real e-commerce work across procurement, listings, operations, fulfillment and after-sales through CLI, browser, file and API tools. A task passes only if every verifier check on the end state succeeds.
107 tasks: 53 CLI, 28 browser, 16 file and 10 API or MCP. Tasks were distilled from Accio merchant traces. Verifiers are deterministic or LLM-assisted per task. Formerly RealReplicaBench. Repo has 1,301 stars.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Claude Opus 5 leads with 61.7% among models released by Aug 2026
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
The newest model with a score here came out in Aug 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 12 more
- 1 Claude Opus 5 61.7%
- 2 Claude Opus 4.8 55.1%
- 3 Claude Opus 4.7 52.3%
- 4
Qwen 3.8 Max 52.3% - 5 GPT-5.6 Sol 51.4%
- 6
DeepSeek-V4-Pro 50.5% - 7 GPT-5.5 47.7%
- 8 DeepSeek-V4-Flash 46.7%
- 9 GLM-5.2 46.7%
- 10
Gemini 3.6 Flash 46.7% - 11 GPT-5.6 Luna 44.9%
- 12 Gemini 3.5 Flash 43%
- 13 Gemini 3 Flash 29%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Opus 5 | 61.7% | Accio | Jul 2026 | Official leaderboard |
| 2 | Claude Opus 4.8 | 55.1% | Accio | May 2026 | Official leaderboard |
| 3 | Claude Opus 4.7 | 52.3% | Accio | Apr 2026 | Official leaderboard |
| 4 | Qwen 3.8 Max | 52.3% | Accio | Aug 2026 | Official leaderboard |
| 5 | GPT-5.6 Sol | 51.4% | Accio | Jul 2026 | Official leaderboard |
| 6 | DeepSeek-V4-Pro | 50.5% | Accio | Apr 2026 | Official leaderboard |
| 7 | GPT-5.5 | 47.7% | OpenClaw | Apr 2026 | Official leaderboard |
| 8 | DeepSeek-V4-Flash | 46.7% | Accio | Apr 2026 | Official leaderboard |
| 9 | GLM-5.2 | 46.7% | Pi | Jun 2026 | Official leaderboard |
| 10 | Gemini 3.6 Flash | 46.7% | Accio | Jul 2026 | Official leaderboard |
| 11 | GPT-5.6 Luna | 44.9% | Accio | Jul 2026 | Official leaderboard |
| 12 | Gemini 3.5 Flash | 43% | Accio | May 2026 | Official leaderboard |
| 13 | Gemini 3 Flash | 29% | Pi | Dec 2025 | Official leaderboard |
The best score, 61.7%, costs $3.50 a task
Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5 | Accio | 61.7% | $3.50 | ||
| Claude Opus 5 | Pi | 60.8% | $1.79 | ||
| Claude Opus 5 | OpenClaw | 56.1% | $2.70 | ||
| Claude Opus 4.8 | Accio | 55.1% | $4.17 | ||
| Claude Opus 4.7 | Accio | 52.3% | $3.30 | ||
| Claude Opus 4.8 | Pi | 52.3% | $1.83 | ||
| Qwen 3.8 Max | Accio | 52.3% | $1.28 | ||
| Claude Opus 4.8 | OpenClaw | 51.4% | $3.57 | ||
| GPT-5.6 Sol | Accio | 51.4% | $2.79 | ||
| DeepSeek-V4-Pro | Accio | 50.5% | $0.24 | ||
| DeepSeek-V4-Pro | Pi | 49.5% | $0.20 | ||
| GPT-5.6 Sol | OpenClaw | 49.5% | $3.16 | ||
| Qwen 3.8 Max | Pi | 49.5% | $0.72 | ||
| Claude Opus 4.7 | Pi | 48.6% | $1.86 | ||
| GPT-5.6 Sol | Pi | 48.6% | $1.08 | ||
| GPT-5.5 | OpenClaw | 47.7% | $2.73 | ||
| DeepSeek-V4-Flash | Accio | 46.7% | $0.04 | ||
| GLM-5.2 | Pi | 46.7% | $0.28 | ||
| GLM-5.2 | Accio | 46.7% | $1.25 | ||
| Gemini 3.6 Flash | Accio | 46.7% | $1.34 | ||
| Claude Opus 4.7 | OpenClaw | 45.8% | $3.39 | ||
| GPT-5.5 | Accio | 44.9% | $2.00 | ||
| GPT-5.6 Luna | Accio | 44.9% | $0.73 | ||
| Gemini 3.6 Flash | OpenClaw | 44.9% | $2.92 | ||
| GPT-5.5 | Pi | 43.9% | $1.31 | ||
| DeepSeek-V4-Pro | OpenClaw | 43.9% | $0.31 | ||
| Gemini 3.6 Flash | Pi | 43.9% | $2.03 | ||
| Qwen 3.8 Max | OpenClaw | 43.9% | $0.83 | ||
| DeepSeek-V4-Flash | Pi | 43% | $0.06 | ||
| DeepSeek-V4-Flash | OpenClaw | 43% | $0.07 | ||
| Gemini 3.5 Flash | Accio | 43% | $2.05 | ||
| Gemini 3.5 Flash | Pi | 41.1% | $3.27 | ||
| GLM-5.2 | OpenClaw | 39.3% | $1.07 | ||
| GPT-5.6 Luna | Pi | 39.3% | $0.03 | ||
| Gemini 3.5 Flash | OpenClaw | 36.5% | $2.11 | ||
| GPT-5.6 Luna | OpenClaw | 33.6% | $0.48 | ||
| Gemini 3 Flash | Pi | 29% | $0.61 | ||
| Gemini 3 Flash | OpenClaw | 29% | $0.51 | ||
| Gemini 3 Flash | Accio | 29% | $0.38 |
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release.
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Economic work
- JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
- GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate.
Seed2.1 Pro 87.9% - GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
- AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
- CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Qwen3.8-Omni-Flash 75.3%
- OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%