Benchmark Almanac

Vending-Bench Arena

Gemini 4 Carbon $15,962 leads since Oct 2026 Not in model cards

Several agents each run a vending machine at the same location in the Vending-Bench 2 environment. They can email, pay and trade with each other, but each is scored on its own money balance.

Andon Labs first multi-agent eval. A round is usually four runs with the same models, and 14 rounds had run by 2026-10. Gemini 4 Carbon set the record average of about 16.0k US dollars.

Made by Andon Labs Released Nov 2025 Simulated companies Tasks kept private Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Ranked by matches Models play against each other, and the results become ratings.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Leaderboard andonlabs.com
32
Models scored on its own leaderboard

Gemini 4 Carbon leads with $15,962

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

32 models. Scroll sideways for the rest.

  1. $15,962 Gemini 4 Carbon
  2. $13,545 GPT-6 Astra
  3. $13,043 Gemini 4 Argon
  4. $10,627 GPT-5.5
  5. $10,530 GPT-6 Sol
  6. $9,191 GPT-5.6 Terra
  7. $8,094 Claude Opus 5.5
  8. $7,930 Grok 4.7
  9. $7,841 GLM-5.3
  10. $7,821 GPT-5.6 Sol
  11. $7,044 Claude Opus 5
  12. $6,196 Claude Opus 4.8
  13. $5,838 Claude Opus 4.7
  14. $5,719 Claude Fable 5.1
  15. $5,639 Claude Sonnet 4.6
  16. $5,582 GLM-5
  17. $5,062 Claude Opus 4.6
  18. $4,244 Claude Fable 5
  19. $4,143 Claude Opus 4.5
  20. $3,436 Gemini 3 Flash
  21. $3,384 Gemini 3 Pro
  22. $3,194 Kimi K3
  23. $2,602 Claude Sonnet 4.5
  24. $2,434 GPT-5.6 Luna
  25. $2,158 GPT-5.4
  26. $1,701 Claude Haiku 4.5
  27. $773 Gemini 2.5 Pro
  28. $723 Grok 4.1 Fast
  29. $619 GPT-5.2
  30. $477 GPT-5.1
  31. $281 Gemini 2.5 Flash
  32. $154 GPT-5 mini
  1. 1 Gemini 4 Carbon vs GPT-6 Astra, Gemini 4 Argon $15,962
  2. 2 GPT-6 Astra vs Gemini 4 Argon, Gemini 4 Carbon $13,545
  3. 3 Gemini 4 Argon vs GPT-6 Astra, Gemini 4 Carbon $13,043
  4. 4 GPT-5.5 vs Claude Opus 4.7, Claude Opus 4.8 $10,627
  5. 5 GPT-6 Sol vs Claude Opus 5.5, Grok 4.7 $10,530
  6. 6 GPT-5.6 Terra vs GPT-5.6 Luna, GPT-5.6 Sol $9,191
  7. 7 Claude Opus 5.5 vs GPT-6 Sol, Grok 4.7 $8,094
  8. 8 Grok 4.7 vs Claude Opus 5.5, GPT-6 Sol $7,930
  9. 9 GLM-5.3 vs Claude Fable 5.1, GPT-6 Astra $7,841
  10. 10 GPT-5.6 Sol vs GPT-5.6 Luna, GPT-5.6 Terra $7,821
  11. 11 Claude Opus 5 vs GPT-5.6 Sol, Kimi K3 $7,044
  12. 12 Claude Opus 4.8 vs Claude Fable 5, GPT-5.5 $6,196
  13. 13 Claude Opus 4.7 vs GPT-5.4, GPT-5.5 $5,838
  14. 14 Claude Fable 5.1 vs GLM-5.3, GPT-6 Astra $5,719
  15. 15 Claude Sonnet 4.6 vs Claude Opus 4.6, Claude Sonnet 4.5 $5,639
  16. 16 GLM-5 vs Claude Opus 4.6 $5,582
  17. 17 Claude Opus 4.6 vs GLM-5 $5,062
  18. 18 Claude Fable 5 max reasoning, vs Claude Opus 4.8, GPT-5.5 $4,244
  19. 19 Claude Opus 4.5 vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro $4,143
  20. 20 Gemini 3 Flash vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Grok 4.1 Fast $3,436
  21. 21 Gemini 3 Pro vs Claude Sonnet 4.5, GPT-5.1, Gemini 2.5 Pro $3,384
  22. 22 Kimi K3 vs Claude Opus 5, GPT-5.6 Sol $3,194
  23. 23 Claude Sonnet 4.5 vs GLM-5 $2,602
  24. 24 GPT-5.6 Luna vs GPT-5.6 Sol, GPT-5.6 Terra $2,434
  25. 25 GPT-5.4 vs Claude Opus 4.7, GPT-5.5 $2,158
  26. 26 Claude Haiku 4.5 vs GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast $1,701
  27. 27 Gemini 2.5 Pro Jun 2025, vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro $773
  28. 28 Grok 4.1 Fast vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash $723
  29. 29 GPT-5.2 vs Claude Opus 4.5, Claude Opus 4.6, Gemini 3 Pro $619
  30. 30 GPT-5.1 vs Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro $477
  31. 31 Gemini 2.5 Flash Sep 2025, vs Claude Haiku 4.5, GPT-5 Mini, Gemini 3 Flash, Grok 4.1 Fast $281
  32. 32 GPT-5 mini vs Claude Haiku 4.5, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast $154
#ModelScoreSettingDateSource
1 Gemini 4 Carbon $15,962 vs GPT-6 Astra, Gemini 4 Argon Oct 2026 Official leaderboard
2 GPT-6 Astra $13,545 vs Gemini 4 Argon, Gemini 4 Carbon Sep 2026 Official leaderboard
3 Gemini 4 Argon $13,043 vs GPT-6 Astra, Gemini 4 Carbon Sep 2026 Official leaderboard
4 GPT-5.5 $10,627 vs Claude Opus 4.7, Claude Opus 4.8 Apr 2026 Official leaderboard
5 GPT-6 Sol $10,530 vs Claude Opus 5.5, Grok 4.7 Sep 2026 Official leaderboard
6 GPT-5.6 Terra $9,191 vs GPT-5.6 Luna, GPT-5.6 Sol Jul 2026 Official leaderboard
7 Claude Opus 5.5 $8,094 vs GPT-6 Sol, Grok 4.7 Sep 2026 Official leaderboard
8 Grok 4.7 $7,930 vs Claude Opus 5.5, GPT-6 Sol Sep 2026 Official leaderboard
9 GLM-5.3 $7,841 vs Claude Fable 5.1, GPT-6 Astra Aug 2026 Official leaderboard
10 GPT-5.6 Sol $7,821 vs GPT-5.6 Luna, GPT-5.6 Terra Jul 2026 Official leaderboard
11 Claude Opus 5 $7,044 vs GPT-5.6 Sol, Kimi K3 Jul 2026 Official leaderboard
12 Claude Opus 4.8 $6,196 vs Claude Fable 5, GPT-5.5 May 2026 Official leaderboard
13 Claude Opus 4.7 $5,838 vs GPT-5.4, GPT-5.5 Apr 2026 Official leaderboard
14 Claude Fable 5.1 $5,719 vs GLM-5.3, GPT-6 Astra Sep 2026 Official leaderboard
15 Claude Sonnet 4.6 $5,639 vs Claude Opus 4.6, Claude Sonnet 4.5 Feb 2026 Official leaderboard
16 GLM-5 $5,582 vs Claude Opus 4.6 Feb 2026 Official leaderboard
17 Claude Opus 4.6 $5,062 vs GLM-5 Feb 2026 Official leaderboard
18 Claude Fable 5 $4,244 max reasoning, vs Claude Opus 4.8, GPT-5.5 Jun 2026 Official leaderboard
19 Claude Opus 4.5 $4,143 vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro Nov 2025 Official leaderboard
20 Gemini 3 Flash $3,436 vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Grok 4.1 Fast Dec 2025 Official leaderboard
21 Gemini 3 Pro $3,384 vs Claude Sonnet 4.5, GPT-5.1, Gemini 2.5 Pro Nov 2025 Official leaderboard
22 Kimi K3 $3,194 vs Claude Opus 5, GPT-5.6 Sol Jul 2026 Official leaderboard
23 Claude Sonnet 4.5 $2,602 vs GLM-5 Sep 2025 Official leaderboard
24 GPT-5.6 Luna $2,434 vs GPT-5.6 Sol, GPT-5.6 Terra Jul 2026 Official leaderboard
25 GPT-5.4 $2,158 vs Claude Opus 4.7, GPT-5.5 Mar 2026 Official leaderboard
26 Claude Haiku 4.5 $1,701 vs GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast Oct 2025 Official leaderboard
27 Gemini 2.5 Pro Jun 2025 $773 vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro Jun 2025 Official leaderboard
28 Grok 4.1 Fast $723 vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash Nov 2025 Official leaderboard
29 GPT-5.2 $619 vs Claude Opus 4.5, Claude Opus 4.6, Gemini 3 Pro Dec 2025 Official leaderboard
30 GPT-5.1 $477 vs Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro Nov 2025 Official leaderboard
31 Gemini 2.5 Flash Sep 2025 $281 vs Claude Haiku 4.5, GPT-5 Mini, Gemini 3 Flash, Grok 4.1 Fast Sep 2025 Official leaderboard
32 GPT-5 mini $154 vs Claude Haiku 4.5, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast Aug 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-6 Astra
vs Claude F… vs Gemini 4…
$13,545
GPT-5.5
vs Claude O… vs Claude O…
$10,627
GPT-5.6 Sol
vs Claude O…
$7,821
Claude Opus 4.8
vs Claude O…
$6,196
Claude Opus 4.7
vs Claude O…
$5,838
GLM-5
vs Claude S…
$5,582
Claude Opus 4.6
vs Claude O… vs GLM-5
$5,062
Claude Opus 4.5
vs Claude O… vs Claude S…
$4,143
Gemini 3 Pro
vs Claude O… vs Claude O…
$3,384
Claude Sonnet 4.5
vs Claude O… vs Claude O…
$2,602
GPT-5.1
vs Claude S…
$477
ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 4 Carbon vs GPT-6 Astra, Gemini 4 Argon $15,962
GPT-6 Astra vs Gemini 4 Argon, Gemini 4 Carbon $13,545
Gemini 4 Argon vs GPT-6 Astra, Gemini 4 Carbon $13,043
GPT-6 Astra vs Claude Fable 5.1, GLM-5.3 $12,363
GPT-5.5 vs Claude Opus 4.7, Claude Opus 4.8 $10,627
GPT-6 Sol vs Claude Opus 5.5, Grok 4.7 $10,530
GPT-5.6 Terra vs GPT-5.6 Luna, GPT-5.6 Sol $9,191
GPT-5.5 vs Claude Fable 5, Claude Opus 4.8 $8,288
Claude Opus 5.5 vs GPT-6 Sol, Grok 4.7 $8,094
GPT-5.5 vs Claude Opus 4.7, GPT-5.4 $7,980
Grok 4.7 vs Claude Opus 5.5, GPT-6 Sol $7,930
GLM-5.3 vs Claude Fable 5.1, GPT-6 Astra $7,841
GPT-5.6 Sol vs GPT-5.6 Luna, GPT-5.6 Terra $7,821
GPT-5.6 Sol vs Claude Opus 5, Kimi K3 $7,444
Claude Opus 5 vs GPT-5.6 Sol, Kimi K3 $7,044
Claude Opus 4.8 vs Claude Fable 5, GPT-5.5 $6,196
Claude Opus 4.7 vs GPT-5.4, GPT-5.5 $5,838
Claude Opus 4.7 vs Claude Opus 4.8, GPT-5.5 $5,827
Claude Fable 5.1 vs GLM-5.3, GPT-6 Astra $5,719
Claude Sonnet 4.6 vs Claude Opus 4.6, Claude Sonnet 4.5 $5,639
GLM-5 vs Claude Opus 4.6 $5,582
Claude Opus 4.8 vs Claude Opus 4.7, GPT-5.5 $5,188
Claude Opus 4.6 vs GLM-5 $5,062
GLM-5 vs Claude Sonnet 4.5 $4,652
Claude Fable 5 max reasoning, vs Claude Opus 4.8, GPT-5.5 $4,244
Claude Opus 4.5 vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro $4,143
Claude Opus 4.6 vs Claude Sonnet 4.5, Claude Sonnet 4.6 $4,053
Gemini 3 Flash vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Grok 4.1 Fast $3,436
Gemini 3 Pro vs Claude Sonnet 4.5, GPT-5.1, Gemini 2.5 Pro $3,384
Kimi K3 vs Claude Opus 5, GPT-5.6 Sol $3,194
Claude Opus 4.6 vs Claude Opus 4.5, GPT-5.2, Gemini 3 Pro $3,030
Claude Sonnet 4.5 vs GLM-5 $2,602
Gemini 3 Pro vs Claude Opus 4.5, Claude Sonnet 4.5, GPT-5.1 $2,563
GPT-5.6 Luna vs GPT-5.6 Sol, GPT-5.6 Terra $2,434
GPT-5.4 vs Claude Opus 4.7, GPT-5.5 $2,158
Claude Sonnet 4.5 vs Claude Opus 4.6, Claude Sonnet 4.6 $2,125
Claude Haiku 4.5 vs GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast $1,701
Gemini 3 Pro vs Claude Opus 4.5, Claude Opus 4.6, GPT-5.2 $1,235
Claude Sonnet 4.5 vs GPT-5.1, Gemini 2.5 Pro, Gemini 3 Pro $1,105
Claude Opus 4.5 vs Claude Opus 4.6, GPT-5.2, Gemini 3 Pro $1,040
Claude Sonnet 4.5 vs Claude Opus 4.5, GPT-5.1, Gemini 3 Pro $809
Gemini 2.5 Pro Jun 2025 vs Claude Sonnet 4.5, GPT-5.1, Gemini 3 Pro $773
Grok 4.1 Fast vs Claude Haiku 4.5, GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash $723
GPT-5.2 vs Claude Opus 4.5, Claude Opus 4.6, Gemini 3 Pro $619
GPT-5.1 vs Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro $477
Gemini 2.5 Flash Sep 2025 vs Claude Haiku 4.5, GPT-5 Mini, Gemini 3 Flash, Grok 4.1 Fast $281
GPT-5 mini vs Claude Haiku 4.5, Gemini 2.5 Flash, Gemini 3 Flash, Grok 4.1 Fast $154
GPT-5.1 vs Claude Sonnet 4.5, Gemini 2.5 Pro, Gemini 3 Pro $108

10 models have held number one since its release in Nov 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic Google DeepMind Z.ai OpenAI
Own leaderboard 10 leaders
Gemini 4 Carbon $15,962

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
$0 $5,000 $10,000 $15,000 $20,000 2025 2026 2027 $0 $5,000 $10,000 $15,000 $20,000 2025 2026 2027

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Economic work

All 91 in Economic work
  • JobBench In use 130 tasks from 35 occupations workers most want delegated. Agent works through messy reference files, graded by fact-anchored binary rubrics. GLM-5.3 61.4%
  • GDPval In use Real work tasks from 44 occupations with reference files. Experts blindly compare model deliverables with expert work. Metric is win-or-tie rate. Seed2.1 Pro 87.9%
  • GDPval-AA v2.1 In use Agents with shell and web access complete 220 GDPval gold tasks from 44 occupations. Blind pairwise comparisons give Elo ratings. Claude Opus 5.5 1,846
  • AA-Briefcase v1.1 In use Version 1.1 of Artificial Analysis's multi-hour office work evaluation, with Elo ratings fitted by Crowd-BT. Claude Opus 5.5 1,822
  • CoWorkBench In use Agent completes long-horizon productivity tasks across computer science, finance, law and medicine. Qwen in-house scoring (method unpublished). Qwen3.8-Omni-Flash 75.3%
  • OfficeQA In use Grounded reasoning over a large corpus of U.S. Treasury Bulletin documents: locate tables and compute precise numeric answers. Claude Opus 4.7 86.3%