Benchmark Almanac

LLM Chess

GPT-6 Astra 1,614 leads since Sep 2026 Not in model cards

Models play chess in multi-turn dialogs against a random player or the Komodo Dragon engine. The board reports wins, Elo and game duration, which tracks instruction following.

Dragon was added as a stronger opponent for Elo anchoring once reasoning models beat the random player. Paper at NeurIPS 2025 FoRLM workshop. About 140 GitHub stars.

Made by Maxim Saplin Released Sep 2024 Puzzles and chess All tasks public Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Ranked by matches Models play against each other, and the results become ratings.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
maxim-saplin/llm_chess 137 stars Leaderboard maxim-saplin.github.io
130
Entries on its own leaderboard, 124 models we track

GPT-6 Astra leads with 1,614

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

130 entries. Scroll sideways for the rest.

  1. 1,614 GPT-6 Astra
  2. 1,612 GPT-6.1 Sol
  3. 1,609 Gemini 4 Argon
  4. 1,575 Claude Opus 5.5
  5. 1,550 GPT-5.6 Sol
  6. 1,546 Gemini 3.8 Flash
  7. 1,545 GPT-6 Sol
  8. 1,532 GPT-5.5
  9. 1,511 Gemini 3.1 Pro
  10. 1,322 GPT-5.4
  11. 1,308 Mercury 3
  12. 1,293 GPT-5.6 Terra
  13. 1,285 Claude Opus 5
  14. 1,219 GPT-6 Luna
  15. 1,195 Claude Sonnet 5.5
  16. 1,185 GPT-5.6 Luna
  17. 1,143 Claude Haiku 5.5
  18. 1,107 Gemini 3.7 Flash
  19. 1,094 Gemini 3.6 Flash
  20. 1,087 DeepSeek V4.1 Flash
  21. 1,086 GPT-5
  22. 1,085 GPT-5.2
  23. 1,053 GPT-5.3 Codex
  24. 1,034 Gemini 3 Pro
  25. 953 Gemini 3.5 Flash
  26. 948 GPT-5.1
  27. 846 Qwen 3.8 27B
  28. 843 Grok 4.20
  29. 822 Grok 4.7
  30. 806 Claude Sonnet 5
  31. 800 Grok 4.1 Fast
  32. 778 o3
  33. 759 GPT-5 mini
  34. 735 Grok 4 Fast
  35. 691 Claude Opus 4.7
  36. 683 GPT-5 nano
  37. 605 GLM-5.3
  38. 572 GLM-5.3-Flash
  39. 544 GPT-5.1-Codex-mini
  40. 504 GPT-5.1-Codex
  41. 450 Claude Opus 4.8
  42. 446 Claude Opus 4.5
  43. 443 Grok 4.6
  44. 440 o4-mini
  45. 430 Grok-3 mini
  46. 416 GPT-5.4 Mini
  47. 403 Ternary-Bonsai-2-27B-gguf:TQ1-0-high
  48. 403 o3-mini
  49. 304 agents-a1
  50. 300 DeepSeek-R1
  51. 280 o1
  52. 270 Qwen3.6 27B
  53. 214 Muse Glimmer
  54. 207 GPT-5.4 Nano
  55. 198 Claude Opus 4.6
  56. 190 Claude Sonnet 4.6
  57. 189 DeepSeek-V3.2
  58. 179 Grok 4.3
  59. 142 GPT-5.2 Instant
  60. 142 non-gemini-25pro-t03-mini41-t00-mini41-t03
  61. 127 Claude Haiku 4.5
  62. 115 Claude Sonnet 4
  63. 113 Claude Sonnet 4.5
  64. 111 Kimi K2.5
  65. 46 o1-preview
  66. 29 Claude Opus 4
  67. 4 non-deepseek-r1-t03-mini41-t10-mini41-t03
  68. -1 Claude 3.7 Sonnet
  69. -26 Qwen3.5-35B-A3B
  70. -37 Qwen3-30B-A3B-Thinking
  71. -38 Claude Opus 4.1
  72. -52 o1-mini
  73. -63 Composer 2
  74. -71 Gemini 2.5 Pro
  75. -87 GPT-5 Chat
  76. -98 gpt-4-32k
  77. -99 Claude 3.5 Sonnet (October 2024)
  78. -99 qwen-max
  79. -100 Llama 4 Maverick
  80. -104 GPT-4 Turbo
  81. -108 GPT-4o
  82. -111 GPT-4.1
  83. -111 GPT-4.5
  84. -120 GPT-4
  85. -120 Claude 3.5 Haiku
  86. -128 Qwen3-Next-80B-A3B
  87. -131 GPT-5.1 Instant
  88. -133 Gemini 3.1 Flash-Lite
  89. -134 GPT-4o
  90. -134 Claude 3.5 Sonnet
  91. -137 Grok 3
  92. -140 Qwen3-235B-A22B-Instruct
  93. -144 Gemini 2.5 Pro
  94. -146 Claude 3 Opus
  95. -147 Gemini 2.5 Pro
  96. -158 GPT-4o
  97. -167 GPT-4.1 mini
  98. -179 Gemini 3 Flash
  99. -200 Nemotron 3 Nano 30B-A3B
  100. -200 Qwen3.5-9B
  101. -212 Composer 1.5
  102. -216 DeepSeek-R1
  103. -217 gpt-oss-120b
  104. -232 gpt-oss-20b
  105. -252 GPT-4o mini
  106. -278 Llama 3-70B
  107. -289 Gemini 2.5 Flash
  108. -299 GLM-5
  109. -311 Gemini 2.0 Flash
  110. -322 Claude 3 Sonnet
  111. -328 glm-4.7-flash
  112. -335 Grok-2
  113. -367 Gemini 1.5 Flash
  114. -402 Llama 4 Scout
  115. -412 Gemma 2 27B
  116. -458 K2-Think
  117. -546 Gemma 2 9B
  118. -558 Qwen3.5-4B
  119. -579 DeepSeek-V3
  120. -608 Llama 3.3 70B
  121. -616 Qwen Plus
  122. -637 Qwen3-4B
  123. -651 Gemini 1.5 Pro
  124. -672 Qwen2.5-72B
  125. -672 Gemini 2.0 Flash
  126. -689 Kimi K2
  127. -796 Llama 3.1-8B
  128. -812 Gemini 2.0 Flash-Lite
  129. -812 Composer 1
  130. -823 DeepSeek-V3
  1. 1 GPT-6 Astra high 1,614
  2. 2 GPT-6.1 Sol xhigh 1,612
  3. 3 Gemini 4 Argon high 1,609
  4. 4 Claude Opus 5.5 high 1,575
  5. 5 GPT-5.6 Sol xhigh 1,550
  6. 6 Gemini 3.8 Flash high 1,546
  7. 7 GPT-6 Sol xhigh 1,545
  8. 8 GPT-5.5 medium 1,532
  9. 9 Gemini 3.1 Pro high 1,511
  10. 10 GPT-5.4 high 1,322
  11. 11 Mercury 3 high 1,308
  12. 12 GPT-5.6 Terra xhigh 1,293
  13. 13 Claude Opus 5 high 1,285
  14. 14 GPT-6 Luna max 1,219
  15. 15 Claude Sonnet 5.5 high 1,195
  16. 16 GPT-5.6 Luna xhigh 1,185
  17. 17 Claude Haiku 5.5 high 1,143
  18. 18 Gemini 3.7 Flash high 1,107
  19. 19 Gemini 3.6 Flash high 1,094
  20. 20 DeepSeek V4.1 Flash high 1,087
  21. 21 GPT-5 medium 1,086
  22. 22 GPT-5.2 medium 1,085
  23. 23 GPT-5.3 Codex medium 1,053
  24. 24 Gemini 3 Pro high 1,034
  25. 25 Gemini 3.5 Flash medium 953
  26. 26 GPT-5.1 medium 948
  27. 27 Qwen 3.8 27B 846
  28. 28 Grok 4.20 reasoning 843
  29. 29 Grok 4.7 high 822
  30. 30 Claude Sonnet 5 high 806
  31. 31 Grok 4.1 Fast reasoning 800
  32. 32 o3 medium 778
  33. 33 GPT-5 mini high 759
  34. 34 Grok 4 Fast reasoning 735
  35. 35 Claude Opus 4.7 high 691
  36. 36 GPT-5 nano high 683
  37. 37 GLM-5.3 high 605
  38. 38 GLM-5.3-Flash high 572
  39. 39 GPT-5.1-Codex-mini high 544
  40. 40 GPT-5.1-Codex low 504
  41. 41 Claude Opus 4.8 high 450
  42. 42 Claude Opus 4.5 thinking 16000 446
  43. 43 Grok 4.6 high 443
  44. 44 o4-mini high 440
  45. 45 Grok-3 mini high 430
  46. 46 GPT-5.4 Mini high 416
  47. 47 Ternary-Bonsai-2-27B-gguf:TQ1-0-high 403
  48. 48 o3-mini 403
  49. 49 agents-a1 304
  50. 50 DeepSeek-R1 May 2025 300
  51. 51 o1 medium 280
  52. 52 Qwen3.6 27B 270
  53. 53 Muse Glimmer 214
  54. 54 GPT-5.4 Nano high 207
  55. 55 Claude Opus 4.6 high 198
  56. 56 Claude Sonnet 4.6 high 190
  57. 57 DeepSeek-V3.2 non-reasoning 189
  58. 58 Grok 4.3 high 179
  59. 59 GPT-5.2 Instant 142
  60. 60 non-gemini-25pro-t03-mini41-t00-mini41-t03 142
  61. 61 Claude Haiku 4.5 thinking 16000 127
  62. 62 Claude Sonnet 4 thinking 16000 115
  63. 63 Claude Sonnet 4.5 thinking 16000 113
  64. 64 Kimi K2.5 111
  65. 65 o1-preview 46
  66. 66 Claude Opus 4 thinking 16000 29
  67. 67 non-deepseek-r1-t03-mini41-t10-mini41-t03 4
  68. 68 Claude 3.7 Sonnet thinking 10000 -1
  69. 69 Qwen3.5-35B-A3B -26
  70. 70 Qwen3-30B-A3B-Thinking Jul 2025 -37
  71. 71 Claude Opus 4.1 -38
  72. 72 o1-mini -52
  73. 73 Composer 2 -63
  74. 74 Gemini 2.5 Pro May 2025 -71
  75. 75 GPT-5 Chat -87
  76. 76 gpt-4-32k -98
  77. 77 Claude 3.5 Sonnet (October 2024) -99
  78. 78 qwen-max -99
  79. 79 Llama 4 Maverick -100
  80. 80 GPT-4 Turbo Apr 2024 -104
  81. 81 GPT-4o Nov 2024 -108
  82. 82 GPT-4.1 -111
  83. 83 GPT-4.5 -111
  84. 84 GPT-4 Jun 2023 -120
  85. 85 Claude 3.5 Haiku -120
  86. 86 Qwen3-Next-80B-A3B -128
  87. 87 GPT-5.1 Instant -131
  88. 88 Gemini 3.1 Flash-Lite -133
  89. 89 GPT-4o Aug 2024 -134
  90. 90 Claude 3.5 Sonnet -134
  91. 91 Grok 3 -137
  92. 92 Qwen3-235B-A22B-Instruct Jul 2025 -140
  93. 93 Gemini 2.5 Pro Mar 2025 -144
  94. 94 Claude 3 Opus -146
  95. 95 Gemini 2.5 Pro Jun 2025 -147
  96. 96 GPT-4o May 2024 -158
  97. 97 GPT-4.1 mini -167
  98. 98 Gemini 3 Flash -179
  99. 99 Nemotron 3 Nano 30B-A3B -200
  100. 100 Qwen3.5-9B thinking -200
  101. 101 Composer 1.5 -212
  102. 102 DeepSeek-R1 -216
  103. 103 gpt-oss-120b medium -217
  104. 104 gpt-oss-20b medium -232
  105. 105 GPT-4o mini -252
  106. 106 Llama 3-70B -278
  107. 107 Gemini 2.5 Flash Jun 2025 -289
  108. 108 GLM-5 -299
  109. 109 Gemini 2.0 Flash Feb 2025 -311
  110. 110 Claude 3 Sonnet -322
  111. 111 glm-4.7-flash -328
  112. 112 Grok-2 Dec 2024 -335
  113. 113 Gemini 1.5 Flash May 2024 -367
  114. 114 Llama 4 Scout -402
  115. 115 Gemma 2 27B -412
  116. 116 K2-Think -458
  117. 117 Gemma 2 9B -546
  118. 118 Qwen3.5-4B thinking -558
  119. 119 DeepSeek-V3 Mar 2025 -579
  120. 120 Llama 3.3 70B -608
  121. 121 Qwen Plus Jan 2025 -616
  122. 122 Qwen3-4B thinking -637
  123. 123 Gemini 1.5 Pro Sep 2024 -651
  124. 124 Qwen2.5-72B -672
  125. 125 Gemini 2.0 Flash Dec 2024 -672
  126. 126 Kimi K2 Jul 2025 -689
  127. 127 Llama 3.1-8B -796
  128. 128 Gemini 2.0 Flash-Lite -812
  129. 129 Composer 1 -812
  130. 130 DeepSeek-V3 -823
#EntryScoreSettingDateSource
1 GPT-6 Astra 1,614 high Sep 2026 Official leaderboard
2 GPT-6.1 Sol 1,612 xhigh Sep 2026 Official leaderboard
3 Gemini 4 Argon 1,609 high Sep 2026 Official leaderboard
4 Claude Opus 5.5 1,575 high Sep 2026 Official leaderboard
5 GPT-5.6 Sol 1,550 xhigh Jul 2026 Official leaderboard
6 Gemini 3.8 Flash 1,546 high Sep 2026 Official leaderboard
7 GPT-6 Sol 1,545 xhigh Sep 2026 Official leaderboard
8 GPT-5.5 1,532 medium Apr 2026 Official leaderboard
9 Gemini 3.1 Pro 1,511 high Feb 2026 Official leaderboard
10 GPT-5.4 1,322 high Mar 2026 Official leaderboard
11 Mercury 3 1,308 high Oct 2026 Official leaderboard
12 GPT-5.6 Terra 1,293 xhigh Jul 2026 Official leaderboard
13 Claude Opus 5 1,285 high Jul 2026 Official leaderboard
14 GPT-6 Luna 1,219 max Sep 2026 Official leaderboard
15 Claude Sonnet 5.5 1,195 high Sep 2026 Official leaderboard
16 GPT-5.6 Luna 1,185 xhigh Jul 2026 Official leaderboard
17 Claude Haiku 5.5 1,143 high Oct 2026 Official leaderboard
18 Gemini 3.7 Flash 1,107 high Aug 2026 Official leaderboard
19 Gemini 3.6 Flash 1,094 high Jul 2026 Official leaderboard
20 DeepSeek V4.1 Flash 1,087 high Sep 2026 Official leaderboard
21 GPT-5 1,086 medium Aug 2025 Official leaderboard
22 GPT-5.2 1,085 medium Dec 2025 Official leaderboard
23 GPT-5.3 Codex 1,053 medium Feb 2026 Official leaderboard
24 Gemini 3 Pro 1,034 high Nov 2025 Official leaderboard
25 Gemini 3.5 Flash 953 medium May 2026 Official leaderboard
26 GPT-5.1 948 medium Nov 2025 Official leaderboard
27 Qwen 3.8 27B 846 Aug 2026 Official leaderboard
28 Grok 4.20 843 reasoning Feb 2026 Official leaderboard
29 Grok 4.7 822 high Sep 2026 Official leaderboard
30 Claude Sonnet 5 806 high Jun 2026 Official leaderboard
31 Grok 4.1 Fast 800 reasoning Nov 2025 Official leaderboard
32 o3 778 medium Apr 2025 Official leaderboard
33 GPT-5 mini 759 high Aug 2025 Official leaderboard
34 Grok 4 Fast 735 reasoning Sep 2025 Official leaderboard
35 Claude Opus 4.7 691 high Apr 2026 Official leaderboard
36 GPT-5 nano 683 high Aug 2025 Official leaderboard
37 GLM-5.3 605 high Aug 2026 Official leaderboard
38 GLM-5.3-Flash 572 high Aug 2026 Official leaderboard
39 GPT-5.1-Codex-mini 544 high Nov 2025 Official leaderboard
40 GPT-5.1-Codex 504 low Nov 2025 Official leaderboard
41 Claude Opus 4.8 450 high May 2026 Official leaderboard
42 Claude Opus 4.5 446 thinking 16000 Nov 2025 Official leaderboard
43 Grok 4.6 443 high Aug 2026 Official leaderboard
44 o4-mini 440 high Apr 2025 Official leaderboard
45 Grok-3 mini 430 high Apr 2025 Official leaderboard
46 GPT-5.4 Mini 416 high Mar 2026 Official leaderboard
47 Ternary-Bonsai-2-27B-gguf:TQ1-0-high 403 Sep 2026 Official leaderboard
48 o3-mini 403 Jan 2025 Official leaderboard
49 agents-a1 304 Jul 2026 Official leaderboard
50 DeepSeek-R1 May 2025 300 May 2025 Official leaderboard
51 o1 280 medium Dec 2024 Official leaderboard
52 Qwen3.6 27B 270 Apr 2026 Official leaderboard
53 Muse Glimmer 214 Aug 2026 Official leaderboard
54 GPT-5.4 Nano 207 high Mar 2026 Official leaderboard
55 Claude Opus 4.6 198 high Feb 2026 Official leaderboard
56 Claude Sonnet 4.6 190 high Feb 2026 Official leaderboard
57 DeepSeek-V3.2 189 non-reasoning Dec 2025 Official leaderboard
58 Grok 4.3 179 high Apr 2026 Official leaderboard
59 GPT-5.2 Instant 142 Dec 2025 Official leaderboard
60 non-gemini-25pro-t03-mini41-t00-mini41-t03 142 Apr 2025 Official leaderboard
61 Claude Haiku 4.5 127 thinking 16000 Oct 2025 Official leaderboard
62 Claude Sonnet 4 115 thinking 16000 May 2025 Official leaderboard
63 Claude Sonnet 4.5 113 thinking 16000 Sep 2025 Official leaderboard
64 Kimi K2.5 111 Jan 2026 Official leaderboard
65 o1-preview 46 Sep 2024 Official leaderboard
66 Claude Opus 4 29 thinking 16000 May 2025 Official leaderboard
67 non-deepseek-r1-t03-mini41-t10-mini41-t03 4 Apr 2025 Official leaderboard
68 Claude 3.7 Sonnet -1 thinking 10000 Feb 2025 Official leaderboard
69 Qwen3.5-35B-A3B -26 Feb 2026 Official leaderboard
70 Qwen3-30B-A3B-Thinking Jul 2025 -37 Jul 2025 Official leaderboard
71 Claude Opus 4.1 -38 Aug 2025 Official leaderboard
72 o1-mini -52 Sep 2024 Official leaderboard
73 Composer 2 -63 Cursor CLI Mar 2026 Official leaderboard
74 Gemini 2.5 Pro May 2025 -71 May 2025 Official leaderboard
75 GPT-5 Chat -87 Aug 2025 Official leaderboard
76 gpt-4-32k -98 Jan 2025 Official leaderboard
77 Claude 3.5 Sonnet (October 2024) -99 Oct 2024 Official leaderboard
78 qwen-max -99 Jan 2025 Official leaderboard
79 Llama 4 Maverick -100 Apr 2025 Official leaderboard
80 GPT-4 Turbo Apr 2024 -104 Apr 2024 Official leaderboard
81 GPT-4o Nov 2024 -108 Nov 2024 Official leaderboard
82 GPT-4.1 -111 Apr 2025 Official leaderboard
83 GPT-4.5 -111 Feb 2025 Official leaderboard
84 GPT-4 Jun 2023 -120 Jun 2023 Official leaderboard
85 Claude 3.5 Haiku -120 Oct 2024 Official leaderboard
86 Qwen3-Next-80B-A3B -128 Sep 2025 Official leaderboard
87 GPT-5.1 Instant -131 Nov 2025 Official leaderboard
88 Gemini 3.1 Flash-Lite -133 Mar 2026 Official leaderboard
89 GPT-4o Aug 2024 -134 Aug 2024 Official leaderboard
90 Claude 3.5 Sonnet -134 Jun 2024 Official leaderboard
91 Grok 3 -137 Apr 2025 Official leaderboard
92 Qwen3-235B-A22B-Instruct Jul 2025 -140 Jul 2025 Official leaderboard
93 Gemini 2.5 Pro Mar 2025 -144 Mar 2025 Official leaderboard
94 Claude 3 Opus -146 Feb 2024 Official leaderboard
95 Gemini 2.5 Pro Jun 2025 -147 Jun 2025 Official leaderboard
96 GPT-4o May 2024 -158 May 2024 Official leaderboard
97 GPT-4.1 mini -167 Apr 2025 Official leaderboard
98 Gemini 3 Flash -179 Dec 2025 Official leaderboard
99 Nemotron 3 Nano 30B-A3B -200 Dec 2025 Official leaderboard
100 Qwen3.5-9B -200 thinking Feb 2026 Official leaderboard
101 Composer 1.5 -212 Cursor CLI Feb 2026 Official leaderboard
102 DeepSeek-R1 -216 Jan 2025 Official leaderboard
103 gpt-oss-120b -217 medium Aug 2025 Official leaderboard
104 gpt-oss-20b -232 medium Aug 2025 Official leaderboard
105 GPT-4o mini -252 Jul 2024 Official leaderboard
106 Llama 3-70B -278 Apr 2024 Official leaderboard
107 Gemini 2.5 Flash Jun 2025 -289 Jun 2025 Official leaderboard
108 GLM-5 -299 Feb 2026 Official leaderboard
109 Gemini 2.0 Flash Feb 2025 -311 Feb 2025 Official leaderboard
110 Claude 3 Sonnet -322 Feb 2024 Official leaderboard
111 glm-4.7-flash -328 Jan 2026 Official leaderboard
112 Grok-2 Dec 2024 -335 Dec 2024 Official leaderboard
113 Gemini 1.5 Flash May 2024 -367 May 2024 Official leaderboard
114 Llama 4 Scout -402 Apr 2025 Official leaderboard
115 Gemma 2 27B -412 Jun 2024 Official leaderboard
116 K2-Think -458 Sep 2025 Official leaderboard
117 Gemma 2 9B -546 Jun 2024 Official leaderboard
118 Qwen3.5-4B -558 thinking Feb 2026 Official leaderboard
119 DeepSeek-V3 Mar 2025 -579 Mar 2025 Official leaderboard
120 Llama 3.3 70B -608 Dec 2024 Official leaderboard
121 Qwen Plus Jan 2025 -616 Jan 2025 Official leaderboard
122 Qwen3-4B -637 thinking Apr 2025 Official leaderboard
123 Gemini 1.5 Pro Sep 2024 -651 Sep 2024 Official leaderboard
124 Qwen2.5-72B -672 Sep 2024 Official leaderboard
125 Gemini 2.0 Flash Dec 2024 -672 Dec 2024 Official leaderboard
126 Kimi K2 Jul 2025 -689 Jul 2025 Official leaderboard
127 Llama 3.1-8B -796 Jul 2024 Official leaderboard
128 Gemini 2.0 Flash-Lite -812 Feb 2025 Official leaderboard
129 Composer 1 -812 Cursor CLI Oct 2025 Official leaderboard
130 DeepSeek-V3 -823 Dec 2024 Official leaderboard

The best score, 1,614, costs $4.64 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 4 Argon at high comes within 4.4 of the best for $1.27 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Google DeepMind Meta OpenAI xAI NVIDIA Z.ai Anthropic Inception Labs Cost frontier One model at each effort level
0 500 1,000 1,500 2,000 $0.0001 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, high Gemini 3.8 Flash, medium Mercury 3, high GPT-6 Luna, max GPT-6 Luna, xhigh GPT-6 Luna, high GPT-5 nano, low 0 500 1,000 1,500 2,000 $0.0001 $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, high GPT-6.1 Sol, high
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra high 1,614 $4.64
GPT-6.1 Sol xhigh 1,612 $1.65
Gemini 4 Argon high 1,609 $1.27
GPT-6 Astra xhigh 1,579 $5.47
Claude Opus 5.5 high 1,575 $1.40
GPT-5.6 Sol xhigh 1,550 $8.23
GPT-5.6 Sol high 1,549 $6.20
Gemini 3.8 Flash high 1,546 $2.19
GPT-6 Sol xhigh 1,545 $3.41
GPT-5.5 medium 1,532 $6.57
GPT-5.5 high 1,528 $8.25
Gemini 3.8 Flash medium 1,527 $0.86
Gemini 3.1 Pro high 1,511 $4.19
GPT-6.1 Sol high 1,454 $0.76
GPT-5.4 high 1,322 $5.12
GPT-6 Sol high 1,311 $1.41
Mercury 3 high 1,308 $0.48
GPT-5.6 Terra xhigh 1,293 $3.32
Claude Opus 5 high 1,285 $7.56
GPT-5.4 medium 1,253 $4.90
GPT-5.6 Sol medium 1,240 $3.71
GPT-6 Luna max 1,219 $0.25
GPT-5.6 Terra high 1,211 $2.66
Claude Sonnet 5.5 high 1,195 $1.53
GPT-5.6 Luna xhigh 1,185 $2.51
Claude Haiku 5.5 high 1,143 $0.08
Gemini 3.7 Flash high 1,107 $0.75
Gemini 3.6 Flash high 1,094 $2.79
DeepSeek V4.1 Flash high 1,087 $0.74
GPT-5 medium 1,086 $6.36
GPT-5.2 medium 1,085 $3.12
GPT-5.3 Codex medium 1,053 $1.84
Gemini 3 Pro high 1,034 $4.23
Gemini 3.5 Flash medium 953 $2.52
GPT-5.1 medium 948 $3.21
GPT-5.6 Terra medium 936 $1.32
Gemini 3.6 Flash medium 921 $1.61
GPT-5.6 Luna high 919 $1.50
GPT-5 low 858 $2.56
GPT-6 Luna xhigh 855 $0.07
Qwen 3.8 27B 846 $2.29
Grok 4.20 reasoning 843
Grok 4.7 high 822 $2.71
Claude Sonnet 5 high 806 $2.77
Grok 4.1 Fast reasoning 800
GPT-5.2 low 783 $1.30
o3 medium 778 $11
GPT-5 mini high 759 $1.28
o3 low 739 $8.16
Grok 4 Fast reasoning 735
GPT-6 Luna high 730 $0.05
GPT-5.6 Luna medium 730 $0.63
Claude Opus 4.7 high 691 $5.57
GPT-5 nano high 683 $0.49
GPT-5.1 low 615 $0.91
GLM-5.3 high 605 $1.12
GLM-5.3-Flash high 572 $0.10
GPT-5.1-Codex-mini high 544 $1.10
GPT-5.1-Codex low 504 $1.38
Gemini 3.6 Flash low 498 $0.85
Claude Opus 4.8 high 450 $4.00
Claude Opus 4.5 thinking 16000 446 $7.34
Grok 4.6 high 443 $0.47
o4-mini high 440 $2.71
Grok-3 mini high 430
GPT-5.4 Mini high 416 $3.23
GPT-5 nano medium 413 $0.19
o3-mini 403 $3.62
GPT-5 mini medium 349 $0.44
GPT-5.4 low 348 $7.69
DeepSeek-R1 May 2025 300 $1.91
o1 medium 280 $22
Claude Sonnet 5 272 $0.57
Qwen3.6 27B 270 $1.46
o4-mini medium 240 $1.11
Muse Glimmer 214 $1.17
GPT-5.4 Nano high 207 $0.74
Claude Opus 4.6 high 198 $8.76
Claude Sonnet 4.6 high 190 $11
DeepSeek-V3.2 non-reasoning 189 $0.09
Grok 4.3 high 179 $0.19
GPT-5.4 Mini low 176 $0.26
o1 low 154 $14
GPT-5.2 Instant 142 $0.77
Claude Haiku 4.5 thinking 16000 127 $1.15
Claude Sonnet 4 thinking 16000 115 $2.33
Claude Sonnet 4.5 thinking 16000 113 $2.20
Kimi K2.5 111 $0.97
Claude Sonnet 4.6 111 $0.63
o4-mini low 110 $0.58
o4-mini low 104 $0.51
o3-mini medium 82 $1.61
Claude Opus 4.5 71 $0.87
o1-preview 46 $23
Claude Opus 4 thinking 16000 29 $7.92
Claude 3.7 Sonnet thinking 10000 -1 $2.08
Claude Opus 4 -11 $3.28
Claude Sonnet 4 -11 $0.64
Qwen3.5-35B-A3B -26 $0.46
Grok-3 mini low -34
Claude Sonnet 4.5 -36 $0.50
Claude 3.7 Sonnet thinking 1024 -36 $1.32
Claude 3.7 Sonnet thinking 5000 -36 $1.92
Grok-3 mini -37
Qwen3-30B-A3B-Thinking Jul 2025 -37 $0.55
Claude Opus 4.1 -38 $2.97
Claude Haiku 4.5 -42 $0.19
o1-mini -52 $2.41
GPT-5 mini low -56 $0.13
Claude 3.7 Sonnet thinking 2048 -62 $1.61
Composer 2 Cursor CLI -63
Gemini 2.5 Pro May 2025 -71 $0.58
Claude 3.7 Sonnet -76 $0.90
GPT-5 Chat -87 $0.12
Claude 3.5 Sonnet (October 2024) -99 $0.56
Llama 4 Maverick -100 $0.06
GPT-4 Turbo Apr 2024 -104 $0.85
GPT-4o Nov 2024 -108 $0.32
GPT-4.1 -111 $0.14
GPT-4.5 -111 $6.48
GPT-5 nano low -117 $0.03
Grok 4.1 Fast non-reasoning -118 $0.02
GPT-4 Jun 2023 -120 $1.90
Claude 3.5 Haiku -120 $0.15
Grok 4.20 non-reasoning -125 $0.11
Qwen3-Next-80B-A3B -128 $0.12
GPT-5.1 Instant -131 $0.12
Gemini 3.1 Flash-Lite -133 $0.02
GPT-4o Aug 2024 -134 $0.21
Claude 3.5 Sonnet -134 $0.57
Grok 3 -137
Qwen3-235B-A22B-Instruct Jul 2025 -140 $0.08
Gemini 2.5 Pro Mar 2025 -144 $0.56
Claude 3 Opus -146 $2.32
Gemini 2.5 Pro Jun 2025 -147 $0.35
GPT-4o May 2024 -158 $0.27
GPT-4.1 mini -167 $0.86
Gemini 3 Flash -179 $0.70
o3-mini low -184 $0.45
Grok 4 Fast non-reasoning -191 $0.02
Nemotron 3 Nano 30B-A3B -200 $0.02
Qwen3.5-9B thinking -200
Composer 1.5 Cursor CLI -212
DeepSeek-R1 -216 $0.94
gpt-oss-120b medium -217 $0.05
gpt-oss-20b medium -232 $0.02
GPT-4o mini -252 $0.02
gpt-oss-120b high -252 $0.04
gpt-oss-20b high -258 $0.02
Qwen3.5-9B -258
Llama 3-70B -278 $0.02
gpt-oss-120b low -284 $0.06
Gemini 2.5 Flash Jun 2025 -289 $0.05
GLM-5 -299 $0.10
gpt-oss-20b low -300 $0.02
Gemini 2.0 Flash Feb 2025 -311 $0.01
Claude 3 Sonnet -322 $0.40
glm-4.7-flash -328 $0.003
Grok-2 Dec 2024 -335
Gemini 1.5 Flash May 2024 -367 $0.003
Llama 4 Scout -402 $0.02
Gemma 2 27B -412 $0.02
K2-Think -458 $0.06
Gemma 2 9B -546 $0.001
Qwen3.5-4B thinking -558
DeepSeek-V3 Mar 2025 -579 $0.05
Llama 3.3 70B -608 $0.01
Qwen Plus Jan 2025 -616 $0.07
Qwen3-4B thinking -637 $0.13
Gemini 1.5 Pro Sep 2024 -651 $0.15
Qwen2.5-72B -672 $0.01
Gemini 2.0 Flash Dec 2024 -672 $0.01
Kimi K2 Jul 2025 -689 $0.06
Llama 3.1-8B -796 $0.0009
Gemini 2.0 Flash-Lite -812 $0.007
Composer 1 Cursor CLI -812
DeepSeek-V3 -823 $0.03

11 models have held number one since its release in Sep 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI xAI Google DeepMind
Own leaderboard 11 leaders
GPT-6 Astra 1,614

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
-1,000 0 1,000 2,000 2023 2024 2025 2026 -1,000 0 1,000 2,000 2023 2024 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Games and long tasks

All 68 in Games and long tasks
  • ARC-AGI-3 In use Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. GPT-6 Astra 100%
  • MineDojo Fading Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Seed2.0 Pro 49%
  • Orak Fading LLM agents play 12 popular video games across most genres through an MCP interface. Scores come from in-game progress or outcomes per title. Nemotron 3 Nano 4B 22.9
  • GameWorld Only in system cards Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. Gemini 3 Flash 41.9%
  • Swarm Bench Only in system cards Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Kimi K3 76.3%
  • TextQuests Only in system cards Agents play 25 classic interactive fiction games. Measures game progress and harmful actions per 100 steps. Muse Spark 2.40