Benchmark Almanac

ALE-Bench

GPT-6 Astra 2,951 leads since Sep 2026 Not in model cards

Agents iteratively write solutions to 40 hard optimization problems from AtCoder Heuristic Contests. Scores convert to an Elo-like performance rating against human contestants.

In the authors’ words

How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements.

Abstract of ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering (arXiv:2506.09050, CC0)
Made by Sakana AI and AtCoder Released Jun 2025 Competitive programming All tasks public Last model added Oct 2026 Scores refreshed 31 Oct 2026
Level
Competition The strongest students, in olympiad-style contests
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Code A program or patch run against tests.
SakanaAI/ALE-Bench 236 stars Paper Dataset Leaderboard sakana.ai
121
Models scored in Epoch AI data

GPT-6 Astra leads with 2,951

Best score per model in Epoch AI’s data. Where a model ran with several settings or harnesses, we keep its best.

121 models. Scroll sideways for the rest.

  1. 2,951 GPT-6 Astra
  2. 2,744 GPT-6.1 Sol
  3. 2,609 GPT-6.1 Terra
  4. 2,462 GPT-6 Sol
  5. 2,177 GPT-5.6 Sol
  6. 2,165 Claude Opus 5
  7. 2,147 Claude Opus 5.5
  8. 2,143 Claude Fable 5.1
  9. 2,100 Gemini 4 Argon
  10. 2,041 Claude Fable 5
  11. 1,951 GPT-5.6 Terra
  12. 1,943 GPT-5.5
  13. 1,933 Kimi K3.1 Pro
  14. 1,819 Claude Sonnet 5.5
  15. 1,688 Muse Spark 1.3
  16. 1,667 GPT-5.6 Luna
  17. 1,655 GPT-5.3 Codex
  18. 1,622 DeepSeek V4.1 Flash 1017
  19. 1,607 GPT-5.4
  20. 1,590 Muse Spark Open
  21. 1,587 Claude Haiku 5.5
  22. 1,577 GPT-6 Luna
  23. 1,564 Claude Opus 4.8
  24. 1,524 Kimi K3
  25. 1,508 Grok 4.6
  26. 1,463 Claude Sonnet 5
  27. 1,403 DeepSeek V4 Pro 0813
  28. 1,367 Gemini 3 Flash
  29. 1,327 Claude Sonnet 4.6
  30. 1,323 Claude Opus 4.7
  31. 1,317 GLM-5.3
  32. 1,309 Grok 4.5
  33. 1,306 DeepSeek V4 Flash 0731
  34. 1,300 GPT-5.2 Codex
  35. 1,294 GPT-5.2
  36. 1,270 Gemini 3.8 Flash
  37. 1,245 GPT-5.1-Codex
  38. 1,244 MiMo-V2.6 Flash
  39. 1,213 Mistral Large 4
  40. 1,209 GPT-5.1-Codex-Max
  41. 1,192 GPT-5.1
  42. 1,189 Qwen3.7-Max
  43. 1,189 GPT-5.4 Mini
  44. 1,177 Gemini 3 Pro
  45. 1,162 GPT-5
  46. 1,161 Gemini 3.1 Pro
  47. 1,158 MiMo-V2.6 Pro
  48. 1,150 Grok 4.20
  49. 1,093 Kimi K2.6
  50. 1,092 DeepSeek V4.1 Flash
  51. 1,047 GLM-5.2
  52. 1,025 Claude Opus 4.5
  53. 1,006 DeepSeek-V4-Pro
  54. 1,005 GPT-5.4 Nano
  55. 997 Claude Opus 4.6
  56. 946 Inkling
  57. 944 Grok 4.3 Beta
  58. 934 o3
  59. 927 Gemma 4 26B A4B
  60. 926 Gemma 4 31B IT
  61. 911 Gemini 3.5 Flash
  62. 904 Gemini 3.7 Flash
  63. 900 MiMo-V2.5-Pro
  64. 887 GLM-5.1
  65. 886 Kimi K2.7 Code
  66. 826 o4-mini
  67. 822 Kimi K2.5
  68. 804 DeepSeek-R1
  69. 800 GPT-5 mini
  70. 798 Gemini 3.1 Flash-Lite
  71. 796 Claude Sonnet 4.5
  72. 786 Mercury 2
  73. 786 Gemini 2.5 Pro
  74. 785 MiMo-V2-Pro
  75. 766 GLM-5
  76. 765 Gemini 3.5 Flash-Lite
  77. 764 Mistral Medium 3.5
  78. 745 DeepSeek-V3.1-Terminus
  79. 738 MiMo-V2-Flash
  80. 719 GPT-5 nano
  81. 716 Gemini 3.6 Flash
  82. 694 Step 3.7 Flash
  83. 678 DeepSeek-V4-Flash
  84. 675 Claude Opus 4.1
  85. 670 Qwen 3.6 Plus
  86. 662 Gemini 2.5 Flash
  87. 655 Claude Sonnet 4
  88. 653 Claude Haiku 4.5
  89. 640 MiniMax-M3
  90. 624 MiniMax-M2.1
  91. 622 Qwen 3.5 Plus (hosted 397B-A17B)
  92. 618 MiniMax-M2.5
  93. 599 MiniMax-M2.7
  94. 598 Kimi K2 Thinking
  95. 588 grok-code-fast-1
  96. 576 gpt-oss-120b
  97. 566 gpt-oss-20b
  98. 558 GPT-4.1
  99. 514 MiMo-V2.5
  100. 498 Mistral Small 4
  101. 461 Qwen3-Coder-480B-A35B
  102. 457 qwen3-coder-plus
  103. 433 Ring-2.6-1T
  104. 399 GLM-4.7
  105. 395 Grok 4.1 Fast
  106. 370 Qwen3-Max
  107. 349 Qwen3.5-27B
  108. 345 GLM-4.5
  109. 341 GLM-4.6
  110. 326 Qwen 3.6 Flash
  111. 326 Gemini 2.5 Flash-Lite
  112. 304 GLM-5.3-Flash
  113. 302 Mercury 2.5
  114. 267 Kimi K2
  115. 265 Mistral Large 3
  116. 236 Amazon Nova Lite
  117. 222 Qwen 3.5 Flash (hosted 35B-A3B)
  118. 214 Nemotron 3 Super
  119. 210 Mistral Medium 3.1
  120. 173 Llama 4 Maverick
  121. 138 Codestral
  1. 1 GPT-6 Astra max 2,951
  2. 2 GPT-6.1 Sol max 2,744
  3. 3 GPT-6.1 Terra max 2,609
  4. 4 GPT-6 Sol max 2,462
  5. 5 GPT-5.6 Sol max 2,177
  6. 6 Claude Opus 5 high 2,165
  7. 7 Claude Opus 5.5 high 2,147
  8. 8 Claude Fable 5.1 high 2,143
  9. 9 Gemini 4 Argon high 2,100
  10. 10 Claude Fable 5 high 2,041
  11. 11 GPT-5.6 Terra max 1,951
  12. 12 GPT-5.5 xhigh 1,943
  13. 13 Kimi K3.1 Pro max 1,933
  14. 14 Claude Sonnet 5.5 high 1,819
  15. 15 Muse Spark 1.3 xhigh 1,688
  16. 16 GPT-5.6 Luna max 1,667
  17. 17 GPT-5.3 Codex xhigh 1,655
  18. 18 DeepSeek V4.1 Flash 1017 max 1,622
  19. 19 GPT-5.4 high 1,607
  20. 20 Muse Spark Open xhigh 1,590
  21. 21 Claude Haiku 5.5 high 1,587
  22. 22 GPT-6 Luna xhigh 1,577
  23. 23 Claude Opus 4.8 high 1,564
  24. 24 Kimi K3 max 1,524
  25. 25 Grok 4.6 xhigh 1,508
  26. 26 Claude Sonnet 5 high 1,463
  27. 27 DeepSeek V4 Pro 0813 max 1,403
  28. 28 Gemini 3 Flash 1,367
  29. 29 Claude Sonnet 4.6 medium 1,327
  30. 30 Claude Opus 4.7 no thinking 1,323
  31. 31 GLM-5.3 high 1,317
  32. 32 Grok 4.5 high 1,309
  33. 33 DeepSeek V4 Flash 0731 max 1,306
  34. 34 GPT-5.2 Codex 1,300
  35. 35 GPT-5.2 high 1,294
  36. 36 Gemini 3.8 Flash high 1,270
  37. 37 GPT-5.1-Codex 1,245
  38. 38 MiMo-V2.6 Flash 1,244
  39. 39 Mistral Large 4 1,213
  40. 40 GPT-5.1-Codex-Max 1,209
  41. 41 GPT-5.1 high 1,192
  42. 42 Qwen3.7-Max 1,189
  43. 43 GPT-5.4 Mini high 1,189
  44. 44 Gemini 3 Pro 1,177
  45. 45 GPT-5 high 1,162
  46. 46 Gemini 3.1 Pro 1,161
  47. 47 MiMo-V2.6 Pro 1,158
  48. 48 Grok 4.20 1,150
  49. 49 Kimi K2.6 1,093
  50. 50 DeepSeek V4.1 Flash max 1,092
  51. 51 GLM-5.2 high 1,047
  52. 52 Claude Opus 4.5 16k thinking 1,025
  53. 53 DeepSeek-V4-Pro high 1,006
  54. 54 GPT-5.4 Nano high 1,005
  55. 55 Claude Opus 4.6 no thinking 997
  56. 56 Inkling 946
  57. 57 Grok 4.3 Beta 944
  58. 58 o3 high 934
  59. 59 Gemma 4 26B A4B 927
  60. 60 Gemma 4 31B IT 926
  61. 61 Gemini 3.5 Flash high 911
  62. 62 Gemini 3.7 Flash high 904
  63. 63 MiMo-V2.5-Pro 900
  64. 64 GLM-5.1 887
  65. 65 Kimi K2.7 Code 886
  66. 66 o4-mini high 826
  67. 67 Kimi K2.5 822
  68. 68 DeepSeek-R1 May 2025 804
  69. 69 GPT-5 mini high 800
  70. 70 Gemini 3.1 Flash-Lite 798
  71. 71 Claude Sonnet 4.5 32k thinking 796
  72. 72 Mercury 2 786
  73. 73 Gemini 2.5 Pro Jun 2025, 32K 786
  74. 74 MiMo-V2-Pro 785
  75. 75 GLM-5 766
  76. 76 Gemini 3.5 Flash-Lite high 765
  77. 77 Mistral Medium 3.5 764
  78. 78 DeepSeek-V3.1-Terminus 745
  79. 79 MiMo-V2-Flash 738
  80. 80 GPT-5 nano high 719
  81. 81 Gemini 3.6 Flash high 716
  82. 82 Step 3.7 Flash 694
  83. 83 DeepSeek-V4-Flash high 678
  84. 84 Claude Opus 4.1 16K 675
  85. 85 Qwen 3.6 Plus 2026-04-02 670
  86. 86 Gemini 2.5 Flash Jun 2025 662
  87. 87 Claude Sonnet 4 32K 655
  88. 88 Claude Haiku 4.5 32K 653
  89. 89 MiniMax-M3 640
  90. 90 MiniMax-M2.1 624
  91. 91 Qwen 3.5 Plus (hosted 397B-A17B) 622
  92. 92 MiniMax-M2.5 618
  93. 93 MiniMax-M2.7 599
  94. 94 Kimi K2 Thinking 598
  95. 95 grok-code-fast-1 588
  96. 96 gpt-oss-120b 576
  97. 97 gpt-oss-20b 566
  98. 98 GPT-4.1 558
  99. 99 MiMo-V2.5 514
  100. 100 Mistral Small 4 498
  101. 101 Qwen3-Coder-480B-A35B 461
  102. 102 qwen3-coder-plus 457
  103. 103 Ring-2.6-1T 433
  104. 104 GLM-4.7 399
  105. 105 Grok 4.1 Fast 395
  106. 106 Qwen3-Max 370
  107. 107 Qwen3.5-27B 349
  108. 108 GLM-4.5 345
  109. 109 GLM-4.6 341
  110. 110 Qwen 3.6 Flash 2026-04-16 326
  111. 111 Gemini 2.5 Flash-Lite Jun 2025 326
  112. 112 GLM-5.3-Flash high 304
  113. 113 Mercury 2.5 high 302
  114. 114 Kimi K2 Sep 2025 267
  115. 115 Mistral Large 3 265
  116. 116 Amazon Nova Lite 236
  117. 117 Qwen 3.5 Flash (hosted 35B-A3B) 222
  118. 118 Nemotron 3 Super 214
  119. 119 Mistral Medium 3.1 210
  120. 120 Llama 4 Maverick FP8 173
  121. 121 Codestral 138
#ModelScoreSettingDateSource
1 GPT-6 Astra 2,951 max Sep 2026 Epoch AI
2 GPT-6.1 Sol 2,744 max Sep 2026 Epoch AI
3 GPT-6.1 Terra 2,609 max Oct 2026 Epoch AI
4 GPT-6 Sol 2,462 max Sep 2026 Epoch AI
5 GPT-5.6 Sol 2,177 max Jul 2026 Epoch AI
6 Claude Opus 5 2,165 high Jul 2026 Epoch AI
7 Claude Opus 5.5 2,147 high Sep 2026 Epoch AI
8 Claude Fable 5.1 2,143 high Sep 2026 Epoch AI
9 Gemini 4 Argon 2,100 high Sep 2026 Epoch AI
10 Claude Fable 5 2,041 high Jun 2026 Epoch AI
11 GPT-5.6 Terra 1,951 max Jul 2026 Epoch AI
12 GPT-5.5 1,943 xhigh Apr 2026 Epoch AI
13 Kimi K3.1 Pro 1,933 max Oct 2026 Epoch AI
14 Claude Sonnet 5.5 1,819 high Sep 2026 Epoch AI
15 Muse Spark 1.3 1,688 xhigh Sep 2026 Epoch AI
16 GPT-5.6 Luna 1,667 max Jul 2026 Epoch AI
17 GPT-5.3 Codex 1,655 xhigh Feb 2026 Epoch AI
18 DeepSeek V4.1 Flash 1017 1,622 max Oct 2026 Epoch AI
19 GPT-5.4 1,607 high Mar 2026 Epoch AI
20 Muse Spark Open 1,590 xhigh Oct 2026 Epoch AI
21 Claude Haiku 5.5 1,587 high Oct 2026 Epoch AI
22 GPT-6 Luna 1,577 xhigh Sep 2026 Epoch AI
23 Claude Opus 4.8 1,564 high May 2026 Epoch AI
24 Kimi K3 1,524 max Jul 2026 Epoch AI
25 Grok 4.6 1,508 xhigh Aug 2026 Epoch AI
26 Claude Sonnet 5 1,463 high Jun 2026 Epoch AI
27 DeepSeek V4 Pro 0813 1,403 max Aug 2026 Epoch AI
28 Gemini 3 Flash 1,367 Dec 2025 Epoch AI
29 Claude Sonnet 4.6 1,327 medium Feb 2026 Epoch AI
30 Claude Opus 4.7 1,323 no thinking Apr 2026 Epoch AI
31 GLM-5.3 1,317 high Aug 2026 Epoch AI
32 Grok 4.5 1,309 high Jul 2026 Epoch AI
33 DeepSeek V4 Flash 0731 1,306 max Jul 2026 Epoch AI
34 GPT-5.2 Codex 1,300 Dec 2025 Epoch AI
35 GPT-5.2 1,294 high Dec 2025 Epoch AI
36 Gemini 3.8 Flash 1,270 high Sep 2026 Epoch AI
37 GPT-5.1-Codex 1,245 Nov 2025 Epoch AI
38 MiMo-V2.6 Flash 1,244 Sep 2026 Epoch AI
39 Mistral Large 4 1,213 Oct 2026 Epoch AI
40 GPT-5.1-Codex-Max 1,209 Nov 2025 Epoch AI
41 GPT-5.1 1,192 high Nov 2025 Epoch AI
42 Qwen3.7-Max 1,189 May 2026 Epoch AI
43 GPT-5.4 Mini 1,189 high Mar 2026 Epoch AI
44 Gemini 3 Pro 1,177 Nov 2025 Epoch AI
45 GPT-5 1,162 high Aug 2025 Epoch AI
46 Gemini 3.1 Pro 1,161 Feb 2026 Epoch AI
47 MiMo-V2.6 Pro 1,158 Sep 2026 Epoch AI
48 Grok 4.20 1,150 Feb 2026 Epoch AI
49 Kimi K2.6 1,093 Apr 2026 Epoch AI
50 DeepSeek V4.1 Flash 1,092 max Sep 2026 Epoch AI
51 GLM-5.2 1,047 high Jun 2026 Epoch AI
52 Claude Opus 4.5 1,025 16k thinking Nov 2025 Epoch AI
53 DeepSeek-V4-Pro 1,006 high Apr 2026 Epoch AI
54 GPT-5.4 Nano 1,005 high Mar 2026 Epoch AI
55 Claude Opus 4.6 997 no thinking Feb 2026 Epoch AI
56 Inkling 946 Jul 2026 Epoch AI
57 Grok 4.3 Beta 944 Apr 2026 Epoch AI
58 o3 934 high Apr 2025 Epoch AI
59 Gemma 4 26B A4B 927 Apr 2026 Epoch AI
60 Gemma 4 31B IT 926 Apr 2026 Epoch AI
61 Gemini 3.5 Flash 911 high May 2026 Epoch AI
62 Gemini 3.7 Flash 904 high Aug 2026 Epoch AI
63 MiMo-V2.5-Pro 900 Apr 2026 Epoch AI
64 GLM-5.1 887 Apr 2026 Epoch AI
65 Kimi K2.7 Code 886 Jun 2026 Epoch AI
66 o4-mini 826 high Apr 2025 Epoch AI
67 Kimi K2.5 822 Jan 2026 Epoch AI
68 DeepSeek-R1 May 2025 804 May 2025 Epoch AI
69 GPT-5 mini 800 high Aug 2025 Epoch AI
70 Gemini 3.1 Flash-Lite 798 Mar 2026 Epoch AI
71 Claude Sonnet 4.5 796 32k thinking Sep 2025 Epoch AI
72 Mercury 2 786 Feb 2026 Epoch AI
73 Gemini 2.5 Pro Jun 2025 786 32K Jun 2025 Epoch AI
74 MiMo-V2-Pro 785 Mar 2026 Epoch AI
75 GLM-5 766 Feb 2026 Epoch AI
76 Gemini 3.5 Flash-Lite 765 high Jul 2026 Epoch AI
77 Mistral Medium 3.5 764 Apr 2026 Epoch AI
78 DeepSeek-V3.1-Terminus 745 Sep 2025 Epoch AI
79 MiMo-V2-Flash 738 Dec 2025 Epoch AI
80 GPT-5 nano 719 high Aug 2025 Epoch AI
81 Gemini 3.6 Flash 716 high Jul 2026 Epoch AI
82 Step 3.7 Flash 694 May 2026 Epoch AI
83 DeepSeek-V4-Flash 678 high Apr 2026 Epoch AI
84 Claude Opus 4.1 675 16K Aug 2025 Epoch AI
85 Qwen 3.6 Plus 670 2026-04-02 Mar 2026 Epoch AI
86 Gemini 2.5 Flash Jun 2025 662 Jun 2025 Epoch AI
87 Claude Sonnet 4 655 32K May 2025 Epoch AI
88 Claude Haiku 4.5 653 32K Oct 2025 Epoch AI
89 MiniMax-M3 640 Jun 2026 Epoch AI
90 MiniMax-M2.1 624 Dec 2025 Epoch AI
91 Qwen 3.5 Plus (hosted 397B-A17B) 622 Feb 2026 Epoch AI
92 MiniMax-M2.5 618 Feb 2026 Epoch AI
93 MiniMax-M2.7 599 Mar 2026 Epoch AI
94 Kimi K2 Thinking 598 Nov 2025 Epoch AI
95 grok-code-fast-1 588 Aug 2025 Epoch AI
96 gpt-oss-120b 576 Aug 2025 Epoch AI
97 gpt-oss-20b 566 Aug 2025 Epoch AI
98 GPT-4.1 558 Apr 2025 Epoch AI
99 MiMo-V2.5 514 Apr 2026 Epoch AI
100 Mistral Small 4 498 Mar 2026 Epoch AI
101 Qwen3-Coder-480B-A35B 461 Jul 2025 Epoch AI
102 qwen3-coder-plus 457 Sep 2025 Epoch AI
103 Ring-2.6-1T 433 May 2026 Epoch AI
104 GLM-4.7 399 Dec 2025 Epoch AI
105 Grok 4.1 Fast 395 Nov 2025 Epoch AI
106 Qwen3-Max 370 Sep 2025 Epoch AI
107 Qwen3.5-27B 349 Feb 2026 Epoch AI
108 GLM-4.5 345 Aug 2025 Epoch AI
109 GLM-4.6 341 Sep 2025 Epoch AI
110 Qwen 3.6 Flash 326 2026-04-16 Apr 2026 Epoch AI
111 Gemini 2.5 Flash-Lite Jun 2025 326 Jun 2025 Epoch AI
112 GLM-5.3-Flash 304 high Aug 2026 Epoch AI
113 Mercury 2.5 302 high Sep 2026 Epoch AI
114 Kimi K2 Sep 2025 267 Sep 2025 Epoch AI
115 Mistral Large 3 265 Dec 2025 Epoch AI
116 Amazon Nova Lite 236 Dec 2024 Epoch AI
117 Qwen 3.5 Flash (hosted 35B-A3B) 222 Feb 2026 Epoch AI
118 Nemotron 3 Super 214 Mar 2026 Epoch AI
119 Mistral Medium 3.1 210 Aug 2025 Epoch AI
120 Llama 4 Maverick 173 FP8 Apr 2025 Epoch AI
121 Codestral 138 Jul 2025 Epoch AI

10 models have held number one since its release in Jun 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind Anthropic
Epoch AI runs 10 leaders
GPT-6 Astra 2,951

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Run or collected by Epoch AI
0 1,000 2,000 3,000 2025 2026 2027 0 1,000 2,000 3,000 2025 2026 2027

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471