Benchmark Almanac

WeirdML

GPT-6 Astra Pro 93.6% leads since Sep 2026 Not in model cards

Models write PyTorch code to solve 17 unusual machine learning tasks, with 5 attempts and feedback each. Scoring is mean best test accuracy.

This record is WeirdML v2 (17 tasks), launched in June 2025 and run on Epoch AI Benchmarking Hub. v1 (6 tasks, December 2024) is archived. WeirdML v3 replaced v2 in September 2026.

Made by Håvard Tveit Ihle Released Jun 2025 Machine learning engineering Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Code A program or patch run against tests.
Leaderboard htihle.github.io
131
Entries on its own leaderboard, 112 models we track

GPT-6 Astra Pro leads with 93.6% among models released by Sep 2026

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Sep 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

131 entries. Scroll sideways for the rest.

  1. 93.6 GPT-6 Astra Pro
  2. 93.3 GPT-6 Astra
  3. 92.9 Claude Fable 5.1
  4. 92.0 Claude Fable 5
  5. 91.8 Claude Opus 5
  6. 89.4 GPT-5.6 Sol
  7. 84.9 GPT-5.5
  8. 84.8 Gemini 3.8 Flash
  9. 82.9 Claude Opus 4.8
  10. 82.6 Kimi K3
  11. 78.3 GPT-5.6 Terra
  12. 78.0 Claude Opus 4.6
  13. 77.9 GPT-5.3 Codex
  14. 77.7 GPT-5.4
  15. 76.4 Claude Opus 4.7
  16. 75.4 GLM-5.3
  17. 75.2 Qwen3.8-2.4T-A95B
  18. 72.2 GPT-5.2
  19. 72.1 Gemini 3.1 Pro
  20. 70.1 GLM-5.2
  21. 69.9 Gemini 3 Pro
  22. 68.8 Claude Sonnet 5
  23. 67.3 Grok 4.6
  24. 66.2 DeepSeek V4 Pro 0813
  25. 66.1 Claude Sonnet 4.6
  26. 63.7 Claude Opus 4.5
  27. 63.0 DeepSeek V4 Flash 0731
  28. 62.6 Gemini 3.5 Flash
  29. 61.6 Gemini 3 Flash
  30. 60.9 GPT-5.6 Luna
  31. 60.8 GPT-5.1
  32. 60.7 GPT-5
  33. 60.4 GPT-5 Pro
  34. 60.3 GPT-5.4 Mini
  35. 60.3 Muse Spark 1.2
  36. 58.2 o3-pro
  37. 57.1 GLM-5.1
  38. 56.1 Gemini 3.6 Flash
  39. 55.9 Kimi K2.6
  40. 54.5 GPT-5-Codex
  41. 54.1 Kimi K2.7 Code
  42. 54.0 Gemini 2.5 Pro
  43. 52.7 GPT-5 mini
  44. 52.6 o4-mini
  45. 52.4 o3
  46. 52.3 x-ai/grok-4.20-beta
  47. 52.3 gemma4:31b-it-q4_K_M
  48. 52.2 Gemini 3.1 Flash-Lite
  49. 49.9 Grok 4.3
  50. 49.2 GPT-5.4 Nano
  51. 48.9 DeepSeek-V4-Pro
  52. 48.2 gpt-oss-120b
  53. 48.2 GLM-5
  54. 47.7 Claude Sonnet 4.5
  55. 47.6 o1-preview
  56. 46.7 DeepSeek-V3.2-Speciale
  57. 46.4 Grok 4.5
  58. 46.1 Claude Sonnet 4
  59. 46.1 o1
  60. 45.9 Claude Opus 4.1
  61. 45.7 Grok 4
  62. 45.6 DeepSeek-V4-Flash
  63. 45.6 Kimi K2.5
  64. 45.4 Claude Haiku 4.5
  65. 43.7 Claude Opus 4
  66. 43.7 mistral-medium-3.5:128b-q4_K_M
  67. 43.7 o3-mini
  68. 43.5 Nemotron 3 Ultra
  69. 43.2 Mercury 2
  70. 42.9 Grok 4 Fast
  71. 42.8 Kimi K2 Thinking
  72. 42.6 Grok-3 mini
  73. 41.9 Gemini 2.5 Flash
  74. 41.6 DeepSeek-R1
  75. 41.2 Qwen3-Coder-480B-A35B
  76. 41.0 Qwen3-235B-A22B-Thinking
  77. 41.0 Gemini 2.5 Flash
  78. 40.9 gpt-oss-20b
  79. 40.6 GLM-4.5
  80. 40.0 Claude 3.5 Sonnet
  81. 39.8 GPT-5 Chat
  82. 39.5 qwen3.5:27b-q4_K_M
  83. 39.5 DeepSeek-V3.2-Exp
  84. 39.4 GPT-4.5
  85. 39.4 Kimi K2
  86. 39.0 GPT-4.1
  87. 39.0 Gemini 3.5 Flash-Lite
  88. 38.7 Qwen3-235B-A22B-Instruct
  89. 38.4 DeepSeek-V3.1
  90. 38.1 GPT-5 nano
  91. 38.0 nemotron-3-super:120b-a12b-q4_K_M
  92. 37.6 GPT-4.1 mini
  93. 37.3 Qwen3-235B-A22B
  94. 37.2 Grok 3
  95. 37.0 MiniMax-M2.7
  96. 36.7 moonshotai/kimi-k2-0905
  97. 36.5 DeepSeek-R1
  98. 36.3 o1-mini
  99. 36.1 deepseek/deepseek-chat-v3-0324
  100. 35.2 Gemini 2.5 Flash-Lite
  101. 35.2 gemma4:26b-a4b-it-q4_K_M
  102. 35.1 grok-code-fast-1
  103. 34.5 qwen3.6:35b-a3b-q4_K_M
  104. 34.4 qwen3-coder-next
  105. 33.1 Mistral Medium 3.1
  106. 32.3 Inkling
  107. 30.7 Claude 3.5 Haiku
  108. 29.8 Qwen3-30B-A3B
  109. 25.8 google/gemini-2.0-flash-001
  110. 25.1 GPT-4o
  111. 24.9 google/gemini-flash-1.5
  112. 24.5 Llama 4 Maverick
  113. 22.2 x-ai/grok-2-1212
  114. 22.2 google/gemini-pro-1.5
  115. 21.4 meta-llama/llama-3.1-405b-instruct
  116. 19.2 Claude 3 Opus
  117. 19.0 GPT-4.1 nano
  118. 18.0 GPT-4 Turbo
  119. 16.0 Qwen2.5-72B
  120. 14.4 Llama 3.3 70B Instruct
  121. 13.0 deepseek-r1-0528-qwen3-8b
  122. 12.4 GPT-4
  123. 11.8 GPT-4o mini
  124. 11.3 qwen2:72b-instruct-q8_0
  125. 10.2 Claude 3 Sonnet
  126. 9.84 Claude 3 Haiku
  127. 8.97 meta-llama/llama-3.1-70b-instruct
  128. 7.06 Claude 2.1
  129. 3.48 openai/gpt-3.5-turbo-16k
  130. 3.17 mistralai/mixtral-8x22b-instruct
  131. 1.73 meta-llama/llama-3.1-8b-instruct
  1. 1 GPT-6 Astra Pro max 93.6%
  2. 2 GPT-6 Astra max 93.3%
  3. 3 Claude Fable 5.1 max 92.9%
  4. 4 Claude Fable 5 max 92%
  5. 5 Claude Opus 5 max 91.8%
  6. 6 GPT-5.6 Sol max 89.4%
  7. 7 GPT-5.5 xhigh 84.9%
  8. 8 Gemini 3.8 Flash high 84.8%
  9. 9 Claude Opus 4.8 xhigh 82.9%
  10. 10 Kimi K3 max 82.6%
  11. 11 GPT-5.6 Terra high 78.3%
  12. 12 Claude Opus 4.6 high 78%
  13. 13 GPT-5.3 Codex xhigh 77.9%
  14. 14 GPT-5.4 xhigh 77.7%
  15. 15 Claude Opus 4.7 high 76.4%
  16. 16 GLM-5.3 max 75.4%
  17. 17 Qwen3.8-2.4T-A95B xhigh 75.2%
  18. 18 GPT-5.2 xhigh 72.2%
  19. 19 Gemini 3.1 Pro high 72.1%
  20. 20 GLM-5.2 max 70.1%
  21. 21 Gemini 3 Pro high 69.9%
  22. 22 Claude Sonnet 5 high 68.8%
  23. 23 Grok 4.6 high 67.3%
  24. 24 DeepSeek V4 Pro 0813 max 66.2%
  25. 25 Claude Sonnet 4.6 medium 66.1%
  26. 26 Claude Opus 4.5 high, 16k 63.7%
  27. 27 DeepSeek V4 Flash 0731 max 63%
  28. 28 Gemini 3.5 Flash high 62.6%
  29. 29 Gemini 3 Flash high 61.6%
  30. 30 GPT-5.6 Luna high 60.9%
  31. 31 GPT-5.1 high 60.8%
  32. 32 GPT-5 high 60.7%
  33. 33 GPT-5 Pro high 60.4%
  34. 34 GPT-5.4 Mini high 60.3%
  35. 35 Muse Spark 1.2 xhigh 60.3%
  36. 36 o3-pro high 58.2%
  37. 37 GLM-5.1 57.1%
  38. 38 Gemini 3.6 Flash high 56.1%
  39. 39 Kimi K2.6 55.9%
  40. 40 GPT-5-Codex high 54.5%
  41. 41 Kimi K2.7 Code 54.1%
  42. 42 Gemini 2.5 Pro Jun 2025, thinking 16k 54%
  43. 43 GPT-5 mini high 52.7%
  44. 44 o4-mini high 52.6%
  45. 45 o3 high 52.4%
  46. 46 x-ai/grok-4.20-beta 52.3%
  47. 47 gemma4:31b-it-q4_K_M 52.3%
  48. 48 Gemini 3.1 Flash-Lite high 52.2%
  49. 49 Grok 4.3 49.9%
  50. 50 GPT-5.4 Nano high 49.2%
  51. 51 DeepSeek-V4-Pro max 48.9%
  52. 52 gpt-oss-120b high 48.2%
  53. 53 GLM-5 thinking 48.2%
  54. 54 Claude Sonnet 4.5 thinking 16k 47.7%
  55. 55 o1-preview 47.6%
  56. 56 DeepSeek-V3.2-Speciale 46.7%
  57. 57 Grok 4.5 46.4%
  58. 58 Claude Sonnet 4 thinking 16k 46.1%
  59. 59 o1 high 46.1%
  60. 60 Claude Opus 4.1 thinking 16k 45.9%
  61. 61 Grok 4 45.7%
  62. 62 DeepSeek-V4-Flash max 45.6%
  63. 63 Kimi K2.5 45.6%
  64. 64 Claude Haiku 4.5 no thinking 45.4%
  65. 65 Claude Opus 4 thinking 16k 43.7%
  66. 66 mistral-medium-3.5:128b-q4_K_M 43.7%
  67. 67 o3-mini high 43.7%
  68. 68 Nemotron 3 Ultra 43.5%
  69. 69 Mercury 2 high 43.2%
  70. 70 Grok 4 Fast 42.9%
  71. 71 Kimi K2 Thinking 42.8%
  72. 72 Grok-3 mini high 42.6%
  73. 73 Gemini 2.5 Flash Sep 2025, thinking 16k 41.9%
  74. 74 DeepSeek-R1 May 2025 41.6%
  75. 75 Qwen3-Coder-480B-A35B 41.2%
  76. 76 Qwen3-235B-A22B-Thinking Jul 2025 41%
  77. 77 Gemini 2.5 Flash Jun 2025, thinking 16k 41%
  78. 78 gpt-oss-20b high 40.9%
  79. 79 GLM-4.5 thinking 40.6%
  80. 80 Claude 3.5 Sonnet 40%
  81. 81 GPT-5 Chat 39.8%
  82. 82 qwen3.5:27b-q4_K_M 39.5%
  83. 83 DeepSeek-V3.2-Exp thinking 39.5%
  84. 84 GPT-4.5 39.4%
  85. 85 Kimi K2 Jul 2025 39.4%
  86. 86 GPT-4.1 39%
  87. 87 Gemini 3.5 Flash-Lite high 39%
  88. 88 Qwen3-235B-A22B-Instruct Jul 2025 38.7%
  89. 89 DeepSeek-V3.1 thinking 38.4%
  90. 90 GPT-5 nano high 38.1%
  91. 91 nemotron-3-super:120b-a12b-q4_K_M 38%
  92. 92 GPT-4.1 mini 37.6%
  93. 93 Qwen3-235B-A22B thinking 37.3%
  94. 94 Grok 3 37.2%
  95. 95 MiniMax-M2.7 37%
  96. 96 moonshotai/kimi-k2-0905 36.7%
  97. 97 DeepSeek-R1 36.5%
  98. 98 o1-mini 36.3%
  99. 99 deepseek/deepseek-chat-v3-0324 36.1%
  100. 100 Gemini 2.5 Flash-Lite Jun 2025, thinking 16k 35.2%
  101. 101 gemma4:26b-a4b-it-q4_K_M 35.2%
  102. 102 grok-code-fast-1 35.1%
  103. 103 qwen3.6:35b-a3b-q4_K_M 34.5%
  104. 104 qwen3-coder-next 34.4%
  105. 105 Mistral Medium 3.1 33.1%
  106. 106 Inkling high 32.3%
  107. 107 Claude 3.5 Haiku 30.7%
  108. 108 Qwen3-30B-A3B thinking 29.8%
  109. 109 google/gemini-2.0-flash-001 25.8%
  110. 110 GPT-4o Nov 2024 25.1%
  111. 111 google/gemini-flash-1.5 24.9%
  112. 112 Llama 4 Maverick 24.5%
  113. 113 x-ai/grok-2-1212 22.2%
  114. 114 google/gemini-pro-1.5 22.2%
  115. 115 meta-llama/llama-3.1-405b-instruct 21.4%
  116. 116 Claude 3 Opus 19.2%
  117. 117 GPT-4.1 nano 19%
  118. 118 GPT-4 Turbo Apr 2024 18%
  119. 119 Qwen2.5-72B 16%
  120. 120 Llama 3.3 70B Instruct 14.4%
  121. 121 deepseek-r1-0528-qwen3-8b 13%
  122. 122 GPT-4 Jun 2023 12.4%
  123. 123 GPT-4o mini 11.8%
  124. 124 qwen2:72b-instruct-q8_0 11.3%
  125. 125 Claude 3 Sonnet 10.2%
  126. 126 Claude 3 Haiku 9.84%
  127. 127 meta-llama/llama-3.1-70b-instruct 8.97%
  128. 128 Claude 2.1 7.06%
  129. 129 openai/gpt-3.5-turbo-16k 3.48%
  130. 130 mistralai/mixtral-8x22b-instruct 3.17%
  131. 131 meta-llama/llama-3.1-8b-instruct 1.73%
#EntryScoreSettingDateSource
1 GPT-6 Astra Pro 93.6% max Sep 2026 Official leaderboard
2 GPT-6 Astra 93.3% max Sep 2026 Official leaderboard
3 Claude Fable 5.1 92.9% max Sep 2026 Official leaderboard
4 Claude Fable 5 92% max Jun 2026 Official leaderboard
5 Claude Opus 5 91.8% max Jul 2026 Official leaderboard
6 GPT-5.6 Sol 89.4% max Jul 2026 Official leaderboard
7 GPT-5.5 84.9% xhigh Apr 2026 Official leaderboard
8 Gemini 3.8 Flash 84.8% high Sep 2026 Official leaderboard
9 Claude Opus 4.8 82.9% xhigh May 2026 Official leaderboard
10 Kimi K3 82.6% max Jul 2026 Official leaderboard
11 GPT-5.6 Terra 78.3% high Jul 2026 Official leaderboard
12 Claude Opus 4.6 78% high Feb 2026 Official leaderboard
13 GPT-5.3 Codex 77.9% xhigh Feb 2026 Official leaderboard
14 GPT-5.4 77.7% xhigh Mar 2026 Official leaderboard
15 Claude Opus 4.7 76.4% high Apr 2026 Official leaderboard
16 GLM-5.3 75.4% max Aug 2026 Official leaderboard
17 Qwen3.8-2.4T-A95B 75.2% xhigh Aug 2026 Official leaderboard
18 GPT-5.2 72.2% xhigh Dec 2025 Official leaderboard
19 Gemini 3.1 Pro 72.1% high Feb 2026 Official leaderboard
20 GLM-5.2 70.1% max Jun 2026 Official leaderboard
21 Gemini 3 Pro 69.9% high Nov 2025 Official leaderboard
22 Claude Sonnet 5 68.8% high Jun 2026 Official leaderboard
23 Grok 4.6 67.3% high Aug 2026 Official leaderboard
24 DeepSeek V4 Pro 0813 66.2% max Aug 2026 Official leaderboard
25 Claude Sonnet 4.6 66.1% medium Feb 2026 Official leaderboard
26 Claude Opus 4.5 63.7% high, 16k Nov 2025 Official leaderboard
27 DeepSeek V4 Flash 0731 63% max Jul 2026 Official leaderboard
28 Gemini 3.5 Flash 62.6% high May 2026 Official leaderboard
29 Gemini 3 Flash 61.6% high Dec 2025 Official leaderboard
30 GPT-5.6 Luna 60.9% high Jul 2026 Official leaderboard
31 GPT-5.1 60.8% high Nov 2025 Official leaderboard
32 GPT-5 60.7% high Aug 2025 Official leaderboard
33 GPT-5 Pro 60.4% high Oct 2025 Official leaderboard
34 GPT-5.4 Mini 60.3% high Mar 2026 Official leaderboard
35 Muse Spark 1.2 60.3% xhigh Aug 2026 Official leaderboard
36 o3-pro 58.2% high Jun 2025 Official leaderboard
37 GLM-5.1 57.1% Apr 2026 Official leaderboard
38 Gemini 3.6 Flash 56.1% high Jul 2026 Official leaderboard
39 Kimi K2.6 55.9% Apr 2026 Official leaderboard
40 GPT-5-Codex 54.5% high Sep 2025 Official leaderboard
41 Kimi K2.7 Code 54.1% Jun 2026 Official leaderboard
42 Gemini 2.5 Pro Jun 2025 54% thinking 16k Jun 2025 Official leaderboard
43 GPT-5 mini 52.7% high Aug 2025 Official leaderboard
44 o4-mini 52.6% high Apr 2025 Official leaderboard
45 o3 52.4% high Apr 2025 Official leaderboard
46 x-ai/grok-4.20-beta 52.3% Mar 2026 Official leaderboard
47 gemma4:31b-it-q4_K_M 52.3% Apr 2026 Official leaderboard
48 Gemini 3.1 Flash-Lite 52.2% high Mar 2026 Official leaderboard
49 Grok 4.3 49.9% Apr 2026 Official leaderboard
50 GPT-5.4 Nano 49.2% high Mar 2026 Official leaderboard
51 DeepSeek-V4-Pro 48.9% max Apr 2026 Official leaderboard
52 gpt-oss-120b 48.2% high Aug 2025 Official leaderboard
53 GLM-5 48.2% thinking Feb 2026 Official leaderboard
54 Claude Sonnet 4.5 47.7% thinking 16k Sep 2025 Official leaderboard
55 o1-preview 47.6% Sep 2024 Official leaderboard
56 DeepSeek-V3.2-Speciale 46.7% Dec 2025 Official leaderboard
57 Grok 4.5 46.4% Jul 2026 Official leaderboard
58 Claude Sonnet 4 46.1% thinking 16k May 2025 Official leaderboard
59 o1 46.1% high Dec 2024 Official leaderboard
60 Claude Opus 4.1 45.9% thinking 16k Aug 2025 Official leaderboard
61 Grok 4 45.7% Jul 2025 Official leaderboard
62 DeepSeek-V4-Flash 45.6% max Apr 2026 Official leaderboard
63 Kimi K2.5 45.6% Jan 2026 Official leaderboard
64 Claude Haiku 4.5 45.4% no thinking Oct 2025 Official leaderboard
65 Claude Opus 4 43.7% thinking 16k May 2025 Official leaderboard
66 mistral-medium-3.5:128b-q4_K_M 43.7% Apr 2026 Official leaderboard
67 o3-mini 43.7% high Jan 2025 Official leaderboard
68 Nemotron 3 Ultra 43.5% Jun 2026 Official leaderboard
69 Mercury 2 43.2% high Feb 2026 Official leaderboard
70 Grok 4 Fast 42.9% Sep 2025 Official leaderboard
71 Kimi K2 Thinking 42.8% Nov 2025 Official leaderboard
72 Grok-3 mini 42.6% high Apr 2025 Official leaderboard
73 Gemini 2.5 Flash Sep 2025 41.9% thinking 16k Sep 2025 Official leaderboard
74 DeepSeek-R1 May 2025 41.6% May 2025 Official leaderboard
75 Qwen3-Coder-480B-A35B 41.2% Jul 2025 Official leaderboard
76 Qwen3-235B-A22B-Thinking Jul 2025 41% Jul 2025 Official leaderboard
77 Gemini 2.5 Flash Jun 2025 41% thinking 16k Jun 2025 Official leaderboard
78 gpt-oss-20b 40.9% high Aug 2025 Official leaderboard
79 GLM-4.5 40.6% thinking Aug 2025 Official leaderboard
80 Claude 3.5 Sonnet 40% Jun 2024 Official leaderboard
81 GPT-5 Chat 39.8% Aug 2025 Official leaderboard
82 qwen3.5:27b-q4_K_M 39.5% Feb 2026 Official leaderboard
83 DeepSeek-V3.2-Exp 39.5% thinking Sep 2025 Official leaderboard
84 GPT-4.5 39.4% Feb 2025 Official leaderboard
85 Kimi K2 Jul 2025 39.4% Jul 2025 Official leaderboard
86 GPT-4.1 39% Apr 2025 Official leaderboard
87 Gemini 3.5 Flash-Lite 39% high Jul 2026 Official leaderboard
88 Qwen3-235B-A22B-Instruct Jul 2025 38.7% Jul 2025 Official leaderboard
89 DeepSeek-V3.1 38.4% thinking Aug 2025 Official leaderboard
90 GPT-5 nano 38.1% high Aug 2025 Official leaderboard
91 nemotron-3-super:120b-a12b-q4_K_M 38% Mar 2026 Official leaderboard
92 GPT-4.1 mini 37.6% Apr 2025 Official leaderboard
93 Qwen3-235B-A22B 37.3% thinking Apr 2025 Official leaderboard
94 Grok 3 37.2% Apr 2025 Official leaderboard
95 MiniMax-M2.7 37% Mar 2026 Official leaderboard
96 moonshotai/kimi-k2-0905 36.7% Sep 2025 Official leaderboard
97 DeepSeek-R1 36.5% Jan 2025 Official leaderboard
98 o1-mini 36.3% Sep 2024 Official leaderboard
99 deepseek/deepseek-chat-v3-0324 36.1% Mar 2025 Official leaderboard
100 Gemini 2.5 Flash-Lite Jun 2025 35.2% thinking 16k Jun 2025 Official leaderboard
101 gemma4:26b-a4b-it-q4_K_M 35.2% Apr 2026 Official leaderboard
102 grok-code-fast-1 35.1% Aug 2025 Official leaderboard
103 qwen3.6:35b-a3b-q4_K_M 34.5% Apr 2026 Official leaderboard
104 qwen3-coder-next 34.4% Feb 2026 Official leaderboard
105 Mistral Medium 3.1 33.1% Aug 2025 Official leaderboard
106 Inkling 32.3% high Jul 2026 Official leaderboard
107 Claude 3.5 Haiku 30.7% Oct 2024 Official leaderboard
108 Qwen3-30B-A3B 29.8% thinking Apr 2025 Official leaderboard
109 google/gemini-2.0-flash-001 25.8% Feb 2025 Official leaderboard
110 GPT-4o Nov 2024 25.1% Nov 2024 Official leaderboard
111 google/gemini-flash-1.5 24.9% Sep 2024 Official leaderboard
112 Llama 4 Maverick 24.5% Apr 2025 Official leaderboard
113 x-ai/grok-2-1212 22.2% Dec 2024 Official leaderboard
114 google/gemini-pro-1.5 22.2% Sep 2024 Official leaderboard
115 meta-llama/llama-3.1-405b-instruct 21.4% Jul 2024 Official leaderboard
116 Claude 3 Opus 19.2% Feb 2024 Official leaderboard
117 GPT-4.1 nano 19% Apr 2025 Official leaderboard
118 GPT-4 Turbo Apr 2024 18% Apr 2024 Official leaderboard
119 Qwen2.5-72B 16% Sep 2024 Official leaderboard
120 Llama 3.3 70B Instruct 14.4% Dec 2024 Official leaderboard
121 deepseek-r1-0528-qwen3-8b 13% May 2025 Official leaderboard
122 GPT-4 Jun 2023 12.4% Jun 2023 Official leaderboard
123 GPT-4o mini 11.8% Jul 2024 Official leaderboard
124 qwen2:72b-instruct-q8_0 11.3% Jun 2024 Official leaderboard
125 Claude 3 Sonnet 10.2% Feb 2024 Official leaderboard
126 Claude 3 Haiku 9.84% Mar 2024 Official leaderboard
127 meta-llama/llama-3.1-70b-instruct 8.97% Jul 2024 Official leaderboard
128 Claude 2.1 7.06% Nov 2023 Official leaderboard
129 openai/gpt-3.5-turbo-16k 3.48% Jan 2024 Official leaderboard
130 mistralai/mixtral-8x22b-instruct 3.17% Apr 2024 Official leaderboard
131 meta-llama/llama-3.1-8b-instruct 1.73% Jul 2024 Official leaderboard

Epoch AI has scores for 130 models. We plot every source in the charts below.

The best score, 93.6%, costs $29 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-5.6 Sol at high comes within 4.8 points of the best for $1.25 a task, 23 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Meta OpenAI Other labs DeepSeek Anthropic Google DeepMind Cost frontier One model at each effort level
0% 20% 40% 60% 80% 100% $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra Pro, max GPT-6 Astra, high GPT-5.6 Sol, high GPT-5.6 Terra, high DeepSeek V4 Pro 0813, max DeepSeek V4 Flash 0731, max DeepSeek V4 Flash 0731, high 0% 20% 40% 60% 80% 100% $0.001 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra Pro, max Gemini 3.8 Flash, high
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra Pro max 93.6% $29 64,161
GPT-6 Astra max 93.3% $7.75 21,941
Claude Fable 5.1 max 92.9% $12 31,644
GPT-6 Astra high 92.9% $2.98 8,056
Claude Fable 5.1 high 92.3% $3.20 8,368
Claude Fable 5 max 92% $8.42 22,153
Claude Opus 5 max 91.8% $4.74 25,400
Claude Opus 5 high 91.6% $2.19 11,841
GPT-5.6 Sol max 89.4% $3.87 20,241
GPT-5.6 Sol high 88.8% $1.25 7,245
Claude Fable 5 high 87.8% $3.11 8,393
GPT-5.6 Sol max 87% $2.31 10,828
Claude Opus 5 no thinking 86.3% $1.42 7,394
GPT-5.5 xhigh 84.9% $2.57 15,144
Gemini 3.8 Flash high 84.8% $0.51 24,698
GPT-5.5 high 83.9% $2.22 12,659
Claude Opus 4.8 xhigh 82.9% $2.35 12,515
Kimi K3 max 82.6% $2.05 16,306
GPT-5.6 Terra high 78.3% $0.25 3,479
Claude Opus 4.6 high 78% $5.78 31,733
GPT-5.3 Codex xhigh 77.9% $2.45 33,792
GPT-5.4 xhigh 77.7% $5.72 71,878
Claude Opus 4.7 high 76.4% $1.82 9,965
Claude Opus 4.7 no thinking 76.4% $0.69 3,594
Claude Opus 4.8 medium 76% $0.85 4,332
Claude Opus 4.7 max 75.5% $7.95 43,183
GLM-5.3 max 75.4% $0.54 16,943
Qwen3.8-2.4T-A95B xhigh 75.2% $1.42 32,618
GPT-5.2 xhigh 72.2% $2.05 27,867
Gemini 3.1 Pro high 72.1% $1.36 21,437
Claude Opus 4.8 no thinking 70.5% $0.48 2,414
GLM-5.2 max 70.1% $0.51 21,907
Gemini 3 Pro high 69.9% $0.53 7,795
Claude Sonnet 5 high 68.8% $0.90 11,593
GLM-5.2 high 67.3% $0.28 11,795
Grok 4.6 high 67.3% $0.33 7,558
GPT-5.5 no thinking 67.2% $0.54 2,493
DeepSeek V4 Pro 0813 max 66.2% $0.19 18,019
Claude Sonnet 4.6 medium 66.1% $2.86 27,707
Claude Opus 4.6 no thinking 65.9% $0.74 3,877
Claude Opus 4.5 high, 16k 63.7% $0.74 3,684
GPT-5.2 medium 63.4% $0.42 4,928
DeepSeek V4 Flash 0731 max 63% $0.03 18,083
Gemini 3.5 Flash high 62.6% $0.76 13,380
Gemini 3 Flash high 61.6% $0.22 13,959
GPT-5.6 Luna high 60.9% $0.04 5,867
GPT-5.1 high 60.8% $0.69 13,207
GPT-5 high 60.7% $0.59 10,675
GPT-5 Pro high 60.4% $7.65 11,491
GPT-5.4 Mini high 60.3% $1.29 54,770
Muse Spark 1.2 xhigh 60.3% $0.38 9,948
o3-pro high 58.2% $2.54 5,243
GPT-5.4 no thinking 57.4% $0.27 2,780
GLM-5.1 57.1% $0.32 12,380
DeepSeek V4 Flash 0731 high 57% $0.03 15,646
Gemini 3.6 Flash high 56.1% $0.62 14,696
Kimi K2.6 55.9% $0.35 16,808
GPT-5-Codex high 54.5% $1.10 21,302
Kimi K2.7 Code 54.1% $0.23 9,342
Gemini 2.5 Pro Jun 2025 thinking 16k 54% $0.41 7,312
GPT-5 mini high 52.7% $0.11 10,290
o4-mini high 52.6% $0.20 7,922
o3 high 52.4% $0.23 4,622
Gemini 3.1 Flash-Lite high 52.2% $0.07 8,128
Grok 4.3 49.9% $0.09 3,525
GPT-5.2 no thinking 49.6% $0.21 2,243
GPT-5.4 Nano high 49.2% $0.05 6,444
DeepSeek-V4-Pro max 48.9% $0.09 16,898
gpt-oss-120b high 48.2% $0.03 9,314
GLM-5 thinking 48.2% $0.08 3,326
Claude Sonnet 4.5 thinking 16k 47.7% $0.33 3,260
o1-preview 47.6% $1.85 5,129
DeepSeek-V3.2-Speciale 46.7% $0.04 11,930
Claude Sonnet 4.5 no thinking 46.7% $0.28 2,406
DeepSeek-V4-Pro high 46.5% $0.03 5,183
Grok 4.5 46.4% $0.08 1,723
Claude Sonnet 4 thinking 16k 46.1% $0.34 3,147
o1 high 46.1% $2.03 5,766
Claude Opus 4.1 thinking 16k 45.9% $1.76 3,180
Grok 4 45.7% $0.45 5,154
DeepSeek-V4-Flash max 45.6% $0.01 7,978
Kimi K2.5 45.6% $0.05 2,892
Claude Haiku 4.5 no thinking 45.4% $0.09 2,286
Claude Haiku 4.5 thinking 16k 44.1% $0.12 3,705
Claude Sonnet 4 no thinking 43.9% $0.31 2,693
DeepSeek-V4-Flash high 43.8% $0.007 3,570
Claude Opus 4 thinking 16k 43.7% $1.74 3,136
o3-mini high 43.7% $0.17 6,930
Nemotron 3 Ultra 43.5% $0.08 4,368
Mercury 2 high 43.2% $0.03 6,449
Grok 4 Fast 42.9% $0.01 4,009
Kimi K2 Thinking 42.8% $0.08 5,090
Grok-3 mini high 42.6% $0.02 4,919
Gemini 2.5 Flash Sep 2025 thinking 16k 41.9% $0.07 5,190
gpt-oss-120b medium 41.9% $0.01 2,346
DeepSeek-R1 May 2025 41.6% $0.07 5,243
Qwen3-Coder-480B-A35B 41.2% $0.07 1,730
Qwen3-235B-A22B-Thinking Jul 2025 41% $0.02 5,771
Gemini 2.5 Flash Jun 2025 thinking 16k 41% $0.11 7,457
gpt-oss-20b high 40.9% $0.01 9,131
GLM-4.5 thinking 40.6% $0.07 4,988
Claude 3.5 Sonnet 40% $0.25 2,121
GPT-5 Chat 39.8% $0.08 1,437
DeepSeek-V3.2-Exp thinking 39.5% $0.02 4,236
GPT-4.5 39.4% $1.57 979
Kimi K2 Jul 2025 39.4% $0.03 1,710
GPT-4.1 39% $0.10 1,723
Gemini 3.5 Flash-Lite high 39% $0.05 3,366
Qwen3-235B-A22B-Instruct Jul 2025 38.7% $0.01 1,961
DeepSeek-V3.1 thinking 38.4% $0.03 4,545
GPT-5 nano high 38.1% $0.03 12,377
GPT-5.4 Nano no thinking 38% $0.02 2,154
GPT-5.4 Mini no thinking 37.9% $0.07 2,133
GPT-4.1 mini 37.6% $0.02 1,532
DeepSeek-V3.1 37.5% $0.02 2,058
Qwen3-235B-A22B thinking 37.3% $0.02 5,575
Grok 3 37.2% $0.25 2,122
MiniMax-M2.7 37% $0.03 4,304
gpt-oss-20b medium 36.8% $0.004 2,593
DeepSeek-R1 36.5% $0.08 5,170
o1-mini 36.3% $0.11 3,877
Gemini 2.5 Flash-Lite Jun 2025 thinking 16k 35.2% $0.03 12,204
grok-code-fast-1 35.1% $0.02 2,850
qwen3-coder-next 34.4% $0.03 2,279
Mistral Medium 3.1 33.1% $0.04 2,428
Inkling high 32.3% $0.13 4,300
Claude 3.5 Sonnet 31% $0.25 1,966
Claude 3.5 Haiku 30.7% $0.06 1,764
Qwen3-30B-A3B thinking 29.8% $0.008 4,614
GPT-5 nano low 25.9% $0.004 1,673
GPT-4o Nov 2024 25.1% $0.08 1,135
Llama 4 Maverick 24.5% $0.006 992
Claude 3 Opus 19.2% $0.86 1,326
GPT-4.1 nano 19% $0.003 945
GPT-4 Turbo Apr 2024 18% $0.32 915
Qwen2.5-72B 16% $0.01 1,380
Llama 3.3 70B Instruct 14.4% $0.001 1,109
deepseek-r1-0528-qwen3-8b 13% $0.001 5,756
GPT-4 Jun 2023 12.4% $0.64 728
GPT-4o mini 11.8% $0.005 1,111
Claude 3 Sonnet 10.2% $0.18 1,389
Claude 3 Haiku 9.84% $0.01 1,275
Claude 2.1 7.06% $0.17 480

12 models have held number one since its release in Jun 2025

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and Epoch AI’s runs.

OpenAI Google DeepMind Anthropic
Own leaderboard Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%
Epoch AI runs Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra 93.6%

The best score passed 85% in Jun 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Run or collected by Epoch AI Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) GPT-6 Astra Pro GPT-5.5 GPT-5.3 Codex GPT-5.2 GPT-5.1 Gemini 2.5 Pro o1-preview 0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) GPT-6 Astra Pro GPT-5.5

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Google DeepMind Anthropic
WeirdML v3 14 leaders
GPT-6 Astra 54%
WeirdML Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%
WeirdML v3 2 leaders
GPT-6 Astra 54%
Replaced by WeirdML v3

More in AI research

All 83 in AI research
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
  • OpenAI PRs Only in system cards Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. GPT-5.2 55%