Benchmark Almanac

LMCA

Claude Sonnet 5.5 68.3% leads since Sep 2026 Not in model cards

Models rate arguments against position texts on philosophy, decision theory, and AI risk. Scored by agreement with expert ratings, with a ceiling near 85.

Same benchmark as the lmca record, from the same arXiv paper 2607.27499.

In the authors’ words

Large language models have improved rapidly on tasks with verifiable answers, such as mathematics and programming. Much less is known about their ability to reason about what we call conceptual questions: questions for which no ground truth is realistically accessible and no widely accepted resolution methodology exists, but on which progress can still be made by debating arguments. Most philosophical questions are of this kind, as are central components of questions in AI safety, decision theory, and social choice. Our approach is based on the view that while bottom-line conclusions on such questions are hard to evaluate, individual contextualized arguments can be evaluated far more reliably. We therefore introduce a dataset of 951 argumentative critiques of 442 position texts, spanning topics from AI safety and decision theory to ethics and politics, with 1,458 ratings by six expert raters along dimensions including centrality, strength, correctness, and clarity. We propose two scoring functions and benchmark a range of models. Performance tracks general capability rankings.

Abstract of A dataset of rated conceptual arguments (arXiv:2607.27499, CC0)
Made by Emery Cooper , Caspar Oesterheld , and collaborators Released Jul 2026 Other Tasks kept private Last model added Oct 2026 Scores refreshed 31 Oct 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
Paper conceptualreasoning.ai
139
Models scored in Epoch AI data

Claude Sonnet 5.5 leads with 68.3%

Best score per model in Epoch AI’s data. Where a model ran with several settings or harnesses, we keep its best.

139 models. Scroll sideways for the rest.

  1. 68.3 Claude Sonnet 5.5
  2. 68.2 Claude Opus 5.5
  3. 65.5 Claude Fable 5.1
  4. 64.5 Claude Opus 5
  5. 64.4 GPT-6 Astra
  6. 63.7 Gemini 4 Argon
  7. 62.8 GPT-6.1 Sol
  8. 61.1 Claude Fable 5
  9. 59.2 GPT-5.6 Sol
  10. 59.1 GPT-6 Sol
  11. 57.5 Claude Opus 4.8
  12. 56.8 Muse Wonder
  13. 55.8 Claude Opus 4.6
  14. 55.5 GLM-5.3
  15. 55.0 GPT-5.6 Terra
  16. 54.3 GPT-5.5
  17. 53.9 GPT-5.5 Pro
  18. 53.9 Muse Spark 1.3
  19. 53.8 Gemini 3.1 Pro
  20. 52.9 Gemini 3.8 Flash
  21. 52.7 Kimi K3
  22. 52.2 Claude Opus 4.7
  23. 52.0 GPT-5.4
  24. 51.1 Claude Haiku 5.5
  25. 50.4 Gemini 3.7 Flash
  26. 50.0 Claude Sonnet 5
  27. 49.9 Muse Spark 1.1
  28. 49.4 Grok 4.7
  29. 48.5 GPT-5.6 Luna
  30. 48.5 Grok 4.6
  31. 48.4 Muse Spark 1.2
  32. 47.1 Gemini 3.5 Flash
  33. 47.0 DeepSeek V4.1 Flash
  34. 46.5 Claude Sonnet 4.6
  35. 46.2 Qwen 3.8 Max
  36. 45.8 GLM-5.2
  37. 45.5 DeepSeek V4 Pro 0813
  38. 45.2 Grok 4.5
  39. 44.9 Gemini 3.6 Flash
  40. 44.5 GPT-6 Luna
  41. 44.5 Claude Opus 4.5
  42. 44.0 Qwen3.7-Max
  43. 43.9 GPT-5.2
  44. 43.9 GPT-5.1
  45. 43.1 Gemini 3 Flash
  46. 42.5 Qwen 3.6 Max (Preview)
  47. 41.7 DeepSeek V4 Flash 0731
  48. 41.4 Qwen 3.8 27B
  49. 41.2 DeepSeek-V4-Pro
  50. 40.8 GPT-5.4 Mini
  51. 40.0 GPT-5
  52. 39.7 o3
  53. 39.3 Gemma 4 31B IT
  54. 38.8 Claude Sonnet 4.5
  55. 38.7 Grok 4.20
  56. 38.5 o3-pro
  57. 38.3 Grok 4.3 Beta
  58. 37.9 Qwen3.5 397B-A17B
  59. 37.6 Gemini 3.5 Flash-Lite
  60. 37.6 Qwen3.7-Plus
  61. 37.6 Inkling
  62. 37.4 Claude Opus 4
  63. 37.3 Kimi K2.6
  64. 37.1 Claude Opus 4.1
  65. 36.9 Nemotron 3 Ultra
  66. 36.9 GPT-5.4 Nano
  67. 36.4 Qwen 3.5 Plus (hosted 397B-A17B)
  68. 35.9 DeepSeek-V4-Flash
  69. 35.8 Mistral Large 4
  70. 35.0 Gemini 3.1 Flash-Lite
  71. 34.8 Gemini 2.5 Pro
  72. 34.5 Qwen3.6 27B
  73. 34.2 GPT-5 mini
  74. 34.0 Qwen3.5-27B
  75. 33.7 MiniMax-M3
  76. 33.1 Qwen 3.6 Plus
  77. 32.2 Qwen3.5-122B-A10B
  78. 31.0 Qwen 3.6 Flash
  79. 30.9 Claude Haiku 4.5
  80. 29.7 Qwen 3.6 35B-A3B
  81. 29.7 Gemma 4 26B A4B
  82. 29.5 MiMo-V2.5-Pro
  83. 29.5 Qwen3.5-35B-A3B
  84. 29.3 Qwen3-235B-A22B-Thinking
  85. 29.1 Qwen 3.5 Flash (hosted 35B-A3B)
  86. 29.1 DeepSeek-V3.2-Exp
  87. 29.0 Claude Sonnet 4
  88. 28.8 DeepSeek-V3.2
  89. 28.6 DeepSeek-V3.1-Terminus
  90. 28.3 Qwen3-Max
  91. 27.5 Gemini 2.5 Flash
  92. 26.5 o4-mini
  93. 26.1 Mistral Medium 3.5
  94. 25.6 GPT-4.1
  95. 25.0 Qwen3-235B-A22B
  96. 24.5 Qwen3.5-9B
  97. 24.3 DeepSeek-V3.1
  98. 24.0 Qwen Plus
  99. 23.5 Qwen3-235B-A22B-Instruct
  100. 22.4 Qwen3-30B-A3B-Instruct
  101. 22.3 o1
  102. 22.1 gpt-oss-120b
  103. 21.1 GPT-4.1 mini
  104. 20.6 Mistral Small 4
  105. 19.5 Qwen3-30B-A3B-Thinking
  106. 19.1 Mistral Medium 3.1
  107. 19.0 o3-mini
  108. 18.2 Qwen3-14B
  109. 18.1 Gemini 2.5 Flash-Lite
  110. 17.5 Llama 3.3 70B
  111. 17.3 Qwen3-32B
  112. 17.1 GPT-4
  113. 17.0 Claude 3 Opus
  114. 16.7 Mistral Large 3
  115. 16.6 GPT-4o
  116. 15.9 Llama 4 Maverick
  117. 15.8 Qwen3-30B-A3B
  118. 15.5 DeepSeek-V3
  119. 14.8 Llama 3.1-70B
  120. 14.5 gpt-oss-20b
  121. 13.4 Qwen2.5-72B
  122. 12.3 Gemma 3 27B
  123. 12.0 Llama 4 Scout
  124. 10.4 GPT-4o mini
  125. 10.3 Cohere Command A
  126. 9.78 GPT-4 Turbo
  127. 9.65 GPT-3.5 Turbo
  128. 9.19 c4ai-command-r-08-2024
  129. 8.84 Claude 3 Haiku
  130. 8.82 Qwen3-8B
  131. 7.93 GPT-5 nano
  132. 7.10 Gemma 2 27B
  133. 6.42 Qwen2.5-7B
  134. 5.48 Ministral 3B
  135. 5.46 GPT-4.1 nano
  136. 5.37 Llama 3.1-8B
  137. 5.01 Command R+
  138. 4.46 Gemma 3 12B
  139. 2.78 Gemma 3 4B
  1. 1 Claude Sonnet 5.5 max 68.3%
  2. 2 Claude Opus 5.5 max 68.2%
  3. 3 Claude Fable 5.1 65.5%
  4. 4 Claude Opus 5 xhigh 64.5%
  5. 5 GPT-6 Astra xhigh 64.4%
  6. 6 Gemini 4 Argon high 63.7%
  7. 7 GPT-6.1 Sol max 62.8%
  8. 8 Claude Fable 5 high 61.1%
  9. 9 GPT-5.6 Sol pro, max 59.2%
  10. 10 GPT-6 Sol max 59.1%
  11. 11 Claude Opus 4.8 max 57.5%
  12. 12 Muse Wonder 56.8%
  13. 13 Claude Opus 4.6 max 55.8%
  14. 14 GLM-5.3 unknown thinking 55.5%
  15. 15 GPT-5.6 Terra max 55%
  16. 16 GPT-5.5 xhigh 54.3%
  17. 17 GPT-5.5 Pro xhigh 53.9%
  18. 18 Muse Spark 1.3 53.9%
  19. 19 Gemini 3.1 Pro high 53.8%
  20. 20 Gemini 3.8 Flash medium 52.9%
  21. 21 Kimi K3 max 52.7%
  22. 22 Claude Opus 4.7 max 52.2%
  23. 23 GPT-5.4 xhigh 52%
  24. 24 Claude Haiku 5.5 max 51.1%
  25. 25 Gemini 3.7 Flash high 50.4%
  26. 26 Claude Sonnet 5 high 50%
  27. 27 Muse Spark 1.1 high 49.9%
  28. 28 Grok 4.7 xhigh 49.4%
  29. 29 GPT-5.6 Luna max 48.5%
  30. 30 Grok 4.6 xhigh 48.5%
  31. 31 Muse Spark 1.2 xhigh 48.4%
  32. 32 Gemini 3.5 Flash high 47.1%
  33. 33 DeepSeek V4.1 Flash 47%
  34. 34 Claude Sonnet 4.6 max 46.5%
  35. 35 Qwen 3.8 Max xhigh 46.2%
  36. 36 GLM-5.2 max 45.8%
  37. 37 DeepSeek V4 Pro 0813 45.5%
  38. 38 Grok 4.5 high 45.2%
  39. 39 Gemini 3.6 Flash high 44.9%
  40. 40 GPT-6 Luna max 44.5%
  41. 41 Claude Opus 4.5 high 44.5%
  42. 42 Qwen3.7-Max 44%
  43. 43 GPT-5.2 xhigh 43.9%
  44. 44 GPT-5.1 high 43.9%
  45. 45 Gemini 3 Flash high 43.1%
  46. 46 Qwen 3.6 Max (Preview) 42.5%
  47. 47 DeepSeek V4 Flash 0731 max 41.7%
  48. 48 Qwen 3.8 27B 41.4%
  49. 49 DeepSeek-V4-Pro max 41.2%
  50. 50 GPT-5.4 Mini xhigh 40.8%
  51. 51 GPT-5 high 40%
  52. 52 o3 high 39.7%
  53. 53 Gemma 4 31B IT 39.3%
  54. 54 Claude Sonnet 4.5 unknown thinking 38.8%
  55. 55 Grok 4.20 38.7%
  56. 56 o3-pro high 38.5%
  57. 57 Grok 4.3 Beta high 38.3%
  58. 58 Qwen3.5 397B-A17B 37.9%
  59. 59 Gemini 3.5 Flash-Lite high 37.6%
  60. 60 Qwen3.7-Plus 37.6%
  61. 61 Inkling xhigh 37.6%
  62. 62 Claude Opus 4 37.4%
  63. 63 Kimi K2.6 37.3%
  64. 64 Claude Opus 4.1 unknown thinking 37.1%
  65. 65 Nemotron 3 Ultra 36.9%
  66. 66 GPT-5.4 Nano xhigh 36.9%
  67. 67 Qwen 3.5 Plus (hosted 397B-A17B) 36.4%
  68. 68 DeepSeek-V4-Flash max 35.9%
  69. 69 Mistral Large 4 35.8%
  70. 70 Gemini 3.1 Flash-Lite high 35%
  71. 71 Gemini 2.5 Pro Jun 2025 34.8%
  72. 72 Qwen3.6 27B 34.5%
  73. 73 GPT-5 mini high 34.2%
  74. 74 Qwen3.5-27B 34%
  75. 75 MiniMax-M3 33.7%
  76. 76 Qwen 3.6 Plus 2026-04-02 33.1%
  77. 77 Qwen3.5-122B-A10B none 32.2%
  78. 78 Qwen 3.6 Flash 2026-04-16 31%
  79. 79 Claude Haiku 4.5 unknown thinking 30.9%
  80. 80 Qwen 3.6 35B-A3B 29.7%
  81. 81 Gemma 4 26B A4B 29.7%
  82. 82 MiMo-V2.5-Pro 29.5%
  83. 83 Qwen3.5-35B-A3B 29.5%
  84. 84 Qwen3-235B-A22B-Thinking Jul 2025 29.3%
  85. 85 Qwen 3.5 Flash (hosted 35B-A3B) 29.1%
  86. 86 DeepSeek-V3.2-Exp thinking 29.1%
  87. 87 Claude Sonnet 4 unknown thinking 29%
  88. 88 DeepSeek-V3.2 Thinking; Novita 28.8%
  89. 89 DeepSeek-V3.1-Terminus 28.6%
  90. 90 Qwen3-Max 28.3%
  91. 91 Gemini 2.5 Flash Jun 2025 27.5%
  92. 92 o4-mini high 26.5%
  93. 93 Mistral Medium 3.5 26.1%
  94. 94 GPT-4.1 25.6%
  95. 95 Qwen3-235B-A22B 25%
  96. 96 Qwen3.5-9B 24.5%
  97. 97 DeepSeek-V3.1 thinking 24.3%
  98. 98 Qwen Plus Apr 2025 24%
  99. 99 Qwen3-235B-A22B-Instruct Jul 2025 23.5%
  100. 100 Qwen3-30B-A3B-Instruct Jul 2025 22.4%
  101. 101 o1 high 22.3%
  102. 102 gpt-oss-120b high 22.1%
  103. 103 GPT-4.1 mini 21.1%
  104. 104 Mistral Small 4 20.6%
  105. 105 Qwen3-30B-A3B-Thinking Jul 2025 19.5%
  106. 106 Mistral Medium 3.1 19.1%
  107. 107 o3-mini high 19%
  108. 108 Qwen3-14B 18.2%
  109. 109 Gemini 2.5 Flash-Lite Jun 2025 18.1%
  110. 110 Llama 3.3 70B 17.5%
  111. 111 Qwen3-32B 17.3%
  112. 112 GPT-4 Jun 2023 17.1%
  113. 113 Claude 3 Opus 17%
  114. 114 Mistral Large 3 16.7%
  115. 115 GPT-4o May 2024 16.6%
  116. 116 Llama 4 Maverick 15.9%
  117. 117 Qwen3-30B-A3B 15.8%
  118. 118 DeepSeek-V3 Mar 2025 15.5%
  119. 119 Llama 3.1-70B 14.8%
  120. 120 gpt-oss-20b high 14.5%
  121. 121 Qwen2.5-72B 13.4%
  122. 122 Gemma 3 27B 12.3%
  123. 123 Llama 4 Scout 12%
  124. 124 GPT-4o mini 10.4%
  125. 125 Cohere Command A 10.3%
  126. 126 GPT-4 Turbo Nov 2023 9.78%
  127. 127 GPT-3.5 Turbo Jan 2024 9.65%
  128. 128 c4ai-command-r-08-2024 9.19%
  129. 129 Claude 3 Haiku 8.84%
  130. 130 Qwen3-8B 8.82%
  131. 131 GPT-5 nano high 7.93%
  132. 132 Gemma 2 27B 7.10%
  133. 133 Qwen2.5-7B 6.42%
  134. 134 Ministral 3B 5.48%
  135. 135 GPT-4.1 nano 5.46%
  136. 136 Llama 3.1-8B 5.37%
  137. 137 Command R+ 5.01%
  138. 138 Gemma 3 12B 4.46%
  139. 139 Gemma 3 4B 2.78%
#ModelScoreSettingDateSource
1 Claude Sonnet 5.5 68.3% max Sep 2026 Epoch AI
2 Claude Opus 5.5 68.2% max Sep 2026 Epoch AI
3 Claude Fable 5.1 65.5% Sep 2026 Epoch AI
4 Claude Opus 5 64.5% xhigh Jul 2026 Epoch AI
5 GPT-6 Astra 64.4% xhigh Sep 2026 Epoch AI
6 Gemini 4 Argon 63.7% high Sep 2026 Epoch AI
7 GPT-6.1 Sol 62.8% max Sep 2026 Epoch AI
8 Claude Fable 5 61.1% high Jun 2026 Epoch AI
9 GPT-5.6 Sol 59.2% pro, max Jul 2026 Epoch AI
10 GPT-6 Sol 59.1% max Sep 2026 Epoch AI
11 Claude Opus 4.8 57.5% max May 2026 Epoch AI
12 Muse Wonder 56.8% Oct 2026 Epoch AI
13 Claude Opus 4.6 55.8% max Feb 2026 Epoch AI
14 GLM-5.3 55.5% unknown thinking Aug 2026 Epoch AI
15 GPT-5.6 Terra 55% max Jul 2026 Epoch AI
16 GPT-5.5 54.3% xhigh Apr 2026 Epoch AI
17 GPT-5.5 Pro 53.9% xhigh Apr 2026 Epoch AI
18 Muse Spark 1.3 53.9% Sep 2026 Epoch AI
19 Gemini 3.1 Pro 53.8% high Feb 2026 Epoch AI
20 Gemini 3.8 Flash 52.9% medium Sep 2026 Epoch AI
21 Kimi K3 52.7% max Jul 2026 Epoch AI
22 Claude Opus 4.7 52.2% max Apr 2026 Epoch AI
23 GPT-5.4 52% xhigh Mar 2026 Epoch AI
24 Claude Haiku 5.5 51.1% max Oct 2026 Epoch AI
25 Gemini 3.7 Flash 50.4% high Aug 2026 Epoch AI
26 Claude Sonnet 5 50% high Jun 2026 Epoch AI
27 Muse Spark 1.1 49.9% high Jul 2026 Epoch AI
28 Grok 4.7 49.4% xhigh Sep 2026 Epoch AI
29 GPT-5.6 Luna 48.5% max Jul 2026 Epoch AI
30 Grok 4.6 48.5% xhigh Aug 2026 Epoch AI
31 Muse Spark 1.2 48.4% xhigh Aug 2026 Epoch AI
32 Gemini 3.5 Flash 47.1% high May 2026 Epoch AI
33 DeepSeek V4.1 Flash 47% Sep 2026 Epoch AI
34 Claude Sonnet 4.6 46.5% max Feb 2026 Epoch AI
35 Qwen 3.8 Max 46.2% xhigh Aug 2026 Epoch AI
36 GLM-5.2 45.8% max Jun 2026 Epoch AI
37 DeepSeek V4 Pro 0813 45.5% Aug 2026 Epoch AI
38 Grok 4.5 45.2% high Jul 2026 Epoch AI
39 Gemini 3.6 Flash 44.9% high Jul 2026 Epoch AI
40 GPT-6 Luna 44.5% max Sep 2026 Epoch AI
41 Claude Opus 4.5 44.5% high Nov 2025 Epoch AI
42 Qwen3.7-Max 44% May 2026 Epoch AI
43 GPT-5.2 43.9% xhigh Dec 2025 Epoch AI
44 GPT-5.1 43.9% high Nov 2025 Epoch AI
45 Gemini 3 Flash 43.1% high Dec 2025 Epoch AI
46 Qwen 3.6 Max (Preview) 42.5% Apr 2026 Epoch AI
47 DeepSeek V4 Flash 0731 41.7% max Jul 2026 Epoch AI
48 Qwen 3.8 27B 41.4% Aug 2026 Epoch AI
49 DeepSeek-V4-Pro 41.2% max Apr 2026 Epoch AI
50 GPT-5.4 Mini 40.8% xhigh Mar 2026 Epoch AI
51 GPT-5 40% high Aug 2025 Epoch AI
52 o3 39.7% high Apr 2025 Epoch AI
53 Gemma 4 31B IT 39.3% Apr 2026 Epoch AI
54 Claude Sonnet 4.5 38.8% unknown thinking Sep 2025 Epoch AI
55 Grok 4.20 38.7% Feb 2026 Epoch AI
56 o3-pro 38.5% high Jun 2025 Epoch AI
57 Grok 4.3 Beta 38.3% high Apr 2026 Epoch AI
58 Qwen3.5 397B-A17B 37.9% Feb 2026 Epoch AI
59 Gemini 3.5 Flash-Lite 37.6% high Jul 2026 Epoch AI
60 Qwen3.7-Plus 37.6% Jun 2026 Epoch AI
61 Inkling 37.6% xhigh Jul 2026 Epoch AI
62 Claude Opus 4 37.4% May 2025 Epoch AI
63 Kimi K2.6 37.3% Apr 2026 Epoch AI
64 Claude Opus 4.1 37.1% unknown thinking Aug 2025 Epoch AI
65 Nemotron 3 Ultra 36.9% Jun 2026 Epoch AI
66 GPT-5.4 Nano 36.9% xhigh Mar 2026 Epoch AI
67 Qwen 3.5 Plus (hosted 397B-A17B) 36.4% Feb 2026 Epoch AI
68 DeepSeek-V4-Flash 35.9% max Apr 2026 Epoch AI
69 Mistral Large 4 35.8% Oct 2026 Epoch AI
70 Gemini 3.1 Flash-Lite 35% high Mar 2026 Epoch AI
71 Gemini 2.5 Pro Jun 2025 34.8% Jun 2025 Epoch AI
72 Qwen3.6 27B 34.5% Apr 2026 Epoch AI
73 GPT-5 mini 34.2% high Aug 2025 Epoch AI
74 Qwen3.5-27B 34% Feb 2026 Epoch AI
75 MiniMax-M3 33.7% Jun 2026 Epoch AI
76 Qwen 3.6 Plus 33.1% 2026-04-02 Mar 2026 Epoch AI
77 Qwen3.5-122B-A10B 32.2% none Feb 2026 Epoch AI
78 Qwen 3.6 Flash 31% 2026-04-16 Apr 2026 Epoch AI
79 Claude Haiku 4.5 30.9% unknown thinking Oct 2025 Epoch AI
80 Qwen 3.6 35B-A3B 29.7% Apr 2026 Epoch AI
81 Gemma 4 26B A4B 29.7% Apr 2026 Epoch AI
82 MiMo-V2.5-Pro 29.5% Apr 2026 Epoch AI
83 Qwen3.5-35B-A3B 29.5% Feb 2026 Epoch AI
84 Qwen3-235B-A22B-Thinking Jul 2025 29.3% Jul 2025 Epoch AI
85 Qwen 3.5 Flash (hosted 35B-A3B) 29.1% Feb 2026 Epoch AI
86 DeepSeek-V3.2-Exp 29.1% thinking Sep 2025 Epoch AI
87 Claude Sonnet 4 29% unknown thinking May 2025 Epoch AI
88 DeepSeek-V3.2 28.8% Thinking; Novita Dec 2025 Epoch AI
89 DeepSeek-V3.1-Terminus 28.6% Sep 2025 Epoch AI
90 Qwen3-Max 28.3% Sep 2025 Epoch AI
91 Gemini 2.5 Flash Jun 2025 27.5% Jun 2025 Epoch AI
92 o4-mini 26.5% high Apr 2025 Epoch AI
93 Mistral Medium 3.5 26.1% Apr 2026 Epoch AI
94 GPT-4.1 25.6% Apr 2025 Epoch AI
95 Qwen3-235B-A22B 25% Apr 2025 Epoch AI
96 Qwen3.5-9B 24.5% Feb 2026 Epoch AI
97 DeepSeek-V3.1 24.3% thinking Aug 2025 Epoch AI
98 Qwen Plus Apr 2025 24% Apr 2025 Epoch AI
99 Qwen3-235B-A22B-Instruct Jul 2025 23.5% Jul 2025 Epoch AI
100 Qwen3-30B-A3B-Instruct Jul 2025 22.4% Jul 2025 Epoch AI
101 o1 22.3% high Dec 2024 Epoch AI
102 gpt-oss-120b 22.1% high Aug 2025 Epoch AI
103 GPT-4.1 mini 21.1% Apr 2025 Epoch AI
104 Mistral Small 4 20.6% Mar 2026 Epoch AI
105 Qwen3-30B-A3B-Thinking Jul 2025 19.5% Jul 2025 Epoch AI
106 Mistral Medium 3.1 19.1% Aug 2025 Epoch AI
107 o3-mini 19% high Jan 2025 Epoch AI
108 Qwen3-14B 18.2% Apr 2025 Epoch AI
109 Gemini 2.5 Flash-Lite Jun 2025 18.1% Jun 2025 Epoch AI
110 Llama 3.3 70B 17.5% Dec 2024 Epoch AI
111 Qwen3-32B 17.3% Apr 2025 Epoch AI
112 GPT-4 Jun 2023 17.1% Jun 2023 Epoch AI
113 Claude 3 Opus 17% Feb 2024 Epoch AI
114 Mistral Large 3 16.7% Dec 2025 Epoch AI
115 GPT-4o May 2024 16.6% May 2024 Epoch AI
116 Llama 4 Maverick 15.9% Apr 2025 Epoch AI
117 Qwen3-30B-A3B 15.8% Apr 2025 Epoch AI
118 DeepSeek-V3 Mar 2025 15.5% Mar 2025 Epoch AI
119 Llama 3.1-70B 14.8% Jul 2024 Epoch AI
120 gpt-oss-20b 14.5% high Aug 2025 Epoch AI
121 Qwen2.5-72B 13.4% Sep 2024 Epoch AI
122 Gemma 3 27B 12.3% Mar 2025 Epoch AI
123 Llama 4 Scout 12% Apr 2025 Epoch AI
124 GPT-4o mini 10.4% Jul 2024 Epoch AI
125 Cohere Command A 10.3% Mar 2025 Epoch AI
126 GPT-4 Turbo Nov 2023 9.78% Nov 2023 Epoch AI
127 GPT-3.5 Turbo Jan 2024 9.65% Jan 2024 Epoch AI
128 c4ai-command-r-08-2024 9.19% Aug 2024 Epoch AI
129 Claude 3 Haiku 8.84% Mar 2024 Epoch AI
130 Qwen3-8B 8.82% Apr 2025 Epoch AI
131 GPT-5 nano 7.93% high Aug 2025 Epoch AI
132 Gemma 2 27B 7.10% Jun 2024 Epoch AI
133 Qwen2.5-7B 6.42% Sep 2024 Epoch AI
134 Ministral 3B 5.48% Oct 2024 Epoch AI
135 GPT-4.1 nano 5.46% Apr 2025 Epoch AI
136 Llama 3.1-8B 5.37% Jul 2024 Epoch AI
137 Command R+ 5.01% Aug 2024 Epoch AI
138 Gemma 3 12B 4.46% Mar 2025 Epoch AI
139 Gemma 3 4B 2.78% Mar 2025 Epoch AI

5 models have held number one since its release in Jul 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Epoch AI runs 5 leaders
Claude Sonnet 5.5 68.3%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Run or collected by Epoch AI Best so far
0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Highest possible score Claude Sonnet 5.5 Claude Fable 5 Claude Opus 4.6 Claude Opus 4.5 GPT-5 GPT-4.1 Llama 3.3 70B 0% 20% 40% 60% 80% 100% 2023 2024 2025 2026 Near the ceiling (85%) Highest possible score Claude Sonnet 5.5 Claude Fable 5

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Other

All 41 in Other
  • Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
  • Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Gemini 4 Carbon 59.0
  • Anthropic ECI (AECI) Only in system cards Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Claude Opus 5.5 169.4
  • Qwen in-house English automatic evaluation Only in system cards Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Qwen2-72B 72.9%
  • Recognition of model-written user turns (Anthropic) Only in system cards Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. Claude Mythos Preview 0.35
  • LiveBench Dropped Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Claude Fable 5.1 85.7%