Benchmark Almanac

Misguided Attention

Grok-3 mini 73.7% leads since Apr 2025 Not in model cards

Models get slight variations of famous riddles, paradoxes and thought experiments. A model fails when it answers the well-known original instead of the problem as stated.

Version 0.3 (2025-01) added an evaluation subset with tagged releases. About 490 GitHub stars.

Made by cpldcpu (GitHub user) Released May 2024 Abstract puzzles and logic All tasks public Last model added Jun 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
cpldcpu/MisguidedAttention 488 stars Leaderboard cpldcpu.github.io
51
Entries on its own leaderboard, 45 models we track

Grok-3 mini leads with 73.7% among models released by Jun 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Jun 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

51 entries. Scroll sideways for the rest.

  1. 73.7 Grok-3 mini
  2. 68.6 Claude Sonnet 4
  3. 67.1 DeepSeek-R1
  4. 64.1 Claude Opus 4
  5. 63.5 Gemini 2.5 Flash
  6. 62.5 Gemini 2.5 Flash
  7. 58.8 Gemini 2.5 Pro
  8. 57.4 Llama-3.3-Nemotron-Super-49B-v1
  9. 57.4 Gemini 2.5 Pro
  10. 56.1 Claude 3.7 Sonnet
  11. 56.1 QwQ-32B
  12. 55.2 Gemini 2.5 Pro
  13. 55.1 DeepSeek-R1
  14. 55.1 Grok 3
  15. 53.5 DeepSeek-V3
  16. 53.5 GLM-Z1-32B-0414
  17. 51.7 Magistral Medium 1.0
  18. 50.6 Mistral Medium 3
  19. 49.4 o4-mini-high-long
  20. 49.0 Qwen3-235B-A22B
  21. 48.7 GPT-4.1 mini
  22. 48.4 DeepHermes-3-Mistral-24B-Pre
  23. 48.4 Phi-4-Reasoning-plus
  24. 47.4 optimus-alpha
  25. 46.8 o3-mini
  26. 44.2 GPT-4.1
  27. 43.6 Qwen3-30B-A3B
  28. 42.9 o4-mini
  29. 42.6 DeepSeek-R1-Distill-Llama-70B
  30. 40.7 quasar-alpha
  31. 37.2 Claude 3.5 Sonnet (October 2024)
  32. 36.5 MiniMax-Text-01
  33. 35.9 Llama 4 Scout
  34. 35.3 Llama 3.3 70B Instruct
  35. 34.6 Gemini 2.0 Pro
  36. 33.7 GPT-4o
  37. 32.1 GPT-4o
  38. 31.4 Hermes-3-Llama-3.1-70B
  39. 31.4 Qwen2.5-Max
  40. 30.1 Cohere Command A
  41. 29.5 Llama 4 Maverick
  42. 28.2 Gemini 2.0 Flash
  43. 27.9 Mistral Small 3
  44. 26.9 DeepSeek-V3
  45. 26.6 GPT-4.1 nano
  46. 26.0 Mistral Small 3.1
  47. 25.0 Jamba 1.6 Large
  48. 23.4 Gemma 3 27B
  49. 22.9 gemini-exp-1206
  50. 22.8 Gemma 3n E4B
  51. 19.2 OLMo 2 32B Instruct
  1. 1 Grok-3 mini thinking 73.7%
  2. 2 Claude Sonnet 4 thinking 68.6%
  3. 3 DeepSeek-R1 May 2025 67.1%
  4. 4 Claude Opus 4 64.1%
  5. 5 Gemini 2.5 Flash May 2025, thinking 63.5%
  6. 6 Gemini 2.5 Flash Jun 2025, high 62.5%
  7. 7 Gemini 2.5 Pro May 2025 58.8%
  8. 8 Llama-3.3-Nemotron-Super-49B-v1 thinking 57.4%
  9. 9 Gemini 2.5 Pro Jun 2025, low 57.4%
  10. 10 Claude 3.7 Sonnet thinking 56.1%
  11. 11 QwQ-32B 56.1%
  12. 12 Gemini 2.5 Pro Mar 2025 55.2%
  13. 13 DeepSeek-R1 55.1%
  14. 14 Grok 3 55.1%
  15. 15 DeepSeek-V3 Mar 2025 53.5%
  16. 16 GLM-Z1-32B-0414 53.5%
  17. 17 Magistral Medium 1.0 thinking 51.7%
  18. 18 Mistral Medium 3 50.6%
  19. 19 o4-mini-high-long 49.4%
  20. 20 Qwen3-235B-A22B thinking 49%
  21. 21 GPT-4.1 mini 48.7%
  22. 22 DeepHermes-3-Mistral-24B-Pre 48.4%
  23. 23 Phi-4-Reasoning-plus 48.4%
  24. 24 optimus-alpha 47.4%
  25. 25 o3-mini 46.8%
  26. 26 GPT-4.1 44.2%
  27. 27 Qwen3-30B-A3B thinking 43.6%
  28. 28 o4-mini high 42.9%
  29. 29 DeepSeek-R1-Distill-Llama-70B 42.6%
  30. 30 quasar-alpha 40.7%
  31. 31 Claude 3.5 Sonnet (October 2024) 37.2%
  32. 32 MiniMax-Text-01 36.5%
  33. 33 Llama 4 Scout 35.9%
  34. 34 Llama 3.3 70B Instruct 35.3%
  35. 35 Gemini 2.0 Pro 34.6%
  36. 36 GPT-4o Nov 2024 33.7%
  37. 37 GPT-4o Jan 2025 32.1%
  38. 38 Hermes-3-Llama-3.1-70B 31.4%
  39. 39 Qwen2.5-Max 31.4%
  40. 40 Cohere Command A 30.1%
  41. 41 Llama 4 Maverick 29.5%
  42. 42 Gemini 2.0 Flash Feb 2025 28.2%
  43. 43 Mistral Small 3 27.9%
  44. 44 DeepSeek-V3 26.9%
  45. 45 GPT-4.1 nano 26.6%
  46. 46 Mistral Small 3.1 26%
  47. 47 Jamba 1.6 Large 25%
  48. 48 Gemma 3 27B 23.4%
  49. 49 gemini-exp-1206 22.9%
  50. 50 Gemma 3n E4B 22.8%
  51. 51 OLMo 2 32B Instruct 19.2%
#EntryScoreSettingDateSource
1 Grok-3 mini 73.7% thinking Apr 2025 Official leaderboard
2 Claude Sonnet 4 68.6% thinking May 2025 Official leaderboard
3 DeepSeek-R1 May 2025 67.1% May 2025 Official leaderboard
4 Claude Opus 4 64.1% May 2025 Official leaderboard
5 Gemini 2.5 Flash May 2025 63.5% thinking May 2025 Official leaderboard
6 Gemini 2.5 Flash Jun 2025 62.5% high Jun 2025 Official leaderboard
7 Gemini 2.5 Pro May 2025 58.8% May 2025 Official leaderboard
8 Llama-3.3-Nemotron-Super-49B-v1 57.4% thinking Mar 2025 Official leaderboard
9 Gemini 2.5 Pro Jun 2025 57.4% low Jun 2025 Official leaderboard
10 Claude 3.7 Sonnet 56.1% thinking Feb 2025 Official leaderboard
11 QwQ-32B 56.1% Mar 2025 Official leaderboard
12 Gemini 2.5 Pro Mar 2025 55.2% Mar 2025 Official leaderboard
13 DeepSeek-R1 55.1% Jan 2025 Official leaderboard
14 Grok 3 55.1% Apr 2025 Official leaderboard
15 DeepSeek-V3 Mar 2025 53.5% Mar 2025 Official leaderboard
16 GLM-Z1-32B-0414 53.5% Apr 2025 Official leaderboard
17 Magistral Medium 1.0 51.7% thinking Jun 2025 Official leaderboard
18 Mistral Medium 3 50.6% May 2025 Official leaderboard
19 o4-mini-high-long 49.4% Official leaderboard
20 Qwen3-235B-A22B 49% thinking Apr 2025 Official leaderboard
21 GPT-4.1 mini 48.7% Apr 2025 Official leaderboard
22 DeepHermes-3-Mistral-24B-Pre 48.4% Official leaderboard
23 Phi-4-Reasoning-plus 48.4% Apr 2025 Official leaderboard
24 optimus-alpha 47.4% Official leaderboard
25 o3-mini 46.8% Jan 2025 Official leaderboard
26 GPT-4.1 44.2% Apr 2025 Official leaderboard
27 Qwen3-30B-A3B 43.6% thinking Apr 2025 Official leaderboard
28 o4-mini 42.9% high Apr 2025 Official leaderboard
29 DeepSeek-R1-Distill-Llama-70B 42.6% Jan 2025 Official leaderboard
30 quasar-alpha 40.7% Official leaderboard
31 Claude 3.5 Sonnet (October 2024) 37.2% Oct 2024 Official leaderboard
32 MiniMax-Text-01 36.5% Jan 2025 Official leaderboard
33 Llama 4 Scout 35.9% Apr 2025 Official leaderboard
34 Llama 3.3 70B Instruct 35.3% Dec 2024 Official leaderboard
35 Gemini 2.0 Pro 34.6% Feb 2025 Official leaderboard
36 GPT-4o Nov 2024 33.7% Nov 2024 Official leaderboard
37 GPT-4o Jan 2025 32.1% Jan 2025 Official leaderboard
38 Hermes-3-Llama-3.1-70B 31.4% Official leaderboard
39 Qwen2.5-Max 31.4% Jan 2025 Official leaderboard
40 Cohere Command A 30.1% Mar 2025 Official leaderboard
41 Llama 4 Maverick 29.5% Apr 2025 Official leaderboard
42 Gemini 2.0 Flash Feb 2025 28.2% Feb 2025 Official leaderboard
43 Mistral Small 3 27.9% Jan 2025 Official leaderboard
44 DeepSeek-V3 26.9% Dec 2024 Official leaderboard
45 GPT-4.1 nano 26.6% Apr 2025 Official leaderboard
46 Mistral Small 3.1 26% Mar 2025 Official leaderboard
47 Jamba 1.6 Large 25% Mar 2025 Official leaderboard
48 Gemma 3 27B 23.4% Mar 2025 Official leaderboard
49 gemini-exp-1206 22.9% Official leaderboard
50 Gemma 3n E4B 22.8% Jun 2025 Official leaderboard
51 OLMo 2 32B Instruct 19.2% Mar 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
thinking
68.6%
Gemini 2.5 Flash
thinking
63.5%
Gemini 2.5 Pro
low
57.4%
Claude 3.7 Sonnet
thinking
56.1%
Qwen3-235B-A22B
non-thinking thinking
49%
Qwen3-30B-A3B
non-thinking thinking
43.6%
ModelEffortHarnessScoreCost per taskOutput tokens
Grok-3 mini thinking 73.7%
Claude Sonnet 4 thinking 68.6%
DeepSeek-R1 May 2025 67.1%
Claude Opus 4 64.1%
Gemini 2.5 Flash May 2025 thinking 63.5%
Gemini 2.5 Flash Jun 2025 high 62.5%
Gemini 2.5 Pro May 2025 58.8%
Llama-3.3-Nemotron-Super-49B-v1 thinking 57.4%
Gemini 2.5 Pro Jun 2025 low 57.4%
Gemini 2.5 Pro Jun 2025 high 57.4%
Claude 3.7 Sonnet thinking 56.1%
QwQ-32B 56.1%
Gemini 2.5 Pro Mar 2025 55.2%
DeepSeek-R1 55.1%
Grok 3 55.1%
DeepSeek-V3 Mar 2025 53.5%
GLM-Z1-32B-0414 53.5%
Magistral Medium 1.0 thinking 51.7%
Mistral Medium 3 50.6%
Qwen3-235B-A22B thinking 49%
GPT-4.1 mini 48.7%
Phi-4-Reasoning-plus 48.4%
o3-mini 46.8%
Claude Sonnet 4 46.5%
Claude 3.7 Sonnet 46.2%
GPT-4.1 44.2%
Qwen3-235B-A22B non-thinking 43.6%
Qwen3-30B-A3B thinking 43.6%
o4-mini high 42.9%
DeepSeek-R1-Distill-Llama-70B 42.6%
Gemini 2.5 Flash May 2025 41.3%
Claude 3.5 Sonnet (October 2024) 37.2%
MiniMax-Text-01 36.5%
Llama 4 Scout 35.9%
Llama 3.3 70B Instruct 35.3%
Gemini 2.0 Pro 34.6%
GPT-4o Nov 2024 33.7%
GPT-4o Jan 2025 32.1%
Qwen2.5-Max 31.4%
Qwen3-30B-A3B non-thinking 31.1%
Cohere Command A 30.1%
Llama 4 Maverick 29.5%
Gemini 2.0 Flash Feb 2025 28.2%
Mistral Small 3 27.9%
DeepSeek-V3 26.9%
GPT-4.1 nano 26.6%
Mistral Small 3.1 26%
Jamba 1.6 Large 25%
Gemma 3 27B 23.4%
Gemma 3n E4B 22.8%
OLMo 2 32B Instruct 19.2%

5 models have held number one since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic DeepSeek NVIDIA xAI
Own leaderboard Not tested since Jun 2025
Grok-3 mini 73.7%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Grok-3 mini Llama-3.3-Nemotron-Super-49B-v1 Claude 3.5 Sonnet (October 2024) 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Grok-3 mini Llama-3.3-Nemotron-Super-49B-v1

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Science and reasoning

All 88 in Science and reasoning
  • GPQA Diamond In use 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Gemini 4 Argon 96.4%
  • Terminal-Bench-Science In use 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. GPT-6 Astra 68.1%
  • FrontierScience In use Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Intern-S2-397B 87%
  • BioMysteryBench In use Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Claude Opus 5 90.1%
  • HiPhO In use Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Gemini 3 Pro 78.3%
  • LABBench2 In use 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. GPT-5.2 60.9%