Benchmark Almanac

HELM Capabilities

GPT-5 mini 81.9 leads since Aug 2025 Not in model cards

Stanford CRFM runs language models on a curated set of scenarios, one per key capability, and averages the scores. Prompt-level results are public.

Successor in spirit to HELM Classic and HELM Lite, which CRFM still hosts as separate leaderboards.

Made by Stanford CRFM Released Mar 2025 Composite indices All tasks public Last model added Nov 2025 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
stanford-crfm/helm 2,961 stars Leaderboard crfm.stanford.edu
66
Entries on its own leaderboard, 60 models we track

GPT-5 mini leads with 81.9 among models released by Nov 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Nov 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

66 entries. Scroll sideways for the rest.

  1. 81.9 GPT-5 mini
  2. 81.2 o4-mini
  3. 81.1 o3
  4. 80.7 GPT-5
  5. 79.9 Gemini 3 Pro
  6. 79.8 Qwen3-235B-A22B-Instruct
  7. 78.5 Grok 4
  8. 78.0 Claude Opus 4
  9. 77.0 gpt-oss-120b
  10. 76.8 Kimi K2
  11. 76.6 Claude Sonnet 4
  12. 76.2 Claude Sonnet 4.5
  13. 74.8 GPT-5 nano
  14. 74.5 Gemini 2.5 Pro
  15. 72.7 Grok 3
  16. 72.7 GPT-4.1
  17. 72.6 Qwen3-235B-A22B
  18. 72.6 GPT-4.1 mini
  19. 71.8 Llama 4 Maverick
  20. 71.7 Claude Haiku 4.5
  21. 70.0 Qwen3-Next-80B-A3B
  22. 69.9 DeepSeek-R1
  23. 69.7 Palmyra X5
  24. 67.9 Grok-3 mini
  25. 67.9 Gemini 2.0 Flash
  26. 67.4 Claude 3.7 Sonnet
  27. 67.4 gpt-oss-20b
  28. 67.0 GLM-4.5-Air
  29. 66.5 DeepSeek-V3
  30. 65.7 Gemini 1.5 Pro
  31. 65.6 GPT-5.1
  32. 65.3 Claude 3.5 Sonnet (October 2024)
  33. 64.4 Llama 4 Scout
  34. 64.2 Gemini 2.0 Flash-Lite
  35. 63.7 Amazon Nova Premier
  36. 63.4 GPT-4o
  37. 62.6 Gemini 2.5 Flash
  38. 61.8 Llama 3.1-405B
  39. 61.6 GPT-4.1 nano
  40. 60.9 Palmyra X4
  41. 60.9 Gemini 1.5 Flash
  42. 59.9 Qwen2.5-72B
  43. 59.8 Mistral Large 2
  44. 59.1 Gemini 2.5 Flash-Lite
  45. 59.1 Amazon Nova Pro
  46. 57.7 palmyra-fin
  47. 57.5 Granite 4.0 H Small
  48. 57.4 Llama 3.1-70B
  49. 56.5 GPT-4o mini
  50. 55.8 Mistral Small 3.1
  51. 55.1 Amazon Nova Lite
  52. 54.9 Claude 3.5 Haiku
  53. 52.9 Qwen2.5-7B
  54. 52.2 Amazon Nova Micro
  55. 48.6 granite-4.0-micro
  56. 47.9 Mixtral 8x22B
  57. 47.5 palmyra-med
  58. 47.5 OLMo 2 32B Instruct
  59. 46.3 Granite 3.3 8B Instruct
  60. 44.4 Llama 3.1-8B
  61. 44.0 olmo-2-1124-13b-instruct
  62. 40.5 olmo-2-1124-7b-instruct
  63. 39.7 Mixtral 8x7B
  64. 37.6 Mistral 7B v0.3
  65. 33.2 olmoe-1b-7b-0125-instruct
  66. 32.5 marin-8b-instruct
  1. 1 GPT-5 mini 81.9
  2. 2 o4-mini 81.2
  3. 3 o3 81.1
  4. 4 GPT-5 80.7
  5. 5 Gemini 3 Pro 79.9
  6. 6 Qwen3-235B-A22B-Instruct Jul 2025 79.8
  7. 7 Grok 4 78.5
  8. 8 Claude Opus 4 thinking 10k 78.0
  9. 9 gpt-oss-120b 77.0
  10. 10 Kimi K2 Jul 2025 76.8
  11. 11 Claude Sonnet 4 thinking 10k 76.6
  12. 12 Claude Sonnet 4.5 76.2
  13. 13 GPT-5 nano 74.8
  14. 14 Gemini 2.5 Pro Jun 2025 74.5
  15. 15 Grok 3 72.7
  16. 16 GPT-4.1 72.7
  17. 17 Qwen3-235B-A22B 72.6
  18. 18 GPT-4.1 mini 72.6
  19. 19 Llama 4 Maverick 71.8
  20. 20 Claude Haiku 4.5 71.7
  21. 21 Qwen3-Next-80B-A3B thinking 70.0
  22. 22 DeepSeek-R1 May 2025 69.9
  23. 23 Palmyra X5 69.7
  24. 24 Grok-3 mini 67.9
  25. 25 Gemini 2.0 Flash Feb 2025 67.9
  26. 26 Claude 3.7 Sonnet 67.4
  27. 27 gpt-oss-20b 67.4
  28. 28 GLM-4.5-Air 67.0
  29. 29 DeepSeek-V3 Mar 2025 66.5
  30. 30 Gemini 1.5 Pro Sep 2024 65.7
  31. 31 GPT-5.1 65.6
  32. 32 Claude 3.5 Sonnet (October 2024) 65.3
  33. 33 Llama 4 Scout 64.4
  34. 34 Gemini 2.0 Flash-Lite 64.2
  35. 35 Amazon Nova Premier 63.7
  36. 36 GPT-4o Nov 2024 63.4
  37. 37 Gemini 2.5 Flash Sep 2025 62.6
  38. 38 Llama 3.1-405B 61.8
  39. 39 GPT-4.1 nano 61.6
  40. 40 Palmyra X4 60.9
  41. 41 Gemini 1.5 Flash Sep 2024 60.9
  42. 42 Qwen2.5-72B 59.9
  43. 43 Mistral Large 2 Nov 2024 59.8
  44. 44 Gemini 2.5 Flash-Lite Sep 2025 59.1
  45. 45 Amazon Nova Pro 59.1
  46. 46 palmyra-fin 57.7
  47. 47 Granite 4.0 H Small 57.5
  48. 48 Llama 3.1-70B 57.4
  49. 49 GPT-4o mini 56.5
  50. 50 Mistral Small 3.1 55.8
  51. 51 Amazon Nova Lite 55.1
  52. 52 Claude 3.5 Haiku 54.9
  53. 53 Qwen2.5-7B 52.9
  54. 54 Amazon Nova Micro 52.2
  55. 55 granite-4.0-micro 48.6
  56. 56 Mixtral 8x22B 47.9
  57. 57 palmyra-med 47.5
  58. 58 OLMo 2 32B Instruct 47.5
  59. 59 Granite 3.3 8B Instruct 46.3
  60. 60 Llama 3.1-8B 44.4
  61. 61 olmo-2-1124-13b-instruct 44.0
  62. 62 olmo-2-1124-7b-instruct 40.5
  63. 63 Mixtral 8x7B 39.7
  64. 64 Mistral 7B v0.3 37.6
  65. 65 olmoe-1b-7b-0125-instruct 33.2
  66. 66 marin-8b-instruct 32.5
#EntryScoreSettingDateSource
1 GPT-5 mini 81.9 Aug 2025 Official leaderboard
2 o4-mini 81.2 Apr 2025 Official leaderboard
3 o3 81.1 Apr 2025 Official leaderboard
4 GPT-5 80.7 Aug 2025 Official leaderboard
5 Gemini 3 Pro 79.9 Nov 2025 Official leaderboard
6 Qwen3-235B-A22B-Instruct Jul 2025 79.8 Jul 2025 Official leaderboard
7 Grok 4 78.5 Jul 2025 Official leaderboard
8 Claude Opus 4 78.0 thinking 10k May 2025 Official leaderboard
9 gpt-oss-120b 77.0 Aug 2025 Official leaderboard
10 Kimi K2 Jul 2025 76.8 Jul 2025 Official leaderboard
11 Claude Sonnet 4 76.6 thinking 10k May 2025 Official leaderboard
12 Claude Sonnet 4.5 76.2 Sep 2025 Official leaderboard
13 GPT-5 nano 74.8 Aug 2025 Official leaderboard
14 Gemini 2.5 Pro Jun 2025 74.5 Jun 2025 Official leaderboard
15 Grok 3 72.7 Apr 2025 Official leaderboard
16 GPT-4.1 72.7 Apr 2025 Official leaderboard
17 Qwen3-235B-A22B 72.6 Apr 2025 Official leaderboard
18 GPT-4.1 mini 72.6 Apr 2025 Official leaderboard
19 Llama 4 Maverick 71.8 Apr 2025 Official leaderboard
20 Claude Haiku 4.5 71.7 Oct 2025 Official leaderboard
21 Qwen3-Next-80B-A3B 70.0 thinking Sep 2025 Official leaderboard
22 DeepSeek-R1 May 2025 69.9 May 2025 Official leaderboard
23 Palmyra X5 69.7 Apr 2025 Official leaderboard
24 Grok-3 mini 67.9 Apr 2025 Official leaderboard
25 Gemini 2.0 Flash Feb 2025 67.9 Feb 2025 Official leaderboard
26 Claude 3.7 Sonnet 67.4 Feb 2025 Official leaderboard
27 gpt-oss-20b 67.4 Aug 2025 Official leaderboard
28 GLM-4.5-Air 67.0 Jul 2025 Official leaderboard
29 DeepSeek-V3 Mar 2025 66.5 Mar 2025 Official leaderboard
30 Gemini 1.5 Pro Sep 2024 65.7 Sep 2024 Official leaderboard
31 GPT-5.1 65.6 Nov 2025 Official leaderboard
32 Claude 3.5 Sonnet (October 2024) 65.3 Oct 2024 Official leaderboard
33 Llama 4 Scout 64.4 Apr 2025 Official leaderboard
34 Gemini 2.0 Flash-Lite 64.2 Feb 2025 Official leaderboard
35 Amazon Nova Premier 63.7 Apr 2025 Official leaderboard
36 GPT-4o Nov 2024 63.4 Nov 2024 Official leaderboard
37 Gemini 2.5 Flash Sep 2025 62.6 Sep 2025 Official leaderboard
38 Llama 3.1-405B 61.8 Jul 2024 Official leaderboard
39 GPT-4.1 nano 61.6 Apr 2025 Official leaderboard
40 Palmyra X4 60.9 Oct 2024 Official leaderboard
41 Gemini 1.5 Flash Sep 2024 60.9 Aug 2024 Official leaderboard
42 Qwen2.5-72B 59.9 Sep 2024 Official leaderboard
43 Mistral Large 2 Nov 2024 59.8 Nov 2024 Official leaderboard
44 Gemini 2.5 Flash-Lite Sep 2025 59.1 Sep 2025 Official leaderboard
45 Amazon Nova Pro 59.1 Dec 2024 Official leaderboard
46 palmyra-fin 57.7 Official leaderboard
47 Granite 4.0 H Small 57.5 Oct 2025 Official leaderboard
48 Llama 3.1-70B 57.4 Jul 2024 Official leaderboard
49 GPT-4o mini 56.5 Jul 2024 Official leaderboard
50 Mistral Small 3.1 55.8 Mar 2025 Official leaderboard
51 Amazon Nova Lite 55.1 Dec 2024 Official leaderboard
52 Claude 3.5 Haiku 54.9 Oct 2024 Official leaderboard
53 Qwen2.5-7B 52.9 Sep 2024 Official leaderboard
54 Amazon Nova Micro 52.2 Dec 2024 Official leaderboard
55 granite-4.0-micro 48.6 Oct 2025 Official leaderboard
56 Mixtral 8x22B 47.9 Apr 2024 Official leaderboard
57 palmyra-med 47.5 Official leaderboard
58 OLMo 2 32B Instruct 47.5 Mar 2025 Official leaderboard
59 Granite 3.3 8B Instruct 46.3 Apr 2025 Official leaderboard
60 Llama 3.1-8B 44.4 Jul 2024 Official leaderboard
61 olmo-2-1124-13b-instruct 44.0 Official leaderboard
62 olmo-2-1124-7b-instruct 40.5 Official leaderboard
63 Mixtral 8x7B 39.7 Dec 2023 Official leaderboard
64 Mistral 7B v0.3 37.6 Sep 2023 Official leaderboard
65 olmoe-1b-7b-0125-instruct 33.2 Official leaderboard
66 marin-8b-instruct 32.5 Official leaderboard

5 models have held number one since its release in Mar 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind Meta xAI OpenAI
Own leaderboard Not tested since Nov 2025
GPT-5 mini 81.9

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
40.0 60.0 80.0 2024 2025 2026 40.0 60.0 80.0 2024 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Other

All 41 in Other
  • Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
  • Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Gemini 4 Carbon 59.0
  • Anthropic ECI (AECI) Only in system cards Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Claude Opus 5.5 169.4
  • Qwen in-house English automatic evaluation Only in system cards Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Qwen2-72B 72.9%
  • Recognition of model-written user turns (Anthropic) Only in system cards Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. Claude Mythos Preview 0.35
  • LiveBench Dropped Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Claude Fable 5.1 85.7%