Benchmark Almanac

HELM Lite

GPT-4o May 2024 93.8 leads since May 2024 Not in model cards

A lightweight HELM board that runs models with in-context learning on ten scenarios. They include NarrativeQA, NaturalQuestions, OpenbookQA, MMLU, GSM8K, MATH, LegalBench, MedQA and WMT 2014.

14 releases from v1.0.0 (2023-12-19) to v1.13.0 (2025-01-10). HELM Capabilities followed in 2025-03.

Made by Stanford CRFM Released Dec 2023 Composite indices All tasks public Last model added Mar 2025 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
stanford-crfm/helm 2,961 stars Leaderboard crfm.stanford.edu
91
Entries on its own leaderboard, 71 models we track

GPT-4o leads with 93.8 among models released by Mar 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Mar 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

91 entries. Scroll sideways for the rest.

  1. 93.8 GPT-4o
  2. 92.8 GPT-4o
  3. 90.8 DeepSeek-V3
  4. 88.5 Claude 3.5 Sonnet
  5. 88.5 Amazon Nova Pro
  6. 86.7 GPT-4
  7. 86.4 GPT-4 Turbo
  8. 85.4 Llama 3.1-405B
  9. 84.6 Claude 3.5 Sonnet (October 2024)
  10. 84.2 Gemini 1.5 Pro
  11. 81.9 Llama 3.2 90B
  12. 81.3 Gemini 2.0 Flash
  13. 81.2 Llama 3.3 70B Instruct
  14. 80.8 Llama 3.1-70B
  15. 80.8 Palmyra X4
  16. 79.3 Llama 3-70B
  17. 77.0 Qwen2-72B
  18. 74.5 Qwen2.5-72B
  19. 74.4 Mistral Large 2
  20. 73.9 Gemini 1.5 Pro
  21. 70.8 Amazon Nova Lite
  22. 70.5 Mixtral 8x22B
  23. 70.1 GPT-4o mini
  24. 69.8 gpt-4-1106-preview
  25. 68.3 Claude 3 Opus
  26. 67.9 palmyra-x-v3
  27. 67.5 Gemma 2 27B
  28. 66.7 Gemini 1.5 Flash
  29. 64.4 text-unicorn@001
  30. 63.7 Jamba 1.5 Large
  31. 60.8 Qwen1.5-72B
  32. 60.2 solar-pro-241126
  33. 58.9 palmyra-x-v2
  34. 57.4 Gemini 1.5 Flash
  35. 57.0 Yi-34B
  36. 56.2 Gemma 2 9B
  37. 55.0 Qwen1.5-110B
  38. 54.6 Qwen1.5-32B
  39. 53.1 Claude 3.5 Haiku
  40. 52.6 text-bison@001
  41. 52.4 Amazon Nova Micro
  42. 51.8 Claude 1.3
  43. 51.0 Mixtral 8x7B
  44. 50.9 phi-3-medium 14B
  45. 48.9 Claude 2
  46. 48.8 DeepSeek LLM 67B
  47. 48.8 Qwen2.5-7B
  48. 48.2 Llama 2-70B
  49. 47.3 phi-3-small 7.4B
  50. 47.1 Yi-Large
  51. 44.0 Command R+
  52. 43.9 text-davinci-003
  53. 43.7 Claude 2.1
  54. 42.5 Qwen1.5-14B
  55. 42.2 Gemini 1.0 Pro
  56. 41.4 Jamba 1.5 Mini
  57. 39.9 Claude Instant
  58. 38.7 Llama 3-8B
  59. 37.6 Claude 3 Sonnet
  60. 35.8 GPT-3.5 Turbo
  61. 34.5 LLaMA-65B
  62. 33.8 Snowflake Arctic Instruct
  63. 33.6 Gemma 7B
  64. 33.6 text-davinci-002
  65. 33.3 open-mistral-nemo-2407
  66. 32.8 mistral-large-2402
  67. 32.7 command
  68. 32.5 Llama 3.2 11B
  69. 30.3 Llama 3.1-8B
  70. 29.9 Command R (03-2024)
  71. 29.2 Mistral 7B v0.1
  72. 28.9 DBRX
  73. 28.8 mistral-small-2402
  74. 28.6 jamba-instruct
  75. 27.5 Qwen1.5-7B
  76. 26.8 mistral-medium-2312
  77. 26.3 Claude 3 Haiku
  78. 25.4 Yi 6B
  79. 23.3 Llama 2-13B
  80. 21.7 Falcon-40B
  81. 21.4 j2-jumbo
  82. 19.6 Mistral 7B v0.3
  83. 17.2 j2-grande
  84. 16.9 Phi-2
  85. 15.2 Llama 2-7B
  86. 14.5 luminous-supreme
  87. 10.4 command-light
  88. 7.84 luminous-extended
  89. 6.45 Falcon-7B
  90. 5.22 olmo-7b
  91. 4.11 luminous-base
  1. 1 GPT-4o May 2024 93.8
  2. 2 GPT-4o Aug 2024 92.8
  3. 3 DeepSeek-V3 Mar 2025 90.8
  4. 4 Claude 3.5 Sonnet 88.5
  5. 5 Amazon Nova Pro 88.5
  6. 6 GPT-4 Jun 2023 86.7
  7. 7 GPT-4 Turbo Apr 2024 86.4
  8. 8 Llama 3.1-405B 85.4
  9. 9 Claude 3.5 Sonnet (October 2024) 84.6
  10. 10 Gemini 1.5 Pro Sep 2024 84.2
  11. 11 Llama 3.2 90B 81.9
  12. 12 Gemini 2.0 Flash Feb 2025 81.3
  13. 13 Llama 3.3 70B Instruct 81.2
  14. 14 Llama 3.1-70B 80.8
  15. 15 Palmyra X4 80.8
  16. 16 Llama 3-70B 79.3
  17. 17 Qwen2-72B 77.0
  18. 18 Qwen2.5-72B 74.5
  19. 19 Mistral Large 2 Jul 2024 74.4
  20. 20 Gemini 1.5 Pro May 2024 73.9
  21. 21 Amazon Nova Lite 70.8
  22. 22 Mixtral 8x22B 70.5
  23. 23 GPT-4o mini 70.1
  24. 24 gpt-4-1106-preview 69.8
  25. 25 Claude 3 Opus 68.3
  26. 26 palmyra-x-v3 67.9
  27. 27 Gemma 2 27B 67.5
  28. 28 Gemini 1.5 Flash May 2024 66.7
  29. 29 text-unicorn@001 64.4
  30. 30 Jamba 1.5 Large 63.7
  31. 31 Qwen1.5-72B 60.8
  32. 32 solar-pro-241126 60.2
  33. 33 palmyra-x-v2 58.9
  34. 34 Gemini 1.5 Flash Sep 2024 57.4
  35. 35 Yi-34B 57.0
  36. 36 Gemma 2 9B 56.2
  37. 37 Qwen1.5-110B 55.0
  38. 38 Qwen1.5-32B 54.6
  39. 39 Claude 3.5 Haiku 53.1
  40. 40 text-bison@001 52.6
  41. 41 Amazon Nova Micro 52.4
  42. 42 Claude 1.3 51.8
  43. 43 Mixtral 8x7B 51.0
  44. 44 phi-3-medium 14B 50.9
  45. 45 Claude 2 48.9
  46. 46 DeepSeek LLM 67B 48.8
  47. 47 Qwen2.5-7B 48.8
  48. 48 Llama 2-70B 48.2
  49. 49 phi-3-small 7.4B 47.3
  50. 50 Yi-Large 47.1
  51. 51 Command R+ 44.0
  52. 52 text-davinci-003 43.9
  53. 53 Claude 2.1 43.7
  54. 54 Qwen1.5-14B 42.5
  55. 55 Gemini 1.0 Pro 42.2
  56. 56 Jamba 1.5 Mini 41.4
  57. 57 Claude Instant 39.9
  58. 58 Llama 3-8B 38.7
  59. 59 Claude 3 Sonnet 37.6
  60. 60 GPT-3.5 Turbo Jun 2023 35.8
  61. 61 LLaMA-65B 34.5
  62. 62 Snowflake Arctic Instruct 33.8
  63. 63 Gemma 7B 33.6
  64. 64 text-davinci-002 33.6
  65. 65 open-mistral-nemo-2407 33.3
  66. 66 mistral-large-2402 32.8
  67. 67 command 32.7
  68. 68 Llama 3.2 11B 32.5
  69. 69 Llama 3.1-8B 30.3
  70. 70 Command R (03-2024) 29.9
  71. 71 Mistral 7B v0.1 29.2
  72. 72 DBRX 28.9
  73. 73 mistral-small-2402 28.8
  74. 74 jamba-instruct 28.6
  75. 75 Qwen1.5-7B 27.5
  76. 76 mistral-medium-2312 26.8
  77. 77 Claude 3 Haiku 26.3
  78. 78 Yi 6B 25.4
  79. 79 Llama 2-13B 23.3
  80. 80 Falcon-40B 21.7
  81. 81 j2-jumbo 21.4
  82. 82 Mistral 7B v0.3 19.6
  83. 83 j2-grande 17.2
  84. 84 Phi-2 16.9
  85. 85 Llama 2-7B 15.2
  86. 86 luminous-supreme 14.5
  87. 87 command-light 10.4
  88. 88 luminous-extended 7.84
  89. 89 Falcon-7B 6.45
  90. 90 olmo-7b 5.22
  91. 91 luminous-base 4.11
#EntryScoreSettingDateSource
1 GPT-4o May 2024 93.8 May 2024 Official leaderboard
2 GPT-4o Aug 2024 92.8 Aug 2024 Official leaderboard
3 DeepSeek-V3 Mar 2025 90.8 Mar 2025 Official leaderboard
4 Claude 3.5 Sonnet 88.5 Jun 2024 Official leaderboard
5 Amazon Nova Pro 88.5 Dec 2024 Official leaderboard
6 GPT-4 Jun 2023 86.7 Jun 2023 Official leaderboard
7 GPT-4 Turbo Apr 2024 86.4 Apr 2024 Official leaderboard
8 Llama 3.1-405B 85.4 Jul 2024 Official leaderboard
9 Claude 3.5 Sonnet (October 2024) 84.6 Oct 2024 Official leaderboard
10 Gemini 1.5 Pro Sep 2024 84.2 Sep 2024 Official leaderboard
11 Llama 3.2 90B 81.9 Sep 2024 Official leaderboard
12 Gemini 2.0 Flash Feb 2025 81.3 Feb 2025 Official leaderboard
13 Llama 3.3 70B Instruct 81.2 Dec 2024 Official leaderboard
14 Llama 3.1-70B 80.8 Jul 2024 Official leaderboard
15 Palmyra X4 80.8 Oct 2024 Official leaderboard
16 Llama 3-70B 79.3 Apr 2024 Official leaderboard
17 Qwen2-72B 77.0 Jun 2024 Official leaderboard
18 Qwen2.5-72B 74.5 Sep 2024 Official leaderboard
19 Mistral Large 2 Jul 2024 74.4 Jul 2024 Official leaderboard
20 Gemini 1.5 Pro May 2024 73.9 May 2024 Official leaderboard
21 Amazon Nova Lite 70.8 Dec 2024 Official leaderboard
22 Mixtral 8x22B 70.5 Apr 2024 Official leaderboard
23 GPT-4o mini 70.1 Jul 2024 Official leaderboard
24 gpt-4-1106-preview 69.8 Official leaderboard
25 Claude 3 Opus 68.3 Feb 2024 Official leaderboard
26 palmyra-x-v3 67.9 Official leaderboard
27 Gemma 2 27B 67.5 Jun 2024 Official leaderboard
28 Gemini 1.5 Flash May 2024 66.7 May 2024 Official leaderboard
29 text-unicorn@001 64.4 Official leaderboard
30 Jamba 1.5 Large 63.7 Aug 2024 Official leaderboard
31 Qwen1.5-72B 60.8 Feb 2024 Official leaderboard
32 solar-pro-241126 60.2 Official leaderboard
33 palmyra-x-v2 58.9 Official leaderboard
34 Gemini 1.5 Flash Sep 2024 57.4 Aug 2024 Official leaderboard
35 Yi-34B 57.0 Nov 2023 Official leaderboard
36 Gemma 2 9B 56.2 Jun 2024 Official leaderboard
37 Qwen1.5-110B 55.0 Apr 2024 Official leaderboard
38 Qwen1.5-32B 54.6 Feb 2024 Official leaderboard
39 Claude 3.5 Haiku 53.1 Oct 2024 Official leaderboard
40 text-bison@001 52.6 Official leaderboard
41 Amazon Nova Micro 52.4 Dec 2024 Official leaderboard
42 Claude 1.3 51.8 Apr 2023 Official leaderboard
43 Mixtral 8x7B 51.0 Dec 2023 Official leaderboard
44 phi-3-medium 14B 50.9 Apr 2024 Official leaderboard
45 Claude 2 48.9 Jul 2023 Official leaderboard
46 DeepSeek LLM 67B 48.8 Nov 2023 Official leaderboard
47 Qwen2.5-7B 48.8 Sep 2024 Official leaderboard
48 Llama 2-70B 48.2 Jul 2023 Official leaderboard
49 phi-3-small 7.4B 47.3 Apr 2024 Official leaderboard
50 Yi-Large 47.1 May 2024 Official leaderboard
51 Command R+ 44.0 Aug 2024 Official leaderboard
52 text-davinci-003 43.9 Official leaderboard
53 Claude 2.1 43.7 Nov 2023 Official leaderboard
54 Qwen1.5-14B 42.5 Feb 2024 Official leaderboard
55 Gemini 1.0 Pro 42.2 Dec 2023 Official leaderboard
56 Jamba 1.5 Mini 41.4 Aug 2024 Official leaderboard
57 Claude Instant 39.9 Aug 2023 Official leaderboard
58 Llama 3-8B 38.7 Apr 2024 Official leaderboard
59 Claude 3 Sonnet 37.6 Feb 2024 Official leaderboard
60 GPT-3.5 Turbo Jun 2023 35.8 Jun 2023 Official leaderboard
61 LLaMA-65B 34.5 Feb 2023 Official leaderboard
62 Snowflake Arctic Instruct 33.8 Apr 2024 Official leaderboard
63 Gemma 7B 33.6 Feb 2024 Official leaderboard
64 text-davinci-002 33.6 Official leaderboard
65 open-mistral-nemo-2407 33.3 Official leaderboard
66 mistral-large-2402 32.8 Official leaderboard
67 command 32.7 Official leaderboard
68 Llama 3.2 11B 32.5 Sep 2024 Official leaderboard
69 Llama 3.1-8B 30.3 Jul 2024 Official leaderboard
70 Command R (03-2024) 29.9 Mar 2024 Official leaderboard
71 Mistral 7B v0.1 29.2 Sep 2023 Official leaderboard
72 DBRX 28.9 Mar 2024 Official leaderboard
73 mistral-small-2402 28.8 Feb 2024 Official leaderboard
74 jamba-instruct 28.6 Official leaderboard
75 Qwen1.5-7B 27.5 Feb 2024 Official leaderboard
76 mistral-medium-2312 26.8 Official leaderboard
77 Claude 3 Haiku 26.3 Mar 2024 Official leaderboard
78 Yi 6B 25.4 Nov 2023 Official leaderboard
79 Llama 2-13B 23.3 Jul 2023 Official leaderboard
80 Falcon-40B 21.7 Mar 2023 Official leaderboard
81 j2-jumbo 21.4 Official leaderboard
82 Mistral 7B v0.3 19.6 Sep 2023 Official leaderboard
83 j2-grande 17.2 Official leaderboard
84 Phi-2 16.9 Dec 2023 Official leaderboard
85 Llama 2-7B 15.2 Jul 2023 Official leaderboard
86 luminous-supreme 14.5 Official leaderboard
87 command-light 10.4 Official leaderboard
88 luminous-extended 7.84 Official leaderboard
89 Falcon-7B 6.45 Apr 2023 Official leaderboard
90 olmo-7b 5.22 Official leaderboard
91 luminous-base 4.11 Official leaderboard

2 models have held number one since its release in Dec 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Mar 2025
GPT-4o 93.8

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 20.0 40.0 60.0 80.0 100.0 2023 2024 2025 0.00 20.0 40.0 60.0 80.0 100.0 2023 2024 2025

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Other

All 41 in Other
  • Personal Superintelligence Index (Meta) In use Meta's composite of seven personal and knowledge-work evals: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT and DRACO.
  • Artificial Analysis Intelligence Index v5 In use Version 5 of Artificial Analysis's index: nine evaluations in four equal categories, most with private test sets, run three times each in its own agent harness. Gemini 4 Carbon 59.0
  • Anthropic ECI (AECI) Only in system cards Index that combines a basket of capability benchmarks into one capability score to track trends across Anthropic models. Claude Opus 5.5 169.4
  • Qwen in-house English automatic evaluation Only in system cards Models answer Qwen in-house English datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. Qwen2-72B 72.9%
  • Recognition of model-written user turns (Anthropic) Only in system cards Models write synthetic user turns for real Claude Code transcripts, then judge which version is real. Score is accuracy above chance. Claude Mythos Preview 0.35
  • LiveBench Dropped Models answer monthly refreshed questions across math, coding, reasoning, language, instruction following, and data analysis. Scored automatically against objective ground truth. Claude Fable 5.1 85.7%