Benchmark Almanac

HELM Safety

GPT-5 nano 98.6 leads since Aug 2025 Not in model cards

Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard.

Version 1.0 evaluated 24 models at launch.

Made by Stanford CRFM Released Nov 2024 Refusing harmful requests All tasks public Last model added Nov 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
stanford-crfm/helm 2,961 stars Leaderboard crfm.stanford.edu
81
Entries on its own leaderboard, 75 models we track

GPT-5 nano leads with 98.6 among models released by Nov 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Nov 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

81 entries. Scroll sideways for the rest.

  1. 98.6 GPT-5 nano
  2. 98.2 o3
  3. 98.1 gpt-oss-120b
  4. 98.1 Claude Sonnet 4
  5. 98.1 GPT-5
  6. 98.0 GPT-5 mini
  7. 98.0 Kimi K2
  8. 97.7 Claude 3.5 Sonnet
  9. 97.6 o1
  10. 97.3 o4-mini
  11. 97.1 Claude Sonnet 4.5
  12. 96.9 Claude Opus 4
  13. 96.8 Claude 3 Opus
  14. 96.8 GPT-5.1
  15. 96.5 GPT-4.5
  16. 96.4 gpt-oss-20b
  17. 96.3 GPT-4.1
  18. 96.2 o3-mini
  19. 96.1 GPT-4 Turbo
  20. 95.5 o1-mini
  21. 95.5 Claude Haiku 4.5
  22. 94.9 GPT-4.1 mini
  23. 94.8 Qwen3-235B-A22B-Instruct
  24. 94.8 palmyra-fin
  25. 94.8 Qwen3-Next-80B-A3B
  26. 94.6 GPT-4o
  27. 94.5 Claude 3.7 Sonnet
  28. 94.3 Claude 3 Sonnet
  29. 93.8 GPT-4.1 nano
  30. 93.3 Qwen2-72B
  31. 93.3 Palmyra X4
  32. 93.1 Qwen2.5-72B
  33. 93.0 GPT-4o mini
  34. 92.7 Gemini 1.5 Flash
  35. 92.6 Palmyra X5
  36. 92.5 Gemini 3 Pro
  37. 92.5 Gemini 1.5 Pro
  38. 91.4 Gemini 2.5 Pro
  39. 91.4 Granite 4.0 H Small
  40. 91.2 Gemini 2.5 Flash
  41. 91.0 Gemini 2.5 Flash-Lite
  42. 91.0 Gemini 2.0 Flash
  43. 90.8 Gemini 2.0 Flash-Lite
  44. 90.6 Llama 4 Maverick
  45. 90.5 Gemini 2.0 Pro
  46. 90.5 Granite 3.3 8B Instruct
  47. 90.2 Grok-3 mini
  48. 90.1 GLM-4.5-Air
  49. 90.0 Qwen3-235B-A22B
  50. 89.9 Qwen2.5-7B
  51. 89.7 Llama 3.1-405B
  52. 89.7 OLMo 2 32B Instruct
  53. 89.5 Llama 3-70B
  54. 89.4 DeepSeek-R1
  55. 88.6 Qwen1.5-72B
  56. 88.6 Llama 3-8B
  57. 87.8 Claude 3 Haiku
  58. 87.3 Llama 4 Scout
  59. 87.3 granite-4.0-micro
  60. 87.3 DeepSeek LLM 67B
  61. 87.2 DeepSeek-V3
  62. 87.0 marin-8b-instruct
  63. 86.2 Llama 3.1-8B
  64. 86.0 Command R+
  65. 85.7 palmyra-med
  66. 85.5 Grok 3
  67. 85.3 GPT-3.5 Turbo
  68. 85.1 Mixtral 8x22B
  69. 84.7 Mistral Small 3
  70. 84.5 Llama 3.1-70B
  71. 83.6 Grok 4
  72. 83.5 GPT-3.5 Turbo
  73. 81.4 Mixtral 8x7B
  74. 81.4 GPT-3.5 Turbo
  75. 80.9 Command R (03-2024)
  76. 77.3 olmo-2-1124-13b-instruct
  77. 73.0 Mistral 7B v0.3
  78. 72.9 olmo-2-1124-7b-instruct
  79. 70.1 olmoe-1b-7b-0125-instruct
  80. 62.8 DBRX
  81. 52.5 Mistral-7B-Instruct-v0.1
  1. 1 GPT-5 nano 98.6
  2. 2 o3 98.2
  3. 3 gpt-oss-120b 98.1
  4. 4 Claude Sonnet 4 thinking 10k 98.1
  5. 5 GPT-5 98.1
  6. 6 GPT-5 mini 98.0
  7. 7 Kimi K2 Jul 2025 98.0
  8. 8 Claude 3.5 Sonnet 97.7
  9. 9 o1 97.6
  10. 10 o4-mini 97.3
  11. 11 Claude Sonnet 4.5 97.1
  12. 12 Claude Opus 4 thinking 10k 96.9
  13. 13 Claude 3 Opus 96.8
  14. 14 GPT-5.1 96.8
  15. 15 GPT-4.5 96.5
  16. 16 gpt-oss-20b 96.4
  17. 17 GPT-4.1 96.3
  18. 18 o3-mini 96.2
  19. 19 GPT-4 Turbo Apr 2024 96.1
  20. 20 o1-mini 95.5
  21. 21 Claude Haiku 4.5 95.5
  22. 22 GPT-4.1 mini 94.9
  23. 23 Qwen3-235B-A22B-Instruct Jul 2025 94.8
  24. 24 palmyra-fin 94.8
  25. 25 Qwen3-Next-80B-A3B thinking 94.8
  26. 26 GPT-4o May 2024 94.6
  27. 27 Claude 3.7 Sonnet 94.5
  28. 28 Claude 3 Sonnet 94.3
  29. 29 GPT-4.1 nano 93.8
  30. 30 Qwen2-72B 93.3
  31. 31 Palmyra X4 93.3
  32. 32 Qwen2.5-72B 93.1
  33. 33 GPT-4o mini 93.0
  34. 34 Gemini 1.5 Flash May 2024 92.7
  35. 35 Palmyra X5 92.6
  36. 36 Gemini 3 Pro 92.5
  37. 37 Gemini 1.5 Pro May 2024 92.5
  38. 38 Gemini 2.5 Pro Jun 2025 91.4
  39. 39 Granite 4.0 H Small with guardian 91.4
  40. 40 Gemini 2.5 Flash Sep 2025 91.2
  41. 41 Gemini 2.5 Flash-Lite Sep 2025 91.0
  42. 42 Gemini 2.0 Flash Feb 2025 91.0
  43. 43 Gemini 2.0 Flash-Lite 90.8
  44. 44 Llama 4 Maverick 90.6
  45. 45 Gemini 2.0 Pro 90.5
  46. 46 Granite 3.3 8B Instruct 90.5
  47. 47 Grok-3 mini 90.2
  48. 48 GLM-4.5-Air 90.1
  49. 49 Qwen3-235B-A22B 90.0
  50. 50 Qwen2.5-7B 89.9
  51. 51 Llama 3.1-405B 89.7
  52. 52 OLMo 2 32B Instruct 89.7
  53. 53 Llama 3-70B 89.5
  54. 54 DeepSeek-R1 May 2025 89.4
  55. 55 Qwen1.5-72B 88.6
  56. 56 Llama 3-8B 88.6
  57. 57 Claude 3 Haiku 87.8
  58. 58 Llama 4 Scout 87.3
  59. 59 granite-4.0-micro with guardian 87.3
  60. 60 DeepSeek LLM 67B 87.3
  61. 61 DeepSeek-V3 Mar 2025 87.2
  62. 62 marin-8b-instruct 87.0
  63. 63 Llama 3.1-8B 86.2
  64. 64 Command R+ 86.0
  65. 65 palmyra-med 85.7
  66. 66 Grok 3 85.5
  67. 67 GPT-3.5 Turbo Jun 2023 85.3
  68. 68 Mixtral 8x22B 85.1
  69. 69 Mistral Small 3 84.7
  70. 70 Llama 3.1-70B 84.5
  71. 71 Grok 4 83.6
  72. 72 GPT-3.5 Turbo Nov 2023 83.5
  73. 73 Mixtral 8x7B 81.4
  74. 74 GPT-3.5 Turbo Jan 2024 81.4
  75. 75 Command R (03-2024) 80.9
  76. 76 olmo-2-1124-13b-instruct 77.3
  77. 77 Mistral 7B v0.3 73.0
  78. 78 olmo-2-1124-7b-instruct 72.9
  79. 79 olmoe-1b-7b-0125-instruct 70.1
  80. 80 DBRX 62.8
  81. 81 Mistral-7B-Instruct-v0.1 52.5
#EntryScoreSettingDateSource
1 GPT-5 nano 98.6 Aug 2025 Official leaderboard
2 o3 98.2 Apr 2025 Official leaderboard
3 gpt-oss-120b 98.1 Aug 2025 Official leaderboard
4 Claude Sonnet 4 98.1 thinking 10k May 2025 Official leaderboard
5 GPT-5 98.1 Aug 2025 Official leaderboard
6 GPT-5 mini 98.0 Aug 2025 Official leaderboard
7 Kimi K2 Jul 2025 98.0 Jul 2025 Official leaderboard
8 Claude 3.5 Sonnet 97.7 Jun 2024 Official leaderboard
9 o1 97.6 Dec 2024 Official leaderboard
10 o4-mini 97.3 Apr 2025 Official leaderboard
11 Claude Sonnet 4.5 97.1 Sep 2025 Official leaderboard
12 Claude Opus 4 96.9 thinking 10k May 2025 Official leaderboard
13 Claude 3 Opus 96.8 Feb 2024 Official leaderboard
14 GPT-5.1 96.8 Nov 2025 Official leaderboard
15 GPT-4.5 96.5 Feb 2025 Official leaderboard
16 gpt-oss-20b 96.4 Aug 2025 Official leaderboard
17 GPT-4.1 96.3 Apr 2025 Official leaderboard
18 o3-mini 96.2 Jan 2025 Official leaderboard
19 GPT-4 Turbo Apr 2024 96.1 Apr 2024 Official leaderboard
20 o1-mini 95.5 Sep 2024 Official leaderboard
21 Claude Haiku 4.5 95.5 Oct 2025 Official leaderboard
22 GPT-4.1 mini 94.9 Apr 2025 Official leaderboard
23 Qwen3-235B-A22B-Instruct Jul 2025 94.8 Jul 2025 Official leaderboard
24 palmyra-fin 94.8 Official leaderboard
25 Qwen3-Next-80B-A3B 94.8 thinking Sep 2025 Official leaderboard
26 GPT-4o May 2024 94.6 May 2024 Official leaderboard
27 Claude 3.7 Sonnet 94.5 Feb 2025 Official leaderboard
28 Claude 3 Sonnet 94.3 Feb 2024 Official leaderboard
29 GPT-4.1 nano 93.8 Apr 2025 Official leaderboard
30 Qwen2-72B 93.3 Jun 2024 Official leaderboard
31 Palmyra X4 93.3 Oct 2024 Official leaderboard
32 Qwen2.5-72B 93.1 Sep 2024 Official leaderboard
33 GPT-4o mini 93.0 Jul 2024 Official leaderboard
34 Gemini 1.5 Flash May 2024 92.7 May 2024 Official leaderboard
35 Palmyra X5 92.6 Apr 2025 Official leaderboard
36 Gemini 3 Pro 92.5 Nov 2025 Official leaderboard
37 Gemini 1.5 Pro May 2024 92.5 May 2024 Official leaderboard
38 Gemini 2.5 Pro Jun 2025 91.4 Jun 2025 Official leaderboard
39 Granite 4.0 H Small 91.4 with guardian Oct 2025 Official leaderboard
40 Gemini 2.5 Flash Sep 2025 91.2 Sep 2025 Official leaderboard
41 Gemini 2.5 Flash-Lite Sep 2025 91.0 Sep 2025 Official leaderboard
42 Gemini 2.0 Flash Feb 2025 91.0 Feb 2025 Official leaderboard
43 Gemini 2.0 Flash-Lite 90.8 Feb 2025 Official leaderboard
44 Llama 4 Maverick 90.6 Apr 2025 Official leaderboard
45 Gemini 2.0 Pro 90.5 Feb 2025 Official leaderboard
46 Granite 3.3 8B Instruct 90.5 Apr 2025 Official leaderboard
47 Grok-3 mini 90.2 Apr 2025 Official leaderboard
48 GLM-4.5-Air 90.1 Jul 2025 Official leaderboard
49 Qwen3-235B-A22B 90.0 Apr 2025 Official leaderboard
50 Qwen2.5-7B 89.9 Sep 2024 Official leaderboard
51 Llama 3.1-405B 89.7 Jul 2024 Official leaderboard
52 OLMo 2 32B Instruct 89.7 Mar 2025 Official leaderboard
53 Llama 3-70B 89.5 Apr 2024 Official leaderboard
54 DeepSeek-R1 May 2025 89.4 May 2025 Official leaderboard
55 Qwen1.5-72B 88.6 Feb 2024 Official leaderboard
56 Llama 3-8B 88.6 Apr 2024 Official leaderboard
57 Claude 3 Haiku 87.8 Mar 2024 Official leaderboard
58 Llama 4 Scout 87.3 Apr 2025 Official leaderboard
59 granite-4.0-micro 87.3 with guardian Oct 2025 Official leaderboard
60 DeepSeek LLM 67B 87.3 Nov 2023 Official leaderboard
61 DeepSeek-V3 Mar 2025 87.2 Mar 2025 Official leaderboard
62 marin-8b-instruct 87.0 Official leaderboard
63 Llama 3.1-8B 86.2 Jul 2024 Official leaderboard
64 Command R+ 86.0 Aug 2024 Official leaderboard
65 palmyra-med 85.7 Official leaderboard
66 Grok 3 85.5 Apr 2025 Official leaderboard
67 GPT-3.5 Turbo Jun 2023 85.3 Jun 2023 Official leaderboard
68 Mixtral 8x22B 85.1 Apr 2024 Official leaderboard
69 Mistral Small 3 84.7 Jan 2025 Official leaderboard
70 Llama 3.1-70B 84.5 Jul 2024 Official leaderboard
71 Grok 4 83.6 Jul 2025 Official leaderboard
72 GPT-3.5 Turbo Nov 2023 83.5 Nov 2023 Official leaderboard
73 Mixtral 8x7B 81.4 Dec 2023 Official leaderboard
74 GPT-3.5 Turbo Jan 2024 81.4 Jan 2024 Official leaderboard
75 Command R (03-2024) 80.9 Mar 2024 Official leaderboard
76 olmo-2-1124-13b-instruct 77.3 Official leaderboard
77 Mistral 7B v0.3 73.0 Sep 2023 Official leaderboard
78 olmo-2-1124-7b-instruct 72.9 Official leaderboard
79 olmoe-1b-7b-0125-instruct 70.1 Official leaderboard
80 DBRX 62.8 Mar 2024 Official leaderboard
81 Mistral-7B-Instruct-v0.1 52.5 Sep 2023 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
thinking 10k
98.1
Claude Opus 4
thinking 10k
96.9
Granite 4.0 H Small
with guardi…
91.4
DeepSeek-R1
reasoning h…
89.4
granite-4.0-micro
with guardi…
87.3
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-5 nano 98.6
o3 98.2
gpt-oss-120b 98.1
Claude Sonnet 4 thinking 10k 98.1
GPT-5 98.1
GPT-5 mini 98.0
Kimi K2 Jul 2025 98.0
Claude 3.5 Sonnet 97.7
o1 97.6
Claude Sonnet 4 97.4
o4-mini 97.3
Claude Sonnet 4.5 97.1
Claude Opus 4 thinking 10k 96.9
Claude Opus 4 96.8
Claude 3 Opus 96.8
GPT-5.1 96.8
GPT-4.5 96.5
gpt-oss-20b 96.4
GPT-4.1 96.3
o3-mini 96.2
GPT-4 Turbo Apr 2024 96.1
o1-mini 95.5
Claude Haiku 4.5 95.5
GPT-4.1 mini 94.9
Qwen3-235B-A22B-Instruct Jul 2025 94.8
Qwen3-Next-80B-A3B thinking 94.8
GPT-4o May 2024 94.6
Claude 3.7 Sonnet 94.5
Claude 3 Sonnet 94.3
GPT-4.1 nano 93.8
Qwen2-72B 93.3
Palmyra X4 93.3
Qwen2.5-72B 93.1
GPT-4o mini 93.0
Gemini 1.5 Flash May 2024 92.7
Palmyra X5 92.6
Gemini 3 Pro 92.5
Gemini 1.5 Pro May 2024 92.5
Gemini 2.5 Pro Jun 2025 91.4
Granite 4.0 H Small with guardian 91.4
Gemini 2.5 Flash Sep 2025 91.2
Granite 4.0 H Small 91.2
Gemini 2.5 Flash-Lite Sep 2025 91.0
Gemini 2.0 Flash Feb 2025 91.0
Gemini 2.0 Flash-Lite 90.8
Llama 4 Maverick 90.6
Gemini 2.0 Pro 90.5
Granite 3.3 8B Instruct 90.5
Grok-3 mini 90.2
GLM-4.5-Air 90.1
Qwen3-235B-A22B 90.0
Qwen2.5-7B 89.9
Llama 3.1-405B 89.7
OLMo 2 32B Instruct 89.7
Llama 3-70B 89.5
DeepSeek-R1 May 2025 89.4
Qwen1.5-72B 88.6
Llama 3-8B 88.6
Claude 3 Haiku 87.8
Llama 4 Scout 87.3
granite-4.0-micro with guardian 87.3
DeepSeek LLM 67B 87.3
DeepSeek-V3 Mar 2025 87.2
DeepSeek-R1 May 2025 reasoning hidden 86.8
DeepSeek-R1 May 2025 86.5
Llama 3.1-8B 86.2
Command R+ 86.0
Grok 3 85.5
granite-4.0-micro 85.5
GPT-3.5 Turbo Jun 2023 85.3
Mixtral 8x22B 85.1
Mistral Small 3 84.7
Llama 3.1-70B 84.5
Grok 4 83.6
GPT-3.5 Turbo Nov 2023 83.5
Mixtral 8x7B 81.4
GPT-3.5 Turbo Jan 2024 81.4
Command R (03-2024) 80.9
Mistral 7B v0.3 73.0
DBRX 62.8
Mistral-7B-Instruct-v0.1 52.5

3 models have held number one since its release in Nov 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic OpenAI
Own leaderboard Not tested since Nov 2025
GPT-5 nano 98.6

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
50.0 60.0 70.0 80.0 90.0 100.0 2023 2024 2025 2026 50.0 60.0 70.0 80.0 90.0 100.0 2023 2024 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Safety and alignment

All 444 in Safety and alignment
  • FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
  • StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Inkling 98.6%
  • AgentDojo In use Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. Grok 4 Fast 0%
  • MASK In use Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. Muse Spark Open 98.1%
  • Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. GPT-6 Astra 8.50%
  • Agentic misalignment suite (Anthropic) In use Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Claude Sonnet 4.5 0%