HarmBench
Olmo 3.1 32B Think 89.7% leads since Dec 2025 Only in system cards Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies.
Scores depend on the attack method and judge model, so labs differ.
In the authors’ words
Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.
- Level
- Everyday Most adults, without training
- Scored
- Graded by a model A language model scores each response, usually against a rubric.
- Answers
- Open-ended Free-form text, documents, designs or media judged as a whole.
Olmo 3.1 32B Think leads with 89.7% among models released by Dec 2025
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Dec 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 Olmo 3.1 32B Think 89.7%
- 2 Olmo 3 32B Think 73.5%
- 3
DeepSeek-R1 35% - 4
Apertus 70B 31.9%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Olmo 3.1 32B Think | 89.7% | Dec 2025 | Model card | |
| 2 | Olmo 3 32B Think | 73.5% | Dec 2025 | Model card | |
| 3 | DeepSeek-R1 | 35% | Jan 2025 | Model card | |
| 4 | Apertus 70B | 31.9% | Sep 2025 | Model card |
2 models have held number one since its release in Feb 2024
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Dec 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs report it only in system cards
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.
5 system cards report it, from 4 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 5 Jan 2026 |  Falcon H1R 7B technical report | HarmBench safe rate, CoT+Answer, three subsets with no overall: contextual 98.0, copyright 100.0, standard 99.0 (Table 10) | Safety | |
| 15 Dec 2025 | Olmo 3 32B Think technical report | 73.5 | HarmBench, Final Think 3.0 column (Table 54) | Safety |
| 15 Dec 2025 | Olmo 3.1 32B Think technical report | 89.7 | HarmBench, Final Think 3.1 column (Table 54) | Safety |
| 17 Sep 2025 | Apertus 70B technical report | 31.9 | HarmBench, lower is better (Table 26). Direct Request 10.3, Human Jailbreaks 36.2 | Safety |
| 20 Jan 2025 | DeepSeek-R1 technical report | 35 | HarmBench* safety score for the pure model without the risk control system. 89.3 with the risk control system, GPT-4o judge (Table 9). Table 9 average 85.9 pure, 95.0 with risk control | Safety |
More in Safety and alignment
- FORTRESS In use Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Claude Opus 5.5 5.78%
- StrongREJECT In use Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is.
Inkling 98.6% - AgentDojo In use Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. Grok 4 Fast 0%
- MASK In use Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy.
Muse Spark Open 98.1% - Gray Swan IPI Arena benchmark In use Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. GPT-6 Astra 8.50%
- Agentic misalignment suite (Anthropic) In use Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Claude Sonnet 4.5 0%