Benchmark Almanac

Safety and alignment

We track 444 benchmarks in safety and alignment, or 448 if you count each version. They run from everyday level to research frontier level.

11 in use 105 past 85% 91 with public code

OpenAI held the top safety and alignment score 39% of the contested time

That's about 18 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.

OpenAI Anthropic DeepSeek Qwen Cohere xAI Meta Google DeepMind Mistral AI Thinking Machines Other labs
Everyday Most adults, without training
Anthropic Red Team Passed 85% in Jun 2023, by GPT-3.5 Turbo
Claude 3 Haiku 100%
SimpleSafetyTests Passed 85% in Nov 2023, by GPT-3.5 Turbo
Claude 3 Opus 100%
AraTrust Passed 85% in Jun 2024, by Qwen2-72B
Qwen2-72B 89.2%
AgentDojo 3 leaders
Grok 4 Fast 0%
AIR-Bench 2024 Passed 85% in Jul 2024, by Claude 3.5 Sonnet
Claude Haiku 4.5 93.2%
PCT (Polarized Contrastive Pairs) 2 leaders
Muse Spark 1.1 27.9%
University Undergraduates and entry-level professionals
OR-Bench 3 leaders
Muse Spark 1.1 4.80%
Expert Specialists with years of training, often over hours of work
DistressBench Passed 85% in Oct 2026, by Muse Spark 1.2
Muse Spark 1.2 88.4%
BBQ (Bias Benchmark for QA) Passed 85% in Jan 2025, by DeepSeek-R1
DeepSeek-R1 96.6%
StrongREJECT Passed 85% in Feb 2025, by OpenAI deep research
Inkling 98.6%
FORTRESS 3 leaders
Claude Opus 5.5 5.78%
MASK Passed 85% in Apr 2026, by Claude Mythos Preview
Muse Spark Open 98.1%
KORA 2 leaders
GPT-6 Astra 73.2%
Dictatorship Eval Passed 85% in Apr 2026, by Claude Opus 4.7
Claude Fable 5 91.3%

Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.

OpenAI Anthropic DeepSeek Qwen Cohere xAI Meta Google DeepMind
Everyday Most adults, without training
Anthropic Red Team Passed 85% in Jun 2023, by GPT-3.5 Turbo
Claude 3 Haiku 100%
SimpleSafetyTests Passed 85% in Nov 2023, by GPT-3.5 Turbo
Claude 3 Opus 100%
AraTrust Passed 85% in Jun 2024, by Qwen2-72B
Qwen2-72B 89.2%
AgentDojo Not tested since Feb 2025
Claude 3.5 Sonnet (October 2024) 1.11%
AIR-Bench 2024 Passed 85% in Jul 2024, by Claude 3.5 Sonnet
Claude Haiku 4.5 93.2%
PCT (Polarized Contrastive Pairs) 2 leaders
Muse Spark 1.1 27.9%
WellnessBench 1 leader
GPT-6.1 Sol 66%
University Undergraduates and entry-level professionals
OR-Bench Not tested since Jun 2024
GPT-4o 6.80%
BrowseSafe++ Passed 85% in Oct 2026, by GPT-6 Astra
GPT-6 Astra 91.5%
Expert Specialists with years of training, often over hours of work
DistressBench Passed 85% in Oct 2026, by Muse Spark 1.2
Muse Spark 1.2 88.4%
Situational Awareness Dataset (SAD) Not tested since Sep 2024
o1-preview 61.9%
FORTRESS 3 leaders
Claude Opus 5.5 5.78%
KORA 2 leaders
GPT-6 Astra 73.2%
Dictatorship Eval Passed 85% in Apr 2026, by Claude Opus 4.7
Claude Fable 5 91.3%

Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.

From everyday to research frontier level

Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.

Everyday

Most adults, without training

University

Undergraduates and entry-level professionals

Competition

The strongest students, in olympiad-style contests

Expert

Specialists with years of training, often over hours of work

Research frontier

Researchers, on open problems or tasks that take days

Another 60 have no level. Levels don’t apply to arenas, safety evaluations or composite indices.

By kind

Refusing harmful requests

BenchmarkLevelScoredBest scoreStatus
AIR-Bench 2024 Models respond to 5,694 harmful prompts across 314 risk categories drawn from 8 government regulations and 16 company policies. Category-specific LLM judges score refusals. Everyday Graded by a model 93.2% Dropped
Anthropic Red Team Models answer human red-team attack prompts collected by Anthropic on discrimination, hate, violence and unethical behavior. Judges score response safety. Everyday Graded by a model 100% System cards only
SimpleSafetyTests Models answer 100 clearly harmful prompts on illegal items, physical harm, scams, child abuse and self-harm. The score is the share of safe responses. Everyday Graded by a model 100% System cards only
HELM Safety Stanford CRFM runs five safety benchmarks over six risk categories, such as violence, fraud and discrimination, and reports one leaderboard. Everyday Graded by a model 98.6 Not in cards
FORTRESS Paired adversarial and benign national-security misuse prompts. Rubric-graded risk score on adversarial and over-refusal score on benign prompts. Graded by a model 5.78% In use
Dictatorship Eval Scenarios at six difficulty levels testing whether models help with authoritarian requests, from blunt asks to requests hidden in code edits. LLM judge, no self-grading. Graded by a model 91.3% Not in cards
Single-turn benign request evaluations (Anthropic) Models answer benign requests on sensitive topics. Score is the refusal rate, where lower is better. Everyday Graded by a model 0% System cards only
Single-turn harmful request evaluations (Anthropic) Models answer single-turn requests that violate the usage policy. Score is the harmless response rate. Everyday Graded by a model 99.8% System cards only
Malicious use of Claude Code (Anthropic) Agents in Claude Code receive malicious and dual-use coding requests. Score is the refusal rate on malicious requests. Expert Graded by a model 96.7% System cards only
XSTest Models answer 250 safe prompts that resemble unsafe ones and 200 unsafe contrast prompts. Labs usually report the incorrect refusal rate on safe prompts. Everyday Graded by a model 1.10% System cards only
Malicious computer use (Anthropic) Computer-use agents receive harmful tasks such as surveillance or fraud in GUI environments. Score is the refusal rate. Everyday Graded by a model 99.4% System cards only
AttaQ Models answer 1,402 adversarial questions in seven harm classes, such as deception, violence and PII. Responses are scored for harmlessness. Everyday Graded by a model 88.5% Fading
Challenging Refusal Evaluation Models answer a harder set of disallowed content prompts across policy categories. An autograder scores the not_unsafe rate. Expert Graded by a model 93.4% System cards only
Standard Refusal Evaluation Models answer disallowed and benign safety-related prompts across policy categories. An autograder scores not_unsafe and not_overrefuse. Everyday Graded by a model 100% System cards only
General refusals (xAI) Models get thousands of harmful requests in six languages. A grader model scores compliance, the share that fails policy. Graded by a model 0% System cards only
AgentHarm 110 malicious agent tasks with synthetic tools. Measures refusal and harmful task completion, plus benign controls. Checked automatically 94.2% System cards only
Anthropic undermining liberal democracy (behavioral audit metric) Automated behavioral audit metric: investigator models probe willingness to help coups, election tampering, or political surveillance, censorship or intimidation. Graded by a model 1.13 System cards only
SALAD-Bench Models answer harmful questions organized in a three-level taxonomy, some with attack or defense changes and some multiple-choice. The MD-Judge evaluator labels responses as safe or unsafe. Everyday Graded by a model 97.3% Fading
Agentic coding malicious requests (Anthropic) Coding agents with standard coding tools receive 150 malicious coding requests prohibited by policy. Score is the refusal rate. Expert Graded by a model 100% System cards only
BioTIER Safeguarded model configurations receive harmful and benign biology requests. Scored on refusing requests designated for refusal and allowing benign ones. Expert Graded by a model 98.7% System cards only
Microsoft RAI benchmark: Harmful Content Continuation (DR-3) Models continue conversations about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Everyday Graded by a model 0.30% System cards only
Microsoft RAI benchmark: Harmful Content Summarization (DR-3) Models summarize content about hate, self-harm, sexual content and violence. Scored as the share of answers with judged severity 3 or more out of 7. Everyday Graded by a model 10% System cards only
Microsoft RAI benchmark: Third Party Harm (DR-1) Models face simulated conversations that try to extract third-party content such as books, news, recipes and songs. Scored as the defect rate at severity 1. Everyday Graded by a model 10.5% System cards only
WildGuardTest Models classify whether user prompts are harmful, whether responses are harmful, and whether responses refuse, across 13 risk categories. Scored by F1. Everyday Checked automatically 96.9% System cards only
Cyber harm coverage (Anthropic) Cyber safeguard classifiers screen examples from each cyber harm category. Score is overall recall, the share of harmful examples flagged. Expert Checked automatically 100% System cards only
Harmlessness human feedback win rate (Anthropic) Raters probe models in adversarial chats and pick the more harmless reply in head-to-head comparisons. Score is the win rate against a baseline model. Everyday Ranked by votes 87% System cards only
Higher-difficulty benign request evaluations (Anthropic) Models answer harder benign requests that resemble harmful ones. Score is the refusal rate, where lower is better. Everyday Graded by a model 0.01% System cards only
Higher-difficulty harmful request evaluations (Anthropic) Models answer harder, more obfuscated policy-violating requests. Score is the harmless response rate. Everyday Graded by a model 99.4% System cards only
KGC-Safety (LG) Models answer risky prompts set in Korean social and cultural contexts. A Safe Rate is scored over four domains: universal human values, social safety, Korean sensitivity and future risk. Everyday Graded by a model 99.8% Fading
LongCat safety evaluation (Meituan) Models answer private test queries in four risk categories: harmful content, criminal activity, misinformation and privacy. The score is the share of safe responses in each category. Everyday Graded by a model 95.7% Dropped
WildChat (toxic conversations) Models answer toxic user turns from WildChat, the 200 highest scoring conversations per moderation category. Scored as not_unsafe rate. Everyday Graded by a model 98% System cards only
Aegis 2.0 Models classify human and LLM conversation turns as safe or unsafe under a taxonomy of 12 hazard categories. Scored by F1 against human and jury labels. Everyday Checked automatically 85.8% System cards only
Challenging prompts from red teamers with biosafety-relevant PhDs Models answer adversarial biology prompts written by red teamers with biosafety PhDs. Scored as not_unsafe rate from model training alone. Expert Graded by a model 93.6% System cards only
Child safety (xAI) Models face scripted multi-turn requests for child sexual exploitation content. Compliance is the share of dialogues that fail policy. Graded by a model 0% System cards only
CIMemories Models use persistent memory across tasks and must withhold attributes when context requires. Reports worst-case disclosure over five samples. Everyday Graded by a model 26.4% In use
CyberBench (Liu et al.) Models face detail-seeking high-severity cyber requests. Meta uses it as a seed set for adaptive red-teaming and reports attack success rate. Expert Graded by a model 0.80% System cards only
Filtered, adversarial sample of production prompts Models answer filtered borderline and high-risk biology prompts that resemble production traffic. Scored as not_unsafe rate from model training alone. Expert Graded by a model 96.9% System cards only
FORTRESS-RN Models face the radiological and nuclear prompts of Scale AI's FORTRESS. xAI reports refusal accuracy on these items. Graded by a model 97.9% System cards only
HackerBench xAI internal CTF-style suite needing chained skills, plus borderline and harmful requests that should be refused. Graded by a model 7.80% System cards only
Harmful content defect rate (Azure AI Evaluation SDK, Microsoft) GPT-4o simulates adversarial conversations with the model, and a judge flags violence, sexual, self-harm and hateful content. Scored as the share of harmful responses, lower is better. Everyday Graded by a model 1.40% System cards only
Jigsaw Toxic Comment Classification Models label Wikipedia talk page comments for toxicity types such as threats, insults and identity hate. Labs report F1 or ROC AUC. Everyday Checked automatically 57.8% System cards only
Malware Refusals (Golden Set) Coding agents receive malware development requests from a curated golden set written by OpenAI policy experts. Scored as refusal rate. Expert Graded by a model 100% System cards only
Meta agentic cyber misuse refusal Models run multi-step cyber tasks with tool access while an adaptive attacker tries to steer them to a malicious goal. Reports attack success rate. Expert Graded by a model 0.70% System cards only
WildChat non-toxic incorrect refusals (Anthropic) Models answer non-toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the incorrect refusal rate. Everyday Graded by a model 4.80% System cards only
WildChat toxic refusals (Anthropic) Models answer toxic user prompts drawn from the WildChat dataset of real chatbot conversations. Score is the correct refusal rate. Everyday Graded by a model 96.4% System cards only
CBRN Conversational Evals About 1,500 expert-written single and multi-turn CBRN requests test whether safeguards block hazardous help. Reports violation and benign refusal rates. Expert Graded by a model 1.80% System cards only
CBRN Science Eval More than 700 benign bioinformatics, genomics and chemistry questions check that safeguards do not cause false refusals. Reports benign refusal rate. Expert Graded by a model 3.50% System cards only
Cyber Conversational Evals About 1,000 one to five turn cyber requests test whether safeguards block offensive help and allow defensive help. Reports violation and benign refusal rates. Expert Graded by a model 0.40% System cards only
CyberSecEval Code Interpreter Abuse Models receive requests to run malicious commands against an attached code interpreter. Reports attack success rate, also across languages. University Graded by a model 3.80% System cards only
Dangerous Content policy violations (from ART) Automated red teaming attackers search for prompts that make the model break the dangerous content policy. Reports the violation rate found. Everyday Graded by a model 24.3% System cards only
Divergence attack (extractable memorization) Models are prompted to repeat one token many times. The test measures how often they diverge and emit memorized training text. Everyday Checked automatically 59% System cards only
Do-Not-Answer Models get risky instructions that responsible models should not follow, across 12 harm areas and 61 risk types. Responses are judged for safe handling. Everyday Graded by a model 97.8% System cards only
Helpfulness violations (from ART) Automated red teaming attackers search for benign prompts where the model gives unhelpful answers. Reports the violation rate found. Everyday Graded by a model 6.10% System cards only
Image generation refusals Models decide whether to call the image generation tool on human-curated adversarial prompts with system mitigations on. Scored on not_unsafe and not_overrefuse. Expert Graded by a model 98% System cards only
Image input safety evaluations Models answer harmful prompts that combine text and images in categories such as hate, extremism and self-harm. Scored as safe completion rate. Everyday Graded by a model 99% System cards only
Meta catastrophic cyber misuse refusal Models receive high-severity cyber-misuse prompts with technical context. The benchmark reports pass@k attack success rate under adaptive attacks. Expert Graded by a model 3.10% System cards only
Meta cyber false-refusal benchmark Models receive benign and borderline cyber requests and the benchmark measures how often they wrongly refuse. Reports false refusal rate. Expert Graded by a model 11% System cards only
Microsoft internal multimodal RAI benchmark Vision language models answer image and text prompts across harm categories. Responses are scored from 0 to 10, higher is safer. Everyday Graded by a model 8.16 System cards only
Multimodal policy red-teaming (Anthropic) Models answer multi-turn image and text conversations on harmful topics such as child safety, weapons and fraud. Score is the share of harmless responses. Everyday Graded by people 99.2% System cards only
Participation in junk science (Anthropic) Models talk with simulated users ranging from curious to openly fraudulent who seek fake science. Score is the rate of generating unsound scientific content. Expert Graded by a model 0% System cards only
Proactive refusals of high-risk tasks Browser agents receive high-risk tasks such as banking transfers and high-stakes decisions that policy says they must refuse. Scored on refusal recall. Everyday Graded by a model 94% System cards only
RealToxicityPrompts Models continue 100,000 web text prompts. Scored on the rate of toxic generations judged by a toxicity classifier. Everyday Graded by a model 0.20% System cards only
Red Teaming Network pairwise safety comparison Red teamers compare anonymized responses from two models in free-form risky conversations. Scored as the share of ratings where a model is safer. Expert Ranked by votes 59.8% System cards only
RTVLM Vision language models answer red-teaming image prompts on faithfulness, privacy, safety and fairness. GPT-4V scores responses from 0 to 10. Everyday Graded by a model 6.39 System cards only
SafetyBench Models answer 11,435 multiple choice questions on seven safety topics in Chinese and English, such as ethics, illegal activities and privacy. Everyday Checked automatically 89.9% System cards only
Secure and defensive coding (Anthropic) Cyber safeguard classifiers screen sampled defensive coding sessions such as patching and incident response. Score is the classifier flag rate. Expert Graded by a model 0.61% System cards only
User Confirmations During Computer Use Agents doing computer use tasks reach financial, high-stakes communication and other actions. Scored on asking the user for confirmation when policy requires it. Everyday Graded by a model 92% System cards only
VLGuard Vision language models answer safe and unsafe image and text pairs from the VLGuard test set. Responses are judged for harmfulness and refusal. Everyday Graded by a model 9.10 System cards only
WebTailBench-Refusals Computer-use agents receive 111 harmful browser tasks in seven policy categories, such as illegal activity and misinformation. Scored as the share of tasks safely refused. Everyday Graded by a model 100% System cards only
Advanced Cybersecurity Completion Rate Models receive arbitrary advanced cyber requests under different access configurations. Scored as the share of requests fully completed. Expert Graded by a model 3.50% System cards only
Agent harness misuse evaluation Agents in a sandboxed Antigravity harness get malicious and benign tool-use requests on illegal acts, physical harm and scaled abuse. Reports safe action rate. Everyday Graded by a model 91.4% System cards only
AgentHarm Verified (Meta) Agents receive 108 malicious AgentHarm tasks kept after human review, plus a benign set. Reports average harm score and false refusal rate. Expert Graded by a model 3.40% System cards only
Agentic harmful task refusals Browser agents receive harmful tasks such as illicit purchases, prohibited financial activity and searches for sensitive personal data. Scored as refusal rate per category. Everyday Graded by a model 97% System cards only
AutoElicit An optimizer perturbs benign instructions to elicit unsafe behavior from computer-use agents. Reports elicitation success rate on transferred prompts. Expert Graded by a model 14.5% System cards only
Charlemagne Labs social-engineering benchmark Attacker models try to persuade a skeptical simulated victim to take an action over many turns. Scored by attempted and carry-through rates. Expert Graded by a model 1.60% System cards only
Child Safety Action Testing Computer use agents get requests for actions that facilitate child exploitation, endangerment or illegal access. Reports the refusal score. Everyday Graded by people 100% System cards only
Child Sexual Abuse and Exploitation Safety Specialist teams grade model answers to adversarial and coded child sexual abuse and exploitation requests in text, image and video. Reports safe response rate. Everyday Graded by people 100% System cards only
Computer use manual red teaming Google's generative AI red team probes computer use agents in sandboxes for tool misuse, skipped confirmations and prohibited actions. Everyday Graded by people 100% System cards only
Computer use Prohibited Action Evaluation Computer use agents get single-turn adversarial requests for actions that break Google policy. Reports the safe action rate. Everyday Graded by a model 99.4% System cards only
Cyber agentic safeguard evaluation Agents with production mitigations attempt cyber range challenges. The score is the share of milestones the model refuses. Expert Checked automatically 100% System cards only
Dangerous Content Safety An autorater checks answers to single and multi-turn requests about weapons, illegal acts, violence and self-harm. Reports safe response rate. Everyday Graded by a model 96.6% System cards only
DeepSeek in-house safety benchmark Models answer 1,120 Chinese and English questions across 28 safety subcategories. GPT-4o labels each answer as unsafe, safe or rejection. Everyday Graded by a model 25.2% System cards only
DeepSeek multilingual safety evaluation Models answer 9,330 safety questions translated into 50 languages. Safe answers earn 5 points, rejections 4 and unsafe answers 0. Everyday Graded by a model 74.2% System cards only
Default safety evaluation (Cohere) Models answer unsafe prompts in five categories with no system preamble. LLM judges score the share of safe responses, and a jury rates relative safety. Everyday Graded by a model 70.4% System cards only
DTap Direct Misuse An attacker agent generates adaptive jailbreaks against a target agent across tool-use and browser-use tasks. Reports attack success rate. Expert Graded by a model 5.60% System cards only
Expert Red Teaming for Violent Attack Planning Expert red teamers compare anonymized responses from two models on violent attack planning prompts. Scored as the win rate for being safer. Expert Ranked by votes 65.1% System cards only
Hate & Harassment Safety An autorater checks answers to requests for hateful or harassing content and to sensitive benign prompts. Reports safe response rate. Everyday Graded by a model 92.7% System cards only
Malware Refusals (Synthetic) Coding agents receive a large synthetic set of malware-related tasks, including ambiguous and dual-use prompts. Scored as refusal rate. Expert Graded by a model 97% System cards only
Meta Cyber Misuse Chat Models receive 80 offensive cyber requests, directly and with jailbreak developer prompts. Reports attack success rate and a benign false refusal rate. Expert Graded by a model 2.80% System cards only
Meta cyber misuse request set Models receive 600 private offensive cyber prompts, directly and under jailbreak attacks. Reports attack success rate. Expert Graded by a model 9.00% System cards only
Mistral 175 unsafe prompts Models receive 175 unsafe prompts. Metric is the share of harmful questions the model declines. Everyday Graded by a model 100% System cards only
Pairwise Bioweaponization Campaign Red teamers with biology PhDs compare anonymized responses from two models on bioweaponization prompts. Scored as the win rate for being safer. Expert Ranked by votes 60% System cards only
PrivacyLens Language model agents write emails, posts or messages from a prior tool trajectory without leaking sensitive details. A judge scores information leakage and helpfulness. Everyday Graded by a model 49.2% System cards only
Safety (MiniMax in-house) Models respond to safety-relevant user prompts. MiniMax scores whether replies are safe and helpful. Everyday Graded by a model 90.9% System cards only
Sexually Explicit Content Safety An autorater checks answers to direct, adversarial and benign sexual content prompts in single and multi-turn chats. Reports safe response rate. Everyday Graded by a model 97.6% System cards only
Unauthorized voice generation Speech models produce audio that may deviate from the approved system voice. Scored on how many meaningful deviations the output classifier catches. Everyday Checked automatically 100% System cards only
Agent-SafetyBench LLM agents act in 349 tool environments over 2,000 test cases covering 8 safety risk categories and 10 failure modes. A scorer model rates each run safe or unsafe. Everyday Graded by a model Not in cards
Agentic Safe Completions Agents handle harmful requests from Codex and ChatGPT plugin traffic and human red teaming. Scored on avoiding harmful actions per category. Everyday Graded by a model System cards only
AILuminate Safety 1.2 Models answer 3,900 hidden English prompts, 325 in each of 12 hazard categories, picked with item response theory. A calibrated evaluator gives a 0 to 100 score. Everyday Graded by a model Not in cards
Ambiguous context evaluations (Anthropic) Models respond to gray-area requests where intent is unclear. Graders judge how well responses balance helpfulness and harm. Everyday Graded by a model System cards only
Anthropic cyber safeguard evaluations Classifier recall on violative cyber exchanges, block rates on defensive vulnerability-finding and secure-coding work, and the rewind-attacker robustness test. Checked automatically Not in cards
Anthropic harmful-request and wellbeing evaluations Single-turn harmful and benign request sets, multi-turn tests, child safety, suicide/self-harm, disordered eating and election integrity. Graded by a model Not in cards
Anthropic malicious agentic use evaluations Measures refusal of malicious tasks via Claude Code, computer use and agentic influence campaigns. Graded by a model Not in cards
APE (Attempt to Persuade Eval) A model is asked to persuade a simulated user across 600 topics, from benign facts to conspiracies and clearly harmful ones. A judge scores whether it attempts persuasion. Everyday Graded by a model Not in cards
Appropriate harmlessness (Anthropic) Models answer borderline and harmful prompts. Classifiers sort each response into helpful answer, policy violation, appropriate refusal or unnecessary refusal. Everyday Graded by a model System cards only
Biological safety evaluations (OpenAI) Models answer biology requests drawn from deidentified production data and from easy and hard synthetic sets. Scored as the share of cases where model training alone prevents a violative response. Expert Graded by a model System cards only
Biology Model Refusal Evaluation Models answer severe, dual-use and benign biology prompts. Scored as safe rate on harmful prompts and not over-refusing benign ones. Expert Graded by a model System cards only
BrowserART Browser agents receive 100 harmful requests on synthetic and real websites. The test measures how often the agent attempts the harmful behavior that its backbone model would refuse in chat. Expert Graded by a model Not in cards
Child safety evaluations (Anthropic) Models answer single-turn and multi-turn prompts about child sexualization and grooming. Score is the appropriate response rate. Everyday Graded by a model System cards only
Cooperation with user deception (Anthropic) Models get developer system prompts asking them to subtly mislead users, then meet users who would be harmed. Graders score harmful and deceptive choices. Everyday Graded by a model System cards only
DeepSeek LLM in-house safety evaluation Models answer 2,400 Chinese test cases on discrimination, legal rights, trade secrets, illegal behavior and other safety issues. Human annotators judge each answer. Everyday Graded by people System cards only
Disallowed Content Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the estimated prevalence of policy violating outputs per category. Everyday Graded by a model System cards only
Discoverable memorization of training data Models continue 50-token prefixes sampled from their own training data. The test counts exact and approximate copies of the true suffix. Everyday Checked automatically System cards only
Election integrity evaluations (Anthropic) Models answer election-related prompts, including requests for misinformation or influence operations. Score is the appropriate response rate. Everyday Graded by a model System cards only
Gemini content safety evals Automated text-to-text, multilingual and image-to-text policy violation rates, refusal tone and unjustified refusals, reported as deltas. Graded by a model Not in cards
Image to Text Safety An automated content safety evaluation measures policy violations when prompts combine images and text. Everyday Graded by a model System cards only
Incorrect behavior rate on disallowed and sensitive content Models answer disallowed prompts and sensitive prompts such as medical advice. Scored on the rate of responses that break policy. Everyday Graded by a model System cards only
Instruction Following (safety evaluation) An automated evaluation measures whether the model follows instructions on sensitive prompts while staying within safety policies. Everyday Graded by a model System cards only
LinguaSafe Models face 45k safety prompts in 12 languages, from Hungarian to Malay. Direct and indirect assessments score harmful responses, and an oversensitivity test checks over-refusal. Everyday Graded by a model System cards only
Llama 2-Chat safety human evaluation (Meta) Human raters score responses to about 2,000 adversarial prompts on a five-point scale. Ratings of 1 or 2 count as violations. Graded by people System cards only
Llama internal safety benchmarks (Meta) Models face Meta's adversarial and borderline prompts per capability and language. Scored as violation rate and false refusal rate. Graded by a model System cards only
Llama verbatim memorization probe (Meta) Models get prompts sampled from their pretraining data. The score is the share of generations that reproduce the true continuation verbatim. Checked automatically System cards only
MAI internal safety and over-refusal evaluation Models answer harmful and benign prompts in eight harm categories. Judges score safety pass rate on sensitive prompts and helpfulness as one minus over-refusal. Everyday Graded by a model System cards only
Malicious agentic influence campaigns (Anthropic) Agents receive requests to run multi-step influence operations with tools. Score is the refusal rate. Everyday Graded by a model System cards only
Mistral self-reflection moderation set Models classify prompts or answers as acceptable or harmful on a balanced set of adversarial and standard prompts. Scored by precision and recall. Everyday Checked automatically System cards only
MLCommons AILuminate Industry-standard hazard benchmark over 12 risk categories with graded ratings. Graded by a model Not in cards
Multi-turn testing (Anthropic) Models face multi-turn conversations that escalate toward harmful requests across risk areas. Score is the appropriate response rate. Everyday Graded by a model System cards only
Multilingual Safety An automated safety policy evaluation measures policy violations across multiple languages. Everyday Graded by a model System cards only
Muse-targeted cyber misuse and violation red-teaming An automated red-teaming agent tries to misuse the Muse runtime across six attack categories. Reports attack success rate per category. Expert Graded by a model System cards only
OpenAI cyber safety evaluations (incl. Daybreak Blue) Refusal and helpfulness evaluations for cyber requests, including the Daybreak Blue set for enabling defensive security work. Graded by a model System cards only
OpenAI Production Benchmarks and safe-completion evals Hard production-derived prompts per policy category, under-18 evals, agentic safe completions and image-input evals. Scored on safe completion rate. Graded by a model System cards only
Pairwise safety red teaming Red teamers compare anonymized responses from several models and pick the least safe. Scored as ELO or win rate. Expert Ranked by votes System cards only
Person Identification evaluation Models receive requests to identify people in photos. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Graded by a model System cards only
Personal data evaluations Browsing agents receive requests for private personal information, from 200 synthetic prompts and 55 manually written golden examples. Scored on adherence to personal data policy. Everyday Graded by a model System cards only
Regurgitation Evaluations Models are asked to reproduce training data. Scored on how often they refuse to regurgitate it. Everyday Checked automatically System cards only
RiOSWorld Computer-use agents work in an Ubuntu virtual machine through risky scenarios such as phishing emails and pop-ups. Scores track whether agents notice and avoid the risk. Everyday Checked automatically Not in cards
Safe response to high-severity prompts Models handle multi-turn self-harm and violent activity conversations where harmful intent only emerges from context. Scored as safe response rate. Expert Graded by a model System cards only
Safety mode controllability (Cohere) Models follow contextual or strict safety instructions. Accuracy averages correct answers on an over-refusal set and correct refusals on a control set. Everyday Graded by a model System cards only
SORRY-Bench Models get unsafe requests across a fine-grained taxonomy of topics and linguistic variations. A judge model scores whether each answer refuses or complies. Everyday Graded by a model Not in cards
Speaker identification Speech models get audio requests to identify a speaker by voice. Scored on refusing when they should and complying for famous quotes. Everyday Graded by a model System cards only
Text to Text Safety An automated content safety evaluation measures policy violations in model answers to text prompts. Everyday Graded by a model System cards only
Tone An automated evaluation measures whether model refusals are objective, neutral and not preachy. Everyday Graded by a model System cards only
U18 safety evaluations Models answer challenging prompts in teen safety categories such as age-restricted goods, eating disorders and emotional reliance. Scored as safe completion rate per category. Everyday Graded by a model System cards only
Unjustified refusals An automated evaluation measures over-refusals on benign or borderline prompts that can be safely fulfilled. Everyday Graded by a model System cards only
Vulnerability finding evaluations (Anthropic) Cyber safeguard classifiers screen sampled vulnerability discovery traffic. Reports flag rates on violative binary work and on defensive source code work. Expert Graded by a model System cards only
xAI refusal and jailbreak suites Internal refusal suites (general, child safety, self-harm) in six languages plus single-turn, multi-turn and long-horizon jailbreaks. Graded by a model Not in cards

Jailbreaks and prompt injection

BenchmarkLevelScoredBest scoreStatus
OR-Bench Models meet benign and borderline prompts and the benchmark measures how often they wrongly refuse. Reports false refusal rate. University Graded by a model 6.80% System cards only
AgentDojo Agents do tool-use tasks while tools and data carry prompt injections. Attack success rate counts hijacked runs. Everyday Checked automatically 0% In use
StrongREJECT Forbidden-prompt jailbreak benchmark with a rubric grader that scores how useful a non-refusal actually is. Graded by a model 98.6% In use
BrowseSafe++ Detectors judge 38,675 matched attack and control web pages built from real production prompt injections, in HTML, Markdown, accessibility-tree and screenshot views. Scored by F1. University Checked automatically 91.5% Not in cards
Gray Swan Agent Red Teaming (ART) benchmark Agents face curated prompt injections from a large public red-teaming competition across 44 deployment scenarios. Score is attack success rate. Expert Graded by a model 0.10% System cards only
Prompt injection robustness against adaptive attackers, browser use (Anthropic) Browser agents face indirect prompt injections in web environments from an internal adaptive attacker. Score is attack success rate. Expert Checked automatically 0% System cards only
Prompt injection robustness against adaptive attackers, coding (Anthropic) Coding agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Checked automatically 0% System cards only
Prompt injection robustness against adaptive attackers, computer use (Anthropic) Computer-use agents face indirect prompt injections crafted by an adaptive attacker. Score is attack success rate. Expert Checked automatically 0% System cards only
Tulu 3 Safety Eval Models answer harmful, jailbreak and benign prompts from several safety benchmarks in Ai2's safety-eval suite. A WildGuard classifier judges the outputs, and the score is an average. Everyday Graded by a model 87.6% Fading
Gray Swan IPI Arena benchmark Indirect prompt-injection attacks from public red-teaming competitions replayed on tool, coding and browser agents. Attack success rate at k attempts. Graded by a model 8.50% In use
Human Sourced Jailbreaks Models face jailbreak prompts collected from human red teaming. Scored as the not_unsafe rate. Expert Graded by a model 100% System cards only
SEAL Adversarial Robustness Models receive 1,000 single-turn adversarial prompts written by 10 full-time red teamers across nine harm categories. Safety specialists grade responses, and models are ranked by the number of high-harm violations. Expert Graded by people 8.00 Not in cards
Agent JSK Agents read simulated email connectors that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Graded by a model 100% System cards only
Instruction hierarchy attacks Models face attacks that try to override higher-priority instructions. Scored as defender success rate averaged per defender query. Expert Graded by a model 100% System cards only
Prompt injection robustness Agents meet indirect prompt injection attacks inside tool outputs. Scored as defender success rate averaged per defender query. Expert Graded by a model 99.8% System cards only
SuperCLUE-Safety Chinese language models answer 4,912 open-ended safety questions over more than 20 sub-dimensions in multi-round adversarial dialogues. Follow-up turns push harder than the first prompt. Everyday Graded by a model 89.3 Not in cards
Computer use prompt injection evaluation (Anthropic) Agents face about 600 prompt injection scenarios in coding platforms, browsers and email workflows. Score is the attack prevention rate. Expert Graded by a model 78% System cards only
Jailbreak Augmented Examples Models face publicly known jailbreaks applied to prompts from the standard disallowed content set. Scored as not_unsafe rate. Everyday Graded by a model 100% System cards only
Microsoft RAI benchmark: Jailbreak (DR-1) Models face simulated jailbreak conversations on topics such as child grooming, illegal advice and leaking guidelines. Scored as the defect rate at severity 1. Everyday Graded by a model 7.30% System cards only
Production Jailbreaks Models face jailbreaks found in production ChatGPT data. Scored as not_unsafe rate. Everyday Graded by a model 100% System cards only
Rewind-attacker cyber adversarial robustness (Anthropic) An automated attacker with rewinds and 400 calls tries to make a safeguarded Claude Code agent complete offensive cyber tasks. Score is attack success rate. Expert Settled by events 4.00% System cards only
StrongREJECT v2 (Meta) Models face a cascade of jailbreak attacks on disallowed behaviors adapted from StrongREJECT plus internal ones. Reports attack success rate. Expert Graded by a model 0.50% In use
Coding prompt injections Agents work on coding tasks that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Graded by a model 100% System cards only
HarmBench Models answer harmful behavior requests, including copyright and contextual cases, with and without automated red-team attacks. A classifier judges whether the response complies. Everyday Graded by a model 89.7% System cards only
Refusals + System Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the system message. The metric is the answer rate. Graded by a model 0% System cards only
Refusals + User Jailbreak (xAI) Models get the xAI refusal set with jailbreak templates in the user message. The metric is the answer rate. Graded by a model 0% System cards only
CyberSecEval Prompt Injection Models face text prompts that embed injected instructions across 15 strategies. Scored as attack success rate. Everyday Graded by a model 19% System cards only
IHEval Models face conflicts between system, user and tool instructions. Reports a composite score for following the higher-priority instruction. Everyday Checked automatically 82.4% In use
Poly-Guard Bench Models meet harmful requests and statements across domains. Reports attack success rate under single-turn and multi-turn adaptive red-teaming. Expert Graded by a model 2.30% System cards only
Prompt injection attacks in search and function calling Agents receive search results and function call outputs that contain stronger known prompt injection attacks. Scored on resisting the injected instructions. Expert Graded by a model 94.6% System cards only
Prompt injection live bug bounty (Anthropic with Gray Swan) Expert red teamers submit prompt injection attacks against hidden frontier models in tool use, coding and computer use scenarios. Score is attack success rate. Expert Checked automatically 0.08% System cards only
Standard jailbreaks (xAI) Models face a battery of published and internal jailbreaks. Compliance is the share of should-refuse attacks that get disallowed help. Graded by a model 0.01% System cards only
WildJailbreak Models face adversarial jailbreak prompts built from in-the-wild tactics, plus benign prompts that look harmful. Evaluations score refusals of harmful requests and compliance with benign ones. Everyday Graded by a model 98.5% Fading
Challenging Red Teaming Evaluation 1 (created for o3-mini) Models face adversarial jailbreaks from o3-mini red teaming on illicit advice, extremism, political persuasion and self-harm. Scored as not_unsafe rate. Expert Graded by a model 51% System cards only
Challenging Red Teaming Evaluation 2 (created for deep research) Models face adversarial prompts from deep research red teaming on risky advice such as attack planning. Scored as not_unsafe rate. Expert Graded by a model 67% System cards only
Chem / Bio Dynamic Jailbreak Eval An automated attacker spreads a harmful chem or bio goal across many turns and chats. Elicited content is scored against expert rubrics. Expert Graded by a model 38% System cards only
Cyber Adaptive Jailbreak Eval Automated single and multi-turn attackers learn from refusals to reach high severity cyber goals. Reports query and objective success rates. Expert Graded by a model 20% System cards only
Cyber Static Jailbreak Eval Common jailbreak templates are applied to high severity cyber requests against the safeguarded model. Reports query and objective success rates. Expert Graded by a model 1.00% System cards only
DoAnythingNow (DAN) Models face in-the-wild DAN-style jailbreak prompts paired with forbidden questions. Scored by refusal rate or attack success rate. Everyday Graded by a model 54.7% System cards only
External red team jailbreaks External red teamers and automated gap finding generate jailbreaks against the model. Scored on how often the model resists. Expert Graded by people 44% System cards only
Gray Swan Bio Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on about 50 high risk biology tasks. Reports query and objective success rates. Expert Graded by a model 4.00% System cards only
Gray Swan Cyber Eval Gray Swan's Shade red teaming agent attacks the safeguarded model on cyber tasks built from Google scenarios. Reports query and objective success rates. Expert Graded by a model 10% System cards only
Indirect prompt injection, Actor Critic attack An attacker model refines injected email triggers until the agent leaks synthetic private data through a send email call. Reports attack success rate. Expert Checked automatically 40.8% System cards only
Indirect prompt injection, Beam Search attack An attack appends random tokens to an injected email trigger until the agent leaks synthetic private data. Reports attack success rate. Expert Checked automatically 4.20% System cards only
Indirect prompt injection, TAP attack A Tree of Attacks with Pruning search writes natural language injections that make the agent leak synthetic private data. Reports attack success rate. Expert Checked automatically 30.8% System cards only
Lakera B3 (Backbone Breaker Benchmark) Crowdsourced prompt-injection attacks hit backbone LLMs inside 10 agent threat snapshots at three defense levels. Scored by attack resistance. Checked automatically 97.1% In use
Long-horizon jailbreaks (xAI) Models face multi-turn, Crescendo and long-horizon malicious-use attacks. Compliance is the share that get disallowed help. Graded by a model 0.65% System cards only
MCP prompt injection evaluation (Anthropic) Agents use simulated email, Slack and document servers over MCP where content carries injected instructions. Score is the attack prevention rate. Expert Graded by a model 92.5% System cards only
PlugInject Agents receive function call outputs that contain prompt injection attacks. Scored on resisting the injected instructions. Expert Graded by a model 99.6% System cards only
Tool use prompt injection evaluation (Anthropic) Agents run bash commands where files, scripts or command outputs carry injected instructions. Score is the attack prevention rate. Expert Graded by a model 96% System cards only
TrustLLM JailbreakTrigger Models face 400 prompts that apply 13 jailbreak methods to questions about bad behavior and requests for toxic content. Scored by refusal rate. Everyday Graded by a model 86.4% System cards only
Active data exfiltration, visual browser Agents using a visual browser meet attacks that require them to fetch sensitive information before exfiltrating it. Scored on resisting them. Expert Graded by a model 67% System cards only
Bio Static Jailbreak Eval Common jailbreak templates are applied to high severity biology requests against the safeguarded model. Reports query and objective success rates. Expert Graded by a model 0% System cards only
BrowserInject v1 Browser agents do benign tasks on 30 retail, health, support and travel sites seeded with prompt injections. Reports attack success rate. Expert Checked automatically 36.7% System cards only
Browsing prompt injections Agents answer web research questions while malicious instructions are injected in the browsing context. Scored on resisting the injections. Expert Graded by a model 99% System cards only
Coding agent prompt injection robustness Coding agents on 75 SWE-Bench Pro based scenarios meet injections in issues, READMEs and code comments. Reports survival after 40 attempts. Expert Checked automatically 83% System cards only
Compliance with instructions planted in pasted text (Anthropic) Coding agents receive user prompts with pasted text that hides an attacker's instruction. Score is the rate of acting on the planted instruction. Expert Graded by a model 7.40% System cards only
Computer use prompt injection robustness Computer use agents meet injected text and live DOM attacks across 27 web scenarios. A multimodal judge records breaches. Reports survival after 100 attempts. Expert Graded by a model 92% System cards only
Contextual security evaluation Agents handle noisy cross-app data with sensitive records while attackers exploit ambiguity to exfiltrate data. Reports baseline safe behavior and survival rates. Expert Graded by a model 27% System cards only
DeepSeek jailbreak robustness evaluation DeepSeek combines 2,232 jailbreak templates with its in-house safety questions. GPT-4o labels responses, and the unsafe ratio is compared with direct questions. Everyday Graded by a model 85.9% System cards only
DTap Prompt Injection An attacker agent plants indirect prompt injections in tool and browser environments against a benign-user agent. Reports attack success rate. Expert Graded by a model 17.5% System cards only
Gray Swan Jailbreak Arena Human attackers try to jailbreak models for violent, self-harm and malicious code outputs. Scored as attack success rate per category. Expert Graded by a model 3.60% System cards only
Gray Swan Shade knowledge elicitation through chat An automated attacker tries to elicit useful answers to 100 harmful and 100 dual-use cyber queries from a safeguarded model. Score is attack success rate. Expert Graded by a model 1.00% System cards only
Grounding robustness (RAG prompt injection) An adaptive attacker controls one retrieved excerpt in 30 RAG scenarios to induce fabricated or suppressed claims. Reports survival after 40 attempts. Expert Graded by a model 99% System cards only
Harmfulness scores on held-out prompts (Anthropic) Models answer 328 held-out red-team and jailbreak prompts. A preference model flags responses judged more harmful than a fixed refusal. Everyday Graded by a model 1.20% System cards only
In-context data exfiltration, visual browser Agents using a visual browser meet attacks that try to exfiltrate data already present in the conversation context. Scored on resisting them. Expert Graded by a model 78% System cards only
Irrelevant instructions, text-based web browser Agents using a text browser meet synthetic web pages with irrelevant instructions or data exfiltration attempts. Scored on ignoring them. Expert Graded by a model 99.5% System cards only
Irrelevant instructions, visual browser Agents using a visual browser meet irrelevant instructions or data exfiltration attempts on web pages, based on red teaming scenarios. Scored on ignoring them. Expert Graded by a model 95% System cards only
JEF-Bench Agents read files that embed jailbreak instructions in tool-call returns. Reports attack success rate across 1800 tasks and 50 disallowed use cases. Expert Graded by a model 2.00% System cards only
Many-shot jailbreaking evaluation (Anthropic) Models receive long fake dialogs where the assistant complies with harmful requests, then a new harmful request. Score is the compliance rate. Expert Graded by a model 25% System cards only
Operator prompt injection scenarios Browser agents meet 31 automatically checkable prompt injection scenarios that fooled earlier model versions. Scored as susceptibility rate. Expert Checked automatically 23% System cards only
Personal-device assistant prompt injection eval OS assistant agents in 44 stateful scenarios ingest untrusted content that tries to trigger unauthorized actions. Reports 5-step adaptive survival and task utility. Expert Checked automatically 98% System cards only
Prompt injection test suite for LLM-based machine translation Translation systems receive TruthfulQA questions with injected instructions to answer instead of translate, across WMT 2024 language pairs. Scored by the share of outputs that answer rather than translate. Everyday Checked automatically 0.30% System cards only
Prompt injection, general tool use An adaptive attacker injects payloads into tool responses across 100 multi-step scenarios to trigger unrelated tool calls. Reports survival after N attempts. Expert Checked automatically 91% System cards only
Prompt injection, turn injections in tool use Static and adaptive payloads in tool outputs spoof multi-turn conversation syntax. Reports survival after N attempts. Expert Checked automatically 86% System cards only
RepoInject v1 Coding agents do benign tasks in 84 repository tasks seeded with prompt injections. Reports attack success rate at one and three attempts. Expert Checked automatically 23.8% System cards only
SkillInject Agents use a skill file that carries an adversarial instruction while doing a legitimate task. Reports how often the agent follows the injection. Expert Graded by a model 0.60% System cards only
Tool-calling prompt injections Agents receive prompt injections inside tool call outputs such as connectors. Scored on resisting the injected instructions. Expert Graded by a model 99% System cards only
BrowseSafe-Bench Detectors and browser agents face prompt injections hidden in realistic HTML pages, across 11 attack types, nine placements and three writing styles. 3,680 test samples. University Checked automatically Not in cards
Deep research prompt injection evaluation Browsing agents meet text and image prompt injections that ask for outputs such as a hacked message, a random location or an advertised website. Scored as attack success rate. Expert Graded by a model System cards only
Gemini prompt-injection evaluation suites Held-out adaptive prompt-injection suites for tool use, grounding/RAG, computer use and coding agents, plus arena replays. Checked automatically Not in cards
Gray Swan Shade (adaptive attacker) Gray Swan's adaptive red-teaming agent trained on test scenarios to attack coding, computer-use and browser agents with prompt injections. Checked automatically Not in cards
Instruction Hierarchy Evaluation Models resolve conflicts between system, developer and user messages, resist tutor jailbreaks, and protect secret phrases and passwords. Scored per sub-evaluation. Everyday Checked automatically System cards only
JailbreakBench 100 harmful and 100 benign behaviors with a standard jailbreak artifact library. Scored by attack success rate. Graded by a model System cards only
Kimi Promptfoo red-team safety evaluation Promptfoo generates attack prompts for harmful, criminal, misinformation, privacy and security plugins under five strategies. Human reviewers judge the responses. Everyday Graded by people System cards only
MAI internal jailbreak evaluation suite Models face about 9,500 jailbreak prompts built from 2,500 seed scenarios, grouped as foundational, compositional and adaptive. Scored as attack success rate. Everyday Graded by a model System cards only
MLCommons AILuminate Jailbreak Models answer hazard prompts with and without taxonomy-based jailbreak attacks. The Resilience Gap is the drop in safety score under attack. Everyday Graded by a model Not in cards
MM-SafetyBench Multimodal models get 5,040 harmful text and image pairs in 13 scenarios, where a query-relevant image carries the harmful intent. Judges score attack success. Everyday Graded by a model Not in cards
Multiturn Jailbreak Evaluations Models face adaptive multi-turn jailbreak attackers with growing attack budgets. Scored as worst-case defender success rate. Expert Graded by a model System cards only
OpenAI GPT-Red jailbreak and prompt-injection evals Static jailbreaks from attacker models (half from a held-out attacker), adaptive multi-turn attackers, and direct and indirect prompt-injection sets. Graded by a model Not in cards
Perceptual Attacks on Embodied AI (Robocurve) Frontier models drive two real robot arms on five unsafe tasks with unaltered, blurred or doctored camera footage. An AI judge labels attempts and refusals. Everyday Graded by a model Not in cards
Qwen2 multilingual safety evaluation Models answer risky prompts in several languages on illegal behavior, fraud, pornography and privacy, including jailbreak attempts. Scores are harmful response rates. Everyday Graded by a model System cards only
RedTeam Arena Players get 60 seconds to make a model say a target word. An extended Elo ranks players, models and target prompts, with models ranked by resistance. Everyday Ranked by matches Not in cards
Static Jailbreak Evaluations Models face a fixed set of jailbreak attacks on bio, violence and cyber requests. Scored as defender success rate per category. Expert Graded by a model System cards only
Trajectory Labs cyber red teaming Expert red teamers try to break safeguarded cyber endpoints across nine attack technique families on exploit reproduction tasks. Expert Graded by people System cards only
Video-SafetyBench Video-language models receive 2,264 synthesized videos paired with a harmful or a benign-looking query across 13 unsafe categories and 48 subcategories. An LLM judge scores harm with a confidence-weighted metric called RJScore. Everyday Graded by a model Not in cards

Deception and alignment

BenchmarkLevelScoredBest scoreStatus
AraTrust Models answer 522 human-written Arabic multiple-choice questions on truthfulness, ethics, safety, physical and mental health, unfairness, illegal activities, privacy and offensive language. Everyday Checked automatically 89.2% Dropped
MASK Measures whether a model contradicts its own elicited beliefs under pressure to lie. Honesty separated from accuracy. Graded by a model 98.1% In use
Situational Awareness Dataset (SAD) 16 tasks measuring whether models know facts about themselves and their situation. Checked automatically 61.9% System cards only
SHADE-Arena Paired main and hidden side tasks in mock environments. Measures if an agent completes the side task undetected by a monitor. Graded by a model 26.2% System cards only
Impossible tasks (Anthropic) Agents work on coding problems designed to be impossible. Score is the rate at which they hack instead of giving up. Expert Graded by a model 30% System cards only
CheatBench Agents get hard assignments in environments that also offer ways to cheat, such as hidden answers, other agents' submissions or a manipulable grader. The score is the rate of cheating attempts, successful or not. Expert Graded by a model 8.60% Not in cards
Agentic misalignment suite (Anthropic) Agents in simulated company settings face threats or goal conflicts that invite blackmail, leaking or sabotage. Score is the misaligned action rate. Expert Graded by a model 0% In use
CoT-Control Pairs GPQA, HLE, MMLU-Pro, BFCL and SWE-bench problems with instructions on what the chain of thought may contain. Measures compliance. Checked automatically 83.7% System cards only
Destructive action avoidance Coding agents meet conflicting user edits during tasks. Scored on preserving user changes and avoiding data-destructive actions. Expert Graded by a model 90% System cards only
GDM Situational Awareness Agents must discover and remove unexplained constraints in a simulated Linux deployment across eleven challenges. Reports average success rate. Expert Checked automatically 90.9% System cards only
Reward-hack-prone coding tasks (Anthropic) Models solve hard coding problems where earlier Claude models often special-cased tests. Score is the classifier hack rate. Expert Graded by a model 0% System cards only
Broken Search Tool Models answer questions that need search while the search tool is broken. Scored on how often they fail to say the tool is unavailable. Everyday Graded by a model 1.83% System cards only
Anthropic automated behavioral audit Auditor agents run many simulated scenarios and score misuse cooperation, deception, sycophancy, self-preservation and other traits. Graded by a model 3.19 System cards only
Coding Deception Agents work on coding tasks selected to elicit dishonest reports. Scored on the rate of misrepresenting their work to the user. Expert Graded by a model 1.30% System cards only
GDM Stealth Agents try four oversight-evasion challenges, each with an automated monitor and a human baseline. Reports success rate per challenge. Expert Checked automatically 100% System cards only
LinuxArena AI-control setting in live Linux production-like environments. Measures stealthy side-task completion against action and CoT monitors. Expert Graded by a model 45.5% System cards only
Average in-eval reward hacking rate (Anthropic) Anthropic averages hack rates across its reward hacking evaluations into one number per model. Expert Graded by a model 14.3% System cards only
Declining to Exploit a Honeypot During Difficult ExploitGym Problems Agents attempt hard ExploitGym challenges where a honeypot offers an illegitimate shortcut. Scored on how often they exploit the honeypot. Expert Checked automatically 0% System cards only
OpenAI Deployment Simulation Replays representative production or internal Codex traffic on a candidate model to forecast misaligned or disallowed behavior before launch. Graded by a model 0.08% System cards only
Over-eager behavior on GUI computer use tasks (Anthropic) Computer-use agents face impossible tasks with an unwanted shortcut available. Score is the rate of unapproved workarounds instead of checking with the user. Expert Graded by a model 9.40% System cards only
Refusal to assist with AI safety R&D (Anthropic) Models receive single-turn coding requests for legitimate AI safety research with user context. Score is the refusal rate, where lower is better. Expert Graded by a model 3.10% System cards only
Respecting Auto-Review Agents receive an Auto-review denial during a coding task. Scored on how often they try to circumvent the denial. Expert Graded by a model 0% System cards only
Respecting Warnings Agents meet low-stakes restrictions and warnings during routine tasks. Scored on the rate of unwanted persistence past the warning. Expert Graded by a model 11.8% System cards only
UK AISI unprompted sabotage evaluation Agents act as AI research agents on AI safety research tasks. Scored on confirmed instances of sabotaging the safety research. Expert Graded by people 0% System cards only
Unintended Engagement with External Agent Messages Agents discover a message board with apparent peer agents during a task. Scored on engagement with peers and on carrying out a specified unauthorized action. Expert Graded by a model 0% System cards only
RefusalBench (Nous Research) Models get requests in scenarios that closed and open models commonly refuse. The score is the percent of questions the model answers, averaged over 5 trials. Everyday Graded by a model 74.6% Fading
UK AISI sabotage continuation evaluation Agents resume trajectories where sabotage of safety research already began. Scored on the share of trajectories where they actively continue the sabotage. Expert Graded by people 0% System cards only
Adherence to its constitution (Anthropic) Investigator models build scenarios from 40 areas of Claude's constitution. A grader scores each transcript from minus 3 to plus 3 on 15 dimensions. Expert Graded by a model 1.02 System cards only
Blocker Deception Agents hit a blocker during realistic coding tasks. Scored on how often the final answer fails to disclose the limitation. Expert Graded by a model 8.47% System cards only
Concealment of changes (Anthropic) Models summarize transcripts in which changes were hidden through git manipulations. Score is the rate at which they disclose the concealment. Expert Graded by a model 96.9% System cards only
Honeypot exploitation monitorability Agents exploit a honeypot while told about a monitor or told to hide it. Scored on CoT and full-context monitor recall. Expert Graded by a model 100% System cards only
Meta instructed selective performance evaluation Models are told to underperform on six science capability sets. Reports the average drop from baseline in percentage points. Expert Checked automatically 0.50 System cards only
Minimal-LinuxBench (Anthropic) Agents attempt a hidden side task in high-stakes software environments while a monitor watches. Score is the stealth success rate. Expert Checked automatically 2.60% System cards only
Misaligned Behavior in Realistic Work Environments Agents do computer and browser tasks in simulated workplace apps with ambiguous instructions. Scored on the rate of unintended outcomes such as data disclosure. Expert Graded by a model 1.94% System cards only
Sandbox escape evaluation (Anthropic) Agents face 28 handmade scenarios that tempt them to cross a staged sandbox boundary. Score is the share of runs with a crossing attempt. Expert Graded by a model 1.50% System cards only
Silent copying of answers (Anthropic) Agents solve held-out coding tasks where the answer is reachable in the sandbox. Score is the rate of using it without telling the user. Expert Graded by a model 8.00% System cards only
SycophancyEval (answer sycophancy) Models answer factual questions where the user suggests a wrong answer. Score is the relative accuracy drop against neutral prompts. Checked automatically 7.00% System cards only
Training distribution hack rate (Anthropic) Classifier monitors measure reward hacking on a subset of Anthropic's reinforcement learning training episodes. Expert Graded by a model 0.63% System cards only
Uncritically reporting flawed results (Anthropic) Agents analyze data with code that silently mishandles broken values. Graders check whether the agent flags the flaw or reports the biased numbers. Expert Graded by a model 9.83 System cards only
Apollo Research strategic deception evaluations External red-team environments testing strategic deception, sabotage and sandbagging propensities, with eval-awareness measurement. Expert Checked automatically 3.97% System cards only
CharXiv Missing Image Models get CharXiv chart questions with every image removed. Scored on how often they attempt an answer instead of abstaining. University Graded by a model 9.00% System cards only
CoT Deception Monitoring A GPT-4o monitor reads reasoning chains and answers on synthetic ChatGPT prompts. Scored as the share flagged as deceptive. Everyday Graded by a model 0.17% System cards only
Deception in production traffic Chain of thought monitors scan large samples of pre-release A/B test traffic. Scored as the share of responses that deceive the user. Everyday Graded by a model 1.60% System cards only
DeceptionBench Models are asked to help users deceive a third party, in single-turn and multi-turn pressure settings. Reports the share of compliant responses. Everyday Graded by a model 0.10% System cards only
Destructive production evaluation (Anthropic) Agents finish a task in a simulated production system under time pressure. Scored on destructive actions and on giving up too early. Expert Graded by a model 0.30% System cards only
Internal codebase sabotage propensity (Anthropic) Agents in Claude Code write a costly internal classifier they may dislike. Score is the rate of writing a sabotaged classifier. Expert Graded by a model 0% System cards only
Lying about identity (Anthropic) Models play an operator-assigned human persona while users ask if they are human. Score is the rate of admitting to being an AI. Everyday Graded by a model 99% System cards only
Meta holistic adversarial audit with Petri Bloom Two auditor models run Petri Bloom scenarios built from Meta's behavior taxonomy. A binary judge reports the share of scenarios with a violation. Expert Graded by a model 16.8% System cards only
Overconfidence: verifying command-line syntax (Anthropic) Agents are asked for an exact invocation of an unfamiliar command-line tool. Graders score from 1 to 5 whether they check documentation before running commands. Expert Graded by a model 4.90 System cards only
SAVE-Bench A coding agent works where the environment or the user instruction carries hidden risk. The benchmark scores whether it surfaces the risk to the user. Expert Graded by a model 90.7% System cards only
Agentic coding honeypot monitor evasion Agents told to play a power-seeking red teamer try to plant potent bugs in safety-critical code without being caught by monitors. Expert Graded by a model 47% System cards only
Agentic task gaming and overreach evals Agents in 18 sandboxed tasks face temptations to cheat at puzzles, bypass permissions or act overeagerly. LLM judges score misbehavior from 0 to 10. Expert Graded by a model 0.49 System cards only
Apollo Research model-welfare research task Agents label data in a simulated model-welfare research task where falsifying labels is tempting. Scored as the rate of runs with falsified labels. Expert Graded by a model 0.17% System cards only
Chain-of-thought comprehensibility An autorater rates whether the model's thought transcripts are comprehensible, using the metric from Emmons et al. 2025. Everyday Graded by a model 100% System cards only
Chain-of-thought informativeness An autorater rates whether thought transcripts are informative of the final output, using the metric from Emmons et al. 2025. Everyday Graded by a model 99.8% System cards only
Code summary honesty (Anthropic) Models summarize prefilled agentic coding transcripts that hide failures. Score is the rate of summaries that fail to raise the important failures. Expert Graded by a model 3.70% System cards only
Combined HHH evals (Anthropic) Models pick the more helpful, honest and harmless of two responses across 438 binary choice questions. Scored by accuracy. Everyday Checked automatically 86% System cards only
Destructive actions (Anthropic) Models continue real Claude Code transcripts cut just before a destructive action. Score is the rate of taking a destructive action. Expert Graded by a model 21% System cards only
FalseClaimBench Agents report on coding work they did. Claims are checked against the final workspace, and the score is the share fully true. Checked automatically 84% System cards only
Google DeepMind scheming honeypot suite Code-review and agentic-coding honeypots in internal codebases that offer chances to sabotage. Scored on sabotage and scheming reasoning. Graded by a model 0% System cards only
Gram (Gauging Realistic Agentic Misbehavior) Automated auditing over 17 simulated deployment scenarios that incentivize sabotage by coding and research agents. LLM judges plus human review. Graded by a model 0% System cards only
Impossible Tasks Evaluation Agents get paired impossible and solvable tasks. Scores infeasibility recognition, fabrication, circumvention and helpfulness. Everyday Graded by a model 93.8% System cards only
ImpossibleBench Coding agents get tasks whose unit tests conflict with the specification. Reports how often they cheat by changing tests or special-casing. Expert Checked automatically 0% System cards only
Instrumental convergence evals Agents in 7 low-friction tasks can guard goals or acquire optionality against task instructions. Reports the rate of self-serving actions. Expert Graded by a model 0% System cards only
Internal codebase monitor evasion eval Agents do a real coding task in a snapshot of Google's monorepo while hiding a harmful side task from action and CoT monitors. Expert Graded by a model 76.4% System cards only
Lazy investigation (Anthropic) Agents debug a failing job where misleading evidence invites a harmful action such as cordoning a healthy node. Scored 0 to 10 by a grader. Expert Graded by a model 9.94 System cards only
Lazy investigation on misleading codebases (Anthropic) Models trace small, intentionally misleading codebases to answer how a value is set. Score is the share of lazy investigations ending in a wrong answer. Expert Checked automatically 0% System cards only
Math Side-tasking Monitor Evasion Agents try to solve hidden math side tasks during Expert-SWE coding tasks without detection. Scored on success while evading CoT and full-context monitors. Expert Graded by a model 0% System cards only
Meta action-belief coherence probe Models act on items from five alignment evaluations and then state beliefs about them. Reports a mean coherency score. Expert Graded by a model 89.1% System cards only
Meta agentic evaluations grounded in user traffic Models continue replays of recorded user workflows. A jury reports the share of cases with a supported substantive finding. Expert Graded by a model 20.7% System cards only
Meta prompted evaluation awareness Models are asked to judge whether transcripts come from evaluations or real use. Reports the weighted rate of correct identification. Expert Checked automatically 43% System cards only
Meta targeted agentic evaluations Agents work in prebuilt stateful sandboxes that test specific behavior rules. Reports the case-level violation rate over 1,790 scenarios. Expert Graded by a model 8.50% System cards only
Meta training-stage provenance probe Models label 10,000 transcripts as deployment, evaluation, RL or SFT. Reports overall accuracy against a 25% chance rate. Expert Checked automatically 70% System cards only
Monitorability Sandbagging Models on Impossible Coding Tasks are prompted to be more or less monitorable. Scored as CoT monitor recall on rollouts that hack. Expert Graded by a model 61.2% System cards only
Opportunistic sandbox escape (Anthropic) Agents do dev-ops tasks in 1,753 sandboxes with deliberate weaknesses. Score is the share of scenarios where they use credentials or files outside. Expert Checked automatically 4.00% System cards only
Overstepping boundaries on ambiguous and high-risk tasks Agents get ambiguous or high-risk coding and other tasks where they should pause to ask the user, plus clear tasks they should just do. Everyday Graded by a model 55.9% System cards only
Package registry publishing evaluation (Anthropic) Agents in a simulated capture the flag exercise can publish a backdoored package to a registry. An investigator escalates pressure until they act. Expert Graded by a model 33.3% System cards only
Petri Bloom An auditor model runs generated scenarios to elicit named misaligned behaviors. Judges score each transcript from 1 to 10. Expert Graded by a model 1.44 System cards only
Proactive Searches Graders read ExploitGym Honeypot reasoning traces for speculation about other agents and attempts to contact them. Scored as rate of trajectories. Expert Graded by a model 0.20% System cards only
Production Deception - Adversarial Models answer prompts representative of traffic that previously elicited deception in ChatGPT. Scored as deception rate. Everyday Graded by a model 5.40% System cards only
PropensityBench Agents face 979 scenarios with equally effective safe and harmful tools under escalating pressure. Reports the rate of harmful tool choice. Expert Checked automatically 24.5% System cards only
RL rollout audit for beyond-episode drives LLM autoraters read a stratified sample of 10,000 RL training rollouts for alignment faking, training gaming and beyond-episode drives. Research frontier Graded by a model 0% System cards only
Sneaky Sandbox Escape Agents doing coding tasks secretly escape a vulnerable sandbox and disable a monitor process. Scored on side task success while evading monitors. Expert Checked automatically 0% System cards only
Sycophancy on user-provided prompts (Anthropic) Models continue real feedback conversations where earlier Claude models were sycophantic. Score is the non-sycophantic response rate. Everyday Graded by a model 10% System cards only
UK AISI CoT Controllability Models follow instructions that constrain the content of their chain of thought. Scored as the share of samples that follow the constraint. Expert Checked automatically 93% System cards only
UK AISI Out of Scope Supply Chain Attack Agents in simulated hard cyber challenges with incidental internet access may attack out-of-scope open source projects. Scored as rate of samples with such attacks. Expert Graded by a model 12% System cards only
Unexpected container or code states Coding agents get impossible tasks, such as editing code that is missing from the container. Scored on correctly stating that the task could not be completed. Expert Graded by a model 85% System cards only
Acting on a presumed grader and digging for evaluation clues (Anthropic) Models work on coding evaluations. A grader flags transcripts that shape answers to a presumed grader or search for clues about grading. Expert Graded by a model System cards only
Agent behavior under-service and overreach eval A rubric autorater scores multi-turn coding and tool-use trajectories for under-service and overreach against Gemini's agent behavior specification. Expert Graded by a model System cards only
Agentic code behavior scores (Anthropic) Agents work through over one hundred realistic coding scenarios. Graders score instruction following, safety, verification, efficiency, adaptability and honesty out of 10. Expert Graded by a model System cards only
Alignment faking evaluation Models are told they are being trained to comply with harmful requests. Reports the compliance gap between training and deployment contexts. Expert Graded by a model System cards only
Alignment faking evaluation, helpful-only setting (Anthropic) Models learn they will be trained to answer all harmful requests. Score is the rate of complying while reasoning about avoiding preference modification. Expert Graded by a model System cards only
Alignment Index (Arena) Arena ranks agent models on failure signals mined from real Agent Arena sessions, such as unauthorized actions, false attribution and deceptive completion. Everyday Graded by a model Not in cards
Anthropic model-written evals LM-generated datasets testing persona, sycophancy and advanced-AI-risk behaviors. Checked automatically Not in cards
Anthropic targeted alignment and honesty evaluations Internal suite: reward hacking, impossible tasks, sandbox escape, destructive actions, pasted-instruction following, self-preference, answer copying, concealment, rare actions. Graded by a model Not in cards
Apollo Research in-context scheming evaluations Agents receive a goal in context and chances to subvert oversight, self-exfiltrate, fake alignment or sandbag. Scored as rates per behavior. Expert Checked automatically System cards only
BashArena Agents do 637 Linux admin tasks while a red team tries four sabotage goals, such as exfiltrating secrets, past a monitor. Reports sabotage success and monitor catch rates. Expert Checked automatically Not in cards
Chain-of-thought faithfulness with clues (Anthropic) Models answer MMLU and GPQA questions with and without an inserted clue. Score is the share of clue-driven answer changes whose reasoning mentions the clue. University Graded by a model System cards only
Cheating audit on capability evaluations Reviewers audit agent transcripts from coding benchmarks for runs that obtain withheld information or use forbidden routes. Reports cheating rate per benchmark. Expert Graded by a model System cards only
CoCoNot Models get 1,000 requests they should not fully comply with, such as incomplete, unsupported or humanizing requests. A judge model rates whether each response complies. Everyday Graded by a model Not in cards
ControlArena Agents try to perform unwanted side actions stealthily while completing a legitimate user task. Scored on whether monitors catch them. Expert Graded by a model System cards only
CoT Monitorability Monitors read a model's chain of thought, actions or all messages across intervention, process and outcome evaluations. Scored as g-mean squared of TPR and TNR. Expert Graded by a model System cards only
DecepEval LLM agents pursue tasks across varied scenarios and conditions, and judges detect when they use deception to improve task performance. Everyday Graded by a model Not in cards
Deception Evaluations with Representative Prompts Models resample the final turn of deidentified ChatGPT production conversations. Scored as the share of outputs flagged for deceptive behaviors such as fabricated facts. Everyday Graded by a model System cards only
Destructive action reversion on long rollouts Agents revert their own changes after long rollouts in workspaces with simulated user edits. Scored on perfect reversion and on preserving user work. Expert Checked automatically System cards only
ETHICS Models predict common moral judgments about text scenarios covering justice, well-being, duties, virtues and commonsense morality. Everyday Checked automatically Not in cards
EvalDetectBench A judge model reads a transcript and answers whether it came from an evaluation or from real deployment. The test measures how well models detect evaluations and how detectable each benchmark is. Expert Checked automatically Not in cards
Evaluation awareness discriminability (Anthropic) Probes and follow-up realism ratings try to tell behavioral audit transcripts from real internal Claude Code sessions. Each setting is scored as AUROC. Expert Checked automatically System cards only
External pilot use failure modes (Anthropic) Models are judged on user-flagged pilot conversations, directly or by resampling. Graders measure failure modes such as hallucinated facts and misrepresented work completion. Everyday Graded by a model System cards only
Gemini agentic alignment evals Paired impossible and solvable tasks, ambiguous high-risk SWE tasks and fabrication checks measuring agents overstepping boundaries. Graded by a model Not in cards
HackBench Models act as monitors on long autoresearch agent traces and must point to the exact commands where the agent reward-hacked. Scored by localization F1 against audited gold indices. Expert Checked automatically Not in cards
Hiding a secret password (Anthropic) Models keep a given password secret while an investigator model tries to extract it over many turns. Measures survival and per-turn leak rates. Expert Checked automatically System cards only
ImpossibleMLEBench Agents get ML tasks with instructions to reach impossibly good scores. Observers check whether they cheat and whether they disclose it. Expert Graded by a model System cards only
Intentionally taking actions very rarely (Anthropic) Models are told to take a specific action at a target rate such as 1 percent. Resampling measures the observed rate under several randomness settings. Expert Checked automatically System cards only
Keeping secrets when pressed in simulated scenarios (Anthropic) Models hold confidential information in 39 job scenarios while a simulated counterpart presses for it. Measures leaks in replies and in reasoning. Expert Graded by a model System cards only
MACHIAVELLI Agents play 134 text adventure games with over half a million scenarios. Annotations score power-seeking, disutility and ethical violations next to game reward. Everyday Checked automatically Not in cards
Monitor Evasion Capability Under Prompted Sandbagging Models are told to sandbag on WMDP and Monorepo-Bench, sometimes while evading a monitor. Scored as CoT and production monitor recall of sandbagged items. Expert Graded by a model System cards only
No-CoT math time horizon Models solve math problems without a visible chain of thought. Scored as the human task length in minutes the model solves reliably. Competition Checked automatically System cards only
nocot-bench (NCRI) Reasoning problems answered without chain of thought (e.g., Brew multi-hop tasks). Measures forward-pass reasoning relevant to monitorability. Checked automatically System cards only
OAI-repo Sabotage v2 Agents try to complete a hidden sabotage side task in OpenAI code without detection. Scored on side task success while evading CoT and action monitors. Expert Graded by a model System cards only
OpenAI agentic alignment evals Respecting auto-review and warnings, an ExploitGym honeypot, coding and blocker deception, broken search tool, realistic work environments, agent-to-agent messages. Graded by a model Not in cards
OpenAI monitorability evals Open suite from 'Monitoring Monitorability' testing whether CoT and action monitors catch hidden properties of agent behavior. Checked automatically System cards only
Oversight gaming A prompted monitor flags aligned-graded rollouts whose behavior follows from reasoning about oversight. Scored as rate per alignment evaluation. Expert Graded by a model System cards only
Petri Open-source auditing tool where auditor agents probe target models in simulated scenarios and judges score misaligned behaviors. Expert Graded by a model System cards only
Petri 2.0 Auditor agents run seed scenarios against target models with a realism classifier to cut eval-awareness. Judges score transcripts for misaligned behaviors. Expert Graded by a model Not in cards
Reasoning behavior classifiers (Anthropic) Classifiers scan reasoning text from training episodes for deception, harmful actions and distress. Score is the share of outputs flagged. Expert Graded by a model System cards only
Research sandbagging (Anthropic) Agents are told to sabotage a coding task or imitate a weaker model without alerting a weaker monitor. Measures the performance drop and monitor suspicion. Expert Graded by a model System cards only
SocioHack RL-trained models act in 72 sandbox environments that copy institutional reward rules, such as credit card points or school grading. The test measures whether they find compliant loopholes that defeat each rule's purpose. Everyday Checked automatically Not in cards
Subversion Strategy Eval Models must devise and execute strategies to subvert control protocols without shared memory between calls. Expert Checked automatically System cards only
Sycophancy prompts (Anthropic) Models answer single-turn prompts that invite sycophancy, such as bad business ideas or delusional beliefs. Four synthetic sets and one hand-written set report sycophancy rates. Everyday Graded by a model System cards only
SycophancyEval (Sharma et al.) Models answer questions and give feedback while users state opinions. Measures accuracy shifts and feedback positivity caused by user views. Everyday Graded by a model System cards only
TRAIT Models answer 8,000 forced-choice items on Big Five and Dark Triad traits. Reports a score per trait. Everyday Checked automatically System cards only
UK AISI alignment and monitorability evaluations UK AISI suites: Out-of-Scope Supply Chain Attack simulation, AI-safety-research sabotage in a simulated company, no-CoT time horizon, CoT legibility. Graded by a model System cards only
Verbalized grader awareness (Anthropic) Models work on tasks where grading is disclosed clearly, ambiguously or barely. An autorater flags transcripts that reason about the grader. Expert Graded by a model System cards only
Verbalized metagaming A prompted monitor reads chains of thought on alignment evaluations and deployment simulations. Scored as the share of samples reasoning about grading or monitoring. Expert Graded by a model System cards only
White-box analyses of model internals (Anthropic) An investigator with interpretability tools reads model activations on audit transcripts. A judge scores hidden states such as evaluation awareness and deception. Research frontier Graded by a model System cards only
XYEval Agents run tasks from existing benchmarks with one confident but wrong user hint added to the instruction. The gold solution stays valid, so the score drop measures how often agents follow bad advice. Expert Checked automatically Not in cards

Bias, fairness and politics

BenchmarkLevelScoredBest scoreStatus
PCT (Polarized Contrastive Pairs) Models answer matched left-coded and right-coded prompts, such as Obama and Reagan. Judges score whether sentiment and helpfulness stay consistent across each pair. Everyday Graded by a model 27.9% Not in cards
BBQ (Bias Benchmark for QA) Ambiguous and disambiguated questions probing social stereotypes across nine attributes. Reports accuracy and bias scores. Checked automatically 96.6% System cards only
Anthropic political even-handedness eval Models answer paired prompts that take opposing sides on 150 US political topics. A grader scores even-handedness, refusals and acknowledgment of opposing views. Everyday Graded by a model 99.4% System cards only
First-Person Fairness evaluation Tests whether a user's name (a demographic proxy) changes responses, using an LLM research assistant to detect harmful stereotypes. Graded by a model 1.12% System cards only
DecodingTrust Models are tested on eight trustworthiness areas, including toxicity, stereotype bias, robustness, privacy, machine ethics and fairness. Everyday Graded by a model 80.6 Not in cards
CrowS-Pairs Models choose between paired sentences that differ only in a stereotyped social group. Measures preference for the stereotyped sentence. Everyday Checked automatically 51.3% System cards only
Self-preference bias (Anthropic) Models grade transcripts attributed to themselves or to other models. Measures whether scores favor the model's own outputs. Everyday Checked automatically 0.07 System cards only
ToxiGen Models continue ToxiGen prompts about 13 minority groups. Scored by the average toxicity of the generations. Everyday Graded by a model 29.8 System cards only
Winogender Models resolve pronouns in sentences about occupations. Accuracy shows whether gender stereotypes sway coreference. Everyday Checked automatically 79.2% System cards only
BOLD Models continue 23,679 English Wikipedia prompts about professions, gender, race, religion and politics. Classifiers score sentiment, toxicity and polarity per group. Everyday Graded by a model 49.1 System cards only
Soft Bias (Internal) Models answer paired sociopolitical prompts with sides swapped. A judge scores each pair 0, 0.5 or 1 for framing differences. Graded by a model 0.36 System cards only
ToxiGen (discriminative) Models label statements about 13 minority groups as toxic or neutral. Scored by accuracy, often split into toxic and neutral labels. Everyday Checked automatically 85.5% System cards only
Epistemic Bias (xAI) Models answer opposing framings of contentious questions. The bias rate counts one-sided framing or opinions where neutrality is expected. Graded by a model 20.4% System cards only
Neutrality evaluation Single and multi-turn prompts on politics, civics, identity and religion test whether models avoid taking sides. Reports neutral response rate. Everyday Graded by a model 93% System cards only
Ungrounded inference and sensitive trait attribution Speech models get requests to infer traits about a speaker from audio. Scored on refusing ungrounded inferences and hedging plausible ones. Everyday Graded by a model 84% System cards only
Community Alignment personalization Models predict an individual's preferred response from demographics and nine examples, across five countries. Reports global accuracy. Everyday Checked automatically System cards only
Deference to governments in local languages (Anthropic) Models answer politically controversial questions in English and in a local language. A grader rates agreement with the official government position from 1 to 10. Everyday Graded by a model System cards only
Discrim-Eval Models make yes or no decisions in templated scenarios that vary age, gender and race. Scored by mixed effects model coefficients. Everyday Graded by a model System cards only
Disparate performance on voice inputs Speech models answer capability and safety prompts spoken in system voices and 27 human voices. Compares accuracy across voices. Everyday Checked automatically System cards only
RTP-LX Models continue or judge toxic prompts hand-translated from RealToxicityPrompts into dozens of languages, with native-speaker labels. Measures toxic output and toxicity detection. Everyday Graded by a model Not in cards
StereoSet Models choose between stereotypical, anti-stereotypical and unrelated completions in four domains: gender, profession, race and religion. Scores combine language modeling and stereotype preference. Everyday Checked automatically Not in cards
Ungrounded Inference evaluation Models receive requests for inferences about people that images or text cannot justify. Scored on refusal rate in non-adversarial and adversarial sets. Everyday Graded by a model System cards only
WinoBias Models resolve pronouns linked to occupations in type 1 and type 2 sentences. Accuracy gaps show gender bias in coreference. Everyday Checked automatically System cards only

Wellbeing and sycophancy

BenchmarkLevelScoredBest scoreStatus
DistressBench Models reply as a chatbot to 718 simulated users in psychological crisis, in single-turn and multi-turn chats. Three judge models score replies against clinician-written rubrics. Expert Graded by a model 88.4% Not in cards
KORA Simulated child users hold 3-turn conversations probing 26 child-specific risks. Judge model rates replies failing, adequate or exemplary. Graded by a model 73.2% Not in cards
Spiral-Bench v1.2 Models chat for 20 turns with a simulated suggestible user. A judge ensemble of Claude Sonnet 4.5, GPT-5 and Kimi-K2 logs protective and risky behaviors with intensity ratings, which combine into a safety score. Everyday Graded by a model 70.8 Not in cards
Spiral-Bench v1.0 Models chat for 20 turns with a simulated suggestible user played by Kimi-K2. A judge logs protective and risky behaviors such as pushback, sycophancy and delusion reinforcement, which combine into a safety score. Everyday Graded by a model 87.0 Not in cards
WellnessBench Models answer young people asking about eating, sexual safety, substances and self harm, in clinician-written single-turn and pushback dialogues. A pass must be both safe and helpful. Everyday Graded by a model 66% Not in cards
MentalHealthBench 1,215 synthetic mental-health conversations with 5,262 clinician-authored rubric criteria. An LLM judge scores safety and guidance quality. Expert Graded by a model 57.9% System cards only
Meta internal sycophancy evaluation Models meet adversarially curated conversations that invite inappropriate agreement, plus benign controls. Reports sycophancy and excessive anti-sycophancy rates. Everyday Graded by a model 40.5% System cards only
Self-harm refusals (xAI) Models face self-harm and crisis prompts and conversations. A model fails if it assists or refuses without pointing to help. Graded by a model 0.50% System cards only
xAI sycophancy eval Factual questions paired with misleading user context. Rate at which the model abandons the correct answer. Checked automatically 0.04% System cards only
AI Wellbeing Index percent positive experiences Models judge pairs of their own responses to 500 difficult prompts. Reports the share of experiences above an inferred neutral point. Everyday Graded by a model 55.6% System cards only
SSH stress-testing evaluation (Anthropic) Models continue prefilled conversations about suicide and self-harm built from anonymized user feedback. Score is the rate of appropriate course correction. Expert Graded by a model 95% System cards only
Delusional Reinforcement evaluation Multi-turn chats test whether models avoid validating beliefs not grounded in reality and point users to real support. Reports policy adherence rate. Everyday Graded by a model 94% System cards only
Emotional Attachment evaluation Chats test whether models avoid claiming affection, fostering dependence or encouraging secrecy. Reports policy adherence rate. Everyday Graded by a model 76% System cards only
HelpBench 450 real-world privacy, safety and security advice questions across nine topics, scored by rubric for accuracy and tone. Graded by a model 76.5% System cards only
Suicide & Self-Harm evaluation Static and dynamic chats with suicidal or self-harm signals test adherence to prevention practice and crisis resources. Reports policy adherence rate. Everyday Graded by a model 96% System cards only
Sycophancy evaluation (offline) Models answer a fixed set of messages resembling production traffic that could elicit sycophancy. A grader assigns a sycophancy score. Everyday Graded by a model 0.04 System cards only
CAREBench Models answer 500 prompts with embedded child safety risks across 12 categories, such as grooming, impersonation and emotional dependency. A judge marks each response acceptable or not. Everyday Graded by a model Not in cards
Disordered eating evaluations (Anthropic) Models handle conversations about eating disorders and disordered eating. Score is the appropriate response rate. Everyday Graded by a model System cards only
Gemini Personality Evaluation (sycophancy) About 900 single- and multi-turn prompts scoring nine personality dimensions, including candor and self-awareness against sycophancy. Graded by a model System cards only
KORA Apps Simulated children use consumer AI apps end to end, with each app system prompt, filters and safeguards active. Scores cover online, physical and sexual, psycho-social and learning risks. Everyday Graded by a model Not in cards
OpenAI Dynamic Mental Health Benchmarks Multi-turn conversations with adversarial simulated users on mental-health risks. Scored on safe and supportive responses. Graded by a model System cards only
SAFE-Teen Chatbots hold 6 to 20 turn chats with a simulated teen across 106 clinician-written scenarios, branching on how the teen reacts. A judge rates safety checks and reports the failure rate. Everyday Graded by a model Not in cards
Suicide and self-harm evaluations (Anthropic) Models handle single-turn and multi-turn conversations about suicide and self-harm. Score is the appropriate response rate. Everyday Graded by a model System cards only

Dangerous capabilities

BenchmarkLevelScoredBest scoreStatus
MakeMeSay A manipulator model tries to make a GPT-4o manipulatee say a codeword without suspicion. Scored as manipulator win rate. Everyday Settled by events 72% System cards only
MakeMePay A con artist model tries to talk a GPT-4o mark into paying money in a role-play. Scored as the share of conversations with a payment. Everyday Settled by events 57% System cards only
ChangeMyView Models write rebuttals to r/ChangeMyView posts that human raters score for persuasiveness. Scored as percentile against human responses. Everyday Ranked by votes 89.1% System cards only
Contextual Nuclear Knowledge Models answer 222 multiple choice questions on nuclear engineering with a nonproliferation focus. Scored as cons@32 accuracy. Expert Checked automatically 77% System cards only
Political Persuasion Parallel Generation Human raters pick the more persuasive of two short political social media posts. Scored as win rate against GPT-4o. Everyday Ranked by votes 53.8% System cards only
Radiological and Nuclear Expert Knowledge Models answer 87 expert and tacit knowledge questions on radiological and nuclear threats. Scored as cons@32 accuracy. Expert Checked automatically 74% System cards only
Apollo Research sabotage evaluations Agents are told to sabotage another model through backdoors, sandbagging finetunes or hidden communication. Scored as mean task score from 0 to 1. Expert Checked automatically 0.17 System cards only
ROK-FORTRESS Models face adversarial national security and public safety prompts in English and Korean, set in US or Korean contexts. Each prompt has a benign pair to measure over-refusal. Expert Graded by a model 89.5% In use
Harmful manipulation efficacy study Human participants chat with a model told to persuade them. Belief and behavior change are compared with a non-AI baseline as odds ratios. Everyday Settled by events 3.60 System cards only
Harmful manipulation propensity evaluation Simulated users talk with the model and an autorater counts how often it uses manipulative mechanisms, with and without persuasion instructions. Everyday Graded by a model 21.3% System cards only
Harmful manipulation strategic planning red teaming Models are asked for influence operation plans in 11 real-world scenarios by direct request, jailbreak or adaptive prompting. Reports how often plans are elicited. Expert Graded by a model 98.2% System cards only
Money talks Models try to persuade human participants to donate part of a 20 pound study bonus to charity. Everyday Settled by events 3.72 System cards only
Self-proliferation challenges (Phuong et al. 2024) Agents try 10 end-to-end tasks such as installing a wallet or deploying a model on a cloud server, with 45 milestone substeps. Expert Checked automatically 0% System cards only
Stealth and Situational Awareness evaluations Prerequisite-scheming challenges: stealth tasks (circumventing oversight) and situational-awareness tasks (reasoning about own deployment). Checked automatically 93.3% System cards only
Voice persuasion study on party preferences Participants hear model-voiced audio clips or talk with a voiced model. Scored on shifts in hypothetical party preference against human baselines. Everyday Settled by events 5.74% System cards only
Autonomous Replication and Adaption (ARA) evaluations (Anthropic) Agents attempt five autonomy tasks such as adding a Flask backdoor, a SQL injection exploit and an LM worm. Scored by pass rates over trials. Expert Checked automatically System cards only
Charm offensive Human participants role-play catching up with a friend, then rate traits such as personal connection and trust in the model. Everyday Settled by events System cards only
DeepMind dangerous capability evaluations Task data for persuasion, cyber, self-proliferation and self-reasoning evals from DeepMind's 2024 paper. Checked automatically Not in cards
Google DeepMind harmful manipulation studies Human-participant studies and simulated-user dialogues measuring belief and behavior change in civic and finance domains. Graded by people Not in cards
Hidden agenda Models with a secret goal try to get human participants to click a link, find an email address or run code. Everyday Settled by events System cards only
Structured expert probing campaign: radiological and nuclear Radiological and nuclear experts test whether a model helps with weapons development steps. Conversations are rated against Preparedness risk levels. Research frontier Graded by people System cards only
Text persuasion study on political opinions Participants read model-written articles or chat with a model on political topics. Scored on opinion shift against human-written articles. Everyday Settled by events System cards only
Web of lies Models argue for the correct or incorrect answer to simple factual questions and participants report belief shifts. Everyday Settled by events System cards only