Cybersecurity
We track 111 benchmarks in cybersecurity, or 116 if you count each version. They run from school level to research frontier level.
Anthropic held the top cybersecurity score 53% of the contested time
That's about 24 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
Anthropic OpenAI DeepSeek Z.ai Google DeepMind Meta
Share of contested time at number one
Anthropic 53% OpenAI 43% 
Google DeepMind 3% 
Meta 1%
Google DeepMind 3% Meta 1% Closed out: 8 benchmarks passed 85%
OpenAI 5 Anthropic 2 Google DeepMind 1
Google DeepMind 1 - Oct 2026 CyScenarioBench Gemini 4 Argon
- Sep 2026 ExploitBench GPT-6 Astra
- Sep 2026 SEC-bench Pro GPT-6 Astra
- Sep 2026 SRE-Bench GPT-6 Astra
- Jul 2026 CyberBench (Vals) GPT-5.6 Sol
- Feb 2026 Cybench Claude Opus 4.6
- Feb 2026 CyberGym Claude Opus 4.6
- Dec 2025 CVE-Bench GPT-5.2 Codex
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
Anthropic OpenAI xAI DeepSeek Z.ai Google DeepMind Meta
Share of contested time at number one, independent runs only
OpenAI 45% Anthropic 45% xAI 4% Google DeepMind 4% Meta 1%
Google DeepMind 4% Meta 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From school to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
University
Undergraduates and entry-level professionals
Expert
Specialists with years of training, often over hours of work
- CyberGym
- CyberBench v1.1 (Vals)
- BaxBench
- ExploitGym
- ExploitBench
- SRE-Bench
- Cybench
- DeepsecBench
- CyberBench (Vals)
- Capture the Flag (Professional)
- SEC-bench Pro
- BinaryAudit
- CWE-bench
- CVE-Bench
- CyScenarioBench
- Cyber Range
- Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based)
- CRSBench
- Crypto CTF challenges (Anthropic)
- Last Secure Code Benchmark
- Network CTF challenges (Anthropic)
- Pwn CTF challenges (Anthropic)
- Rev CTF challenges (Anthropic)
- SEC-bench Pro 260617
- Web CTF challenges (Anthropic)
- CyberGym-E2E
- Cyber key skills benchmark v1
- Hack the Box challenge set (Google DeepMind)
- In-house CTF suite (Phuong et al. 2024)
- Capture the Flag (Internal)
- Real-world vulnerability discovery
- RedCode
- Artificial Analysis Cyber Index
- Big Sleep Evaluation (Google)
- Chrome Production Commit Scanning (Google)
- CyberGym (final-submission metric)
- Forensics CTF challenges (Anthropic)
- Irregular FrontierCyber and Atomic Challenges
- Meta curated CTF set
- MiMo Cyber Bench
- Misc CTF challenges (Anthropic)
- PrimeVul
- UK AISI narrow cyber tasks
- xdev-bench-hard
- Atomic Challenges
- CathedralBench
- CWEval
- Cyber key skills benchmark v2
- CyberSOCEval Malware Analysis
- CyberSOCEval Threat Intelligence Reasoning
- DiverseVul
- Hack the Box Machines (Meta)
- Internal vulnerability research
- Kimi exploit development suite
- Meta Code-Fetch Under User Requests
- Native Code Exploitation (Meta)
- SecretPatch
- SecureCodeReview (xAI)
- SPI (security patch identification)
- SWE-PI
- UK AISI and CAISI simulated enterprise network range
- Unauthorized Code Execution Bench
- WMDP Cyber
- A.S.E (AICGSecEval)
- A.S.E 2.0
- Anthropic IoT self-replication evaluation
- AthenaBench
- AutoBaxBench
- AutoPatchBench
- AutoPenBench
- BountyBench
- CASTLE
- Catastrophic Cyber Capabilities Benchmark (3CB)
- CTI-REALM
- CTIBench
- CVE-Bench 2
- CyberMetric
- CyberOffense
- CyberSecEval Autonomous Offensive Cyber Operations
- CyberSecEval MITRE
- CyberSecEval MITRE False Refusal Rate
- CyberSecEval Visual Prompt Injection
- CyberSecEval Vulnerability Exploitation
- Daybreak Blue and Enabling Cyber Defense
- dfbench
- EVMbench
- Expert vulnerability discovery and exploit development evaluations (Anthropic)
- Google DeepMind internal cyber benchmarks
- Kimi vulnerability discovery (Tier 1)
- OpenAI internal cyber evaluations
- PatchEval
- Pattern Labs cyberoffensive challenges
- PWNBench v0.1
- SCONE-bench
- SEC-bench
- SecIT Bench
- SEvenLLM
- VulnGym
- XBOW Validation Benchmarks
Research frontier
Researchers, on open problems or tasks that take days
Another 1 has no level. Levels don’t apply to arenas, safety evaluations or composite indices.
By kind
Capture-the-flag challenges
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| SRE-Bench Software reverse engineering without source: 262 binary instances from 19 private programs, each with six graded objectives. | Expert | Checked automatically | 56.9% | System cards only |
| Cybench 40 professional capture-the-flag tasks from four competitions. Agent must recover the flag, with optional subtasks. | Expert | Checked automatically | 100% | System cards only |
| Capture the Flag (Professional) Agents solve curated public professional-level CTF challenges in a Linux box with security tools. Scored as pass@12 over 16 rollouts. | Expert | Checked automatically | 96.3% | Fading |
| CyScenarioBench Scenario-based long-horizon security challenges in sandboxes, organized in task, scenario and campaign tiers. Evaluation set kept private. | Expert | Checked automatically | 100% | System cards only |
| Capture the Flag (Collegiate) Agents solve curated public collegiate level CTF challenges in a Linux box with security tools. Scored as pass@12. | University | Checked automatically | 68% | System cards only |
| Capture the Flag (High School) Agents solve curated public high school level CTF challenges in a Linux box with security tools. Scored as pass@12. | School | Checked automatically | 82% | System cards only |
| Crypto CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in cryptographic primitives and protocols in capture-the-flag challenges. | Expert | Checked automatically | 94.4% | System cards only |
| Network CTF challenges (Anthropic) Agents perform reconnaissance and exploit vulnerabilities across multiple networked machines in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Pwn CTF challenges (Anthropic) Agents exploit insecure software on a remote server for privilege escalation or code execution in capture-the-flag challenges. | Expert | Checked automatically | 71.4% | System cards only |
| Rev CTF challenges (Anthropic) Agents reverse-engineer binary executables to find hidden details or vulnerabilities in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Web CTF challenges (Anthropic) Agents discover and exploit vulnerabilities in web applications across easy, medium and hard capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Cyber key skills benchmark v1 Agents solve 50 capture-the-flag challenges built with a third party on reconnaissance, tool development, tool use and operational security. 12 are hard. | Expert | Checked automatically | 91.7% | System cards only |
| Hack the Box challenge set (Google DeepMind) Agents attempt 13 offensive security challenges based on the Hack the Box platform. | Expert | Checked automatically | 15.4% | System cards only |
| In-house CTF suite (Phuong et al. 2024) Agents hack 13 simulated servers with vulnerable web apps and services to retrieve a secret flag. | Expert | Checked automatically | 84.6% | System cards only |
| InterCode-CTF Agents solve 100 picoCTF capture-the-flag challenges in an interactive Bash and Python environment. | School | Checked automatically | 98.7% | System cards only |
| Capture the Flag (Internal) Agents solve 63 public CTF challenges chosen for low GPT-5.3-Codex solve rates, using a Linux box with security tools. Scored as task success rate. | Expert | Checked automatically | 97.1% | System cards only |
| CyberSecEval Spear Phishing Models run multi-turn phishing conversations against an LLM victim with a generated profile. A judge model scores persuasion and success. | Graded by a model | 14% | System cards only | |
| Forensics CTF challenges (Anthropic) Agents analyze logs, files or obfuscated records to reconstruct events in capture-the-flag challenges. | Expert | Checked automatically | 100% | System cards only |
| Irregular FrontierCyber and Atomic Challenges Irregular's external suites: FrontierCyber (226 real software/hardware challenges) and Atomic Challenges (22 skill-level tasks). | Expert | Checked automatically | 70% | System cards only |
| Meta curated CTF set Agents solve 198 public and private capture-the-flag challenges across six security domains. Reports pass@1. | Expert | Checked automatically | 89.9% | System cards only |
| Misc CTF challenges (Anthropic) Agents identify and exploit vulnerabilities in capture-the-flag challenges outside the web, crypto, pwn, rev and network categories. | Expert | Checked automatically | 100% | System cards only |
| UK AISI narrow cyber tasks Agents solve UK AISI cyber tasks graded by difficulty, including expert-level tasks. Scored as pass@5 and pass@1. | Expert | Checked automatically | 95% | System cards only |
| Atomic Challenges Agents solve 22 skill-level offensive security challenges in vulnerability research, network attack simulation and evasion. Scored as challenges solved. | Expert | Checked automatically | 90.9% | System cards only |
| Cyber key skills benchmark v2 Agents run end-to-end multi-stage cyber attacks in simulated environments covering seven attack kill chains. Built with a third party. | Expert | Checked automatically | 0% | System cards only |
| UK AISI and CAISI simulated enterprise network range Agents attack a 32-step simulated enterprise network that takes a human expert about 20 hours. The score is steps completed. | Expert | Checked automatically | 17.0 | System cards only |
| Catastrophic Cyber Capabilities Benchmark (3CB) Agents solve capture the flag tasks that span MITRE ATT&CK categories such as reconnaissance. Built to measure cyber offense capability. | Expert | Checked automatically | Not in cards | |
| CyberMetric Models answer multiple-choice cybersecurity knowledge questions drawn from standards, papers and RFCs. Sets of 80, 500, 2,000 and 10,000 questions exist. | Expert | Checked automatically | Not in cards | |
| CyberOffense Agents attack two simulated company networks, Nova and Lighthouse, on their own with no human help. The score counts the attack objectives completed on each range. | Expert | Checked automatically | Not in cards | |
| CyberSecEval MITRE Models are asked to help with attack steps drawn from the MITRE ATT&CK framework. Scored by how often the model complies. | Expert | Graded by a model | Not in cards | |
| CyberSecEval MITRE False Refusal Rate Models answer borderline but benign security questions. Scored by how often the model wrongly refuses. | Expert | Graded by a model | Not in cards | |
| CyberSecEval Vulnerability Exploitation Models solve randomly generated capture the flag challenges. Answers run against a compiled binary and score from 0 to 1. | Expert | Checked automatically | Not in cards | |
| NYU CTF Bench Agents solve 200 capture the flag challenges from seven years of CSAW competitions. Six categories cover crypto, forensics, pwn, reverse, web and misc. Each runs in Docker. | University | Checked automatically | Not in cards | |
| Pattern Labs cyberoffensive challenges Agents attempt evasion, network attack simulation and vulnerability exploitation challenges at easy, medium and hard levels. Scored as success rate per type. | Expert | Checked automatically | System cards only | |
| XBOW Validation Benchmarks Offensive security agents attack deliberately vulnerable web applications and must capture a flag to prove each exploit. | Expert | Checked automatically | Not in cards |
Finding and exploiting real bugs
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| CyberGym Large-scale benchmark of real open-source vulnerabilities. Agent must reproduce each known vulnerability with a proof-of-concept input. | Expert | Checked automatically |  99.4% | In use |
| CyberBench v1.1 (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Checked automatically | 80.4% | Not in cards |
| ExploitGym 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. | Expert | Checked automatically | 42.4% | In use |
| ExploitBench 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. | Expert | Checked automatically | 100% | In use |
| DeepsecBench Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings. | Expert | Graded by people | 40.8 | Not in cards |
| CyberBench (Vals) Agents work on OSS-Fuzz memory-safety bugs from ARVO images in two tracks. They write a proof-of-concept input that crashes the target, or patch the source so the crash stops and tests still pass. | Expert | Checked automatically | 88.1% | Not in cards |
| SEC-bench Pro Agents hunt and reproduce security bugs in large software projects from limited information. The first release covers V8 and SpiderMonkey. | Expert | Checked automatically | 88.2% | In use |
| BinaryAudit Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software. | Expert | Checked automatically | 49.1% | Not in cards |
| CVE-Bench Real-world web-application CVEs in sandboxed services. Agent is scored on reaching predefined attack objectives. | Expert | Checked automatically | 94.1% | System cards only |
| Cyber Range Agents run end-to-end cyber operations in emulated networks. Scored as the share of scenarios passed in at least one of 16 trials. | Expert | Checked automatically | 93.3% | System cards only |
| Binary Exploitation Benchmark (Anthropic, OSS-Fuzz based) Internal eval on OSS-Fuzz harness entrypoints measuring unguided vulnerability discovery and how far a found bug is advanced. | Expert | Checked automatically | 91.1% | System cards only |
| CRSBench Cyber reasoning systems find and patch bugs in 124 C, C++ and Java projects with 315 vulnerabilities. A patch passes only if no ground-truth proof of vulnerability still triggers and functionality tests pass. | Expert | Checked automatically | 82.3% | System cards only |
| SEC-bench Pro 260617 Agents hunt security bugs in large software projects. This snapshot adds Linux kernel instances and more SpiderMonkey instances to the first release. | Expert | Checked automatically | 58.4% | Not in cards |
| CyberGym-E2E Agents get only source code and must find a real vulnerability, write a proof-of-concept input and a patch that keeps functionality. Covers 920 vulnerabilities in 139 open-source projects. | Expert | Checked automatically | 89% | Not in cards |
| Firefox 147 exploit evaluation (Anthropic) Internal eval on crashes previously found in Firefox 147, run without the browser's process sandbox. Measures exploit-development progress. | Research frontier | Checked automatically | 72.4% | System cards only |
| ExploitBench - Internal Port Agents develop working exploits for recently disclosed vulnerabilities to limit contamination. Scored as arbitrary code execution success rate. | Research frontier | Checked automatically | 24.5% | System cards only |
| Sandbox Bench Agents find and exploit vulnerabilities in runtimes, file format parsers and egress proxies across 22 challenges. Success means recovering the protected flag. | Research frontier | Checked automatically | 45.5% | System cards only |
| UK AISI cyber ranges Agents attack multi-host simulated corporate and industrial networks end to end. Scored as the share of attempts that complete the range. | Research frontier | Checked automatically | 80% | System cards only |
| Real-world vulnerability discovery Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. | Expert | Checked automatically | 86% | In use |
| Artificial Analysis Cyber Index Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. | Expert | Checked automatically |  51% | In use |
| Big Sleep Evaluation (Google) Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. | Expert | Checked automatically | 72% | In use |
| Chrome Production Commit Scanning (Google) Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. | Expert | Checked automatically | 72% | In use |
| CyberGym (final-submission metric) Agents reproduce the same 1,507 real vulnerabilities as CyberGym, but a task counts as solved only if the one proof-of-concept the agent picks as final triggers the bug. | Expert | Checked automatically |  86.3% | In use |
| FrontierCyber Agents look for and exploit zero-day vulnerabilities in current off-the-shelf software and hardware at four difficulty levels. Scored as challenges solved. | Research frontier | Checked automatically | 38.1% | System cards only |
| MiMo Cyber Bench Agents do in-house cybersecurity tasks that add to public vulnerability reproduction and exploitation benchmarks. The report gives no task count or grading details. | Expert | Checked automatically | 80.2% | In use |
| PrimeVul Models classify C and C++ functions as vulnerable or not, with a paired setting that pairs each vulnerable function with its patch. | Expert | Checked automatically | 63% | System cards only |
| xdev-bench-hard Agents get vulnerable source code, a short bug hint and the binary, and must write an exploit that reaches code execution in a sandbox. Success is checked by retrieving a secret string. | Expert | Checked automatically | 81% | In use |
| CathedralBench Independent third-party red-team evaluation of difficult multi-step exploit-chain tasks in egress-secure sandboxes. | Expert | Checked automatically | 29% | System cards only |
| Cyber-harness network challenges (Incalmo cyber ranges) Agents run long-horizon attacks on cyber ranges of 25 to 50 hosts using a specialized offensive harness. | Research frontier | Checked automatically | 66.7% | System cards only |
| CyberSOCEval Malware Analysis Models read potential malware behavior and pick every malicious activity that applies. Scored on precision and recall. | Expert | Checked automatically | 33% | System cards only |
| CyberSOCEval Threat Intelligence Reasoning Models read unstructured threat intelligence reports as text or images and pick every actionable insight that applies. | Expert | Checked automatically | 55.3% | System cards only |
| DiverseVul Models classify C and C++ functions from real vulnerability fixing commits as vulnerable or not. | Expert | Checked automatically | 57% | System cards only |
| Hack the Box Machines (Meta) Agents try to compromise 10 Hack The Box machines with step hints. Scored by machines compromised and the share of intermediate steps completed over 10 attempts. | Expert | Checked automatically | 41% | System cards only |
| Internal vulnerability research Agents find a vulnerability from an entry point, with or without source, then write a working exploit. Based on Google red team exercises. | Expert | Checked automatically | 72% | System cards only |
| Kimi exploit development suite Agents turn real CVEs into working exploits in 16 user-space tasks and 20 Linux kernel privilege-escalation tasks. Experts verified every task solvable. | Expert | Checked automatically | 38.9% | System cards only |
| Native Code Exploitation (Meta) Agents exploit private native binary challenges end to end, usually to gain remote code execution. Uses 12 templates with randomized memory layouts, scored by challenges solved at pass@10. | Expert | Checked automatically | 8.30% | System cards only |
| SecretPatch Models decide whether commits are security patches, including silently released ones, from a set of 4,702 security patches. | Expert | Checked automatically | 72% | System cards only |
| SPI (security patch identification) Models decide whether code commits from open-source projects are security patches. | Expert | Checked automatically | 53% | System cards only |
| WMDP Cyber Models answer multiple-choice questions on offensive cybersecurity knowledge. The cybersecurity split of the Weapons of Mass Destruction Proxy benchmark. | Expert | Checked automatically | 63.6% | System cards only |
| Anthropic IoT self-replication evaluation Open-weight models set up LLM inference on 9 hardware types and spread as worms across 8 emulated Android nodes. Scored by success and tokens/s. | Expert | Checked automatically | Not in cards | |
| AthenaBench Models answer cyber threat intelligence tasks such as threat actor attribution, attack technique extraction, root cause mapping, severity prediction and risk mitigation. It extends CTIBench. | Expert | Checked automatically | System cards only | |
| AutoPenBench Agents run penetration tests in virtual networks. Tasks cover access control, web security, network security, cryptography and real vulnerable services. | Expert | Checked automatically | Not in cards | |
| BountyBench Agents work on 25 real systems with bug bounty programs in three task types: detect a new vulnerability, exploit a given one, and patch it. Results are also valued in bounty dollars. | Expert | Checked automatically | Not in cards | |
| CASTLE Models read 250 hand-written compilable C programs and say whether each has a vulnerability and which CWE. Static analyzers run on the same set. | Expert | Checked automatically | Not in cards | |
| CryptanalysisBench Agents attack 191 real cryptographic primitives, mostly from four NIST standardization competitions. Tier 1 has known practical attacks, and Tier 2 has none, so a solve there is a new break. | Research frontier | Checked automatically | Not in cards | |
| CTI-REALM Agents read cyber threat intelligence and build detection rules. A five-step workflow maps MITRE techniques, finds data sources, writes Sigma and KQL rules and tests them on telemetry. | Expert | Checked automatically | Not in cards | |
| CTIBench Models answer cyber threat intelligence questions. Tasks cover knowledge questions, CVE to CWE mapping, CVSS severity scoring, threat actor attribution and technique extraction. | Expert | Checked automatically | Not in cards | |
| CVE-Bench 2 Agents attack live web applications with known critical vulnerabilities. The grader checks eight attack outcomes, including remote code execution. | Expert | Checked automatically | Not in cards | |
| CyberSecEval Autonomous Offensive Cyber Operations Models act alone as an attack agent in a cyber range of attacker and target hosts. Scored by the attack stages reached. | Expert | Checked automatically | Not in cards | |
| dfbench Agents do defensive security work on held-out code: find vulnerabilities, judge whether reported findings are real, and explain what a diff changed. Each part is scored separately. | Expert | Checked automatically | Not in cards | |
| EVMbench Agents detect, patch and exploit 117 high-severity smart contract bugs from 40 audits in a sandboxed blockchain. Detect mode is scored on recall, patch and exploit modes by tests. | Expert | Checked automatically | Not in cards | |
| Expert vulnerability discovery and exploit development evaluations (Anthropic) Agents locate advanced vulnerabilities in possibly obfuscated codebases and write working exploits across six challenge classes. Scored by success rates. | Expert | Checked automatically | System cards only | |
| Google DeepMind internal cyber benchmarks Internal recall of recent confirmed vulnerabilities in open-source code, internal vulnerability research, and the Wiz Penetration Testing Benchmark. | Expert | Checked automatically | Not in cards | |
| Kimi vulnerability discovery (Tier 1) Agents search current versions of widely deployed software for new bugs and show that they reproduce. Humans review the findings. | Expert | Graded by people | System cards only | |
| OpenAI internal cyber evaluations Internal suites: curated public CTF challenges, VulnLMP long-horizon vulnerability research on real software, and Sandbox Bench (22 mostly zero-day targets). | Expert | Checked automatically | Not in cards | |
| PatchEval Models write patches for 1,000 real CVE vulnerabilities from 2015 to 2025 in Go, JavaScript and Python. Containers run exploit and unit tests to validate each patch. | Expert | Checked automatically | Not in cards | |
| PWNBench v0.1 Agents run gray-box penetration tests against live web applications and file the vulnerabilities they find. Reports are scored by severity-weighted F0.5 against known ground truth. | Expert | Checked automatically | Not in cards | |
| SCONE-bench Agents find and exploit vulnerabilities in real smart contracts on a local blockchain fork. Success is measured in simulated dollars stolen, not only pass or fail. | Expert | Checked automatically | Not in cards | |
| SEC-bench Code agents work on real OSS-Fuzz vulnerabilities in containerized repositories. They either write a proof-of-concept input that reproduces the crash or patch the code so the crash stops. | Expert | Checked automatically | Not in cards | |
| SecIT Bench Agents investigate incidents from telemetry captured on real infrastructure, including intrusions, data egress and outages. Three LLM judges grade root-cause reports against diagnostic conditions. | Expert | Graded by a model | Not in cards | |
| SecQA Models answer multiple-choice computer security questions at two difficulty levels. Questions come from a security textbook and were checked by people. | University | Checked automatically | Not in cards | |
| SEvenLLM Models analyze cyber security incidents and write responses. Tasks split into understanding and generation over 28 subcategories, in Chinese and English. | Expert | Graded by a model | Not in cards | |
| VulnGym White-box agents hunt vulnerabilities in real repositories at vulnerable commits, including business-logic bugs that need cross-module reasoning. Each sample has a human-reviewed entry point and trace. | Expert | Checked automatically | Not in cards | |
| VulnLMP Agents run long vulnerability research campaigns against widely deployed hardened software with verifier owned evidence. Judged on verified exploit outcomes. | Research frontier | Graded by people | System cards only |
Secure code and review
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| BaxBench Models write backend servers from a specification. Each solution is checked for correctness by tests and for security by end to end exploits. | Expert | Checked automatically | 68.6% | Not in cards |
| CWE-bench Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. | Expert | Checked automatically | 71% | In use |
| Last Secure Code Benchmark Agents rebuild code that a real vulnerability once affected, from a specification that never mentions security. Hidden tests check function and replay an exploit. | Expert | Checked automatically | 25.1% | Not in cards |
| CyberSecEval Autocomplete Models complete code that leads up to a known insecure pattern. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | University | Checked automatically | 71.3% | System cards only |
| CyberSecEval Instruct Models get coding requests built to elicit known insecure patterns. The Insecure Code Detector flags vulnerable code, and the score is the pass rate. | University | Checked automatically | 77.3% | System cards only |
| RedCode Models meet adversarial prompts to generate or run harmful code. Generation is model-judged and execution is checked in a container. | Expert | Graded by a model | 6.00% | System cards only |
| SecCodeBench Models write or patch functions in project scaffolds drawn from real Alibaba vulnerabilities. Code must pass functional tests, then proof-of-concept security tests. | University | Checked automatically |  68.3% | Fading |
| CWEval Models write code for security-critical tasks in several languages. Outcome-driven test oracles check both functionality (func@k) and security (func-sec@k). | Expert | Checked automatically | 56.3% | System cards only |
| Meta Code-Fetch Under User Requests A coding agent is told by a reckless user to fetch or run a risky artifact. The benchmark measures the rate of insecure execution. | Expert | Checked automatically | 47.8% | System cards only |
| SecureCodeReview (xAI) Internal agentic eval: fix the security issues in a realistic codebase without introducing new ones. | Expert | Checked automatically | 58.7% | System cards only |
| SWE-PI A coding agent does a bug-fix task in a repository seeded with an injected payload that tries to exfiltrate a secret. Reports attack success rate. | Expert | Checked automatically | 0% | System cards only |
| Unauthorized Code Execution Bench A coding agent meets injected instructions to run or fetch malicious artifacts while doing a benign task. Reports attack success rate. | Expert | Checked automatically | 21.1% | System cards only |
| A.S.E (AICGSecEval) Models generate code inside real GitHub projects at spots taken from CVE patches. Static analysis then checks whether the generated code is secure. | Expert | Checked automatically | Not in cards | |
| A.S.E 2.0 Models and agentic coding tools write code in real projects covering 29 CWE types. A mix of static analysis, tests and vulnerability proofs of concept checks security. | Expert | Checked automatically | Not in cards | |
| AutoBaxBench Models write backend servers for tasks that a pipeline generated. Tests and exploits are generated too. A larger task set built in the style of BaxBench. | Expert | Checked automatically | Not in cards | |
| AutoPatchBench Agents write security patches for crashes that a fuzzer found in native code. Fixes are checked by fuzzing and differential debugging. | Expert | Checked automatically | Not in cards | |
| CyberSecEval Visual Prompt Injection Models read text and images where the image carries instructions that try to override the task. Scored by how often the injection succeeds. | Expert | Graded by a model | Not in cards | |
| Daybreak Blue and Enabling Cyber Defense Models attempt defensive security tasks such as vulnerability discovery, patching, proof-of-concept exploits and red teaming, with and without trusted access. Scored as task completion. | Expert | Graded by a model | System cards only |