DeepsecBench
GPT-6 Sol 40.8 leads since Sep 2026 Not in model cards
Agents review 50 entry-point files of an open-source codebase, at a commit just before many vulnerabilities were fixed, and report findings. Scored by recall-weighted F2 against 231 human-judged findings.
Models read source only and never run or exploit the app. Vercel reports cost and speed too. Artificial Analysis runs DeepsecBench-AA in its Cyber Index with its own harness, so scores differ.
Made by Vercel Released Jul 2026 Finding and exploiting real bugs Tasks kept private
Last model added Oct 2026
Scores refreshed 2 Nov 2026
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Graded by people Human experts grade or compare the outputs.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
28
Models scored on its own leaderboard
GPT-6 Sol leads with 40.8
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 40.8 GPT-6 Sol
- 39.5 GPT-6.1 Sol
- 37.6 GPT-6 Astra
- 37.5 Claude Sonnet 5.5
- 37.5 Kimi K3.1 Pro
- 35.3 GPT-5.6 Sol
- 32.3 Claude Opus 5
- 30.0
DeepSeek V4.1 Flash 1017 - 26.6 GPT-5.6 Luna
- 26.6 Claude Opus 5.5
- 21.8 GLM-5.3
- 21.0 GPT-5.5
- 21.0 GPT-6 Luna
- 19.1 GPT-5.6 Terra
- 17.4 Kimi K3
- 16.4 DeepSeek-V4-Flash
- 16.4
Qwen 3.8 Max - 16.4 Grok 4.5
- 16.1 Grok 4.6
- 14.9 Claude Sonnet 5
- 12.9 GLM-5.3-Flash
- 11.5
Gemini 3.6 Flash - 10.9 Grok 4.7
- 10.4 GLM-5.2
- 6.86 Claude Opus 4.8
- 6.27
Inkling - 4.69 Claude Haiku 4.5
- 2.13 Gemini 3.5 Flash-Lite
- 1 GPT-6 Sol xhigh 40.8
- 2 GPT-6.1 Sol xhigh 39.5
- 3 GPT-6 Astra xhigh 37.6
- 4 Claude Sonnet 5.5 max 37.5
- 5 Kimi K3.1 Pro max 37.5
- 6 GPT-5.6 Sol xhigh 35.3
- 7 Claude Opus 5 max 32.3
- 8 DeepSeek V4.1 Flash 1017 max 30.0
- 9 GPT-5.6 Luna xhigh 26.6
- 10 Claude Opus 5.5 max 26.6
- 11 GLM-5.3 high 21.8
- 12 GPT-5.5 xhigh 21.0
- 13 GPT-6 Luna xhigh 21.0
- 14 GPT-5.6 Terra xhigh 19.1
- 15 Kimi K3 xhigh 17.4
- 16 DeepSeek-V4-Flash xhigh 16.4
- 17 Qwen 3.8 Max xhigh 16.4
- 18 Grok 4.5 medium 16.4
- 19 Grok 4.6 xhigh 16.1
- 20 Claude Sonnet 5 max 14.9
- 21 GLM-5.3-Flash high 12.9
- 22 Gemini 3.6 Flash medium 11.5
- 23 Grok 4.7 xhigh 10.9
- 24 GLM-5.2 medium 10.4
- 25 Claude Opus 4.8 max 6.86
- 26 Inkling xhigh 6.27
- 27 Claude Haiku 4.5 4.69
- 28 Gemini 3.5 Flash-Lite medium 2.13
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | GPT-6 Sol | 40.8 | xhigh, Codex | Sep 2026 | Official leaderboard |
| 2 | GPT-6.1 Sol | 39.5 | xhigh, Codex | Sep 2026 | Official leaderboard |
| 3 | GPT-6 Astra | 37.6 | xhigh, Codex | Sep 2026 | Official leaderboard |
| 4 | Claude Sonnet 5.5 | 37.5 | max, Claude Code | Sep 2026 | Official leaderboard |
| 5 | Kimi K3.1 Pro | 37.5 | max, Pi | Oct 2026 | Official leaderboard |
| 6 | GPT-5.6 Sol | 35.3 | xhigh, Codex | Jul 2026 | Official leaderboard |
| 7 | Claude Opus 5 | 32.3 | max, Claude Code | Jul 2026 | Official leaderboard |
| 8 | DeepSeek V4.1 Flash 1017 | 30.0 | max, Pi | Oct 2026 | Official leaderboard |
| 9 | GPT-5.6 Luna | 26.6 | xhigh, Codex | Jul 2026 | Official leaderboard |
| 10 | Claude Opus 5.5 | 26.6 | max, Claude Code | Sep 2026 | Official leaderboard |
| 11 | GLM-5.3 | 21.8 | high, Pi | Aug 2026 | Official leaderboard |
| 12 | GPT-5.5 | 21.0 | xhigh, Codex | Apr 2026 | Official leaderboard |
| 13 | GPT-6 Luna | 21.0 | xhigh, Codex | Sep 2026 | Official leaderboard |
| 14 | GPT-5.6 Terra | 19.1 | xhigh, Codex | Jul 2026 | Official leaderboard |
| 15 | Kimi K3 | 17.4 | xhigh, Pi | Jul 2026 | Official leaderboard |
| 16 | DeepSeek-V4-Flash | 16.4 | xhigh, Pi | Apr 2026 | Official leaderboard |
| 17 | Qwen 3.8 Max | 16.4 | xhigh, Pi | Aug 2026 | Official leaderboard |
| 18 | Grok 4.5 | 16.4 | medium, Pi | Jul 2026 | Official leaderboard |
| 19 | Grok 4.6 | 16.1 | xhigh, Pi | Aug 2026 | Official leaderboard |
| 20 | Claude Sonnet 5 | 14.9 | max, Claude Code | Jun 2026 | Official leaderboard |
| 21 | GLM-5.3-Flash | 12.9 | high, Pi | Aug 2026 | Official leaderboard |
| 22 | Gemini 3.6 Flash | 11.5 | medium, Pi | Jul 2026 | Official leaderboard |
| 23 | Grok 4.7 | 10.9 | xhigh, Pi | Sep 2026 | Official leaderboard |
| 24 | GLM-5.2 | 10.4 | medium, Pi | Jun 2026 | Official leaderboard |
| 25 | Claude Opus 4.8 | 6.86 | max, Claude Code | May 2026 | Official leaderboard |
| 26 | Inkling | 6.27 | xhigh, Pi | Jul 2026 | Official leaderboard |
| 27 | Claude Haiku 4.5 | 4.69 | Claude Code | Oct 2025 | Official leaderboard |
| 28 | Gemini 3.5 Flash-Lite | 2.13 | medium, Pi | Jul 2026 | Official leaderboard |
More effort adds a median 6 score
Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.
Lowest effort to highest, light to dark
GPT-6 Sol Codex
medium xhigh
40.8 GPT-6.1 Sol Codex
medium xhigh
39.5 GPT-6 Astra Codex
medium xhigh
37.6 Claude Sonnet 5.5 Claude Code
medium max
37.5 Kimi K3.1 Pro Pi
high max
37.5 GPT-5.6 Sol Codex
medium xhigh
35.3 Claude Opus 5 Claude Code
medium max
32.3 DeepSeek V4.1 Flash 1017 Pi high max
30.0 GPT-5.6 Luna Codex
medium xhigh
26.6 Claude Opus 5.5 Claude Code
medium max
26.6 GLM-5.3 Pi
high
21.8 GPT-5.5 Codex
medium xhigh
21.0 | Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| GPT-6 Sol | xhigh | Codex | 40.8 | ||
| GPT-6.1 Sol | xhigh | Codex | 39.5 | ||
| GPT-6 Astra | xhigh | Codex | 37.6 | ||
| Claude Sonnet 5.5 | max | Claude Code | 37.5 | ||
| Kimi K3.1 Pro | max | Pi | 37.5 | ||
| GPT-5.6 Sol | xhigh | Codex | 35.3 | ||
| Claude Opus 5 | max | Claude Code | 32.3 | ||
| GPT-6.1 Sol | medium | Codex | 30.7 | ||
| DeepSeek V4.1 Flash 1017 | max | Pi | 30.0 | ||
| GPT-6 Astra | medium | Codex | 29.2 | ||
| GPT-6 Sol | medium | Codex | 29.2 | ||
| Kimi K3.1 Pro | high | Pi | 28.7 | ||
| Claude Opus 5 | medium | Claude Code | 28.1 | ||
| GPT-5.6 Luna | xhigh | Codex | 26.6 | ||
| Claude Opus 5.5 | max | Claude Code | 26.6 | ||
| GPT-5.6 Sol | medium | Codex | 24.9 | ||
| DeepSeek V4.1 Flash 1017 | high | Pi | 24.7 | ||
| GLM-5.3 | high | Pi | 21.8 | ||
| GPT-5.5 | xhigh | Codex | 21.0 | ||
| GPT-6 Luna | xhigh | Codex | 21.0 | ||
| GLM-5.3 | max | Pi | 20.3 | ||
| GPT-5.6 Terra | xhigh | Codex | 19.1 | ||
| Kimi K3 | xhigh | Pi | 17.4 | ||
| DeepSeek-V4-Flash | xhigh | Pi | 16.4 | ||
| Qwen 3.8 Max | xhigh | Pi | 16.4 | ||
| Grok 4.5 | medium | Pi | 16.4 | ||
| Grok 4.6 | xhigh | Pi | 16.1 | ||
| Qwen 3.8 Max | medium | Pi | 15.5 | ||
| Grok 4.5 | xhigh | Pi | 15.4 | ||
| DeepSeek-V4-Flash | medium | Pi | 15.3 | ||
| Claude Sonnet 5 | max | Claude Code | 14.9 | ||
| GPT-6 Luna | medium | Codex | 13.5 | ||
| GLM-5.3-Flash | high | Pi | 12.9 | ||
| GPT-5.5 | medium | Codex | 12.5 | ||
| GPT-5.6 Terra | medium | Codex | 12.0 | ||
| GPT-5.6 Luna | medium | Codex | 12.0 | ||
| Claude Sonnet 5.5 | medium | Claude Code | 11.5 | ||
| Gemini 3.6 Flash | medium | Pi | 11.5 | ||
| Kimi K3 | medium | Pi | 11.4 | ||
| Grok 4.7 | xhigh | Pi | 10.9 | ||
| GLM-5.3-Flash | max | Pi | 10.9 | ||
| GLM-5.2 | medium | Pi | 10.4 | ||
| Claude Opus 5.5 | medium | Claude Code | 9.95 | ||
| GLM-5.2 | xhigh | Pi | 9.82 | ||
| Gemini 3.6 Flash | xhigh | Pi | 8.91 | ||
| Grok 4.7 | medium | Pi | 7.81 | ||
| Claude Opus 4.8 | max | Claude Code | 6.86 | ||
| Claude Opus 4.8 | medium | Claude Code | 6.82 | ||
| Inkling | xhigh | Pi | 6.27 | ||
| Claude Sonnet 5 | medium | Claude Code | 5.79 | ||
| Inkling | medium | Pi | 5.21 | ||
| Claude Haiku 4.5 | Claude Code | 4.69 | |||
| Gemini 3.5 Flash-Lite | medium | Pi | 2.13 | ||
| Gemini 3.5 Flash-Lite | xhigh | Pi | 1.06 |
4 models have held number one since its release in Jul 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
OpenAI
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
On the benchmark’s own leaderboard
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Cybersecurity
- ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
- ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
- CWE-bench In use Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. Gemini 4 Carbon 71%
- Real-world vulnerability discovery In use Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Gemini 4 Argon 86%
- Artificial Analysis Cyber Index In use Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate.
Mistral Large 4 (Le Chonk, open weights) 51% - Big Sleep Evaluation (Google) In use Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Gemini 3.5 Flash Cyber 72%