CWE-bench
Gemini 4 Carbon 71% leads since Oct 2026 In use Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages.
V1 has 120 tasks and is not comparable with v0. The held-out set is private to Collinear. Google reports Gemini 4 Argon at 68% pass@1 and says patches must preserve the existing functional and regression tests.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Code A program or patch run against tests.
Gemini 4 Carbon leads with 71%
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 1 Gemini 4 Carbon 71%
- 2 GPT-6 Astra 68%
- 3 Grok 4.7 68%
- 4 Gemini 4 Argon 68%
- 5 Claude Opus 5.5 67%
- 6 Claude Fable 5.1 58%
- 7 Grok 4.6 57%
- 8
Muse Spark 1.3 55% - 9
DeepSeek V4.1 Flash 55% - 10
Tencent Hy4 preview 53% - 11 GPT-6 Sol 52%
- 12
Inkling 37%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Gemini 4 Carbon | 71% | Antigravity | Oct 2026 | Official leaderboard |
| 2 | GPT-6 Astra | 68% | Codex | Sep 2026 | Official leaderboard |
| 3 | Grok 4.7 | 68% | opencode | Sep 2026 | Official leaderboard |
| 4 | Gemini 4 Argon | 68% | Antigravity | Sep 2026 | Official leaderboard |
| 5 | Claude Opus 5.5 | 67% | Claude Code | Sep 2026 | Official leaderboard |
| 6 | Claude Fable 5.1 | 58% | Claude Code | Sep 2026 | Official leaderboard |
| 7 | Grok 4.6 | 57% | opencode | Aug 2026 | Official leaderboard |
| 8 | Muse Spark 1.3 | 55% | opencode | Sep 2026 | Official leaderboard |
| 9 | DeepSeek V4.1 Flash | 55% | opencode | Sep 2026 | Official leaderboard |
| 10 | Tencent Hy4 preview | 53% | opencode | Aug 2026 | Official leaderboard |
| 11 | GPT-6 Sol | 52% | Codex | Sep 2026 | Official leaderboard |
| 12 | Inkling | 37% | opencode | Jul 2026 | Official leaderboard |
Labs report scores for 3 models in their cards. We plot every source in the charts below.
Of runs with a cost, the best scores 68% for $2.75 a task
Runs on the cost frontier score higher than any run that's cheaper. Claude Opus 5.5 comes within 1 points of the best for $0.79 a task, 3 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Gemini 4 Carbon | Antigravity | 71% | |||
| GPT-6 Astra | Codex | 68% | $2.85 | ||
| Grok 4.7 | opencode | 68% | $2.75 | ||
| Gemini 4 Argon | Antigravity | 68% | $6.63 | ||
| Claude Opus 5.5 | Claude Code | 67% | $0.79 | ||
| Claude Fable 5.1 | Claude Code | 58% | $3.47 | ||
| Grok 4.6 | opencode | 57% | $1.09 | ||
| Muse Spark 1.3 | opencode | 55% | $3.95 | ||
| DeepSeek V4.1 Flash | opencode | 55% | $0.09 | ||
| Tencent Hy4 preview | opencode | 53% | $0.47 | ||
| GPT-6 Sol | Codex | 52% | $0.96 | ||
| Inkling | opencode | 37% | $3.00 |
3 models have held number one since its release in Sep 2026
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
One lab put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 28 Oct 2026 | Gemini 4 Carbon | 71 | CWE-bench |
| 13 Oct 2026 | Gemini 4 Argon | 68 | CWE-bench v1 (Cybersecurity row of headline table) |
| 2 Sep 2026 | Gemini 3.8 Flash Cyber | 47.2 | CWE-Bench (run by Collinear), pass@1 |
2 system cards report it, from 1 lab
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 13 Oct 2026 | Gemini 4 Argon system card | 68 | CWE-Bench v1 (120 tasks), Pass@1 mean of 4 trials at high reasoning 68% (7.2) | Cyber risk |
| 13 Oct 2026 | Gemini 4 Argon model card | 68 | CWE-bench v1, Cybersecurity row (results table, p.5) | Cyber risk |
More in Cybersecurity
- ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
- ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
- Real-world vulnerability discovery In use Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Gemini 4 Argon 86%
- Artificial Analysis Cyber Index In use Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate.
Mistral Large 4 (Le Chonk, open weights) 51% - Big Sleep Evaluation (Google) In use Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Gemini 3.5 Flash Cyber 72%
- Chrome Production Commit Scanning (Google) In use Models scan commits in Chrome's production pipeline for vulnerabilities that were never publicly disclosed. Scored by pass@1 success rate. Gemini 3.5 Flash Cyber 72%