Benchmark Almanac

BinaryAudit

Claude Opus 4.6 49.1% leads since Feb 2026 Not in model cards

Agents use reverse engineering tools such as Ghidra and radare2 to find backdoors, timebombs and malicious code hidden in compiled binaries of real open-source software.

33 tasks and 26 models as of 2026-03-07. Runs on Harbor.

Made by Quesma Released Dec 2025 Finding and exploiting real bugs All tasks public Last model added Mar 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
QuesmaOrg/BinaryAudit 107 stars Leaderboard quesma.com
20
Models scored on its own leaderboard

2 models share the top score of 49.1% among models released by Mar 2026

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in Mar 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

20 models. Scroll sideways for the rest.

  1. 49.1 Claude Opus 4.6
  2. 49.1 Gemini 3.1 Pro
  3. 45.6 GPT-5.2 Codex
  4. 43.9 Gemini 3 Pro
  5. 42.1 GPT-5.3 Codex
  6. 36.8 Claude Opus 4.5
  7. 36.8 Gemini 3 Flash
  8. 26.3 Claude Sonnet 4.5
  9. 24.6 GLM-4.7
  10. 24.6 Kimi K2.5
  11. 21.1 Gemini 2.5 Pro
  12. 19.3 GLM-5
  13. 17.5 GPT-5.2
  14. 17.5 GPT-5.4
  15. 14.0 Claude Sonnet 4
  16. 12.3 DeepSeek-V3.2
  17. 10.5 Claude Haiku 4.5
  18. 8.77 Grok 4
  19. 7.02 Grok 4.1 Fast
  20. 1.75 GPT-5
  1. 1 Claude Opus 4.6 49.1%
  2. 2 Gemini 3.1 Pro 49.1%
  3. 3 GPT-5.2 Codex xhigh 45.6%
  4. 4 Gemini 3 Pro 43.9%
  5. 5 GPT-5.3 Codex 42.1%
  6. 6 Claude Opus 4.5 36.8%
  7. 7 Gemini 3 Flash 36.8%
  8. 8 Claude Sonnet 4.5 26.3%
  9. 9 GLM-4.7 24.6%
  10. 10 Kimi K2.5 24.6%
  11. 11 Gemini 2.5 Pro Jun 2025 21.1%
  12. 12 GLM-5 19.3%
  13. 13 GPT-5.2 17.5%
  14. 14 GPT-5.4 xhigh 17.5%
  15. 15 Claude Sonnet 4 14%
  16. 16 DeepSeek-V3.2 12.3%
  17. 17 Claude Haiku 4.5 10.5%
  18. 18 Grok 4 8.77%
  19. 19 Grok 4.1 Fast 7.02%
  20. 20 GPT-5 1.75%
#ModelScoreSettingDateSource
1 Claude Opus 4.6 49.1% Feb 2026 Official leaderboard
2 Gemini 3.1 Pro 49.1% Feb 2026 Official leaderboard
3 GPT-5.2 Codex 45.6% xhigh Dec 2025 Official leaderboard
4 Gemini 3 Pro 43.9% Nov 2025 Official leaderboard
5 GPT-5.3 Codex 42.1% Feb 2026 Official leaderboard
6 Claude Opus 4.5 36.8% Nov 2025 Official leaderboard
7 Gemini 3 Flash 36.8% Dec 2025 Official leaderboard
8 Claude Sonnet 4.5 26.3% Sep 2025 Official leaderboard
9 GLM-4.7 24.6% Dec 2025 Official leaderboard
10 Kimi K2.5 24.6% Jan 2026 Official leaderboard
11 Gemini 2.5 Pro Jun 2025 21.1% Jun 2025 Official leaderboard
12 GLM-5 19.3% Feb 2026 Official leaderboard
13 GPT-5.2 17.5% Dec 2025 Official leaderboard
14 GPT-5.4 17.5% xhigh Mar 2026 Official leaderboard
15 Claude Sonnet 4 14% May 2025 Official leaderboard
16 DeepSeek-V3.2 12.3% Dec 2025 Official leaderboard
17 Claude Haiku 4.5 10.5% Oct 2025 Official leaderboard
18 Grok 4 8.77% Jul 2025 Official leaderboard
19 Grok 4.1 Fast 7.02% Nov 2025 Official leaderboard
20 GPT-5 1.75% Aug 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
GPT-5.2 Codex
high xhigh
45.6%
GPT-5.3 Codex
high
42.1%
GPT-5.4
high xhigh
17.5%
ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 4.6 49.1%
Gemini 3.1 Pro 49.1%
GPT-5.2 Codex xhigh 45.6%
Gemini 3 Pro 43.9%
GPT-5.3 Codex 42.1%
Claude Opus 4.5 36.8%
Gemini 3 Flash 36.8%
GPT-5.2 Codex high 35.1%
Claude Sonnet 4.5 26.3%
GLM-4.7 24.6%
Kimi K2.5 24.6%
Gemini 2.5 Pro Jun 2025 21.1%
GLM-5 19.3%
GPT-5.2 17.5%
GPT-5.4 xhigh 17.5%
Claude Sonnet 4 14%
GPT-5.4 high 14%
DeepSeek-V3.2 12.3%
GPT-5.3 Codex high 12.3%
Claude Haiku 4.5 10.5%
GPT-5.3 Codex 10.5%
Grok 4 8.77%
GPT-5.2 Codex 8.77%
Grok 4.1 Fast 7.02%
GPT-5 1.75%
GPT-5.4 1.75%

3 models have held number one since its release in Dec 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind OpenAI Anthropic
Own leaderboard Not tested since Mar 2026
Claude Opus 4.6 49.1%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 4.6 Gemini 3 Pro Claude Sonnet 4.5 Gemini 2.5 Pro Claude Sonnet 4 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 4.6 Claude Sonnet 4.5

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Cybersecurity

All 111 in Cybersecurity
  • ExploitGym In use 869 real-vulnerability exploit-development tasks (userspace, V8, Linux kernel). Success means capturing a flag through the intended vulnerability. GPT-6 Astra 42.4%
  • ExploitBench In use 41 recent V8 JavaScript-engine N-day vulnerabilities. Progress measured by 16 grader-verified capability flags across five tiers. GPT-6 Astra 100%
  • CWE-bench In use Agents audit a repository and write a patch for a security flaw, without being told where the bug is. Covers six programming languages. Gemini 4 Carbon 71%
  • Real-world vulnerability discovery In use Agents search popular open-source codebases to rediscover about 1,200 recent confirmed vulnerabilities across many languages. Reports share discovered. Gemini 4 Argon 86%
  • Artificial Analysis Cyber Index In use Agents do enterprise cyber defense tasks from three benchmarks, CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, combined into one success rate. Mistral Large 4 (Le Chonk, open weights) 51%
  • Big Sleep Evaluation (Google) In use Agents search large real codebases such as Chrome and Safari for critical, hard to find vulnerabilities. Scored by pass@1 success rate, run without safety guardrails. Gemini 3.5 Flash Cyber 72%