Benchmark Almanac

CompileBench

Claude Opus 4.1 100% leads since Aug 2025 Not in model cards

Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code.

About 60 GitHub stars. Quesma also contributed to Terminal-Bench 3.0.

Made by Quesma Released Aug 2025 Terminal tasks All tasks public Last model added Oct 2025 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
QuesmaOrg/CompileBench 61 stars Leaderboard quesma.com
20
Models scored on its own leaderboard

Claude Opus 4.1 leads with 100% among models released by Oct 2025

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

The newest model with a score here came out in Oct 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

20 models. Scroll sideways for the rest.

  1. 100.0 Claude Opus 4.1
  2. 93.3 Claude Sonnet 4
  3. 93.3 GPT-5
  4. 93.3 GPT-5-Codex
  5. 86.7 Grok 4
  6. 86.7 Claude Sonnet 4.5
  7. 86.7 Claude Haiku 4.5
  8. 80.0 GPT-5 mini
  9. 80.0 DeepSeek-V3.1
  10. 80.0 Kimi K2
  11. 73.3 GLM-4.5
  12. 73.3 grok-code-fast-1
  13. 73.3 Grok 4 Fast
  14. 73.3 DeepSeek-V3.1-Terminus
  15. 66.7 GPT-4.1 mini
  16. 66.7 Qwen3-Max
  17. 60.0 GPT-4.1
  18. 60.0 Gemini 2.5 Pro
  19. 60.0 gpt-oss-120b
  20. 60.0 Gemini 2.5 Flash
  1. 1 Claude Opus 4.1 thinking 16k 100%
  2. 2 Claude Sonnet 4 thinking 16k 93.3%
  3. 3 GPT-5 high 93.3%
  4. 4 GPT-5-Codex high 93.3%
  5. 5 Grok 4 86.7%
  6. 6 Claude Sonnet 4.5 86.7%
  7. 7 Claude Haiku 4.5 thinking 16k 86.7%
  8. 8 GPT-5 mini high 80%
  9. 9 DeepSeek-V3.1 80%
  10. 10 Kimi K2 Sep 2025 80%
  11. 11 GLM-4.5 73.3%
  12. 12 grok-code-fast-1 73.3%
  13. 13 Grok 4 Fast 73.3%
  14. 14 DeepSeek-V3.1-Terminus 73.3%
  15. 15 GPT-4.1 mini 66.7%
  16. 16 Qwen3-Max 66.7%
  17. 17 GPT-4.1 60%
  18. 18 Gemini 2.5 Pro Jun 2025 60%
  19. 19 gpt-oss-120b high 60%
  20. 20 Gemini 2.5 Flash Sep 2025 60%
#ModelScoreSettingDateSource
1 Claude Opus 4.1 100% thinking 16k Aug 2025 Official leaderboard
2 Claude Sonnet 4 93.3% thinking 16k May 2025 Official leaderboard
3 GPT-5 93.3% high Aug 2025 Official leaderboard
4 GPT-5-Codex 93.3% high Sep 2025 Official leaderboard
5 Grok 4 86.7% Jul 2025 Official leaderboard
6 Claude Sonnet 4.5 86.7% Sep 2025 Official leaderboard
7 Claude Haiku 4.5 86.7% thinking 16k Oct 2025 Official leaderboard
8 GPT-5 mini 80% high Aug 2025 Official leaderboard
9 DeepSeek-V3.1 80% Aug 2025 Official leaderboard
10 Kimi K2 Sep 2025 80% Sep 2025 Official leaderboard
11 GLM-4.5 73.3% Aug 2025 Official leaderboard
12 grok-code-fast-1 73.3% Aug 2025 Official leaderboard
13 Grok 4 Fast 73.3% Sep 2025 Official leaderboard
14 DeepSeek-V3.1-Terminus 73.3% Sep 2025 Official leaderboard
15 GPT-4.1 mini 66.7% Apr 2025 Official leaderboard
16 Qwen3-Max 66.7% Sep 2025 Official leaderboard
17 GPT-4.1 60% Apr 2025 Official leaderboard
18 Gemini 2.5 Pro Jun 2025 60% Jun 2025 Official leaderboard
19 gpt-oss-120b 60% high Aug 2025 Official leaderboard
20 Gemini 2.5 Flash Sep 2025 60% Sep 2025 Official leaderboard

More effort adds a median 33 points

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Claude Sonnet 4
thinking 16k
93.3%
GPT-5
minimal high
93.3%
Claude Sonnet 4.5
thinking 16k
86.7%
Claude Haiku 4.5
thinking 16k
86.7%
GPT-5 mini
minimal high
80%
Gemini 2.5 Flash
thinking
60%
ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 4.1 thinking 16k 100%
Claude Sonnet 4 thinking 16k 93.3%
GPT-5 high 93.3%
GPT-5-Codex high 93.3%
Grok 4 86.7%
Claude Sonnet 4.5 86.7%
Claude Sonnet 4.5 thinking 16k 86.7%
Claude Haiku 4.5 thinking 16k 86.7%
Claude Haiku 4.5 86.7%
Claude Sonnet 4 80%
GPT-5 minimal 80%
GPT-5 mini high 80%
DeepSeek-V3.1 80%
Kimi K2 Sep 2025 80%
GLM-4.5 73.3%
grok-code-fast-1 73.3%
Grok 4 Fast 73.3%
DeepSeek-V3.1-Terminus 73.3%
GPT-4.1 mini 66.7%
Qwen3-Max 66.7%
GPT-4.1 60%
Gemini 2.5 Pro Jun 2025 60%
gpt-oss-120b high 60%
Gemini 2.5 Flash Sep 2025 60%
Gemini 2.5 Flash Sep 2025 thinking 53.3%
GPT-5 mini minimal 46.7%

2 models have held number one since its release in Aug 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Passed 85% in Aug 2025, by Claude Sonnet 4
Claude Opus 4.1 100%

The best score passed 85% in May 2025

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude Opus 4.1 Claude Sonnet 4 GPT-4.1 mini 0% 20% 40% 60% 80% 100% 2025 Near the ceiling (85%) Claude Opus 4.1 Claude Sonnet 4

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Terminal agents

All 18 in Terminal agents
  • Terminal-Bench 4.0 In use Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Claude Opus 5.5 64.8%
  • SkillsBench In use 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. GPT-5.5 67.3%
  • Terminal-Bench Hard In use Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Gemma 4 31B IT 36%
  • Harbor-Index In use Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. GPT-5.5 28.1%
  • Long-Horizon-Terminal-Bench 2.0 Not in model cards Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. GPT-6 Astra 69.2%
  • ITBench Not in model cards Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. Gemini 3 Pro 58.3%