Benchmark Almanac

Terminal-Bench Hard

Gemma 4 31B IT 36% leads since Jul 2026 In use

Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats.

Artificial Analysis retired it in June 2026 in favor of Terminal-Bench 2.1.

Made by Artificial Analysis Released Sep 2025 Terminal tasks All tasks public Last in a launch post Jul 2026 Last model added Jul 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
harbor-framework/terminal-bench-1 2,613 stars Leaderboard artificialanalysis.ai
11
Models scored in lab cards
9
Launch posts list it, from 7 labs
7
System cards report it, from 5 labs

Gemma 4 31B IT leads with 36% among models released by Jul 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Jul 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

  1. 36.0 Gemma 4 31B IT
  2. 33.3 GLM-4.7
  3. 31.1 North Mini Code 1.0
  4. 30.5 MiMo-V2-Flash
  5. 28.3 Solar Open 2
  6. 25.8 Nemotron 3 Super
  7. 25.0 Command A+
  8. 24.0 MiniMax-M2
  9. 14.0 Gemma 4 26B A4B
  10. 8.51 Nemotron 3 Nano 30B-A3B
  11. 4.96 Falcon H1R 7B
  1. 1 Gemma 4 31B IT 36%
  2. 2 GLM-4.7 33.3%
  3. 3 North Mini Code 1.0 31.1%
  4. 4 MiMo-V2-Flash 30.5%
  5. 5 Solar Open 2 28.3%
  6. 6 Nemotron 3 Super 25.8%
  7. 7 Command A+ 25%
  8. 8 MiniMax-M2 24%
  9. 9 Gemma 4 26B A4B 14%
  10. 10 Nemotron 3 Nano 30B-A3B 8.51%
  11. 11 Falcon H1R 7B 4.96%
#ModelScoreSettingDateSource
1 Gemma 4 31B IT 36% Jul 2026 Model card
2 GLM-4.7 33.3% Dec 2025 Model card
3 North Mini Code 1.0 31.1% Jun 2026 Model card
4 MiMo-V2-Flash 30.5% Dec 2025 Model card
5 Solar Open 2 28.3% Jul 2026 Model card
6 Nemotron 3 Super 25.8% Mar 2026 Model card
7 Command A+ 25% May 2026 Model card
8 MiniMax-M2 24% Oct 2025 Model card
9 Gemma 4 26B A4B 14% Jul 2026 Model card
10 Nemotron 3 Nano 30B-A3B 8.51% Dec 2025 Model card
11 Falcon H1R 7B 4.96% Jan 2026 Model card

4 models have held number one since its release in Sep 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

MiniMax Xiaomi Corp Z.ai Google DeepMind
Lab claims Not tested since Jul 2026
Gemma 4 31B IT 36%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Gemma 4 31B IT GLM-4.7 MiniMax-M2 0% 20% 40% 60% 80% 100% 2026 Near the ceiling (85%) Gemma 4 31B IT MiniMax-M2

2 labs put it in a launch post in the last six months

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

MiniMax
NVIDIA
Xiaomi Corp
Z.ai
Technology Innovation Institute
Cohere
Upstage
DateLaunch postScoreAs the post writes it
22 Jul 2026 Solar Open 2 28.3 Terminal Bench Hard
9 Jun 2026 North Mini Code 1.0 31.1 Terminal-Bench Hard (Terminus-2)
20 May 2026 Command A+ 25 Terminal-Bench Hard (Terminus-2, Artificial Analysis method)
11 Mar 2026 Nemotron 3 Super 25.8 Terminal Bench (hard subset)
5 Jan 2026 Falcon H1R 7B 4.96 TB Hard
22 Dec 2025 GLM-4.7 33.3 Terminal Bench Hard
16 Dec 2025 MiMo-V2-Flash 30.5 Terminal-Bench Hard
15 Dec 2025 Nemotron 3 Nano 30B-A3B 8.5 Terminal Bench (hard subset)
27 Oct 2025 MiniMax-M2 24 Terminal-Bench-Hard (AA)

7 system cards report it, from 5 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
24 Jul 2026 Gemma 4 31B IT technical report 36 Terminal Bench Hard, thinking (Table 5) Capabilities
24 Jul 2026 Gemma 4 26B A4B technical report 14 Terminal Bench Hard, thinking (Table 5) Capabilities
22 Jul 2026 Solar Open 2 technical report 28.3 Terminal Bench Hard (Table 3) Capabilities
11 Mar 2026 Nemotron 3 Super technical report 25.8 Terminal Bench (hard subset), 48 tasks (Table 5) Capabilities
6 Jan 2026 MiMo-V2-Flash technical report 30.5 Terminal-Bench Hard (Table 9) Capabilities
5 Jan 2026 Falcon H1R 7B technical report 4.9 TB Hard (Table 5, 4.96 on the launch page) Capabilities
23 Dec 2025 Nemotron 3 Nano 30B-A3B technical report 8.51 Terminal Bench (hard subset) (Table 3) Capabilities

More in Terminal agents

All 18 in Terminal agents
  • Terminal-Bench 4.0 In use Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Claude Opus 5.5 64.8%
  • SkillsBench In use 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. GPT-5.5 67.3%
  • Harbor-Index In use Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. GPT-5.5 28.1%
  • CompileBench Not in model cards Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. Claude Opus 4.1 100%
  • Long-Horizon-Terminal-Bench 2.0 Not in model cards Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. GPT-6 Astra 69.2%
  • ITBench Not in model cards Agents handle IT automation tasks in Kubernetes environments: site reliability incidents, compliance and security checks, and cloud cost work. Gemini 3 Pro 58.3%