o11y-bench
Agents query metrics, logs and traces in a live Grafana stack, investigate incidents and fix dashboards. Grading checks claims against facts read from the same environment.
64 tasks built on Harbor, scored pass^3. Version 1.1 (2026-10-28) made grading more grounded and reproducible, so check comparability with 1.0 scores.
- Level
- Expert Specialists with years of training, often over hours of work
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Claude Opus 5.5 leads with 93.8%
Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.
- 1 Claude Opus 5.5 medium 93.8%
- 2 Claude Sonnet 5.5 xhigh 90.6%
- 3 GPT-6 Astra low 87.5%
- 4 GPT-6.1 Sol medium 87.5%
- 5
Gemini 4 Argon low 87.5% - 6 Claude Haiku 5.5 xhigh 87.5%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | 93.8% | medium | Sep 2026 | Official leaderboard |
| 2 | Claude Sonnet 5.5 | 90.6% | xhigh | Sep 2026 | Official leaderboard |
| 3 | GPT-6 Astra | 87.5% | low | Sep 2026 | Official leaderboard |
| 4 | GPT-6.1 Sol | 87.5% | medium | Sep 2026 | Official leaderboard |
| 5 | Gemini 4 Argon | 87.5% | low | Sep 2026 | Official leaderboard |
| 6 | Claude Haiku 5.5 | 87.5% | xhigh | Oct 2026 | Official leaderboard |
The best score, 93.8%, costs $0.26 a task
Runs on the cost frontier score higher than any run that's cheaper. Claude Sonnet 5.5 at high comes within 4.7 points of the best for $0.11 a task, about half the cost. The costs are the leaderboard's own estimates, so compare them only within this benchmark.
| Model | Effort | Harness | Score | Cost per task | Output tokens |
|---|---|---|---|---|---|
| Claude Opus 5.5 | medium | 93.8% | $0.26 | ||
| Claude Sonnet 5.5 | xhigh | 90.6% | $0.29 | ||
| Claude Sonnet 5.5 | high | 89.1% | $0.11 | ||
| GPT-6 Astra | low | 87.5% | $0.20 | ||
| Claude Opus 5.5 | max | 87.5% | $1.69 | ||
| GPT-6.1 Sol | medium | 87.5% | $0.04 | ||
| Gemini 4 Argon | low | 87.5% | $0.27 | ||
| Claude Haiku 5.5 | xhigh | 87.5% | $0.06 | ||
| Claude Opus 5.5 | low | 85.9% | $0.13 | ||
| Claude Haiku 5.5 | high | 85.9% | $0.02 |
2 models have held number one since its release in Apr 2026
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Sep 2026
All the scores we have, one dot each, so a model tested at several settings appears more than once.
No lab model card reports it yet
We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.
More in Terminal agents
- Terminal-Bench 4.0 In use Agents complete 66 hard tasks in containerized terminals across software, ML, science, security, and operations. Test scripts verify each result, scored as resolution rate. Claude Opus 5.5 64.8%
- SkillsBench In use 86-87 containerized expert tasks in 11 domains, run with and without Agent Skills. Verifier tests give pass rate and skill lift. GPT-5.5 67.3%
- Terminal-Bench Hard In use Agents complete 44 hard tasks from Terminal-Bench Core in the Terminus 2 harness. Artificial Analysis scores pass@1 averaged over 3 repeats. Gemma 4 31B IT 36%
- Harbor-Index In use Agents solve about 80 hard tasks drawn from 29 benchmarks run through Harbor adapters, such as HLE, GSO and SWE-bench Pro. Each task keeps its source verifier. GPT-5.5 28.1%
- CompileBench Not in model cards Agents build real open-source projects through dependency problems, legacy toolchains and complex build systems, for example compiling 22-year-old code. Claude Opus 4.1 100%
- Long-Horizon-Terminal-Bench 2.0 Not in model cards Agents work for up to 3 hours per task on 77 container tasks, including 30 tasks that rebuild a scientific tool from its outputs. Hidden verifiers give partial-credit rewards. GPT-6 Astra 69.2%