PinchBench
KAT-Coder-Pro V2.5 94.9% leads since Jul 2026 In use Real-world tasks (scheduling, email triage, coding, research, files) for an LLM driving an OpenClaw agent. Graded by automated checks and LLM judge.
v1.x grew from 23 to 53 tasks. PinchBench 2.0 (2026-05-06) expanded to 148 tasks and changed the default judge, so it has its own record.
- Level
- Everyday Most adults, without training
- Scored
- Graded by a model A language model scores each response, usually against a rubric.
- Answers
- Agent task Many steps in a terminal, browser, desktop, app, game or robot.
KAT-Coder-Pro V2.5 leads with 94.9%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 KAT-Coder-Pro V2.5 94.9%
- 2 KAT-Coder-V2.5-Dev 93.4%
- 3
Trinity Large Thinking 91.9% - 4
Muse Glimmer 90.5% - 5
Nemotron 3 Ultra 90% - 6 Nemotron 3.5 Bolt 74B-A9B 87.8%
- 7
Ring-2.6-1T 87.6% - 8 Ling-2.6-1T 87.4%
- 9 Nemotron 3.5 Lightning 30B-A3B 85.4%
- 10 LFM2.5-24B-A2B 83.2%
- 11 MiMo-V2-Omni 81.2%
- 12 MiMo-V2-Pro 81%
- 13 LFM2.5-2.6B 68.2%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | KAT-Coder-Pro V2.5 | 94.9% | Jul 2026 | Model card | |
| 2 | KAT-Coder-V2.5-Dev | 93.4% | Jul 2026 | Model card | |
| 3 | Trinity Large Thinking | 91.9% | Apr 2026 | Model card | |
| 4 | Muse Glimmer | 90.5% | Oct 2026 | Model card | |
| 5 | Nemotron 3 Ultra | 90% | Jun 2026 | Model card | |
| 6 | Nemotron 3.5 Bolt 74B-A9B | 87.8% | Oct 2026 | Model card | |
| 7 | Ring-2.6-1T | 87.6% | May 2026 | Model card | |
| 8 | Ling-2.6-1T | 87.4% | Apr 2026 | Model card | |
| 9 | Nemotron 3.5 Lightning 30B-A3B | 85.4% | Aug 2026 | Model card | |
| 10 | LFM2.5-24B-A2B | 83.2% | Oct 2026 | Model card | |
| 11 | MiMo-V2-Omni | 81.2% | Mar 2026 | Model card | |
| 12 | MiMo-V2-Pro | 81% | Mar 2026 | Model card | |
| 13 | LFM2.5-2.6B | 68.2% | Aug 2026 | Model card |
The benchmark's own leaderboard has 49 entries, 48 of them models we track. We plot every source in the charts below.
Number one over time
Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.
The best score passed 85% in Dec 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
3 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 15 Oct 2026 | Nemotron 3.5 Bolt 74B-A9B | 87.8 | PinchBench |
| 13 Oct 2026 | LFM2.5-24B-A2B | 83.2 | PinchBench |
| 11 Aug 2026 | Nemotron 3.5 Lightning 30B-A3B | 85.4 | PinchBench |
| 4 Aug 2026 | LFM2.5-2.6B | 68.2 | PinchBench |
| 23 Jul 2026 | KAT-Coder-V2.5-Dev | 93.4 | PinchBench |
| 6 Jul 2026 | KAT-Coder-Pro V2.5 | 94.9 | PinchBench(Avg) |
| 4 Jun 2026 | Nemotron 3 Ultra | 90 | PinchBench |
| 14 May 2026 | Ring-2.6-1T | 87.6 | PinchBench (high) |
| 29 Apr 2026 | Ling-2.6-1T | 87.4 | PinchBench (Avg. Score) |
| 1 Apr 2026 | Trinity Large Thinking | 91.9 | PinchBench |
| 18 Mar 2026 | MiMo-V2-Omni | 81.2 | PinchBench (avg.) |
| 18 Mar 2026 | MiMo-V2-Pro | 81 | PinchBench (avg.) |
6 system cards report it, from 3 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 19 Oct 2026 | Muse Glimmer technical report | 90.5 | PinchBench, 24 tasks, high (Table 5) | Capabilities |
| 15 Oct 2026 | Nemotron 3.5 Lightning 30B-A3B technical report | 83.9 | PinchBench, BF16 acc. (Table 22) | Capabilities |
| 15 Oct 2026 | Nemotron 3.5 Bolt 74B-A9B technical report | 87.8 | PinchBench, released 24GB checkpoint (Table 7). BF16 90.35 | Capabilities |
| 13 Jun 2026 | Ring-2.6-1T technical report | 87.6 | PinchBench (Avg@3), high, no xhigh value (Table 6) | Capabilities |
| 13 Jun 2026 | Ling-2.6-1T technical report | 85.2 | PinchBench, Mean@5 (Table 5) | Capabilities |
| 9 Jun 2026 | Nemotron 3 Ultra technical report | 90 | PinchBench, held-out gate (Table 10) | Capabilities |
Versions
We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.
More in Tool use
- AutomationBench In use Agent completes end-to-end business workflows inside a simulated company exposing dozens of REST APIs. Graded on resulting state.
Gemini 4 Carbon 53.9% - MCP Atlas In use Multi-step workflows over real MCP servers. Scores tool-use competency on realistic tasks. Muse Spark 1.1 88.1%
- BFCL V4 In use Models call functions and tools, including multi-turn agentic tasks with web search and memory. Checked against expected calls and outcomes.
Qwen3.7-Max 75% - τ³-bench Banking In use Agents act as banking customer-service reps over many turns with a simulated user, following policy. Pass^1 on the final database state. Qwen 3.8 Max 55.1%
- Claw-Eval In use 300 human-verified agent tasks (service orchestration, multimodal, multi-turn dialogue) audited via traces and snapshots. Rubric scores with Pass^3 consistency. Claude Sonnet 4.6 81.4%
- DeepSearchQA In use Questions that need multi-step search and synthesis. Final answers graded against references. Kimi K3 95%