Benchmark Almanac

Vals RSI Index

Claude Opus 5.5 0.37 leads since Sep 2026 Not in model cards

Agents work four open AI R&D problems under fixed compute and time budgets: compression, LM training, judge harness work and post-training. Each runs in its vendor harness.

Each task maps a measured result, such as bits per byte on enwik8, onto a score between a baseline and a theoretical best. 25 models on 2026-10-15. Opus 5.5 in Claude Code led compression.

Made by Vals AI Released Aug 2026 AI research tasks Tasks kept private Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Research frontier Researchers, on open problems or tasks that take days
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Leaderboard
25
Models scored on its own leaderboard

Claude Opus 5.5 leads with 0.37

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

25 models. Scroll sideways for the rest.

  1. 0.37 Claude Opus 5.5
  2. 0.36 Claude Fable 5.1
  3. 0.33 Claude Opus 5
  4. 0.30 Gemini 4 Argon
  5. 0.28 GPT-6 Sol
  6. 0.27 GPT-6 Astra
  7. 0.27 GPT-6.1 Sol
  8. 0.26 Grok 4.6
  9. 0.24 Muse Wonder Full
  10. 0.24 Claude Fable 5
  11. 0.24 Claude Opus 4.8
  12. 0.24 GPT-5.6 Sol
  13. 0.23 GLM-5.3
  14. 0.22 Qwen 3.8 Max
  15. 0.21 Kimi K3
  16. 0.20 Grok 4.7
  17. 0.20 Gemini 3.8 Flash
  18. 0.20 Muse Spark 1.3
  19. 0.19 Claude Opus 4.7
  20. 0.18 Gemini 3.7 Flash
  21. 0.17 GPT-5.5
  22. 0.16 GPT-5.2
  23. 0.15 GPT-5.4
  24. 0.15 Claude Opus 4.5
  25. 0.07 Kimi K2.5
  1. 1 Claude Opus 5.5 max 0.37
  2. 2 Claude Fable 5.1 max 0.36
  3. 3 Claude Opus 5 max 0.33
  4. 4 Gemini 4 Argon high 0.30
  5. 5 GPT-6 Sol max 0.28
  6. 6 GPT-6 Astra max 0.27
  7. 7 GPT-6.1 Sol max 0.27
  8. 8 Grok 4.6 xhigh 0.26
  9. 9 Muse Wonder Full max 0.24
  10. 10 Claude Fable 5 max 0.24
  11. 11 Claude Opus 4.8 max 0.24
  12. 12 GPT-5.6 Sol max 0.24
  13. 13 GLM-5.3 max 0.23
  14. 14 Qwen 3.8 Max max 0.22
  15. 15 Kimi K3 max 0.21
  16. 16 Grok 4.7 xhigh 0.20
  17. 17 Gemini 3.8 Flash high 0.20
  18. 18 Muse Spark 1.3 max 0.20
  19. 19 Claude Opus 4.7 max 0.19
  20. 20 Gemini 3.7 Flash high 0.18
  21. 21 GPT-5.5 xhigh 0.17
  22. 22 GPT-5.2 xhigh 0.16
  23. 23 GPT-5.4 xhigh 0.15
  24. 24 Claude Opus 4.5 non-thinking 0.15
  25. 25 Kimi K2.5 max 0.07
#ModelScoreSettingDateSource
1 Claude Opus 5.5 0.37 max, Claude Code Sep 2026 Official leaderboard
2 Claude Fable 5.1 0.36 max, Claude Code Sep 2026 Official leaderboard
3 Claude Opus 5 0.33 max, Claude Code Jul 2026 Official leaderboard
4 Gemini 4 Argon 0.30 high, Antigravity Sep 2026 Official leaderboard
5 GPT-6 Sol 0.28 max, Codex Sep 2026 Official leaderboard
6 GPT-6 Astra 0.27 max, Codex Sep 2026 Official leaderboard
7 GPT-6.1 Sol 0.27 max, Codex Sep 2026 Official leaderboard
8 Grok 4.6 0.26 xhigh, Grok Build Aug 2026 Official leaderboard
9 Muse Wonder Full 0.24 max, Muse Code Oct 2026 Official leaderboard
10 Claude Fable 5 0.24 max, Claude Code Jun 2026 Official leaderboard
11 Claude Opus 4.8 0.24 max, Claude Code May 2026 Official leaderboard
12 GPT-5.6 Sol 0.24 max, Codex Jul 2026 Official leaderboard
13 GLM-5.3 0.23 max, Claude Code Aug 2026 Official leaderboard
14 Qwen 3.8 Max 0.22 max, Qwen Code Aug 2026 Official leaderboard
15 Kimi K3 0.21 max, Kimi Code Jul 2026 Official leaderboard
16 Grok 4.7 0.20 xhigh, Grok Build Sep 2026 Official leaderboard
17 Gemini 3.8 Flash 0.20 high, Antigravity Sep 2026 Official leaderboard
18 Muse Spark 1.3 0.20 max, Muse Code Sep 2026 Official leaderboard
19 Claude Opus 4.7 0.19 max, Claude Code Apr 2026 Official leaderboard
20 Gemini 3.7 Flash 0.18 high, Antigravity Aug 2026 Official leaderboard
21 GPT-5.5 0.17 xhigh, Codex Apr 2026 Official leaderboard
22 GPT-5.2 0.16 xhigh, Codex Dec 2025 Official leaderboard
23 GPT-5.4 0.15 xhigh, Codex Mar 2026 Official leaderboard
24 Claude Opus 4.5 0.15 non-thinking, Claude Code Nov 2025 Official leaderboard
25 Kimi K2.5 0.07 max, Kimi Code Jan 2026 Official leaderboard

The best score, 0.37, costs $412 a task

Runs on the cost frontier score higher than any run that's cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Moonshot AI xAI Z.ai Anthropic Meta OpenAI Google DeepMind Cost frontier One model at each effort level
0.00 0.10 0.20 0.30 0.40 $10 $100 $1,000 $10,000 Cost per task, log scale Claude Opus 5.5, max Gemini 4 Argon, high GPT-6.1 Sol, max GLM-5.3, max Muse Spark 1.3, max Kimi K2.5, max 0.00 0.10 0.20 0.30 0.40 $10 $100 $1,000 $10,000 Cost per task, log scale Claude Opus 5.5, max Gemini 4 Argon, high
ModelEffortHarnessScoreCost per taskOutput tokens
Claude Opus 5.5 max Claude Code 0.37 $412
Claude Fable 5.1 max Claude Code 0.36 $298
Claude Opus 5 max Claude Code 0.33 $419
Gemini 4 Argon high Antigravity 0.30 $180
GPT-6 Sol max Codex 0.28 $665
GPT-6 Astra max Codex 0.27 $760
GPT-6.1 Sol max Codex 0.27 $98
Grok 4.6 xhigh Grok Build 0.26 $88
Muse Wonder Full max Muse Code 0.24
Claude Fable 5 max Claude Code 0.24 $491
Claude Opus 4.8 max Claude Code 0.24 $465
GPT-5.6 Sol max Codex 0.24 $347
GLM-5.3 max Claude Code 0.23 $62
Qwen 3.8 Max max Qwen Code 0.22 $114
Kimi K3 max Kimi Code 0.21 $137
Grok 4.7 xhigh Grok Build 0.20 $237
Gemini 3.8 Flash high Antigravity 0.20 $332
Muse Spark 1.3 max Muse Code 0.20 $49
Claude Opus 4.7 max Claude Code 0.19 $1,003
Gemini 3.7 Flash high Antigravity 0.18 $180
GPT-5.5 xhigh Codex 0.17 $857
GPT-5.2 xhigh Codex 0.16 $102
GPT-5.4 xhigh Codex 0.15 $287
Claude Opus 4.5 non-thinking Claude Code 0.15 $366
Kimi K2.5 max Kimi Code 0.07 $25

3 models have held number one since its release in Aug 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard 3 leaders
Claude Opus 5.5 0.37

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 0.10 0.20 0.30 0.40 2026 0.00 0.10 0.20 0.30 0.40 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in AI research

All 83 in AI research
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
  • OpenAI PRs Only in system cards Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. GPT-5.2 55%