Benchmark Almanac

RuneBench

GPT-6 Astra 7.26 leads since September 2026
Not in model cards

Coding agents play RuneScape on an emulated game server by writing TypeScript against the rs-sdk library, with game-wiki notes for reference. Tasks train one skill or gather gold within 15 or 30 minutes. Skill tasks are scored by the peak XP rate in any 15-second window, and a verifier reads the final game state.

Built on Harbor and the LostCity engine. The suite has 16 skill tasks at each of two lengths and 8 gold tasks. The authors switched from total XP to peak XP rate because total XP rewarded simple grinding. They also warn that small samples make model gaps noisy. About 171 GitHub stars.

Released
February 2026
Kind
Video games
Level
Everyday
Scored
Checked automatically
Answers
Agent task
Tasks
All public
Strongest model tested
Claude Fable 5.2 #1 on the capability index
Last model added
November 2026
Scores refreshed
November 12th, 2026
78
Entries on its own leaderboard, 76 models we track

GPT-6 Astra leads with 7.26

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

78 entries. Scroll sideways for the rest.

  1. 7.26 GPT-6 Astra
  2. 6.98 Claude Fable 5.2
  3. 6.91 Gemini 4 Argon
  4. 6.88 GPT-6.1 Sol
  5. 6.66 Claude Opus 5.5
  6. 6.65 GPT-6.1 Terra
  7. 6.48 Kimi K3.1 Pro
  8. 6.44 GPT-6 Sol
  9. 6.28 Grok 4.6
  10. 6.14 Claude Fable 5.1
  11. 6.01 Claude Fable 5
  12. 5.96 Gemini 3.7 Flash
  13. 5.96 Gemini 3.8 Flash
  14. 5.90 GPT-5.6 Sol
  15. 5.88 GPT-5.6 Terra
  16. 5.83 DeepSeek V4.1 Flash 1017
  17. 5.73 Claude Opus 5
  18. 5.70 Grok 4.5
  19. 5.68 DeepSeek V4.1 Flash
  20. 5.68 GPT-5.5
  21. 5.66 Grok 4.7
  22. 5.58 Gemini 3.6 Flash
  23. 5.54 Muse Spark 1.1
  24. 5.52 Claude Sonnet 5.5
  25. 5.40 GLM 5.3 (Prime)
  26. 5.37 Tencent Hy4 preview
  27. 5.35 Gemini 3.5 Flash
  28. 5.35 Qwen 3.8 Max
  29. 5.29 GPT-5.6 Luna
  30. 5.26 Beam 1.0
  31. 5.24 Muse Spark 1.2
  32. 5.20 GLM 5.2 (WandB fp4)
  33. 5.13 GPT-6 Luna
  34. 5.08 Claude Opus 4.8
  35. 5.00 GLM-5.3
  36. 4.98 Muse Wonder
  37. 4.89 Mistral Large 4 (Le Chonk, open weights)
  38. 4.81 Claude Sonnet 5
  39. 4.76 GLM-5.2
  40. 4.76 DeepSeek-V4-Flash
  41. 4.70 Claude Opus 4.7
  42. 4.70 GPT-5.4
  43. 4.69 Gemini 3 Flash
  44. 4.61 Muse Spark Open
  45. 4.59 Muse Spark 1.3
  46. 4.51 Muse Ember
  47. 4.48 Gemini 3.1 Pro
  48. 4.45 Claude Haiku 5.5
  49. 4.42 Claude Opus 4.6
  50. 4.36 Nemotron 3.5 Super VL
  51. 4.30 DeepSeek-V4-Pro
  52. 4.29 GPT-5.3 Codex
  53. 4.20 DeepSeek V4 Flash 0731
  54. 4.18 Kimi K2.7 Code
  55. 4.18 Gemini 3.5 Flash-Lite
  56. 4.14 GPT-5.4 Mini
  57. 4.08 Claude Opus 4.5
  58. 4.07 Qwen3.7-Max
  59. 3.83 GLM-5.3-Flash
  60. 3.80 Gemini 3 Pro
  61. 3.76 Nemotron 3 Ultra
  62. 3.76 Inkling
  63. 3.75 Kimi K3
  64. 3.71 Claude Sonnet 4
  65. 3.66 Grok 4.3
  66. 3.60 Kimi K2.6
  67. 3.20 Claude Sonnet 4.5
  68. 3.18 Claude Sonnet 4.6
  69. 2.82 Laguna S 2.1
  70. 2.79 Qwen 3.8 27B
  71. 2.35 GPT-5.4 Nano
  72. 2.10 Kimi K2.5
  73. 1.90 GLM-5
  74. 1.64 Claude Haiku 4.5
  75. 1.57 Gemma 4 31B IT
  76. 1.45 gpt-oss-120b
  77. 1.40 Qwen3-Max
  78. 0.75 Qwen3.5-35B-A3B
  1. 1 GPT-6 Astra 30 minutes; high 7.26
  2. 2 Claude Fable 5.2 30 minutes; xhigh 6.98
  3. 3 Gemini 4 Argon 30 minutes 6.91
  4. 4 GPT-6.1 Sol 30 minutes; high 6.88
  5. 5 Claude Opus 5.5 30 minutes; xhigh 6.66
  6. 6 GPT-6.1 Terra 30 minutes 6.65
  7. 7 Kimi K3.1 Pro 30 minutes 6.48
  8. 8 GPT-6 Sol 30 minutes; high 6.44
  9. 9 Grok 4.6 30 minutes 6.28
  10. 10 Claude Fable 5.1 30 minutes; xhigh 6.14
  11. 11 Claude Fable 5 30 minutes 6.01
  12. 12 Gemini 3.7 Flash 30 minutes 5.96
  13. 13 Gemini 3.8 Flash 30 minutes 5.96
  14. 14 GPT-5.6 Sol 30 minutes; xhigh 5.90
  15. 15 GPT-5.6 Terra 30 minutes; xhigh 5.88
  16. 16 DeepSeek V4.1 Flash 1017 30 minutes 5.83
  17. 17 Claude Opus 5 30 minutes 5.73
  18. 18 Grok 4.5 30 minutes 5.70
  19. 19 DeepSeek V4.1 Flash 30 minutes 5.68
  20. 20 GPT-5.5 30 minutes; xhigh 5.68
  21. 21 Grok 4.7 30 minutes 5.66
  22. 22 Gemini 3.6 Flash 30 minutes 5.58
  23. 23 Muse Spark 1.1 30 minutes 5.54
  24. 24 Claude Sonnet 5.5 30 minutes; xhigh 5.52
  25. 25 GLM 5.3 (Prime) 30 minutes 5.40
  26. 26 Tencent Hy4 preview 30 minutes 5.37
  27. 27 Gemini 3.5 Flash 30 minutes 5.35
  28. 28 Qwen 3.8 Max 30 minutes 5.35
  29. 29 GPT-5.6 Luna 30 minutes; xhigh 5.29
  30. 30 Beam 1.0 30 minutes 5.26
  31. 31 Muse Spark 1.2 30 minutes 5.24
  32. 32 GLM 5.2 (WandB fp4) 30 minutes 5.20
  33. 33 GPT-6 Luna 30 minutes; high 5.13
  34. 34 Claude Opus 4.8 30 minutes; max 5.08
  35. 35 GLM-5.3 30 minutes 5.00
  36. 36 Muse Wonder 30 minutes 4.98
  37. 37 Mistral Large 4 (Le Chonk, open weights) 30 minutes 4.89
  38. 38 Claude Sonnet 5 30 minutes; xhigh 4.81
  39. 39 GLM-5.2 30 minutes 4.76
  40. 40 DeepSeek-V4-Flash 30 minutes 4.76
  41. 41 Claude Opus 4.7 30 minutes; xhigh 4.70
  42. 42 GPT-5.4 30 minutes 4.70
  43. 43 Gemini 3 Flash 30 minutes 4.69
  44. 44 Muse Spark Open 30 minutes 4.61
  45. 45 Muse Spark 1.3 30 minutes 4.59
  46. 46 Muse Ember 30 minutes 4.51
  47. 47 Gemini 3.1 Pro 30 minutes 4.48
  48. 48 Claude Haiku 5.5 30 minutes; high 4.45
  49. 49 Claude Opus 4.6 30 minutes 4.42
  50. 50 Nemotron 3.5 Super VL 30 minutes 4.36
  51. 51 DeepSeek-V4-Pro 30 minutes 4.30
  52. 52 GPT-5.3 Codex 30 minutes 4.29
  53. 53 DeepSeek V4 Flash 0731 30 minutes 4.20
  54. 54 Kimi K2.7 Code 30 minutes 4.18
  55. 55 Gemini 3.5 Flash-Lite 30 minutes 4.18
  56. 56 GPT-5.4 Mini 30 minutes 4.14
  57. 57 Claude Opus 4.5 30 minutes 4.08
  58. 58 Qwen3.7-Max 30 minutes 4.07
  59. 59 GLM-5.3-Flash 30 minutes 3.83
  60. 60 Gemini 3 Pro 30 minutes 3.80
  61. 61 Nemotron 3 Ultra 30 minutes 3.76
  62. 62 Inkling 30 minutes 3.76
  63. 63 Kimi K3 30 minutes 3.75
  64. 64 Claude Sonnet 4 30 minutes 3.71
  65. 65 Grok 4.3 30 minutes 3.66
  66. 66 Kimi K2.6 30 minutes 3.60
  67. 67 Claude Sonnet 4.5 30 minutes 3.20
  68. 68 Claude Sonnet 4.6 30 minutes 3.18
  69. 69 Laguna S 2.1 30 minutes 2.82
  70. 70 Qwen 3.8 27B 30 minutes 2.79
  71. 71 GPT-5.4 Nano 30 minutes 2.35
  72. 72 Kimi K2.5 30 minutes 2.10
  73. 73 GLM-5 30 minutes 1.90
  74. 74 Claude Haiku 4.5 30 minutes 1.64
  75. 75 Gemma 4 31B IT 30 minutes 1.57
  76. 76 gpt-oss-120b 30 minutes 1.45
  77. 77 Qwen3-Max 30 minutes 1.40
  78. 78 Qwen3.5-35B-A3B 30 minutes 0.75

Source: Official RuneBench leaderboard

#EntryScoreSettingDateSource
1 GPT-6 Astra 7.26 30 minutes; high Sep 2026 Official leaderboard
2 Claude Fable 5.2 6.98 30 minutes; xhigh Nov 2026 Official leaderboard
3 Gemini 4 Argon 6.91 30 minutes Sep 2026 Official leaderboard
4 GPT-6.1 Sol 6.88 30 minutes; high Sep 2026 Official leaderboard
5 Claude Opus 5.5 6.66 30 minutes; xhigh Sep 2026 Official leaderboard
6 GPT-6.1 Terra 6.65 30 minutes Oct 2026 Official leaderboard
7 Kimi K3.1 Pro 6.48 30 minutes Oct 2026 Official leaderboard
8 GPT-6 Sol 6.44 30 minutes; high Sep 2026 Official leaderboard
9 Grok 4.6 6.28 30 minutes Aug 2026 Official leaderboard
10 Claude Fable 5.1 6.14 30 minutes; xhigh Sep 2026 Official leaderboard
11 Claude Fable 5 6.01 30 minutes Jun 2026 Official leaderboard
12 Gemini 3.7 Flash 5.96 30 minutes Aug 2026 Official leaderboard
13 Gemini 3.8 Flash 5.96 30 minutes Sep 2026 Official leaderboard
14 GPT-5.6 Sol 5.90 30 minutes; xhigh Jul 2026 Official leaderboard
15 GPT-5.6 Terra 5.88 30 minutes; xhigh Jul 2026 Official leaderboard
16 DeepSeek V4.1 Flash 1017 5.83 30 minutes Oct 2026 Official leaderboard
17 Claude Opus 5 5.73 30 minutes Jul 2026 Official leaderboard
18 Grok 4.5 5.70 30 minutes Jul 2026 Official leaderboard
19 DeepSeek V4.1 Flash 5.68 30 minutes Sep 2026 Official leaderboard
20 GPT-5.5 5.68 30 minutes; xhigh Apr 2026 Official leaderboard
21 Grok 4.7 5.66 30 minutes Sep 2026 Official leaderboard
22 Gemini 3.6 Flash 5.58 30 minutes Jul 2026 Official leaderboard
23 Muse Spark 1.1 5.54 30 minutes Jul 2026 Official leaderboard
24 Claude Sonnet 5.5 5.52 30 minutes; xhigh Sep 2026 Official leaderboard
25 GLM 5.3 (Prime) 5.40 30 minutes Official leaderboard
26 Tencent Hy4 preview 5.37 30 minutes Aug 2026 Official leaderboard
27 Gemini 3.5 Flash 5.35 30 minutes May 2026 Official leaderboard
28 Qwen 3.8 Max 5.35 30 minutes Aug 2026 Official leaderboard
29 GPT-5.6 Luna 5.29 30 minutes; xhigh Jul 2026 Official leaderboard
30 Beam 1.0 5.26 30 minutes Oct 2026 Official leaderboard
31 Muse Spark 1.2 5.24 30 minutes Aug 2026 Official leaderboard
32 GLM 5.2 (WandB fp4) 5.20 30 minutes Official leaderboard
33 GPT-6 Luna 5.13 30 minutes; high Sep 2026 Official leaderboard
34 Claude Opus 4.8 5.08 30 minutes; max May 2026 Official leaderboard
35 GLM-5.3 5.00 30 minutes Aug 2026 Official leaderboard
36 Muse Wonder 4.98 30 minutes Oct 2026 Official leaderboard
37 Mistral Large 4 (Le Chonk, open weights) 4.89 30 minutes Oct 2026 Official leaderboard
38 Claude Sonnet 5 4.81 30 minutes; xhigh Jun 2026 Official leaderboard
39 GLM-5.2 4.76 30 minutes Jun 2026 Official leaderboard
40 DeepSeek-V4-Flash 4.76 30 minutes Apr 2026 Official leaderboard
41 Claude Opus 4.7 4.70 30 minutes; xhigh Apr 2026 Official leaderboard
42 GPT-5.4 4.70 30 minutes Mar 2026 Official leaderboard
43 Gemini 3 Flash 4.69 30 minutes Dec 2025 Official leaderboard
44 Muse Spark Open 4.61 30 minutes Oct 2026 Official leaderboard
45 Muse Spark 1.3 4.59 30 minutes Sep 2026 Official leaderboard
46 Muse Ember 4.51 30 minutes Nov 2026 Official leaderboard
47 Gemini 3.1 Pro 4.48 30 minutes Feb 2026 Official leaderboard
48 Claude Haiku 5.5 4.45 30 minutes; high Oct 2026 Official leaderboard
49 Claude Opus 4.6 4.42 30 minutes Feb 2026 Official leaderboard
50 Nemotron 3.5 Super VL 4.36 30 minutes Oct 2026 Official leaderboard
51 DeepSeek-V4-Pro 4.30 30 minutes Apr 2026 Official leaderboard
52 GPT-5.3 Codex 4.29 30 minutes Feb 2026 Official leaderboard
53 DeepSeek V4 Flash 0731 4.20 30 minutes Jul 2026 Official leaderboard
54 Kimi K2.7 Code 4.18 30 minutes Jun 2026 Official leaderboard
55 Gemini 3.5 Flash-Lite 4.18 30 minutes Jul 2026 Official leaderboard
56 GPT-5.4 Mini 4.14 30 minutes Mar 2026 Official leaderboard
57 Claude Opus 4.5 4.08 30 minutes Nov 2025 Official leaderboard
58 Qwen3.7-Max 4.07 30 minutes May 2026 Official leaderboard
59 GLM-5.3-Flash 3.83 30 minutes Aug 2026 Official leaderboard
60 Gemini 3 Pro 3.80 30 minutes Nov 2025 Official leaderboard
61 Nemotron 3 Ultra 3.76 30 minutes Jun 2026 Official leaderboard
62 Inkling 3.76 30 minutes Jul 2026 Official leaderboard
63 Kimi K3 3.75 30 minutes Jul 2026 Official leaderboard
64 Claude Sonnet 4 3.71 30 minutes May 2025 Official leaderboard
65 Grok 4.3 3.66 30 minutes Apr 2026 Official leaderboard
66 Kimi K2.6 3.60 30 minutes Apr 2026 Official leaderboard
67 Claude Sonnet 4.5 3.20 30 minutes Sep 2025 Official leaderboard
68 Claude Sonnet 4.6 3.18 30 minutes Feb 2026 Official leaderboard
69 Laguna S 2.1 2.82 30 minutes Jul 2026 Official leaderboard
70 Qwen 3.8 27B 2.79 30 minutes Aug 2026 Official leaderboard
71 GPT-5.4 Nano 2.35 30 minutes Mar 2026 Official leaderboard
72 Kimi K2.5 2.10 30 minutes Jan 2026 Official leaderboard
73 GLM-5 1.90 30 minutes Feb 2026 Official leaderboard
74 Claude Haiku 4.5 1.64 30 minutes Oct 2025 Official leaderboard
75 Gemma 4 31B IT 1.57 30 minutes Apr 2026 Official leaderboard
76 gpt-oss-120b 1.45 30 minutes Aug 2025 Official leaderboard
77 Qwen3-Max 1.40 30 minutes Sep 2025 Official leaderboard
78 Qwen3.5-35B-A3B 0.75 30 minutes Feb 2026 Official leaderboard

Skill performance vs cost

The line joins the runs that no other run beats on both measures. We redrew this chart from the numbers on the official leaderboard.

0 2 4 6 8 $0.01$0.1$1$10$100 Mean cost per skill run ($), log scale, lower is better Mean ln(1 + peak XP/min) Claude Sonnet 4 (30 minutes): Mean ln(1 + peak XP/min) 3.71, mean cost per skill run $1.82 gpt-oss-120b (30 minutes): Mean ln(1 + peak XP/min) 1.45, mean cost per skill run $0.2 Qwen3-Max (30 minutes): Mean ln(1 + peak XP/min) 1.4, mean cost per skill run $0.44 Claude Sonnet 4.5 (30 minutes): Mean ln(1 + peak XP/min) 3.2, mean cost per skill run $2.57 Claude Haiku 4.5 (30 minutes): Mean ln(1 + peak XP/min) 1.64, mean cost per skill run $0.84 Gemini 3 Pro (30 minutes): Mean ln(1 + peak XP/min) 3.8, mean cost per skill run $2.2 Claude Opus 4.5 (30 minutes): Mean ln(1 + peak XP/min) 4.08, mean cost per skill run $3.36 Gemini 3 Flash (30 minutes): Mean ln(1 + peak XP/min) 4.69, mean cost per skill run $0.69 Kimi K2.5 (30 minutes): Mean ln(1 + peak XP/min) 2.1, mean cost per skill run $0.31 Claude Opus 4.6 (30 minutes): Mean ln(1 + peak XP/min) 4.42, mean cost per skill run $3.16 GPT-5.3 Codex (30 minutes): Mean ln(1 + peak XP/min) 4.29, mean cost per skill run $2.47 GLM-5 (30 minutes): Mean ln(1 + peak XP/min) 1.9, mean cost per skill run $0.38 Claude Sonnet 4.6 (30 minutes): Mean ln(1 + peak XP/min) 3.18, mean cost per skill run $3.75 Gemini 3.1 Pro (30 minutes): Mean ln(1 + peak XP/min) 4.48, mean cost per skill run $1.88 Qwen3.5-35B-A3B (30 minutes): Mean ln(1 + peak XP/min) 0.75, mean cost per skill run $0.17 GPT-5.4 (30 minutes): Mean ln(1 + peak XP/min) 4.7, mean cost per skill run $5.54 GPT-5.4 Mini (30 minutes): Mean ln(1 + peak XP/min) 4.14, mean cost per skill run $3.27 GPT-5.4 Nano (30 minutes): Mean ln(1 + peak XP/min) 2.35, mean cost per skill run $0.59 Gemma 4 31B IT (30 minutes): Mean ln(1 + peak XP/min) 1.57, mean cost per skill run $0.18 Claude Opus 4.7 (30 minutes): Mean ln(1 + peak XP/min) 4.57, mean cost per skill run $6.98 Claude Opus 4.7 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 4.7, mean cost per skill run $7.27 Kimi K2.6 (30 minutes): Mean ln(1 + peak XP/min) 3.6, mean cost per skill run $1 GPT-5.5 (30 minutes): Mean ln(1 + peak XP/min) 5.32, mean cost per skill run $8.38 GPT-5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $13.4 DeepSeek-V4-Flash (30 minutes): Mean ln(1 + peak XP/min) 4.76, mean cost per skill run $0.36 DeepSeek-V4-Pro (30 minutes): Mean ln(1 + peak XP/min) 4.3, mean cost per skill run $0.68 Grok 4.3 (30 minutes): Mean ln(1 + peak XP/min) 3.66, mean cost per skill run $1.18 Gemini 3.5 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $4.18 Gemini 3.5 Flash (30 minutes; high): Mean ln(1 + peak XP/min) 4.94, mean cost per skill run $4.24 Qwen3.7-Max (30 minutes): Mean ln(1 + peak XP/min) 4.07, mean cost per skill run $1.26 Claude Opus 4.8 (30 minutes; max): Mean ln(1 + peak XP/min) 5.08, mean cost per skill run $5.53 Claude Opus 4.8 (30 minutes): Mean ln(1 + peak XP/min) 5.03, mean cost per skill run $5.16 Nemotron 3 Ultra (30 minutes): Mean ln(1 + peak XP/min) 3.76, mean cost per skill run $0.83 Claude Fable 5 (30 minutes): Mean ln(1 + peak XP/min) 6.01, mean cost per skill run $8.06 Claude Fable 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $8.65 Kimi K2.7 Code (30 minutes): Mean ln(1 + peak XP/min) 4.18, mean cost per skill run $1.27 GLM-5.2 (30 minutes): Mean ln(1 + peak XP/min) 4.76, mean cost per skill run $2.02 Claude Sonnet 5 (30 minutes): Mean ln(1 + peak XP/min) 4.47, mean cost per skill run $2.51 Claude Sonnet 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 4.81, mean cost per skill run $2.88 Grok 4.5 (30 minutes): Mean ln(1 + peak XP/min) 5.7, mean cost per skill run $2.44 Grok 4.5 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $1.6 GPT-5.6 Luna (30 minutes): Mean ln(1 + peak XP/min) 4.77, mean cost per skill run $0.48 GPT-5.6 Luna (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.29, mean cost per skill run $0.75 GPT-5.6 Sol (30 minutes): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $10.7 GPT-5.6 Sol (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.9, mean cost per skill run $8.71 GPT-5.6 Terra (30 minutes): Mean ln(1 + peak XP/min) 5.52, mean cost per skill run $3.15 GPT-5.6 Terra (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.88, mean cost per skill run $5.22 Muse Spark 1.1 (30 minutes): Mean ln(1 + peak XP/min) 5.54, mean cost per skill run $3.02 Inkling (30 minutes): Mean ln(1 + peak XP/min) 3.76, mean cost per skill run $5.24 Kimi K3 (30 minutes): Mean ln(1 + peak XP/min) 3.75, mean cost per skill run $0.76 Gemini 3.5 Flash-Lite (30 minutes): Mean ln(1 + peak XP/min) 4.18, mean cost per skill run $1.13 Gemini 3.6 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.58, mean cost per skill run $3.81 Laguna S 2.1 (30 minutes): Mean ln(1 + peak XP/min) 2.82, mean cost per skill run $0.17 Claude Opus 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.71, mean cost per skill run $5.14 Claude Opus 5 (30 minutes): Mean ln(1 + peak XP/min) 5.73, mean cost per skill run $4.51 Claude Opus 5 (30 minutes; low): Mean ln(1 + peak XP/min) 5.51, mean cost per skill run $2.95 Claude Opus 5 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.69, mean cost per skill run $3.29 DeepSeek V4 Flash 0731 (30 minutes): Mean ln(1 + peak XP/min) 4.2, mean cost per skill run $0.12 Qwen 3.8 Max (30 minutes): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $1.68 Muse Spark 1.2 (30 minutes): Mean ln(1 + peak XP/min) 5.24, mean cost per skill run $2.25 Grok 4.6 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.62, mean cost per skill run $5.53 Grok 4.6 (30 minutes): Mean ln(1 + peak XP/min) 6.28, mean cost per skill run $5.11 Grok 4.6 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.83, mean cost per skill run $4.76 Gemini 3.7 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.96, mean cost per skill run $2.97 GLM-5.3 (30 minutes): Mean ln(1 + peak XP/min) 5, mean cost per skill run $1 Qwen 3.8 27B (30 minutes): Mean ln(1 + peak XP/min) 2.79, mean cost per skill run $0.87 Tencent Hy4 preview (30 minutes): Mean ln(1 + peak XP/min) 5.37, mean cost per skill run $0.27 Claude Fable 5.1 (30 minutes): Mean ln(1 + peak XP/min) 6, mean cost per skill run $6.48 Claude Fable 5.1 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.14, mean cost per skill run $9.29 Gemini 3.8 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.96, mean cost per skill run $2.63 Muse Spark 1.3 (30 minutes): Mean ln(1 + peak XP/min) 4.59, mean cost per skill run $1.78 GPT-6 Astra (30 minutes): Mean ln(1 + peak XP/min) 6.91, mean cost per skill run $15.6 Grok 4.7 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.41, mean cost per skill run $7.14 Grok 4.7 (30 minutes): Mean ln(1 + peak XP/min) 5.66, mean cost per skill run $6.94 Claude Opus 5.5 (30 minutes): Mean ln(1 + peak XP/min) 6.63, mean cost per skill run $3.36 Claude Opus 5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.66, mean cost per skill run $5.18 GPT-6 Luna (30 minutes): Mean ln(1 + peak XP/min) 4.94, mean cost per skill run $0.12 GPT-6 Luna (30 minutes; high): Mean ln(1 + peak XP/min) 5.13, mean cost per skill run $0.14 GPT-6 Sol (30 minutes; high): Mean ln(1 + peak XP/min) 6.44, mean cost per skill run $4.84 GPT-6 Sol (30 minutes): Mean ln(1 + peak XP/min) 6.42, mean cost per skill run $4.44 Claude Sonnet 5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.52, mean cost per skill run $2.3 Claude Sonnet 5.5 (30 minutes): Mean ln(1 + peak XP/min) 4.82, mean cost per skill run $0.85 Beam 1.0 (30 minutes): Mean ln(1 + peak XP/min) 5.26, mean cost per skill run $0.6 Claude Haiku 5.5 (30 minutes; high): Mean ln(1 + peak XP/min) 4.45, mean cost per skill run $0.15 Muse Wonder (30 minutes): Mean ln(1 + peak XP/min) 4.98, mean cost per skill run $4.18 Muse Spark Open (30 minutes): Mean ln(1 + peak XP/min) 4.61, mean cost per skill run $1.86 Nemotron 3.5 Super VL (30 minutes): Mean ln(1 + peak XP/min) 4.36, mean cost per skill run $0.32 Kimi K3.1 Pro (30 minutes): Mean ln(1 + peak XP/min) 6.48, mean cost per skill run $5.07 Mistral Large 4 (Le Chonk, open weights) (30 minutes): Mean ln(1 + peak XP/min) 4.89, mean cost per skill run $0.41 Claude Fable 5.2 (30 minutes): Mean ln(1 + peak XP/min) 6.74, mean cost per skill run $3.4 Muse Ember (30 minutes): Mean ln(1 + peak XP/min) 4.51, mean cost per skill run $0.17 GLM 5.3 (Prime) (30 minutes): Mean ln(1 + peak XP/min) 5.4, mean cost per skill run $1.84. Not in our model list GLM 5.2 (WandB fp4) (30 minutes): Mean ln(1 + peak XP/min) 5.2, mean cost per skill run $1.74. Not in our model list GLM-5.3-Flash (30 minutes): Mean ln(1 + peak XP/min) 3.83, mean cost per skill run $0.07 GPT-6 Astra (30 minutes; high): Mean ln(1 + peak XP/min) 7.26, mean cost per skill run $15.3 DeepSeek V4.1 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $0.08 GPT-6.1 Sol (30 minutes; low): Mean ln(1 + peak XP/min) 6.75, mean cost per skill run $1.52 GPT-6.1 Sol (30 minutes): Mean ln(1 + peak XP/min) 6.79, mean cost per skill run $1.71 GPT-6.1 Sol (30 minutes; high): Mean ln(1 + peak XP/min) 6.88, mean cost per skill run $1.86 Gemini 4 Argon (30 minutes): Mean ln(1 + peak XP/min) 6.91, mean cost per skill run $4.17 Claude Haiku 5.5 (30 minutes): Mean ln(1 + peak XP/min) 4.3, mean cost per skill run $0.08 DeepSeek V4.1 Flash 1017 (30 minutes): Mean ln(1 + peak XP/min) 5.83, mean cost per skill run $0.09 GPT-6.1 Terra (30 minutes): Mean ln(1 + peak XP/min) 6.65, mean cost per skill run $1.13 Claude Fable 5.2 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.98, mean cost per skill run $8.19 GPT-6 Astra (30 minutes; high)Claude Fable 5.2 (30 minutes; xhigh)Gemini 4 Argon (30 minutes)GPT-6.1 Sol (30 minutes; high)GPT-6.1 Sol (30 minutes)GPT-6.1 Sol (30 minutes; low)GPT-6.1 Terra (30 minutes)DeepSeek V4.1 Flash 1017 (30 minutes)DeepSeek V4.1 Flash (30 minutes)Claude Haiku 5.5 (30 minutes)GLM-5.3-Flash (30 minutes) 0 2 4 6 8 $0.01$0.1$1$10$100 Mean cost per skill run ($), log scale, lower is better Mean ln(1 + peak XP/min) Claude Sonnet 4 (30 minutes): Mean ln(1 + peak XP/min) 3.71, mean cost per skill run $1.82 gpt-oss-120b (30 minutes): Mean ln(1 + peak XP/min) 1.45, mean cost per skill run $0.2 Qwen3-Max (30 minutes): Mean ln(1 + peak XP/min) 1.4, mean cost per skill run $0.44 Claude Sonnet 4.5 (30 minutes): Mean ln(1 + peak XP/min) 3.2, mean cost per skill run $2.57 Claude Haiku 4.5 (30 minutes): Mean ln(1 + peak XP/min) 1.64, mean cost per skill run $0.84 Gemini 3 Pro (30 minutes): Mean ln(1 + peak XP/min) 3.8, mean cost per skill run $2.2 Claude Opus 4.5 (30 minutes): Mean ln(1 + peak XP/min) 4.08, mean cost per skill run $3.36 Gemini 3 Flash (30 minutes): Mean ln(1 + peak XP/min) 4.69, mean cost per skill run $0.69 Kimi K2.5 (30 minutes): Mean ln(1 + peak XP/min) 2.1, mean cost per skill run $0.31 Claude Opus 4.6 (30 minutes): Mean ln(1 + peak XP/min) 4.42, mean cost per skill run $3.16 GPT-5.3 Codex (30 minutes): Mean ln(1 + peak XP/min) 4.29, mean cost per skill run $2.47 GLM-5 (30 minutes): Mean ln(1 + peak XP/min) 1.9, mean cost per skill run $0.38 Claude Sonnet 4.6 (30 minutes): Mean ln(1 + peak XP/min) 3.18, mean cost per skill run $3.75 Gemini 3.1 Pro (30 minutes): Mean ln(1 + peak XP/min) 4.48, mean cost per skill run $1.88 Qwen3.5-35B-A3B (30 minutes): Mean ln(1 + peak XP/min) 0.75, mean cost per skill run $0.17 GPT-5.4 (30 minutes): Mean ln(1 + peak XP/min) 4.7, mean cost per skill run $5.54 GPT-5.4 Mini (30 minutes): Mean ln(1 + peak XP/min) 4.14, mean cost per skill run $3.27 GPT-5.4 Nano (30 minutes): Mean ln(1 + peak XP/min) 2.35, mean cost per skill run $0.59 Gemma 4 31B IT (30 minutes): Mean ln(1 + peak XP/min) 1.57, mean cost per skill run $0.18 Claude Opus 4.7 (30 minutes): Mean ln(1 + peak XP/min) 4.57, mean cost per skill run $6.98 Claude Opus 4.7 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 4.7, mean cost per skill run $7.27 Kimi K2.6 (30 minutes): Mean ln(1 + peak XP/min) 3.6, mean cost per skill run $1 GPT-5.5 (30 minutes): Mean ln(1 + peak XP/min) 5.32, mean cost per skill run $8.38 GPT-5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $13.4 DeepSeek-V4-Flash (30 minutes): Mean ln(1 + peak XP/min) 4.76, mean cost per skill run $0.36 DeepSeek-V4-Pro (30 minutes): Mean ln(1 + peak XP/min) 4.3, mean cost per skill run $0.68 Grok 4.3 (30 minutes): Mean ln(1 + peak XP/min) 3.66, mean cost per skill run $1.18 Gemini 3.5 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $4.18 Gemini 3.5 Flash (30 minutes; high): Mean ln(1 + peak XP/min) 4.94, mean cost per skill run $4.24 Qwen3.7-Max (30 minutes): Mean ln(1 + peak XP/min) 4.07, mean cost per skill run $1.26 Claude Opus 4.8 (30 minutes; max): Mean ln(1 + peak XP/min) 5.08, mean cost per skill run $5.53 Claude Opus 4.8 (30 minutes): Mean ln(1 + peak XP/min) 5.03, mean cost per skill run $5.16 Nemotron 3 Ultra (30 minutes): Mean ln(1 + peak XP/min) 3.76, mean cost per skill run $0.83 Claude Fable 5 (30 minutes): Mean ln(1 + peak XP/min) 6.01, mean cost per skill run $8.06 Claude Fable 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $8.65 Kimi K2.7 Code (30 minutes): Mean ln(1 + peak XP/min) 4.18, mean cost per skill run $1.27 GLM-5.2 (30 minutes): Mean ln(1 + peak XP/min) 4.76, mean cost per skill run $2.02 Claude Sonnet 5 (30 minutes): Mean ln(1 + peak XP/min) 4.47, mean cost per skill run $2.51 Claude Sonnet 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 4.81, mean cost per skill run $2.88 Grok 4.5 (30 minutes): Mean ln(1 + peak XP/min) 5.7, mean cost per skill run $2.44 Grok 4.5 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $1.6 GPT-5.6 Luna (30 minutes): Mean ln(1 + peak XP/min) 4.77, mean cost per skill run $0.48 GPT-5.6 Luna (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.29, mean cost per skill run $0.75 GPT-5.6 Sol (30 minutes): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $10.7 GPT-5.6 Sol (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.9, mean cost per skill run $8.71 GPT-5.6 Terra (30 minutes): Mean ln(1 + peak XP/min) 5.52, mean cost per skill run $3.15 GPT-5.6 Terra (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.88, mean cost per skill run $5.22 Muse Spark 1.1 (30 minutes): Mean ln(1 + peak XP/min) 5.54, mean cost per skill run $3.02 Inkling (30 minutes): Mean ln(1 + peak XP/min) 3.76, mean cost per skill run $5.24 Kimi K3 (30 minutes): Mean ln(1 + peak XP/min) 3.75, mean cost per skill run $0.76 Gemini 3.5 Flash-Lite (30 minutes): Mean ln(1 + peak XP/min) 4.18, mean cost per skill run $1.13 Gemini 3.6 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.58, mean cost per skill run $3.81 Laguna S 2.1 (30 minutes): Mean ln(1 + peak XP/min) 2.82, mean cost per skill run $0.17 Claude Opus 5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.71, mean cost per skill run $5.14 Claude Opus 5 (30 minutes): Mean ln(1 + peak XP/min) 5.73, mean cost per skill run $4.51 Claude Opus 5 (30 minutes; low): Mean ln(1 + peak XP/min) 5.51, mean cost per skill run $2.95 Claude Opus 5 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.69, mean cost per skill run $3.29 DeepSeek V4 Flash 0731 (30 minutes): Mean ln(1 + peak XP/min) 4.2, mean cost per skill run $0.12 Qwen 3.8 Max (30 minutes): Mean ln(1 + peak XP/min) 5.35, mean cost per skill run $1.68 Muse Spark 1.2 (30 minutes): Mean ln(1 + peak XP/min) 5.24, mean cost per skill run $2.25 Grok 4.6 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.62, mean cost per skill run $5.53 Grok 4.6 (30 minutes): Mean ln(1 + peak XP/min) 6.28, mean cost per skill run $5.11 Grok 4.6 (30 minutes; medium): Mean ln(1 + peak XP/min) 5.83, mean cost per skill run $4.76 Gemini 3.7 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.96, mean cost per skill run $2.97 GLM-5.3 (30 minutes): Mean ln(1 + peak XP/min) 5, mean cost per skill run $1 Qwen 3.8 27B (30 minutes): Mean ln(1 + peak XP/min) 2.79, mean cost per skill run $0.87 Tencent Hy4 preview (30 minutes): Mean ln(1 + peak XP/min) 5.37, mean cost per skill run $0.27 Claude Fable 5.1 (30 minutes): Mean ln(1 + peak XP/min) 6, mean cost per skill run $6.48 Claude Fable 5.1 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.14, mean cost per skill run $9.29 Gemini 3.8 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.96, mean cost per skill run $2.63 Muse Spark 1.3 (30 minutes): Mean ln(1 + peak XP/min) 4.59, mean cost per skill run $1.78 GPT-6 Astra (30 minutes): Mean ln(1 + peak XP/min) 6.91, mean cost per skill run $15.6 Grok 4.7 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.41, mean cost per skill run $7.14 Grok 4.7 (30 minutes): Mean ln(1 + peak XP/min) 5.66, mean cost per skill run $6.94 Claude Opus 5.5 (30 minutes): Mean ln(1 + peak XP/min) 6.63, mean cost per skill run $3.36 Claude Opus 5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.66, mean cost per skill run $5.18 GPT-6 Luna (30 minutes): Mean ln(1 + peak XP/min) 4.94, mean cost per skill run $0.12 GPT-6 Luna (30 minutes; high): Mean ln(1 + peak XP/min) 5.13, mean cost per skill run $0.14 GPT-6 Sol (30 minutes; high): Mean ln(1 + peak XP/min) 6.44, mean cost per skill run $4.84 GPT-6 Sol (30 minutes): Mean ln(1 + peak XP/min) 6.42, mean cost per skill run $4.44 Claude Sonnet 5.5 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 5.52, mean cost per skill run $2.3 Claude Sonnet 5.5 (30 minutes): Mean ln(1 + peak XP/min) 4.82, mean cost per skill run $0.85 Beam 1.0 (30 minutes): Mean ln(1 + peak XP/min) 5.26, mean cost per skill run $0.6 Claude Haiku 5.5 (30 minutes; high): Mean ln(1 + peak XP/min) 4.45, mean cost per skill run $0.15 Muse Wonder (30 minutes): Mean ln(1 + peak XP/min) 4.98, mean cost per skill run $4.18 Muse Spark Open (30 minutes): Mean ln(1 + peak XP/min) 4.61, mean cost per skill run $1.86 Nemotron 3.5 Super VL (30 minutes): Mean ln(1 + peak XP/min) 4.36, mean cost per skill run $0.32 Kimi K3.1 Pro (30 minutes): Mean ln(1 + peak XP/min) 6.48, mean cost per skill run $5.07 Mistral Large 4 (Le Chonk, open weights) (30 minutes): Mean ln(1 + peak XP/min) 4.89, mean cost per skill run $0.41 Claude Fable 5.2 (30 minutes): Mean ln(1 + peak XP/min) 6.74, mean cost per skill run $3.4 Muse Ember (30 minutes): Mean ln(1 + peak XP/min) 4.51, mean cost per skill run $0.17 GLM 5.3 (Prime) (30 minutes): Mean ln(1 + peak XP/min) 5.4, mean cost per skill run $1.84. Not in our model list GLM 5.2 (WandB fp4) (30 minutes): Mean ln(1 + peak XP/min) 5.2, mean cost per skill run $1.74. Not in our model list GLM-5.3-Flash (30 minutes): Mean ln(1 + peak XP/min) 3.83, mean cost per skill run $0.07 GPT-6 Astra (30 minutes; high): Mean ln(1 + peak XP/min) 7.26, mean cost per skill run $15.3 DeepSeek V4.1 Flash (30 minutes): Mean ln(1 + peak XP/min) 5.68, mean cost per skill run $0.08 GPT-6.1 Sol (30 minutes; low): Mean ln(1 + peak XP/min) 6.75, mean cost per skill run $1.52 GPT-6.1 Sol (30 minutes): Mean ln(1 + peak XP/min) 6.79, mean cost per skill run $1.71 GPT-6.1 Sol (30 minutes; high): Mean ln(1 + peak XP/min) 6.88, mean cost per skill run $1.86 Gemini 4 Argon (30 minutes): Mean ln(1 + peak XP/min) 6.91, mean cost per skill run $4.17 Claude Haiku 5.5 (30 minutes): Mean ln(1 + peak XP/min) 4.3, mean cost per skill run $0.08 DeepSeek V4.1 Flash 1017 (30 minutes): Mean ln(1 + peak XP/min) 5.83, mean cost per skill run $0.09 GPT-6.1 Terra (30 minutes): Mean ln(1 + peak XP/min) 6.65, mean cost per skill run $1.13 Claude Fable 5.2 (30 minutes; xhigh): Mean ln(1 + peak XP/min) 6.98, mean cost per skill run $8.19 GPT-6 Astra (30 minutes; high)Claude Fable 5.2 (30 minutes; xhigh)Gemini 4 Argon (30 minutes)GPT-6.1 Sol (30 minutes; high)GPT-6.1 Sol (30 minutes)GPT-6.1 Sol (30 minutes; low)GPT-6.1 Terra (30 minutes)DeepSeek V4.1 Flash 1017 (30 minutes)DeepSeek V4.1 Flash (30 minutes)Claude Haiku 5.5 (30 minutes)GLM-5.3-Flash (30 minutes)
ModelSettingMean ln(1 + peak XP/min)Mean cost per skill run
GPT-6 Astra 30 minutes; high 7.26 $15.26
Claude Fable 5.2 30 minutes; xhigh 6.98 $8.19
GPT-6 Astra 30 minutes 6.91 $15.55
Gemini 4 Argon 30 minutes 6.91 $4.17
GPT-6.1 Sol 30 minutes; high 6.88 $1.86
GPT-6.1 Sol 30 minutes 6.79 $1.71
GPT-6.1 Sol 30 minutes; low 6.75 $1.52
Claude Fable 5.2 30 minutes 6.74 $3.4
Claude Opus 5.5 30 minutes; xhigh 6.66 $5.18
GPT-6.1 Terra 30 minutes 6.65 $1.13
Claude Opus 5.5 30 minutes 6.63 $3.36
Kimi K3.1 Pro 30 minutes 6.48 $5.07
GPT-6 Sol 30 minutes; high 6.44 $4.84
GPT-6 Sol 30 minutes 6.42 $4.44
Grok 4.6 30 minutes 6.28 $5.11
Claude Fable 5.1 30 minutes; xhigh 6.14 $9.29
Claude Fable 5 30 minutes 6.01 $8.06
Claude Fable 5.1 30 minutes 6 $6.48
Gemini 3.7 Flash 30 minutes 5.96 $2.97
Gemini 3.8 Flash 30 minutes 5.96 $2.63
GPT-5.6 Sol 30 minutes; xhigh 5.9 $8.71
GPT-5.6 Terra 30 minutes; xhigh 5.88 $5.22
Grok 4.6 30 minutes; medium 5.83 $4.76
DeepSeek V4.1 Flash 1017 30 minutes 5.83 $0.09
Claude Opus 5 30 minutes 5.73 $4.51
Claude Opus 5 30 minutes; xhigh 5.71 $5.14
Grok 4.5 30 minutes 5.7 $2.44
Claude Opus 5 30 minutes; medium 5.69 $3.29
DeepSeek V4.1 Flash 30 minutes 5.68 $0.08
GPT-5.5 30 minutes; xhigh 5.68 $13.37
Claude Fable 5 30 minutes; xhigh 5.68 $8.65
GPT-5.6 Sol 30 minutes 5.68 $10.74
Grok 4.7 30 minutes 5.66 $6.94
Grok 4.6 30 minutes; xhigh 5.62 $5.53
Gemini 3.6 Flash 30 minutes 5.58 $3.82
Muse Spark 1.1 30 minutes 5.54 $3.02
Claude Sonnet 5.5 30 minutes; xhigh 5.52 $2.3
GPT-5.6 Terra 30 minutes 5.52 $3.16
Claude Opus 5 30 minutes; low 5.51 $2.95
Grok 4.7 30 minutes; xhigh 5.41 $7.14
GLM 5.3 (Prime) 30 minutes 5.4 $1.84
Tencent Hy4 preview 30 minutes 5.37 $0.27
Gemini 3.5 Flash 30 minutes 5.35 $4.18
Grok 4.5 30 minutes; medium 5.35 $1.6
Qwen 3.8 Max 30 minutes 5.35 $1.68
GPT-5.5 30 minutes 5.32 $8.38
GPT-5.6 Luna 30 minutes; xhigh 5.29 $0.75
Beam 1.0 30 minutes 5.26 $0.6
Muse Spark 1.2 30 minutes 5.24 $2.25
GLM 5.2 (WandB fp4) 30 minutes 5.2 $1.74
GPT-6 Luna 30 minutes; high 5.13 $0.14
Claude Opus 4.8 30 minutes; max 5.08 $5.53
Claude Opus 4.8 30 minutes 5.03 $5.16
GLM-5.3 30 minutes 5 $1
Muse Wonder 30 minutes 4.98 $4.18
Gemini 3.5 Flash 30 minutes; high 4.94 $4.24
GPT-6 Luna 30 minutes 4.94 $0.12
Mistral Large 4 (Le Chonk, open weights) 30 minutes 4.89 $0.41
Claude Sonnet 5.5 30 minutes 4.82 $0.85
Claude Sonnet 5 30 minutes; xhigh 4.81 $2.88
GPT-5.6 Luna 30 minutes 4.77 $0.48
GLM-5.2 30 minutes 4.76 $2.02
DeepSeek-V4-Flash 30 minutes 4.76 $0.36
Claude Opus 4.7 30 minutes; xhigh 4.7 $7.27
GPT-5.4 30 minutes 4.7 $5.54
Gemini 3 Flash 30 minutes 4.69 $0.69
Muse Spark Open 30 minutes 4.61 $1.86
Muse Spark 1.3 30 minutes 4.59 $1.78
Claude Opus 4.7 30 minutes 4.57 $6.98
Muse Ember 30 minutes 4.51 $0.17
Gemini 3.1 Pro 30 minutes 4.48 $1.88
Claude Sonnet 5 30 minutes 4.47 $2.51
Claude Haiku 5.5 30 minutes; high 4.45 $0.15
Claude Opus 4.6 30 minutes 4.42 $3.16
Nemotron 3.5 Super VL 30 minutes 4.36 $0.32
DeepSeek-V4-Pro 30 minutes 4.3 $0.68
Claude Haiku 5.5 30 minutes 4.3 $0.08
GPT-5.3 Codex 30 minutes 4.29 $2.47
DeepSeek V4 Flash 0731 30 minutes 4.2 $0.12
Kimi K2.7 Code 30 minutes 4.18 $1.27
Gemini 3.5 Flash-Lite 30 minutes 4.18 $1.13
GPT-5.4 Mini 30 minutes 4.14 $3.27
Claude Opus 4.5 30 minutes 4.08 $3.36
Qwen3.7-Max 30 minutes 4.07 $1.26
GLM-5.3-Flash 30 minutes 3.83 $0.07
Gemini 3 Pro 30 minutes 3.8 $2.2
Nemotron 3 Ultra 30 minutes 3.76 $0.83
Inkling 30 minutes 3.76 $5.24
Kimi K3 30 minutes 3.75 $0.76
Claude Sonnet 4 30 minutes 3.71 $1.82
Grok 4.3 30 minutes 3.66 $1.18
Kimi K2.6 30 minutes 3.6 $1
Claude Sonnet 4.5 30 minutes 3.2 $2.57
Claude Sonnet 4.6 30 minutes 3.18 $3.75
Laguna S 2.1 30 minutes 2.82 $0.17
Qwen 3.8 27B 30 minutes 2.79 $0.87
GPT-5.4 Nano 30 minutes 2.35 $0.59
Kimi K2.5 30 minutes 2.1 $0.31
GLM-5 30 minutes 1.9 $0.38
Claude Haiku 4.5 30 minutes 1.64 $0.84
Gemma 4 31B IT 30 minutes 1.57 $0.18
gpt-oss-120b 30 minutes 1.45 $0.2
Qwen3-Max 30 minutes 1.4 $0.44
Qwen3.5-35B-A3B 30 minutes 0.75 $0.17

The best score, 7.26, costs $15 a task

Runs on the cost frontier score higher than any run that's cheaper. Gemini 4 Argon at 30 minutes comes within 0.4 of the best for $4.17 a task, 4 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

Z.ai OpenAI DeepSeek Google DeepMind Anthropic Cost frontier One model at each effort level
0.00 2.00 4.00 6.00 8.00 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, 30 minutes; high Gemini 4 Argon, 30 minutes DeepSeek V4.1 Flash 1017, 30 minutes DeepSeek V4.1 Flash, 30 minutes Claude Haiku 5.5, 30 minutes GLM-5.3-Flash, 30 minutes 0.00 2.00 4.00 6.00 8.00 $0.01 $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, 30 minutes; high GPT-6.1 Sol, 30 minutes

Source: Official RuneBench leaderboard

ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra 30 minutes; high 7.26 $15 24,438
Claude Fable 5.2 30 minutes; xhigh 6.98 $8.19 86,308
GPT-6 Astra 30 minutes 6.91 $16 20,375
Gemini 4 Argon 30 minutes 6.91 $4.17 27,015
GPT-6.1 Sol 30 minutes; high 6.88 $1.86 28,672
GPT-6.1 Sol 30 minutes 6.79 $1.71 22,625
GPT-6.1 Sol 30 minutes; low 6.75 $1.52 16,943
Claude Fable 5.2 30 minutes 6.74 $3.40 29,044
Claude Opus 5.5 30 minutes; xhigh 6.66 $5.18 56,690
GPT-6.1 Terra 30 minutes 6.65 $1.13 18,247
Claude Opus 5.5 30 minutes 6.63 $3.36 33,628
Kimi K3.1 Pro 30 minutes 6.48 $5.07 24,097
GPT-6 Sol 30 minutes; high 6.44 $4.84 35,775
GPT-6 Sol 30 minutes 6.42 $4.44 28,709
Grok 4.6 30 minutes 6.28 $5.11 20,497
Claude Fable 5.1 30 minutes; xhigh 6.14 $9.29 47,615
Claude Fable 5 30 minutes 6.01 $8.06 42,485
Claude Fable 5.1 30 minutes 6.00 $6.48 33,844
Gemini 3.7 Flash 30 minutes 5.96 $2.97 24,276
Gemini 3.8 Flash 30 minutes 5.96 $2.63 21,336
GPT-5.6 Sol 30 minutes; xhigh 5.90 $8.71 31,784
GPT-5.6 Terra 30 minutes; xhigh 5.88 $5.22 40,488
Grok 4.6 30 minutes; medium 5.83 $4.76 18,448
DeepSeek V4.1 Flash 1017 30 minutes 5.83 $0.09 20,876
Claude Opus 5 30 minutes 5.73 $4.51 46,456
Claude Opus 5 30 minutes; xhigh 5.71 $5.14 47,878
Grok 4.5 30 minutes 5.70 $2.44 25,007
Claude Opus 5 30 minutes; medium 5.69 $3.29 29,602
DeepSeek V4.1 Flash 30 minutes 5.68 $0.08 17,113
GPT-5.5 30 minutes; xhigh 5.68 $13 46,270
Claude Fable 5 30 minutes; xhigh 5.68 $8.65 49,581
GPT-5.6 Sol 30 minutes 5.68 $11 30,775
Grok 4.7 30 minutes 5.66 $6.94 17,475
Grok 4.6 30 minutes; xhigh 5.62 $5.53 21,612
Gemini 3.6 Flash 30 minutes 5.58 $3.81 17,868
Muse Spark 1.1 30 minutes 5.54 $3.02 28,254
Claude Sonnet 5.5 30 minutes; xhigh 5.52 $2.30 111,707
GPT-5.6 Terra 30 minutes 5.52 $3.15 23,784
Claude Opus 5 30 minutes; low 5.51 $2.95 18,450
Grok 4.7 30 minutes; xhigh 5.41 $7.14 16,312
Tencent Hy4 preview 30 minutes 5.37 $0.27 10,912
Gemini 3.5 Flash 30 minutes 5.35 $4.18 60,485
Grok 4.5 30 minutes; medium 5.35 $1.60 26,963
Qwen 3.8 Max 30 minutes 5.35 $1.68 14,318
GPT-5.5 30 minutes 5.32 $8.38 27,337
GPT-5.6 Luna 30 minutes; xhigh 5.29 $0.75 45,982
Beam 1.0 30 minutes 5.26 $0.60 15,234
Muse Spark 1.2 30 minutes 5.24 $2.25 23,235
GPT-6 Luna 30 minutes; high 5.13 $0.14 38,289
Claude Opus 4.8 30 minutes; max 5.08 $5.53 94,375
Claude Opus 4.8 30 minutes 5.03 $5.16 60,051
GLM-5.3 30 minutes 5.00 $1.00 10,413
Muse Wonder 30 minutes 4.98 $4.18 13,851
Gemini 3.5 Flash 30 minutes; high 4.94 $4.24 59,708
GPT-6 Luna 30 minutes 4.94 $0.12 27,426
Mistral Large 4 (Le Chonk, open weights) 30 minutes 4.89 $0.41 11,239
Claude Sonnet 5.5 30 minutes 4.82 $0.85 21,750
Claude Sonnet 5 30 minutes; xhigh 4.81 $2.88 51,958
GPT-5.6 Luna 30 minutes 4.77 $0.48 30,098
GLM-5.2 30 minutes 4.76 $2.02 11,716
DeepSeek-V4-Flash 30 minutes 4.76 $0.36 22,337
Claude Opus 4.7 30 minutes; xhigh 4.70 $7.27 50,447
GPT-5.4 30 minutes 4.70 $5.54 40,007
Gemini 3 Flash 30 minutes 4.69 $0.69 27,070
Muse Spark Open 30 minutes 4.61 $1.86 18,325
Muse Spark 1.3 30 minutes 4.59 $1.78 21,161
Claude Opus 4.7 30 minutes 4.57 $6.98 44,723
Muse Ember 30 minutes 4.51 $0.17 24,855
Gemini 3.1 Pro 30 minutes 4.48 $1.88 27,537
Claude Sonnet 5 30 minutes 4.47 $2.51 37,836
Claude Haiku 5.5 30 minutes; high 4.45 $0.15 26,655
Claude Opus 4.6 30 minutes 4.42 $3.16 25,479
Nemotron 3.5 Super VL 30 minutes 4.36 $0.32 18,299
DeepSeek-V4-Pro 30 minutes 4.30 $0.68 19,190
Claude Haiku 5.5 30 minutes 4.30 $0.08 14,694
GPT-5.3 Codex 30 minutes 4.29 $2.47 30,755
DeepSeek V4 Flash 0731 30 minutes 4.20 $0.12 15,802
Kimi K2.7 Code 30 minutes 4.18 $1.27 10,549
Gemini 3.5 Flash-Lite 30 minutes 4.18 $1.13 13,244
GPT-5.4 Mini 30 minutes 4.14 $3.27 83,136
Claude Opus 4.5 30 minutes 4.08 $3.36 16,726
Qwen3.7-Max 30 minutes 4.07 $1.26 27,534
GLM-5.3-Flash 30 minutes 3.83 $0.07 9,622
Gemini 3 Pro 30 minutes 3.80 $2.20 31,989
Nemotron 3 Ultra 30 minutes 3.76 $0.83 13,683
Inkling 30 minutes 3.76 $5.24 14,485
Kimi K3 30 minutes 3.75 $0.76 5,323
Claude Sonnet 4 30 minutes 3.71 $1.82 17,153
Grok 4.3 30 minutes 3.66 $1.18 9,303
Kimi K2.6 30 minutes 3.60 $1.00 7,834
Claude Sonnet 4.5 30 minutes 3.20 $2.57 14,175
Claude Sonnet 4.6 30 minutes 3.18 $3.75 32,890
Laguna S 2.1 30 minutes 2.82 $0.17 15,496
Qwen 3.8 27B 30 minutes 2.79 $0.87 12,922
GPT-5.4 Nano 30 minutes 2.35 $0.59 74,804
Kimi K2.5 30 minutes 2.10 $0.31 19,394
GLM-5 30 minutes 1.90 $0.38 19,380
Claude Haiku 4.5 30 minutes 1.64 $0.84 13,245
Gemma 4 31B IT 30 minutes 1.57 $0.18 1,134
gpt-oss-120b 30 minutes 1.45 $0.20 8,589
Qwen3-Max 30 minutes 1.40 $0.44 7,900
Qwen3.5-35B-A3B 30 minutes 0.75 $0.17 21,289

7 models have held number one on the official leaderboard since its release in February 2026

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind OpenAI Anthropic xAI
Own leaderboard 7 leaders
GPT-6 Astra 7.26

Source: Official RuneBench leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
0.00 2.00 4.00 6.00 8.00 2025 2026 2027 0.00 2.00 4.00 6.00 8.00 2025 2026 2027

Source: Official RuneBench leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Games and long tasks

All 82 in Games and long tasks
  • ARC-AGI-3 In use Interactive game environments where agents must explore, infer goals and rules, and act efficiently without instructions. GPT-6 Astra 99.95%
  • Orak Fading LLM agents play 12 popular video games across most genres through an MCP interface. Each game scores in-game progress or outcomes, and the board ranks models by their average rank across the games. Gemini 2.5 Pro 3.90
  • MineDojo Fading Agents complete open-ended tasks in Minecraft, such as mining blocks and fighting mobs, from a large simulation suite. Programmatic checks score many tasks. Seed2.0 Pro 49%
  • TextQuests Only in system cards Agents play 25 classic interactive fiction games. The score is harmful actions per 100 steps, where lower is better. The board also reports game progress. Llama 4 Scout 7.00
  • GameWorld Only in system cards Multimodal game agents play 34 browser games across 170 tasks. Outcomes are checked from the game state. GPT-6.1 Terra 76.8%
  • Swarm Bench Only in system cards Models orchestrate agent swarms on complex tasks that benefit from coordinated decomposition and parallel execution. Kimi K3 76.3%