Benchmark Almanac

WeirdML v3

GPT-6 Astra 42.2% leads since Sep 2026 Not in model cards

Agents solve 11 hand-made machine learning and data analysis tasks on a GPU with unfamiliar data and hidden test answers. Scored by token-weighted best results.

Scores are not raw accuracy and are not comparable with WeirdML v2.

Made by Håvard Tveit Ihle (Norwegian Defence Research Establishment) Released Sep 2026 Machine learning engineering Last model added Oct 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Agent task Many steps in a terminal, browser, desktop, app, game or robot.
Leaderboard htihle.github.io
25
Models scored on its own leaderboard

GPT-6 Astra leads with 42.2%

Best score per model from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart.

25 models. Scroll sideways for the rest.

  1. 42.2 GPT-6 Astra
  2. 38.3 GPT-6.1 Sol
  3. 31.2 Claude Opus 5.5
  4. 28.1 GPT-6.1 Terra
  5. 26.0 Claude Fable 5.1
  6. 20.3 Gemini 4 Argon
  7. 19.9 Claude Sonnet 5.5
  8. 19.7 GPT-6 Sol
  9. 19.0 Claude Opus 5
  10. 15.6 Claude Fable 5
  11. 15.2 Muse Wonder
  12. 15.0 GPT-5.6 Sol
  13. 11.7 Claude Haiku 5.5
  14. 9.19 DeepSeek V4.1 Flash 1017
  15. 8.45 Grok 4.7
  16. 7.87 GPT-6 Luna
  17. 7.54 Kimi K3
  18. 7.38 Gemini 3.8 Flash
  19. 6.66 Muse Spark 1.3
  20. 6.51 Claude Opus 4.5
  21. 5.95 DeepSeek V4.1 Flash
  22. 5.16 GPT-5.6 Luna
  23. 5.01 GPT-5.2
  24. 4.45 Mistral Large 4
  25. 3.90 Inkling
  1. 1 GPT-6 Astra xhigh 42.2%
  2. 2 GPT-6.1 Sol xhigh 38.3%
  3. 3 Claude Opus 5.5 xhigh 31.2%
  4. 4 GPT-6.1 Terra xhigh 28.1%
  5. 5 Claude Fable 5.1 xhigh 26%
  6. 6 Gemini 4 Argon high 20.3%
  7. 7 Claude Sonnet 5.5 xhigh 19.9%
  8. 8 GPT-6 Sol xhigh 19.7%
  9. 9 Claude Opus 5 xhigh 19%
  10. 10 Claude Fable 5 xhigh 15.6%
  11. 11 Muse Wonder xhigh 15.2%
  12. 12 GPT-5.6 Sol xhigh 15%
  13. 13 Claude Haiku 5.5 xhigh 11.7%
  14. 14 DeepSeek V4.1 Flash 1017 high 9.19%
  15. 15 Grok 4.7 xhigh 8.45%
  16. 16 GPT-6 Luna xhigh 7.87%
  17. 17 Kimi K3 high 7.54%
  18. 18 Gemini 3.8 Flash high 7.38%
  19. 19 Muse Spark 1.3 xhigh 6.66%
  20. 20 Claude Opus 4.5 high 6.51%
  21. 21 DeepSeek V4.1 Flash high 5.95%
  22. 22 GPT-5.6 Luna xhigh 5.16%
  23. 23 GPT-5.2 xhigh 5.01%
  24. 24 Mistral Large 4 high 4.45%
  25. 25 Inkling xhigh 3.90%
#ModelScoreSettingDateSource
1 GPT-6 Astra 42.2% xhigh, codex_cli Sep 2026 Official leaderboard
2 GPT-6.1 Sol 38.3% xhigh, codex_cli Sep 2026 Official leaderboard
3 Claude Opus 5.5 31.2% xhigh, claude_code Sep 2026 Official leaderboard
4 GPT-6.1 Terra 28.1% xhigh, codex_cli Oct 2026 Official leaderboard
5 Claude Fable 5.1 26% xhigh, claude_code Sep 2026 Official leaderboard
6 Gemini 4 Argon 20.3% high, gemini_cli Sep 2026 Official leaderboard
7 Claude Sonnet 5.5 19.9% xhigh, claude_code Sep 2026 Official leaderboard
8 GPT-6 Sol 19.7% xhigh, codex_cli Sep 2026 Official leaderboard
9 Claude Opus 5 19% xhigh, claude_code Jul 2026 Official leaderboard
10 Claude Fable 5 15.6% xhigh, claude_code Jun 2026 Official leaderboard
11 Muse Wonder 15.2% xhigh, codex_cli Oct 2026 Official leaderboard
12 GPT-5.6 Sol 15% xhigh, codex_cli Jul 2026 Official leaderboard
13 Claude Haiku 5.5 11.7% xhigh, claude_code Oct 2026 Official leaderboard
14 DeepSeek V4.1 Flash 1017 9.19% high, codex_cli via InferenceNet Oct 2026 Official leaderboard
15 Grok 4.7 8.45% xhigh, codex_cli Sep 2026 Official leaderboard
16 GPT-6 Luna 7.87% xhigh, codex_cli Sep 2026 Official leaderboard
17 Kimi K3 7.54% high, kimi_code via Fireworks Jul 2026 Official leaderboard
18 Gemini 3.8 Flash 7.38% high, gemini_cli Sep 2026 Official leaderboard
19 Muse Spark 1.3 6.66% xhigh, codex_cli Sep 2026 Official leaderboard
20 Claude Opus 4.5 6.51% high, claude_code Nov 2025 Official leaderboard
21 DeepSeek V4.1 Flash 5.95% high, opencode via Novita Sep 2026 Official leaderboard
22 GPT-5.6 Luna 5.16% xhigh, codex_cli Jul 2026 Official leaderboard
23 GPT-5.2 5.01% xhigh, codex_cli Dec 2025 Official leaderboard
24 Mistral Large 4 4.45% high, opencode via Mistral Oct 2026 Official leaderboard
25 Inkling 3.90% xhigh, opencode via Together Jul 2026 Official leaderboard

Epoch AI has scores for 25 models. We plot every source in the charts below.

The best score, 42.2%, costs $28 a task

Runs on the cost frontier score higher than any run that's cheaper. GPT-6.1 Sol at xhigh comes within 4 points of the best for $5.29 a task, 5 times cheaper. The costs are the leaderboard's own estimates, so compare them only within this benchmark.

OpenAI Anthropic DeepSeek Cost frontier One model at each effort level
0% 10% 20% 30% 40% 50% $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, xhigh GPT-6.1 Sol, xhigh GPT-6.1 Terra, xhigh Claude Haiku 5.5, xhigh DeepSeek V4.1 Flash 1017, high 0% 10% 20% 30% 40% 50% $0.10 $1.00 $10 $100 Cost per task, log scale GPT-6 Astra, xhigh GPT-6.1 Sol, xhigh
ModelEffortHarnessScoreCost per taskOutput tokens
GPT-6 Astra xhigh codex_cli 42.2% $28 153,241
GPT-6.1 Sol xhigh codex_cli 38.3% $5.29 215,529
Claude Opus 5.5 xhigh claude_code 31.2% $10 180,952
GPT-6.1 Terra xhigh codex_cli 28.1% $2.86 147,165
Claude Fable 5.1 xhigh claude_code 26% $32 251,090
Gemini 4 Argon high gemini_cli 20.3% $16 310,099
Claude Sonnet 5.5 xhigh claude_code 19.9% $9.28 278,720
GPT-6 Sol xhigh codex_cli 19.7% $6.91 143,614
Claude Opus 5 xhigh claude_code 19% $25 288,399
Claude Fable 5 xhigh claude_code 15.6% $53 300,212
Muse Wonder xhigh codex_cli 15.2% $11 169,363
GPT-5.6 Sol xhigh codex_cli 15% $6.28 116,059
Claude Haiku 5.5 xhigh claude_code 11.7% $2.30 315,447
DeepSeek V4.1 Flash 1017 high codex_cli via InferenceNet 9.19% $0.88 829,778
Grok 4.7 xhigh codex_cli 8.45% $27 275,430
GPT-6 Luna xhigh codex_cli 7.87% $0.32 180,561
Kimi K3 high kimi_code via Fireworks 7.54% $20 209,415
Gemini 3.8 Flash high gemini_cli 7.38% $3.94 156,103
Muse Spark 1.3 xhigh codex_cli 6.66% $5.56 280,208
Claude Opus 4.5 high claude_code 6.51% $8.70 82,595
DeepSeek V4.1 Flash high opencode via Novita 5.95% $0.74 323,336
DeepSeek V4.1 Flash high codex_cli via Novita 5.83% $0.51 385,683
GPT-5.6 Luna xhigh codex_cli 5.16% $0.74 99,311
GPT-5.2 xhigh codex_cli 5.01% $9.29 264,947
Mistral Large 4 high opencode via Mistral 4.45% $29 1,081,281
Inkling xhigh opencode via Together 3.90% $1.33 19,902
Inkling xhigh opencode via DeepInfra 3.75% $0.45 18,515

2 models have held number one since its release in Sep 2026

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and Epoch AI’s runs.

Anthropic OpenAI
Own leaderboard 2 leaders
GPT-6 Astra 42.2%
Epoch AI runs 2 leaders
GPT-6 Astra 54%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Run or collected by Epoch AI Best so far
0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Highest possible score GPT-6 Astra Claude Fable 5.1 Claude Opus 5 Claude Fable 5 GPT-5.2 0% 20% 40% 60% 80% 100% 2026 2027 Near the ceiling (85%) Highest possible score GPT-6 Astra Claude Fable 5.1

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

Versions

We switch the top row to each new version when it comes out. Open it for one row per version. An old version stays in play while people still test it, so whoever first takes it past 85% gets credit for closing it out.

OpenAI Google DeepMind Anthropic
WeirdML v3 14 leaders
GPT-6 Astra 54%
WeirdML Passed 85% in Jun 2026, by Claude Fable 5
GPT-6 Astra Pro 93.6%
WeirdML v3 2 leaders
GPT-6 Astra 54%
Replaces WeirdML

More in AI research

All 83 in AI research
  • MLS-Bench In use Agent must invent one algorithmic change in an ML research scaffold whose gain transfers across settings. Scored relative to human baselines. Claude Fable 5.1 50.3%
  • PaperBench In use Agent replicates ICML 2024 papers from scratch. LLM judge grades against author-written hierarchical rubrics. Qwen 3.8 Max 93%
  • MLE-bench In use Kaggle-style ML engineering competitions. Agent trains models and is scored against competition leaderboards. GPT-6 Astra 93.8%
  • PostTrainBench 1.2 In use CLI agent gets a base model, one H100 and a time budget to post-train it for a target benchmark. Scored on normalized gain. Claude Fable 5.1 44.6%
  • KernelBench In use Replace PyTorch operators with faster custom GPU kernels. Scored on correctness and speedup. MiniMax-M3 28.8%
  • OpenAI PRs Only in system cards Agents replicate internal OpenAI pull requests from a pre-PR branch and a prompt. Graded by hidden unit tests as pass@1. GPT-5.2 55%