Benchmark Almanac

SEAL Coding

o1-mini 1,237 leads since Sep 2024 Not in model cards

Models answer 1,000 private coding prompts across common languages and tasks. Human annotators rate adherence, correctness, efficiency and readability, then compare responses side by side to produce Elo-style scores.

One of the four launch SEAL boards. Contributors could not use public code or questions. Scale marked it deprecated in 2025-03 with 28 models, led by o1-mini.

Made by Scale AI Released May 2024 Function-level problems Tasks kept private Last model added Jan 2025 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Graded by people Human experts grade or compare the outputs.
Answers
Code A program or patch run against tests.
Leaderboard scale.com
28
Entries on its own leaderboard, 26 models we track

o1-mini leads with 1,237 among models released by Jan 2025

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Jan 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

28 entries. Scroll sideways for the rest. Bars start at 400.

  1. 1,237 o1-mini
  2. 1,137 o3-mini
  3. 1,132 GPT-4o
  4. 1,123 o1-preview
  5. 1,111 Gemini 2.0 Flash
  6. 1,109 Gemini 2.0 Pro (December 2024)
  7. 1,108 Gemini 2.0 Flash Thinking
  8. 1,100 DeepSeek-R1
  9. 1,083 o1
  10. 1,079 Claude 3.5 Sonnet
  11. 1,045 GPT-4o
  12. 1,036 GPT-4o
  13. 1,034 GPT-4 Turbo
  14. 1,029 Mistral Large 2
  15. 1,022 Llama 3.1-405B
  16. 1,007 Gemini 1.5 Pro (August 27, 2024)
  17. 994 Gemini 1.5 Pro
  18. 992 GPT-4 Turbo
  19. 985 DeepSeek-V3
  20. 984 Llama 3.2 90B
  21. 959 Claude 3 Opus
  22. 943 Gemini 1.5 Flash
  23. 891 Gemini 1.5 Pro
  24. 879 Claude 3 Sonnet
  25. 871 Llama 3-70B
  26. 811 Mistral Large
  27. 685 Gemini 1.0 Pro
  28. 598 Code Llama 34B Instruct

Bars start at 400.

  1. 1 o1-mini 1,237
  2. 2 o3-mini high 1,137
  3. 3 GPT-4o Nov 2024 1,132
  4. 4 o1-preview 1,123
  5. 5 Gemini 2.0 Flash Dec 2024 1,111
  6. 6 Gemini 2.0 Pro (December 2024) 1,109
  7. 7 Gemini 2.0 Flash Thinking Jan 2025 1,108
  8. 8 DeepSeek-R1 1,100
  9. 9 o1 1,083
  10. 10 Claude 3.5 Sonnet 1,079
  11. 11 GPT-4o Aug 2024 1,045
  12. 12 GPT-4o May 2024 1,036
  13. 13 GPT-4 Turbo Apr 2024 1,034
  14. 14 Mistral Large 2 Jul 2024 1,029
  15. 15 Llama 3.1-405B 1,022
  16. 16 Gemini 1.5 Pro (August 27, 2024) 1,007
  17. 17 Gemini 1.5 Pro May 2024 994
  18. 18 GPT-4 Turbo Nov 2023 992
  19. 19 DeepSeek-V3 985
  20. 20 Llama 3.2 90B 984
  21. 21 Claude 3 Opus 959
  22. 22 Gemini 1.5 Flash May 2024 943
  23. 23 Gemini 1.5 Pro Feb 2024 891
  24. 24 Claude 3 Sonnet 879
  25. 25 Llama 3-70B 871
  26. 26 Mistral Large 811
  27. 27 Gemini 1.0 Pro 685
  28. 28 Code Llama 34B Instruct 598
#EntryScoreSettingDateSource
1 o1-mini 1,237 Sep 2024 Official leaderboard
2 o3-mini 1,137 high Jan 2025 Official leaderboard
3 GPT-4o Nov 2024 1,132 Nov 2024 Official leaderboard
4 o1-preview 1,123 Sep 2024 Official leaderboard
5 Gemini 2.0 Flash Dec 2024 1,111 Dec 2024 Official leaderboard
6 Gemini 2.0 Pro (December 2024) 1,109 Official leaderboard
7 Gemini 2.0 Flash Thinking Jan 2025 1,108 Jan 2025 Official leaderboard
8 DeepSeek-R1 1,100 Jan 2025 Official leaderboard
9 o1 1,083 Dec 2024 Official leaderboard
10 Claude 3.5 Sonnet 1,079 Jun 2024 Official leaderboard
11 GPT-4o Aug 2024 1,045 Aug 2024 Official leaderboard
12 GPT-4o May 2024 1,036 May 2024 Official leaderboard
13 GPT-4 Turbo Apr 2024 1,034 Apr 2024 Official leaderboard
14 Mistral Large 2 Jul 2024 1,029 Jul 2024 Official leaderboard
15 Llama 3.1-405B 1,022 Jul 2024 Official leaderboard
16 Gemini 1.5 Pro (August 27, 2024) 1,007 Official leaderboard
17 Gemini 1.5 Pro May 2024 994 May 2024 Official leaderboard
18 GPT-4 Turbo Nov 2023 992 Nov 2023 Official leaderboard
19 DeepSeek-V3 985 Dec 2024 Official leaderboard
20 Llama 3.2 90B 984 Sep 2024 Official leaderboard
21 Claude 3 Opus 959 Feb 2024 Official leaderboard
22 Gemini 1.5 Flash May 2024 943 May 2024 Official leaderboard
23 Gemini 1.5 Pro Feb 2024 891 Feb 2024 Official leaderboard
24 Claude 3 Sonnet 879 Feb 2024 Official leaderboard
25 Llama 3-70B 871 Apr 2024 Official leaderboard
26 Mistral Large 811 Feb 2024 Official leaderboard
27 Gemini 1.0 Pro 685 Dec 2023 Official leaderboard
28 Code Llama 34B Instruct 598 Aug 2023 Official leaderboard

4 models have held number one since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Anthropic
Own leaderboard Not tested since Jan 2025
o1-mini 1,237

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
400 600 800 1,000 1,200 1,400 2024 2025 400 600 800 1,000 1,200 1,400 2024 2025

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Coding

All 243 in Coding
  • SWE-bench Verified In use 500 human-validated GitHub issues from 12 Python repos. Agent writes a patch that must pass hidden tests. Claude Opus 4.7 83.5%
  • SWE-bench Multilingual In use 300 GitHub-issue tasks across nine programming languages in SWE-bench format. Patch graded by repository tests. Claude Opus 5.5 93.9%
  • LiveCodeBench v6 In use Competitive programming problems collected after model cutoffs from LeetCode, AtCoder, Codeforces. Scored pass@1 on hidden tests. DeepSeek-V4-Pro 93.5%
  • DeepSWE In use 113 original long-horizon engineering tasks across 91 repos and five languages, written from scratch. Hand-written verifiers accept alternative solutions. GPT-6 Astra 74.1%
  • SciCode In use Scientist-curated research coding problems split into subproblems across natural sciences. Code checked against test cases. Seed2.1 Pro 59.8%
  • Codeforces rating In use Model solves recent Codeforces contest problems. An Elo-style rating is computed from simulated contest performance. DeepSeek V4.1 Flash 3,471