Benchmark Almanac

KOR-Bench

o1-preview 72.9% leads since Oct 2024 Fading

Models read a new rule set and apply it to questions in five task types: operation, logic, cipher, puzzle and counterfactual. Scored by accuracy against an answer key.

The rules are designed to be independent of domain knowledge. ByteDance Seed reports it in Seed-OSS and Seed 1.6 launch posts.

In the authors’ words

In this paper, we introduce Knowledge-Orthogonal Reasoning (KOR), a concept aimed at minimizing reliance on domain-specific knowledge, enabling more accurate evaluation of models' reasoning abilities in out-of-distribution settings. Based on this concept, we propose the Knowledge-Orthogonal Reasoning Benchmark (KOR-Bench), encompassing five task categories: Operation, Logic, Cipher, Puzzle, and Counterfactual. KOR-Bench emphasizes models' effectiveness in applying new rule descriptions to solve novel rule-driven questions. O1-Preview and O1-Mini achieve accuracies of 72.88% and 70.16%, surpassing Claude-3.5-Sonnet and GPT-4o (58.96% and 58.00%), highlighting the effectiveness of KOR-Bench. We perform detailed analyses, identifying bottlenecks in the Cipher task with Stepwise Prompting, where two rounds of Self-Correction yield optimal results. We evaluate performance across three integrated tasks, explore the impact of Tricks on the Puzzle task, and visualize rule-focused attention. Additionally, we conduct an ablation study on dataset size, benchmark correlations, and zero-shot and three-shot "only questions" experiments. KOR-Bench aims to enhance reasoning evaluation and support further research in this area.

Abstract of KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Tasks (arXiv:2410.06526, CC0)
Made by Multimodal Art Projection (M-A-P) with ByteDance and 01.AI Released Oct 2024 Abstract puzzles and logic All tasks public Last in a launch post Feb 2026 , dropped before it saturated Last model added Mar 2026 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
KOR-Bench/KOR-Bench 19 stars Paper Leaderboard kor-bench.github.io
38
Entries on its own leaderboard, 33 models we track
4
Launch posts list it, from 2 labs
2
System cards report it, from 1 lab

o1-preview leads with 72.9% among models released by Sep 2024

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Sep 2024, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

38 entries. Scroll sideways for the rest.

  1. 72.9 o1-preview
  2. 70.2 o1-mini
  3. 59.0 Claude 3.5 Sonnet
  4. 58.0 GPT-4o
  5. 55.4 Llama 3.1-405B
  6. 54.7 Qwen2.5-32B
  7. 53.5 GPT-4 Turbo
  8. 53.1 Mistral Large 2
  9. 52.2 Qwen2.5-72B
  10. 50.0 Yi-Large
  11. 50.0 Llama 3.1-70B
  12. 49.4 Qwen2.5-14B
  13. 49.2 Llama 3-70B
  14. 48.1 Doubao-Pro-128k
  15. 47.8 DeepSeek-V2.5
  16. 47.0 Qwen2-72B
  17. 44.5 Gemma 2 27B
  18. 43.9 Phi-3.5-MoE
  19. 43.4 Gemini 1.5 Pro
  20. 41.6 Gemma 2 9B
  21. 39.8 Yi-1.5-34B
  22. 39.0 Phi-3.5-mini
  23. 38.6 Qwen2.5-7B
  24. 37.2 Llama 3.1-8B
  25. 35.2 Yi-1.5-9B-Chat
  26. 32.8 Llama 3-8B
  27. 32.7 Command R+
  28. 32.5 Yi-1.5-6B-Chat
  29. 31.1 c4ai-command-r-08-2024
  30. 30.7 Qwen2-7B-Instruct
  31. 24.3 Gemma 2 2B IT
  32. 24.2 Mistral 7B v0.3
  33. 20.4 Qwen2.5-1.5B-Instruct
  34. 18.5 OLMo-7B-0724-Instruct-hf
  35. 18.2 MAP-Neo-7B-Instruct-v0.1
  36. 14.3 Qwen2-1.5B-Instruct
  37. 9.04 Qwen2.5-0.5B-Instruct
  38. 3.52 Qwen2-0.5B-Instruct
  1. 1 o1-preview 72.9%
  2. 2 o1-mini 70.2%
  3. 3 Claude 3.5 Sonnet 59%
  4. 4 GPT-4o May 2024 58%
  5. 5 Llama 3.1-405B 55.4%
  6. 6 Qwen2.5-32B 54.7%
  7. 7 GPT-4 Turbo Apr 2024 53.5%
  8. 8 Mistral Large 2 Jul 2024 53.1%
  9. 9 Qwen2.5-72B 52.2%
  10. 10 Yi-Large 50%
  11. 11 Llama 3.1-70B 50%
  12. 12 Qwen2.5-14B 49.4%
  13. 13 Llama 3-70B 49.2%
  14. 14 Doubao-Pro-128k 48.1%
  15. 15 DeepSeek-V2.5 Sep 2024 47.8%
  16. 16 Qwen2-72B 47%
  17. 17 Gemma 2 27B 44.5%
  18. 18 Phi-3.5-MoE 43.9%
  19. 19 Gemini 1.5 Pro May 2024 43.4%
  20. 20 Gemma 2 9B 41.6%
  21. 21 Yi-1.5-34B 39.8%
  22. 22 Phi-3.5-mini 39%
  23. 23 Qwen2.5-7B 38.6%
  24. 24 Llama 3.1-8B 37.2%
  25. 25 Yi-1.5-9B-Chat 35.2%
  26. 26 Llama 3-8B 32.8%
  27. 27 Command R+ 32.7%
  28. 28 Yi-1.5-6B-Chat 32.5%
  29. 29 c4ai-command-r-08-2024 31.1%
  30. 30 Qwen2-7B-Instruct 30.7%
  31. 31 Gemma 2 2B IT 24.3%
  32. 32 Mistral 7B v0.3 24.2%
  33. 33 Qwen2.5-1.5B-Instruct 20.4%
  34. 34 OLMo-7B-0724-Instruct-hf 18.5%
  35. 35 MAP-Neo-7B-Instruct-v0.1 18.2%
  36. 36 Qwen2-1.5B-Instruct 14.3%
  37. 37 Qwen2.5-0.5B-Instruct 9.04%
  38. 38 Qwen2-0.5B-Instruct 3.52%
#EntryScoreSettingDateSource
1 o1-preview 72.9% Sep 2024 Official leaderboard
2 o1-mini 70.2% Sep 2024 Official leaderboard
3 Claude 3.5 Sonnet 59% Jun 2024 Official leaderboard
4 GPT-4o May 2024 58% May 2024 Official leaderboard
5 Llama 3.1-405B 55.4% Jul 2024 Official leaderboard
6 Qwen2.5-32B 54.7% Sep 2024 Official leaderboard
7 GPT-4 Turbo Apr 2024 53.5% Apr 2024 Official leaderboard
8 Mistral Large 2 Jul 2024 53.1% Jul 2024 Official leaderboard
9 Qwen2.5-72B 52.2% Sep 2024 Official leaderboard
10 Yi-Large 50% May 2024 Official leaderboard
11 Llama 3.1-70B 50% Jul 2024 Official leaderboard
12 Qwen2.5-14B 49.4% Sep 2024 Official leaderboard
13 Llama 3-70B 49.2% Apr 2024 Official leaderboard
14 Doubao-Pro-128k 48.1% Jun 2024 Official leaderboard
15 DeepSeek-V2.5 Sep 2024 47.8% Sep 2024 Official leaderboard
16 Qwen2-72B 47% Jun 2024 Official leaderboard
17 Gemma 2 27B 44.5% Jun 2024 Official leaderboard
18 Phi-3.5-MoE 43.9% Aug 2024 Official leaderboard
19 Gemini 1.5 Pro May 2024 43.4% May 2024 Official leaderboard
20 Gemma 2 9B 41.6% Jun 2024 Official leaderboard
21 Yi-1.5-34B 39.8% May 2024 Official leaderboard
22 Phi-3.5-mini 39% Aug 2024 Official leaderboard
23 Qwen2.5-7B 38.6% Sep 2024 Official leaderboard
24 Llama 3.1-8B 37.2% Jul 2024 Official leaderboard
25 Yi-1.5-9B-Chat 35.2% May 2024 Official leaderboard
26 Llama 3-8B 32.8% Apr 2024 Official leaderboard
27 Command R+ 32.7% Aug 2024 Official leaderboard
28 Yi-1.5-6B-Chat 32.5% May 2024 Official leaderboard
29 c4ai-command-r-08-2024 31.1% Aug 2024 Official leaderboard
30 Qwen2-7B-Instruct 30.7% Jun 2024 Official leaderboard
31 Gemma 2 2B IT 24.3% Jul 2024 Official leaderboard
32 Mistral 7B v0.3 24.2% Sep 2023 Official leaderboard
33 Qwen2.5-1.5B-Instruct 20.4% Sep 2024 Official leaderboard
34 OLMo-7B-0724-Instruct-hf 18.5% Sep 2024 Official leaderboard
35 MAP-Neo-7B-Instruct-v0.1 18.2% Sep 2024 Official leaderboard
36 Qwen2-1.5B-Instruct 14.3% Jun 2024 Official leaderboard
37 Qwen2.5-0.5B-Instruct 9.04% Sep 2024 Official leaderboard
38 Qwen2-0.5B-Instruct 3.52% Jun 2024 Official leaderboard

Labs report scores for 4 models in their cards. We plot every source in the charts below.

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release. We draw one row each for the benchmark’s own leaderboard and what labs claim in their cards.

OpenAI Shanghai AI Lab ByteDance Seed
Own leaderboard Not tested since Oct 2024
o1-preview 72.9%
Lab claims Not tested since Mar 2026
Seed2.0 Pro 77.5%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.0 Pro o1-preview Claude 3.5 Sonnet GPT-4 Turbo Mistral 7B v0.3 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Seed2.0 Pro Claude 3.5 Sonnet

Labs headline it less often now, last in Feb 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Shanghai AI Lab
ByteDance Seed
DateLaunch postScoreAs the post writes it
14 Feb 2026 Seed2.0 Pro 77.5 KORBench
18 Dec 2025 Seed1.8 76.2 KOR-Bench
20 Aug 2025 seed-oss-36b-instruct 70.6 KORBench
13 Jan 2025 InternLM3-8B-Instruct 56.4 KOR-Bench(0-shot)

2 system cards report it, from 1 lab

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
21 Mar 2026 Seed1.8 model card 76.2 KOR-Bench (Table 1) Capabilities
14 Feb 2026 Seed2.0 Pro model card 77.5 KORBench (Table 3). Table 13 prints 77.2 Capabilities

More in Science and reasoning

All 88 in Science and reasoning
  • GPQA Diamond In use 198 graduate-level 'Google-proof' multiple-choice questions in biology, chemistry and physics. Gemini 4 Argon 96.4%
  • Terminal-Bench-Science In use 70 scientific research-workflow tasks across five science areas, run in Harbor containers. Hidden tests give the resolution rate. GPT-6 Astra 68.1%
  • FrontierScience In use Models solve physics, chemistry, and biology problems in an Olympiad track graded by short answer and a Research track graded by 10-point rubrics. Intern-S2-397B 87%
  • BioMysteryBench In use Bioinformatics problems on anonymized real datasets that require real analysis with tools. Rubric-graded, with human-solvable and human-difficult splits. Claude Opus 5 90.1%
  • HiPhO In use Models solve theory problems from 2024 and 2025 physics olympiad exams, some with diagrams. Official marking schemes grade answers and steps, mapped to medal levels. Gemini 3 Pro 78.3%
  • LABBench2 In use 11 biology research sub-tasks (literature, figures, tables, patents, trials and more) with tool access. Macro-averaged accuracy. GPT-5.2 60.9%