Benchmark Almanac

ArbMMLU-HT

Mushari440 Qwen3-8B-SFT-V2 85.7% tops its leaderboard Qwen2.5-72B 72.2% leads the models we track Not in model cards

Models answer a human translation into Arabic of the 57 MMLU subjects.

It differs from ArabicMMLU, which uses native Arabic school exam questions. The OALL v2 blog says Inception curated the translation for Jais.

Made by Inception and MBZUAI (Jais project) Released Sep 2024 Knowledge in many languages All tasks public Last model added Aug 2025 Scores refreshed 2 Nov 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
Dataset Leaderboard huggingface.co
151
Entries on its own leaderboard, 39 models we track

Mushari440 Qwen3-8B-SFT-V2 tops the leaderboard with 85.7%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2.5-72B, at 72.2%.

The newest model with a score here came out in Aug 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

151 entries. Scroll sideways for the rest.

  1. 85.7 Mushari440 Qwen3-8B-SFT-V2
  2. 78.0 Applied-Innovation-Center Karnak
  3. 74.3 AbdulmalekDS qwen72b-ar-lora
  4. 74.0 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2
  5. 74.0 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1
  6. 73.9 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1
  7. 73.9 Lina-Z qwen_arabic_ft
  8. 73.9 Sakalti Ultiima-72B
  9. 73.9 rombodawg Rombos-LLM-V2.5-Qwen-72b
  10. 73.3 Lina-Z arabic_llm_test
  11. 73.1 MaziyarPanahi calme-2.1-qwen2.5-72b
  12. 73.0 MaziyarPanahi calme-2.2-qwen2.5-72b
  13. 72.2 Qwen2.5-72B
  14. 68.9 Ocelotr Qwen-8B-SFT
  15. 68.3 FreedomIntelligence AceGPT-v2-70B-Chat
  16. 68.1 Qwen2-72B
  17. 67.9 Applied-Innovation-Center AIC-1
  18. 67.8 Llama 3.3 70B Instruct
  19. 67.5 Mushari440 qwen3-8B-SFT
  20. 66.9 Gemma 3 27B
  21. 66.8 tanliboy lambda-qwen2.5-32b-dpo-test
  22. 66.7 maldv Qwentile2.5-32B-Instruct
  23. 66.4 maldv Awqward2.5-32B-Instruct
  24. 65.7 Qwen2.5-32B
  25. 65.7 FreedomIntelligence AceGPT-v2-70B
  26. 65.7 TIGER-Lab Qwen2.5-32B-Instruct-CFT
  27. 65.7 zetasepic Qwen2.5-32B-Instruct-abliterated-v2
  28. 65.6 huihui-ai Qwen2.5-32B-Instruct-abliterated
  29. 64.7 Daemontatox Cogito-R1
  30. 64.6 Sakalti Saka-14B
  31. 64.1 Sakalti SakaMoe-3x14B-Instruct
  32. 63.3 TarjamaN Pronoia-14b-community
  33. 62.5 freewheelin free-evo-qwen72b-v0.8-re
  34. 62.5 Cran-May tempmotacilla-cinerea-0308
  35. 61.7 Qwen2.5-14B
  36. 61.6 Mushari440 qwen-3-8B-sft-grpo
  37. 61.5 hammh0a Hala-9B
  38. 61.5 Qwen3-8B
  39. 61.2 FreedomIntelligence AceGPT-v2-32B-Chat
  40. 61.2 Gemma 3 12B
  41. 61.2 Gemma 2 27B
  42. 60.7 AXCXEPT EZO-Qwen2.5-32B-Instruct
  43. 59.5 QCRI/Fanar-1-9B
  44. 58.9 tanliboy lambda-qwen2.5-14b-dpo-test
  45. 58.9 Aya Expanse 32B
  46. 58.6 QCRI/Fanar-1-9B-Instruct
  47. 57.9 Ocelotr Qwen3-8B-GAE
  48. 57.0 JasperV13 Yehia-7B-SFT-Reasoning-preview
  49. 57.0 rombodawg Rombos-LLM-V2.6-Qwen-14b
  50. 56.6 Slim205 Barka-9b-it
  51. 56.4 recoilme recoilme-gemma-2-9B-v0.4
  52. 56.4 AIDC-AI Marco-LLM-AR-V4
  53. 56.4 v000000 Qwen2.5-Lumen-14B
  54. 56.1 v000000 Qwen2.5-14B-Gutenberg-1e-Delta
  55. 55.9 inceptionai/jais-adapted-70b-chat
  56. 55.3 airev-ai emirati-14b-v2
  57. 55.3 JasperV13 Yehia-7B-Reasoning-preview
  58. 55.2 AIDC-AI Marco-LLM-AR-V2
  59. 55.1 JasperV13 Yehia-7B-DPO-Reasoning-preview
  60. 54.0 CohereForAI/aya-23-35B
  61. 53.7 Qwen3-4B
  62. 53.4 Cran-May T.E-8.1
  63. 53.4 Navid-AI Yehia-7B-preview
  64. 52.9 asas-ai AceGPT-v2-32B
  65. 52.9 FreedomIntelligence AceGPT-v2-32B
  66. 52.8 ALLaM-AI/ALLaM-7B-Instruct-preview
  67. 52.7 inceptionai/jais-family-30b-16k-chat
  68. 52.5 Orion-zhen Qwen2.5-7B-Instruct-Uncensored
  69. 51.9 inceptionai/jais-family-30b-8k-chat
  70. 51.7 Qwen2.5-7B
  71. 51.3 arcee-ai Arcee-Spark
  72. 51.3 Qwen2-7B-Instruct
  73. 50.9 FreedomIntelligence AceGPT-v2-8B-Chat
  74. 50.1 CohereForAI/c4ai-command-r7b-arabic-02-2025
  75. 49.2 Aya Expanse 8B
  76. 48.7 SeaLLMs SeaLLM-7B-v2.5
  77. 48.6 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct
  78. 48.2 SeaLLMs SeaLLMs-v3-7B-Chat
  79. 48.0 inceptionai/jais-adapted-70b
  80. 47.9 Llama 3.1-8B
  81. 47.8 Qwen3-14B
  82. 47.5 AIDC-AI Marco-LLM-AR
  83. 47.0 01-ZeroOne SUHAIL-14B-preview
  84. 47.0 inceptionai/jais-family-13b-chat
  85. 46.9 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2
  86. 46.5 FreedomIntelligence AceGPT-v1.5-13B-Chat
  87. 46.4 amine-khelif BARS
  88. 45.5 inceptionai/jais-adapted-13b-chat
  89. 45.3 Qwen1.5-32B
  90. 45.2 Gemma 3 4B
  91. 44.0 recoilme recoilme-gemma-2-9B-v0.2
  92. 43.9 Qwen3-1.7B
  93. 43.0 inceptionai/jais-family-30b-16k
  94. 42.8 inceptionai/jais-family-30b-8k
  95. 42.3 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2
  96. 42.2 Phi-4 Mini
  97. 40.4 Slim205 Barka-2b-it
  98. 40.3 silma-ai SILMA-9B-Instruct-v1.0
  99. 39.8 yellowtown 7B-v0.2
  100. 39.3 FreedomIntelligence AceGPT-13B-chat
  101. 39.1 Phi-4
  102. 38.0 Qwen1.5-14B
  103. 38.0 inceptionai/jais-adapted-7b-chat
  104. 37.4 Qwen3-30B-A3B-Instruct
  105. 37.4 Qwen2-1.5B-Instruct
  106. 37.1 upstage SOLAR-10.7B-v1.0
  107. 36.8 huihui-ai Qwen2.5-7B-Instruct-abliterated
  108. 35.9 Qwen2.5-3B
  109. 35.9 Qwen2.5-1.5B
  110. 35.8 silma-ai SILMA-Kashif-2B-Instruct-v1.0
  111. 35.7 FreedomIntelligence AceGPT-v1.5-13B
  112. 35.5 Hagrass LLama3-3.2-instruct-trained
  113. 34.5 FreedomIntelligence AceGPT-7B-chat
  114. 34.4 Qwen1.5-7B
  115. 33.5 Sakalti Saka-1.5B
  116. 33.2 INSAIT-Institute BgGPT-7B-Instruct-v0.2
  117. 33.1 inceptionai/jais-adapted-13b
  118. 32.2 01-ai/Yi-1.5-34B-32K
  119. 32.1 FreedomIntelligence AceGPT-13B
  120. 31.6 AIDC-AI Marco-LLM-AR-V3
  121. 31.4 01-ai/Yi-1.5-9B
  122. 31.0 inceptionai/jais-family-13b
  123. 31.0 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo
  124. 30.4 01-ai/Yi-1.5-9B-32K
  125. 29.8 cognitivecomputations Dolphin3.0-R1-Mistral-24B
  126. 29.8 Qwen/Qwen1.5-4B
  127. 29.6 Qwen2-0.5B-Instruct
  128. 29.5 Qwen3-32B
  129. 29.2 FreedomIntelligence AceGPT-7B
  130. 28.2 tiiuae/Falcon3-10B-Base
  131. 28.0 speakleash Bielik-11B-v2
  132. 27.9 Qwen2.5-0.5B
  133. 27.7 Sakalti oxyge1-33B
  134. 27.4 inceptionai/jais-adapted-7b
  135. 26.0 Llama 3.2 3B
  136. 25.9 tiiuae/Falcon3-7B-Base
  137. 25.6 riotu-lab ArabianGPT-01B
  138. 25.5 Naseej noon-7b
  139. 25.5 Qwen3-0.6B
  140. 25.0 Gemma 2 9B
  141. 24.1 DeepSeek-R1-Distill-Qwen-14B
  142. 24.0 stabilityai ar-stablelm-2-base
  143. 23.9 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1
  144. 23.9 Llama 3.2 1B
  145. 23.6 Qwen/Qwen1.5-1.8B
  146. 23.5 qwen3-4b-instruct-2507
  147. 23.4 Gemma 2 2B IT
  148. 23.4 QwQ-32B
  149. 23.4 DeepSeek-R1-Distill-Qwen-32B
  150. 23.3 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1
  151. 23.3 TIGER-Lab Qwen2.5-Math-7B-CFT
  1. 1 Mushari440 Qwen3-8B-SFT-V2 85.7%
  2. 2 Applied-Innovation-Center Karnak 78%
  3. 3 AbdulmalekDS qwen72b-ar-lora 74.3%
  4. 4 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 74%
  5. 5 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 74%
  6. 6 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 73.9%
  7. 7 Lina-Z qwen_arabic_ft 73.9%
  8. 8 Sakalti Ultiima-72B 73.9%
  9. 9 rombodawg Rombos-LLM-V2.5-Qwen-72b 73.9%
  10. 10 Lina-Z arabic_llm_test 73.3%
  11. 11 MaziyarPanahi calme-2.1-qwen2.5-72b 73.1%
  12. 12 MaziyarPanahi calme-2.2-qwen2.5-72b 73%
  13. 13 Qwen2.5-72B 72.2%
  14. 14 Ocelotr Qwen-8B-SFT 68.9%
  15. 15 FreedomIntelligence AceGPT-v2-70B-Chat 68.3%
  16. 16 Qwen2-72B base model 68.1%
  17. 17 Applied-Innovation-Center AIC-1 67.9%
  18. 18 Llama 3.3 70B Instruct 67.8%
  19. 19 Mushari440 qwen3-8B-SFT 67.5%
  20. 20 Gemma 3 27B 66.9%
  21. 21 tanliboy lambda-qwen2.5-32b-dpo-test 66.8%
  22. 22 maldv Qwentile2.5-32B-Instruct 66.7%
  23. 23 maldv Awqward2.5-32B-Instruct 66.4%
  24. 24 Qwen2.5-32B 65.7%
  25. 25 FreedomIntelligence AceGPT-v2-70B 65.7%
  26. 26 TIGER-Lab Qwen2.5-32B-Instruct-CFT 65.7%
  27. 27 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 65.7%
  28. 28 huihui-ai Qwen2.5-32B-Instruct-abliterated 65.6%
  29. 29 Daemontatox Cogito-R1 64.7%
  30. 30 Sakalti Saka-14B 64.6%
  31. 31 Sakalti SakaMoe-3x14B-Instruct 64.1%
  32. 32 TarjamaN Pronoia-14b-community 63.3%
  33. 33 freewheelin free-evo-qwen72b-v0.8-re 62.5%
  34. 34 Cran-May tempmotacilla-cinerea-0308 62.5%
  35. 35 Qwen2.5-14B base model 61.7%
  36. 36 Mushari440 qwen-3-8B-sft-grpo 61.6%
  37. 37 hammh0a Hala-9B 61.5%
  38. 38 Qwen3-8B base model 61.5%
  39. 39 FreedomIntelligence AceGPT-v2-32B-Chat 61.2%
  40. 40 Gemma 3 12B base model 61.2%
  41. 41 Gemma 2 27B 61.2%
  42. 42 AXCXEPT EZO-Qwen2.5-32B-Instruct 60.7%
  43. 43 QCRI/Fanar-1-9B 59.5%
  44. 44 tanliboy lambda-qwen2.5-14b-dpo-test 58.9%
  45. 45 Aya Expanse 32B 58.9%
  46. 46 QCRI/Fanar-1-9B-Instruct 58.6%
  47. 47 Ocelotr Qwen3-8B-GAE 57.9%
  48. 48 JasperV13 Yehia-7B-SFT-Reasoning-preview 57%
  49. 49 rombodawg Rombos-LLM-V2.6-Qwen-14b 57%
  50. 50 Slim205 Barka-9b-it 56.6%
  51. 51 recoilme recoilme-gemma-2-9B-v0.4 56.4%
  52. 52 AIDC-AI Marco-LLM-AR-V4 56.4%
  53. 53 v000000 Qwen2.5-Lumen-14B 56.4%
  54. 54 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 56.1%
  55. 55 inceptionai/jais-adapted-70b-chat 55.9%
  56. 56 airev-ai emirati-14b-v2 55.3%
  57. 57 JasperV13 Yehia-7B-Reasoning-preview 55.3%
  58. 58 AIDC-AI Marco-LLM-AR-V2 55.2%
  59. 59 JasperV13 Yehia-7B-DPO-Reasoning-preview 55.1%
  60. 60 CohereForAI/aya-23-35B 54%
  61. 61 Qwen3-4B base model 53.7%
  62. 62 Cran-May T.E-8.1 53.4%
  63. 63 Navid-AI Yehia-7B-preview 53.4%
  64. 64 asas-ai AceGPT-v2-32B 52.9%
  65. 65 FreedomIntelligence AceGPT-v2-32B 52.9%
  66. 66 ALLaM-AI/ALLaM-7B-Instruct-preview 52.8%
  67. 67 inceptionai/jais-family-30b-16k-chat 52.7%
  68. 68 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 52.5%
  69. 69 inceptionai/jais-family-30b-8k-chat 51.9%
  70. 70 Qwen2.5-7B base model 51.7%
  71. 71 arcee-ai Arcee-Spark 51.3%
  72. 72 Qwen2-7B-Instruct 51.3%
  73. 73 FreedomIntelligence AceGPT-v2-8B-Chat 50.9%
  74. 74 CohereForAI/c4ai-command-r7b-arabic-02-2025 50.1%
  75. 75 Aya Expanse 8B 49.2%
  76. 76 SeaLLMs SeaLLM-7B-v2.5 48.7%
  77. 77 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 48.6%
  78. 78 SeaLLMs SeaLLMs-v3-7B-Chat 48.2%
  79. 79 inceptionai/jais-adapted-70b 48%
  80. 80 Llama 3.1-8B 47.9%
  81. 81 Qwen3-14B base model 47.8%
  82. 82 AIDC-AI Marco-LLM-AR 47.5%
  83. 83 01-ZeroOne SUHAIL-14B-preview 47%
  84. 84 inceptionai/jais-family-13b-chat 47%
  85. 85 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 46.9%
  86. 86 FreedomIntelligence AceGPT-v1.5-13B-Chat 46.5%
  87. 87 amine-khelif BARS 46.4%
  88. 88 inceptionai/jais-adapted-13b-chat 45.5%
  89. 89 Qwen1.5-32B base model 45.3%
  90. 90 Gemma 3 4B base model 45.2%
  91. 91 recoilme recoilme-gemma-2-9B-v0.2 44%
  92. 92 Qwen3-1.7B base model 43.9%
  93. 93 inceptionai/jais-family-30b-16k 43%
  94. 94 inceptionai/jais-family-30b-8k 42.8%
  95. 95 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 42.3%
  96. 96 Phi-4 Mini 42.2%
  97. 97 Slim205 Barka-2b-it 40.4%
  98. 98 silma-ai SILMA-9B-Instruct-v1.0 40.3%
  99. 99 yellowtown 7B-v0.2 39.8%
  100. 100 FreedomIntelligence AceGPT-13B-chat 39.3%
  101. 101 Phi-4 39.1%
  102. 102 Qwen1.5-14B base model 38%
  103. 103 inceptionai/jais-adapted-7b-chat 38%
  104. 104 Qwen3-30B-A3B-Instruct Jul 2025 37.4%
  105. 105 Qwen2-1.5B-Instruct 37.4%
  106. 106 upstage SOLAR-10.7B-v1.0 37.1%
  107. 107 huihui-ai Qwen2.5-7B-Instruct-abliterated 36.8%
  108. 108 Qwen2.5-3B 35.9%
  109. 109 Qwen2.5-1.5B 35.9%
  110. 110 silma-ai SILMA-Kashif-2B-Instruct-v1.0 35.8%
  111. 111 FreedomIntelligence AceGPT-v1.5-13B 35.7%
  112. 112 Hagrass LLama3-3.2-instruct-trained 35.5%
  113. 113 FreedomIntelligence AceGPT-7B-chat 34.5%
  114. 114 Qwen1.5-7B base model 34.4%
  115. 115 Sakalti Saka-1.5B 33.5%
  116. 116 INSAIT-Institute BgGPT-7B-Instruct-v0.2 33.2%
  117. 117 inceptionai/jais-adapted-13b 33.1%
  118. 118 01-ai/Yi-1.5-34B-32K base model 32.2%
  119. 119 FreedomIntelligence AceGPT-13B 32.1%
  120. 120 AIDC-AI Marco-LLM-AR-V3 31.6%
  121. 121 01-ai/Yi-1.5-9B base model 31.4%
  122. 122 inceptionai/jais-family-13b 31%
  123. 123 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 31%
  124. 124 01-ai/Yi-1.5-9B-32K base model 30.4%
  125. 125 cognitivecomputations Dolphin3.0-R1-Mistral-24B 29.8%
  126. 126 Qwen/Qwen1.5-4B base model 29.8%
  127. 127 Qwen2-0.5B-Instruct 29.6%
  128. 128 Qwen3-32B 29.5%
  129. 129 FreedomIntelligence AceGPT-7B 29.2%
  130. 130 tiiuae/Falcon3-10B-Base base model 28.2%
  131. 131 speakleash Bielik-11B-v2 28%
  132. 132 Qwen2.5-0.5B 27.9%
  133. 133 Sakalti oxyge1-33B 27.7%
  134. 134 inceptionai/jais-adapted-7b 27.4%
  135. 135 Llama 3.2 3B 26%
  136. 136 tiiuae/Falcon3-7B-Base base model 25.9%
  137. 137 riotu-lab ArabianGPT-01B 25.6%
  138. 138 Naseej noon-7b 25.5%
  139. 139 Qwen3-0.6B 25.5%
  140. 140 Gemma 2 9B 25%
  141. 141 DeepSeek-R1-Distill-Qwen-14B 24.1%
  142. 142 stabilityai ar-stablelm-2-base 24%
  143. 143 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 23.9%
  144. 144 Llama 3.2 1B base model 23.9%
  145. 145 Qwen/Qwen1.5-1.8B base model 23.6%
  146. 146 qwen3-4b-instruct-2507 23.5%
  147. 147 Gemma 2 2B IT 23.4%
  148. 148 QwQ-32B 23.4%
  149. 149 DeepSeek-R1-Distill-Qwen-32B 23.4%
  150. 150 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 23.3%
  151. 151 TIGER-Lab Qwen2.5-Math-7B-CFT 23.3%
#EntryScoreSettingDateSource
1 Mushari440 Qwen3-8B-SFT-V2 85.7% Mar 2026 Official leaderboard
2 Applied-Innovation-Center Karnak 78% Feb 2026 Official leaderboard
3 AbdulmalekDS qwen72b-ar-lora 74.3% Sep 2025 Official leaderboard
4 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 74% Sep 2025 Official leaderboard
5 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 74% Jul 2025 Official leaderboard
6 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 73.9% Dec 2025 Official leaderboard
7 Lina-Z qwen_arabic_ft 73.9% Jan 2026 Official leaderboard
8 Sakalti Ultiima-72B 73.9% Feb 2025 Official leaderboard
9 rombodawg Rombos-LLM-V2.5-Qwen-72b 73.9% Feb 2025 Official leaderboard
10 Lina-Z arabic_llm_test 73.3% Nov 2025 Official leaderboard
11 MaziyarPanahi calme-2.1-qwen2.5-72b 73.1% Jan 2025 Official leaderboard
12 MaziyarPanahi calme-2.2-qwen2.5-72b 73% Jan 2025 Official leaderboard
13 Qwen2.5-72B 72.2% Sep 2024 Official leaderboard
14 Ocelotr Qwen-8B-SFT 68.9% Dec 2025 Official leaderboard
15 FreedomIntelligence AceGPT-v2-70B-Chat 68.3% Jan 2025 Official leaderboard
16 Qwen2-72B 68.1% base model Jun 2024 Official leaderboard
17 Applied-Innovation-Center AIC-1 67.9% Sep 2025 Official leaderboard
18 Llama 3.3 70B Instruct 67.8% Dec 2024 Official leaderboard
19 Mushari440 qwen3-8B-SFT 67.5% Feb 2026 Official leaderboard
20 Gemma 3 27B 66.9% Mar 2025 Official leaderboard
21 tanliboy lambda-qwen2.5-32b-dpo-test 66.8% Jan 2025 Official leaderboard
22 maldv Qwentile2.5-32B-Instruct 66.7% Feb 2025 Official leaderboard
23 maldv Awqward2.5-32B-Instruct 66.4% Feb 2025 Official leaderboard
24 Qwen2.5-32B 65.7% Sep 2024 Official leaderboard
25 FreedomIntelligence AceGPT-v2-70B 65.7% Jan 2025 Official leaderboard
26 TIGER-Lab Qwen2.5-32B-Instruct-CFT 65.7% Mar 2025 Official leaderboard
27 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 65.7% Mar 2025 Official leaderboard
28 huihui-ai Qwen2.5-32B-Instruct-abliterated 65.6% Jan 2025 Official leaderboard
29 Daemontatox Cogito-R1 64.7% Feb 2025 Official leaderboard
30 Sakalti Saka-14B 64.6% Feb 2025 Official leaderboard
31 Sakalti SakaMoe-3x14B-Instruct 64.1% Feb 2025 Official leaderboard
32 TarjamaN Pronoia-14b-community 63.3% Feb 2025 Official leaderboard
33 freewheelin free-evo-qwen72b-v0.8-re 62.5% Jan 2025 Official leaderboard
34 Cran-May tempmotacilla-cinerea-0308 62.5% Mar 2025 Official leaderboard
35 Qwen2.5-14B 61.7% base model Sep 2024 Official leaderboard
36 Mushari440 qwen-3-8B-sft-grpo 61.6% Feb 2026 Official leaderboard
37 hammh0a Hala-9B 61.5% Feb 2026 Official leaderboard
38 Qwen3-8B 61.5% base model Apr 2025 Official leaderboard
39 FreedomIntelligence AceGPT-v2-32B-Chat 61.2% Jan 2025 Official leaderboard
40 Gemma 3 12B 61.2% base model Mar 2025 Official leaderboard
41 Gemma 2 27B 61.2% Jun 2024 Official leaderboard
42 AXCXEPT EZO-Qwen2.5-32B-Instruct 60.7% Jan 2025 Official leaderboard
43 QCRI/Fanar-1-9B 59.5% Jun 2025 Official leaderboard
44 tanliboy lambda-qwen2.5-14b-dpo-test 58.9% Jan 2025 Official leaderboard
45 Aya Expanse 32B 58.9% Oct 2024 Official leaderboard
46 QCRI/Fanar-1-9B-Instruct 58.6% Jun 2025 Official leaderboard
47 Ocelotr Qwen3-8B-GAE 57.9% Jan 2026 Official leaderboard
48 JasperV13 Yehia-7B-SFT-Reasoning-preview 57% May 2025 Official leaderboard
49 rombodawg Rombos-LLM-V2.6-Qwen-14b 57% Jan 2025 Official leaderboard
50 Slim205 Barka-9b-it 56.6% Jun 2025 Official leaderboard
51 recoilme recoilme-gemma-2-9B-v0.4 56.4% Jan 2025 Official leaderboard
52 AIDC-AI Marco-LLM-AR-V4 56.4% Mar 2025 Official leaderboard
53 v000000 Qwen2.5-Lumen-14B 56.4% Jan 2025 Official leaderboard
54 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 56.1% Jan 2025 Official leaderboard
55 inceptionai/jais-adapted-70b-chat 55.9% Jan 2025 Official leaderboard
56 airev-ai emirati-14b-v2 55.3% Jan 2025 Official leaderboard
57 JasperV13 Yehia-7B-Reasoning-preview 55.3% May 2025 Official leaderboard
58 AIDC-AI Marco-LLM-AR-V2 55.2% Mar 2025 Official leaderboard
59 JasperV13 Yehia-7B-DPO-Reasoning-preview 55.1% May 2025 Official leaderboard
60 CohereForAI/aya-23-35B 54% Jan 2025 Official leaderboard
61 Qwen3-4B 53.7% base model Apr 2025 Official leaderboard
62 Cran-May T.E-8.1 53.4% Jan 2025 Official leaderboard
63 Navid-AI Yehia-7B-preview 53.4% Mar 2025 Official leaderboard
64 asas-ai AceGPT-v2-32B 52.9% Jan 2025 Official leaderboard
65 FreedomIntelligence AceGPT-v2-32B 52.9% Jan 2025 Official leaderboard
66 ALLaM-AI/ALLaM-7B-Instruct-preview 52.8% Feb 2025 Official leaderboard
67 inceptionai/jais-family-30b-16k-chat 52.7% Jan 2025 Official leaderboard
68 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 52.5% Jan 2025 Official leaderboard
69 inceptionai/jais-family-30b-8k-chat 51.9% Jan 2025 Official leaderboard
70 Qwen2.5-7B 51.7% base model Sep 2024 Official leaderboard
71 arcee-ai Arcee-Spark 51.3% Jan 2025 Official leaderboard
72 Qwen2-7B-Instruct 51.3% Jun 2024 Official leaderboard
73 FreedomIntelligence AceGPT-v2-8B-Chat 50.9% Jan 2025 Official leaderboard
74 CohereForAI/c4ai-command-r7b-arabic-02-2025 50.1% Feb 2025 Official leaderboard
75 Aya Expanse 8B 49.2% Oct 2024 Official leaderboard
76 SeaLLMs SeaLLM-7B-v2.5 48.7% Jan 2025 Official leaderboard
77 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 48.6% Jan 2025 Official leaderboard
78 SeaLLMs SeaLLMs-v3-7B-Chat 48.2% Mar 2025 Official leaderboard
79 inceptionai/jais-adapted-70b 48% Jan 2025 Official leaderboard
80 Llama 3.1-8B 47.9% Jul 2024 Official leaderboard
81 Qwen3-14B 47.8% base model Apr 2025 Official leaderboard
82 AIDC-AI Marco-LLM-AR 47.5% Feb 2025 Official leaderboard
83 01-ZeroOne SUHAIL-14B-preview 47% Jul 2025 Official leaderboard
84 inceptionai/jais-family-13b-chat 47% Jan 2025 Official leaderboard
85 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 46.9% Jan 2025 Official leaderboard
86 FreedomIntelligence AceGPT-v1.5-13B-Chat 46.5% Jan 2025 Official leaderboard
87 amine-khelif BARS 46.4% Sep 2025 Official leaderboard
88 inceptionai/jais-adapted-13b-chat 45.5% Jan 2025 Official leaderboard
89 Qwen1.5-32B 45.3% base model Feb 2024 Official leaderboard
90 Gemma 3 4B 45.2% base model Mar 2025 Official leaderboard
91 recoilme recoilme-gemma-2-9B-v0.2 44% Jan 2025 Official leaderboard
92 Qwen3-1.7B 43.9% base model Apr 2025 Official leaderboard
93 inceptionai/jais-family-30b-16k 43% Jan 2025 Official leaderboard
94 inceptionai/jais-family-30b-8k 42.8% Jan 2025 Official leaderboard
95 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 42.3% Jan 2025 Official leaderboard
96 Phi-4 Mini 42.2% Feb 2025 Official leaderboard
97 Slim205 Barka-2b-it 40.4% Jun 2025 Official leaderboard
98 silma-ai SILMA-9B-Instruct-v1.0 40.3% Jan 2025 Official leaderboard
99 yellowtown 7B-v0.2 39.8% Jan 2025 Official leaderboard
100 FreedomIntelligence AceGPT-13B-chat 39.3% Jan 2025 Official leaderboard
101 Phi-4 39.1% Dec 2024 Official leaderboard
102 Qwen1.5-14B 38% base model Feb 2024 Official leaderboard
103 inceptionai/jais-adapted-7b-chat 38% Jan 2025 Official leaderboard
104 Qwen3-30B-A3B-Instruct Jul 2025 37.4% Jul 2025 Official leaderboard
105 Qwen2-1.5B-Instruct 37.4% Jun 2024 Official leaderboard
106 upstage SOLAR-10.7B-v1.0 37.1% Jan 2025 Official leaderboard
107 huihui-ai Qwen2.5-7B-Instruct-abliterated 36.8% Jan 2025 Official leaderboard
108 Qwen2.5-3B 35.9% Sep 2024 Official leaderboard
109 Qwen2.5-1.5B 35.9% Sep 2024 Official leaderboard
110 silma-ai SILMA-Kashif-2B-Instruct-v1.0 35.8% Feb 2025 Official leaderboard
111 FreedomIntelligence AceGPT-v1.5-13B 35.7% Jan 2025 Official leaderboard
112 Hagrass LLama3-3.2-instruct-trained 35.5% Aug 2025 Official leaderboard
113 FreedomIntelligence AceGPT-7B-chat 34.5% Jan 2025 Official leaderboard
114 Qwen1.5-7B 34.4% base model Feb 2024 Official leaderboard
115 Sakalti Saka-1.5B 33.5% Feb 2025 Official leaderboard
116 INSAIT-Institute BgGPT-7B-Instruct-v0.2 33.2% Jan 2025 Official leaderboard
117 inceptionai/jais-adapted-13b 33.1% Jan 2025 Official leaderboard
118 01-ai/Yi-1.5-34B-32K 32.2% base model Jan 2025 Official leaderboard
119 FreedomIntelligence AceGPT-13B 32.1% Jan 2025 Official leaderboard
120 AIDC-AI Marco-LLM-AR-V3 31.6% Mar 2025 Official leaderboard
121 01-ai/Yi-1.5-9B 31.4% base model Jan 2025 Official leaderboard
122 inceptionai/jais-family-13b 31% Jan 2025 Official leaderboard
123 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 31% Jan 2025 Official leaderboard
124 01-ai/Yi-1.5-9B-32K 30.4% base model Jan 2025 Official leaderboard
125 cognitivecomputations Dolphin3.0-R1-Mistral-24B 29.8% Feb 2025 Official leaderboard
126 Qwen/Qwen1.5-4B 29.8% base model Jan 2025 Official leaderboard
127 Qwen2-0.5B-Instruct 29.6% Jun 2024 Official leaderboard
128 Qwen3-32B 29.5% Apr 2025 Official leaderboard
129 FreedomIntelligence AceGPT-7B 29.2% Jan 2025 Official leaderboard
130 tiiuae/Falcon3-10B-Base 28.2% base model Feb 2025 Official leaderboard
131 speakleash Bielik-11B-v2 28% Jan 2025 Official leaderboard
132 Qwen2.5-0.5B 27.9% Sep 2024 Official leaderboard
133 Sakalti oxyge1-33B 27.7% Mar 2025 Official leaderboard
134 inceptionai/jais-adapted-7b 27.4% Jan 2025 Official leaderboard
135 Llama 3.2 3B 26% Sep 2024 Official leaderboard
136 tiiuae/Falcon3-7B-Base 25.9% base model Feb 2025 Official leaderboard
137 riotu-lab ArabianGPT-01B 25.6% Oct 2025 Official leaderboard
138 Naseej noon-7b 25.5% Mar 2025 Official leaderboard
139 Qwen3-0.6B 25.5% Apr 2025 Official leaderboard
140 Gemma 2 9B 25% Jun 2024 Official leaderboard
141 DeepSeek-R1-Distill-Qwen-14B 24.1% Jan 2025 Official leaderboard
142 stabilityai ar-stablelm-2-base 24% Jun 2025 Official leaderboard
143 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 23.9% Feb 2025 Official leaderboard
144 Llama 3.2 1B 23.9% base model Sep 2024 Official leaderboard
145 Qwen/Qwen1.5-1.8B 23.6% base model Jan 2025 Official leaderboard
146 qwen3-4b-instruct-2507 23.5% Aug 2025 Official leaderboard
147 Gemma 2 2B IT 23.4% Jul 2024 Official leaderboard
148 QwQ-32B 23.4% Mar 2025 Official leaderboard
149 DeepSeek-R1-Distill-Qwen-32B 23.4% Jan 2025 Official leaderboard
150 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 23.3% Feb 2025 Official leaderboard
151 TIGER-Lab Qwen2.5-Math-7B-CFT 23.3% Mar 2025 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen2.5-72B
base model
72.2%
Gemma 3 27B
base model
66.9%
Qwen2.5-32B
base model
65.7%
Qwen2.5-14B
base model
61.7%
Qwen3-8B
base model
61.5%
Gemma 2 27B
base model
61.2%
Qwen3-4B
base model
53.7%
Qwen2.5-7B
base model
51.7%
Llama 3.1-8B
base model
47.9%
Qwen3-14B
base model
47.8%
Qwen2.5-3B
base model
35.9%
Qwen2.5-1.5B
base model
35.9%
ModelEffortHarnessScoreCost per taskOutput tokens
Qwen2.5-72B 72.2%
Qwen2-72B base model 68.1%
Llama 3.3 70B Instruct 67.8%
Gemma 3 27B 66.9%
Gemma 3 27B base model 66.6%
Qwen2.5-72B base model 66.4%
Qwen2.5-32B 65.7%
Qwen2.5-14B base model 61.7%
Qwen3-8B base model 61.5%
Gemma 3 12B base model 61.2%
Gemma 2 27B 61.2%
Aya Expanse 32B 58.9%
Qwen2.5-32B base model 56.2%
Qwen2.5-14B 55.7%
Qwen3-4B base model 53.7%
Qwen2.5-7B base model 51.7%
Qwen2-7B-Instruct 51.3%
Aya Expanse 8B 49.2%
Llama 3.1-8B 47.9%
Qwen3-14B base model 47.8%
Qwen1.5-32B base model 45.3%
Gemma 3 4B base model 45.2%
Qwen3-1.7B base model 43.9%
Llama 3.1-8B base model 42.7%
Phi-4 Mini 42.2%
Qwen2.5-7B 40.3%
Phi-4 39.1%
Qwen1.5-14B base model 38%
Qwen3-30B-A3B-Instruct Jul 2025 37.4%
Qwen2-1.5B-Instruct 37.4%
Qwen2.5-3B 35.9%
Qwen2.5-1.5B 35.9%
Qwen1.5-7B base model 34.4%
Qwen2.5-3B base model 31.2%
Qwen3-14B 29.9%
Qwen2-0.5B-Instruct 29.6%
Qwen3-32B 29.5%
Qwen2.5-0.5B 27.9%
Qwen3-8B 27.3%
Qwen3-4B 26.6%
Llama 3.2 3B 26%
Qwen2.5-1.5B base model 25.6%
Qwen3-0.6B 25.5%
Gemma 2 9B 25%
DeepSeek-R1-Distill-Qwen-14B 24.1%
Llama 3.2 1B base model 23.9%
qwen3-4b-instruct-2507 23.5%
Gemma 2 2B IT 23.4%
QwQ-32B 23.4%
DeepSeek-R1-Distill-Qwen-32B 23.4%
Qwen2.5-0.5B base model 23.4%
Gemma 2 27B base model 23.3%

2 models have held number one since its release in Sep 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Own leaderboard Not tested since Aug 2025
Qwen2.5-72B 72.2%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Qwen2.5-72B Qwen2-72B Qwen1.5-32B 0% 20% 40% 60% 80% 100% 2024 2025 Near the ceiling (85%) Qwen2.5-72B Qwen1.5-32B

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%