Benchmark Almanac

AlGhafa

Mushari440 Qwen3-8B-SFT-V2 84.8% tops its leaderboard Qwen2-72B 78.6% leads the models we track Only in system cards

Models answer Arabic multiple-choice tasks on reading comprehension, sentiment, facts and question answering. Scored by normalized log-likelihood accuracy.

It started with 11 native Arabic sets and later added 11 translated ones. OALL v2 keeps only the native sets and fixed a scoring bug, so v1 and v2 AlGhafa scores differ.

In the authors’ words

Recent advances in the space of Arabic large language models have opened up a wealth of potential practical applications. From optimal training strategies, large scale data acquisition and continuously increasing NLP resources, the Arabic LLM landscape has improved in a very short span of time, despite being plagued by training data scarcity and limited evaluation resources compared to English. In line with contributing towards this ever-growing field, we introduce AlGhafa, a new multiple-choice evaluation benchmark for Arabic LLMs. For showcasing purposes, we train a new suite of models, including a 14 billion parameter model, the largest monolingual Arabic decoder-only model to date. We use a collection of publicly available datasets, as well as a newly introduced HandMade dataset consisting of 8 billion tokens. Finally, we explore the quantitative and qualitative toxicity of several Arabic models, comparing our models to existing public Arabic LLMs.

Abstract of AlGhafa Evaluation Benchmark for Arabic Language Models (ACL 2023.arabicnlp-1.21, CC BY 4.0)
Made by Technology Innovation Institute (Almazrouei et al.) Released Dec 2023 Knowledge in many languages All tasks public Last in a system card Nov 2025 Last model added Nov 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
Code Paper Leaderboard
151
Entries on its own leaderboard, 39 models we track
1
System card reports it, from 1 lab

Mushari440 Qwen3-8B-SFT-V2 tops the leaderboard with 84.8%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Qwen2-72B, at 78.6%.

The newest model with a score here came out in Aug 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

151 entries. Scroll sideways for the rest.

  1. 84.8 Mushari440 Qwen3-8B-SFT-V2
  2. 82.8 Mushari440 qwen3-8B-SFT
  3. 79.0 Applied-Innovation-Center Karnak
  4. 79.0 Applied-Innovation-Center AIC-1
  5. 78.7 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1
  6. 78.6 AbdulmalekDS qwen72b-ar-lora
  7. 78.6 Qwen2-72B
  8. 78.4 Lina-Z arabic_llm_test
  9. 78.2 Qwen2.5-72B
  10. 78.2 rombodawg Rombos-LLM-V2.5-Qwen-72b
  11. 78.2 tanliboy lambda-qwen2.5-32b-dpo-test
  12. 78.2 Sakalti Ultiima-72B
  13. 78.2 hammh0a Hala-9B
  14. 78.2 Lina-Z qwen_arabic_ft
  15. 78.1 MaziyarPanahi calme-2.2-qwen2.5-72b
  16. 78.1 Qwen2.5-32B
  17. 78.1 MaziyarPanahi calme-2.1-qwen2.5-72b
  18. 78.0 huihui-ai Qwen2.5-32B-Instruct-abliterated
  19. 78.0 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1
  20. 78.0 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2
  21. 78.0 maldv Awqward2.5-32B-Instruct
  22. 78.0 maldv Qwentile2.5-32B-Instruct
  23. 78.0 zetasepic Qwen2.5-32B-Instruct-abliterated-v2
  24. 78.0 TIGER-Lab Qwen2.5-32B-Instruct-CFT
  25. 77.9 Gemma 3 27B
  26. 77.7 Daemontatox Cogito-R1
  27. 77.6 Aya Expanse 32B
  28. 77.3 FreedomIntelligence AceGPT-v2-32B-Chat
  29. 77.2 Qwen2.5-14B
  30. 76.6 FreedomIntelligence AceGPT-v2-70B-Chat
  31. 76.4 QCRI/Fanar-1-9B-Instruct
  32. 76.3 freewheelin free-evo-qwen72b-v0.8-re
  33. 76.3 Sakalti Saka-14B
  34. 75.9 Sakalti SakaMoe-3x14B-Instruct
  35. 75.9 CohereForAI/aya-23-35B
  36. 75.5 Cran-May tempmotacilla-cinerea-0308
  37. 75.5 Ocelotr Qwen3-8B-GAE
  38. 75.2 JasperV13 Yehia-7B-Reasoning-preview
  39. 75.2 inceptionai/jais-adapted-70b-chat
  40. 75.1 JasperV13 Yehia-7B-DPO-Reasoning-preview
  41. 75.0 AIDC-AI Marco-LLM-AR-V4
  42. 75.0 AXCXEPT EZO-Qwen2.5-32B-Instruct
  43. 74.9 Mushari440 qwen-3-8B-sft-grpo
  44. 74.8 Qwen3-8B
  45. 74.8 CohereForAI/c4ai-command-r7b-arabic-02-2025
  46. 74.8 tanliboy lambda-qwen2.5-14b-dpo-test
  47. 74.6 airev-ai emirati-14b-v2
  48. 74.4 recoilme recoilme-gemma-2-9B-v0.4
  49. 74.1 rombodawg Rombos-LLM-V2.6-Qwen-14b
  50. 73.8 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct
  51. 73.5 FreedomIntelligence AceGPT-v2-8B-Chat
  52. 73.2 Qwen2-7B-Instruct
  53. 73.2 Cran-May T.E-8.1
  54. 72.9 arcee-ai Arcee-Spark
  55. 72.8 amine-khelif BARS
  56. 72.8 JasperV13 Yehia-7B-SFT-Reasoning-preview
  57. 72.7 v000000 Qwen2.5-Lumen-14B
  58. 72.5 recoilme recoilme-gemma-2-9B-v0.2
  59. 72.5 v000000 Qwen2.5-14B-Gutenberg-1e-Delta
  60. 72.4 AIDC-AI Marco-LLM-AR-V2
  61. 72.2 Gemma 2 27B
  62. 72.2 Qwen2.5-7B
  63. 72.0 Orion-zhen Qwen2.5-7B-Instruct-Uncensored
  64. 71.4 SeaLLMs SeaLLM-7B-v2.5
  65. 71.3 inceptionai/jais-family-30b-8k-chat
  66. 71.2 inceptionai/jais-family-30b-16k-chat
  67. 71.1 SeaLLMs SeaLLMs-v3-7B-Chat
  68. 70.9 Llama 3.1-8B
  69. 70.8 Navid-AI Yehia-7B-preview
  70. 70.7 Ocelotr Qwen-8B-SFT
  71. 70.7 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2
  72. 70.3 Sakalti oxyge1-33B
  73. 70.2 QCRI/Fanar-1-9B
  74. 70.2 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2
  75. 70.0 TarjamaN Pronoia-14b-community
  76. 69.5 ALLaM-AI/ALLaM-7B-Instruct-preview
  77. 69.3 Phi-4 Mini
  78. 68.5 Qwen3-4B
  79. 67.3 inceptionai/jais-adapted-13b-chat
  80. 66.7 Aya Expanse 8B
  81. 66.5 inceptionai/jais-family-13b-chat
  82. 66.4 Qwen1.5-32B
  83. 65.7 Qwen1.5-14B
  84. 63.7 Llama 3.3 70B Instruct
  85. 63.6 huihui-ai Qwen2.5-7B-Instruct-abliterated
  86. 63.4 inceptionai/jais-adapted-7b-chat
  87. 61.8 Slim205 Barka-2b-it
  88. 61.1 Qwen2.5-3B
  89. 60.7 Gemma 3 12B
  90. 60.7 Slim205 Barka-9b-it
  91. 60.2 INSAIT-Institute BgGPT-7B-Instruct-v0.2
  92. 59.7 silma-ai SILMA-Kashif-2B-Instruct-v1.0
  93. 59.4 upstage SOLAR-10.7B-v1.0
  94. 59.2 FreedomIntelligence AceGPT-13B-chat
  95. 58.7 FreedomIntelligence AceGPT-v2-70B
  96. 57.6 inceptionai/jais-family-30b-16k
  97. 57.1 Qwen3-14B
  98. 56.8 Qwen3-1.7B
  99. 56.8 Hagrass LLama3-3.2-instruct-trained
  100. 54.9 AIDC-AI Marco-LLM-AR
  101. 54.9 asas-ai AceGPT-v2-32B
  102. 54.9 FreedomIntelligence AceGPT-v2-32B
  103. 54.8 inceptionai/jais-adapted-70b
  104. 54.8 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo
  105. 54.7 cognitivecomputations Dolphin3.0-R1-Mistral-24B
  106. 54.5 01-ZeroOne SUHAIL-14B-preview
  107. 54.3 inceptionai/jais-family-30b-8k
  108. 54.2 Gemma 3 4B
  109. 54.0 FreedomIntelligence AceGPT-v1.5-13B-Chat
  110. 53.2 AIDC-AI Marco-LLM-AR-V3
  111. 53.1 Qwen2-1.5B-Instruct
  112. 52.3 Qwen1.5-7B
  113. 51.4 Sakalti Saka-1.5B
  114. 51.2 01-ai/Yi-1.5-9B
  115. 50.9 Qwen3-30B-A3B-Instruct
  116. 49.5 FreedomIntelligence AceGPT-7B-chat
  117. 49.0 01-ai/Yi-1.5-9B-32K
  118. 48.5 yellowtown 7B-v0.2
  119. 48.4 Qwen2.5-1.5B
  120. 48.2 FreedomIntelligence AceGPT-13B
  121. 48.0 speakleash Bielik-11B-v2
  122. 47.9 Gemma 2 9B
  123. 47.7 01-ai/Yi-1.5-34B-32K
  124. 46.8 Qwen3-32B
  125. 46.5 DeepSeek-R1-Distill-Qwen-32B
  126. 44.1 Qwen3-0.6B
  127. 44.1 tiiuae/Falcon3-10B-Base
  128. 44.0 FreedomIntelligence AceGPT-v1.5-13B
  129. 42.9 Llama 3.2 3B
  130. 42.9 inceptionai/jais-family-13b
  131. 42.4 Qwen/Qwen1.5-4B
  132. 42.3 Phi-4
  133. 40.6 inceptionai/jais-adapted-13b
  134. 40.6 Qwen2-0.5B-Instruct
  135. 40.4 Qwen2.5-0.5B
  136. 37.9 tiiuae/Falcon3-7B-Base
  137. 36.7 Naseej noon-7b
  138. 36.5 qwen3-4b-instruct-2507
  139. 36.4 FreedomIntelligence AceGPT-7B
  140. 34.1 Gemma 2 2B IT
  141. 34.0 silma-ai SILMA-9B-Instruct-v1.0
  142. 33.9 Llama 3.2 1B
  143. 33.5 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1
  144. 33.1 stabilityai ar-stablelm-2-base
  145. 32.9 inceptionai/jais-adapted-7b
  146. 32.8 TIGER-Lab Qwen2.5-Math-7B-CFT
  147. 32.7 Qwen/Qwen1.5-1.8B
  148. 32.7 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1
  149. 32.6 DeepSeek-R1-Distill-Qwen-14B
  150. 32.5 riotu-lab ArabianGPT-01B
  151. 31.3 QwQ-32B
  1. 1 Mushari440 Qwen3-8B-SFT-V2 84.8%
  2. 2 Mushari440 qwen3-8B-SFT 82.8%
  3. 3 Applied-Innovation-Center Karnak 79%
  4. 4 Applied-Innovation-Center AIC-1 79%
  5. 5 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 78.7%
  6. 6 AbdulmalekDS qwen72b-ar-lora 78.6%
  7. 7 Qwen2-72B base model 78.6%
  8. 8 Lina-Z arabic_llm_test 78.4%
  9. 9 Qwen2.5-72B 78.2%
  10. 10 rombodawg Rombos-LLM-V2.5-Qwen-72b 78.2%
  11. 11 tanliboy lambda-qwen2.5-32b-dpo-test 78.2%
  12. 12 Sakalti Ultiima-72B 78.2%
  13. 13 hammh0a Hala-9B 78.2%
  14. 14 Lina-Z qwen_arabic_ft 78.2%
  15. 15 MaziyarPanahi calme-2.2-qwen2.5-72b 78.1%
  16. 16 Qwen2.5-32B 78.1%
  17. 17 MaziyarPanahi calme-2.1-qwen2.5-72b 78.1%
  18. 18 huihui-ai Qwen2.5-32B-Instruct-abliterated 78%
  19. 19 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 78%
  20. 20 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 78%
  21. 21 maldv Awqward2.5-32B-Instruct 78%
  22. 22 maldv Qwentile2.5-32B-Instruct 78%
  23. 23 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 78%
  24. 24 TIGER-Lab Qwen2.5-32B-Instruct-CFT 78%
  25. 25 Gemma 3 27B 77.9%
  26. 26 Daemontatox Cogito-R1 77.7%
  27. 27 Aya Expanse 32B 77.6%
  28. 28 FreedomIntelligence AceGPT-v2-32B-Chat 77.3%
  29. 29 Qwen2.5-14B base model 77.2%
  30. 30 FreedomIntelligence AceGPT-v2-70B-Chat 76.6%
  31. 31 QCRI/Fanar-1-9B-Instruct 76.4%
  32. 32 freewheelin free-evo-qwen72b-v0.8-re 76.3%
  33. 33 Sakalti Saka-14B 76.3%
  34. 34 Sakalti SakaMoe-3x14B-Instruct 75.9%
  35. 35 CohereForAI/aya-23-35B 75.9%
  36. 36 Cran-May tempmotacilla-cinerea-0308 75.5%
  37. 37 Ocelotr Qwen3-8B-GAE 75.5%
  38. 38 JasperV13 Yehia-7B-Reasoning-preview 75.2%
  39. 39 inceptionai/jais-adapted-70b-chat 75.2%
  40. 40 JasperV13 Yehia-7B-DPO-Reasoning-preview 75.1%
  41. 41 AIDC-AI Marco-LLM-AR-V4 75%
  42. 42 AXCXEPT EZO-Qwen2.5-32B-Instruct 75%
  43. 43 Mushari440 qwen-3-8B-sft-grpo 74.9%
  44. 44 Qwen3-8B base model 74.8%
  45. 45 CohereForAI/c4ai-command-r7b-arabic-02-2025 74.8%
  46. 46 tanliboy lambda-qwen2.5-14b-dpo-test 74.8%
  47. 47 airev-ai emirati-14b-v2 74.6%
  48. 48 recoilme recoilme-gemma-2-9B-v0.4 74.4%
  49. 49 rombodawg Rombos-LLM-V2.6-Qwen-14b 74.1%
  50. 50 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 73.8%
  51. 51 FreedomIntelligence AceGPT-v2-8B-Chat 73.5%
  52. 52 Qwen2-7B-Instruct 73.2%
  53. 53 Cran-May T.E-8.1 73.2%
  54. 54 arcee-ai Arcee-Spark 72.9%
  55. 55 amine-khelif BARS 72.8%
  56. 56 JasperV13 Yehia-7B-SFT-Reasoning-preview 72.8%
  57. 57 v000000 Qwen2.5-Lumen-14B 72.7%
  58. 58 recoilme recoilme-gemma-2-9B-v0.2 72.5%
  59. 59 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 72.5%
  60. 60 AIDC-AI Marco-LLM-AR-V2 72.4%
  61. 61 Gemma 2 27B 72.2%
  62. 62 Qwen2.5-7B base model 72.2%
  63. 63 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 72%
  64. 64 SeaLLMs SeaLLM-7B-v2.5 71.4%
  65. 65 inceptionai/jais-family-30b-8k-chat 71.3%
  66. 66 inceptionai/jais-family-30b-16k-chat 71.2%
  67. 67 SeaLLMs SeaLLMs-v3-7B-Chat 71.1%
  68. 68 Llama 3.1-8B 70.9%
  69. 69 Navid-AI Yehia-7B-preview 70.8%
  70. 70 Ocelotr Qwen-8B-SFT 70.7%
  71. 71 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 70.7%
  72. 72 Sakalti oxyge1-33B 70.3%
  73. 73 QCRI/Fanar-1-9B 70.2%
  74. 74 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 70.2%
  75. 75 TarjamaN Pronoia-14b-community 70%
  76. 76 ALLaM-AI/ALLaM-7B-Instruct-preview 69.5%
  77. 77 Phi-4 Mini 69.3%
  78. 78 Qwen3-4B base model 68.5%
  79. 79 inceptionai/jais-adapted-13b-chat 67.3%
  80. 80 Aya Expanse 8B 66.7%
  81. 81 inceptionai/jais-family-13b-chat 66.5%
  82. 82 Qwen1.5-32B base model 66.4%
  83. 83 Qwen1.5-14B base model 65.7%
  84. 84 Llama 3.3 70B Instruct 63.7%
  85. 85 huihui-ai Qwen2.5-7B-Instruct-abliterated 63.6%
  86. 86 inceptionai/jais-adapted-7b-chat 63.4%
  87. 87 Slim205 Barka-2b-it 61.8%
  88. 88 Qwen2.5-3B 61.1%
  89. 89 Gemma 3 12B base model 60.7%
  90. 90 Slim205 Barka-9b-it 60.7%
  91. 91 INSAIT-Institute BgGPT-7B-Instruct-v0.2 60.2%
  92. 92 silma-ai SILMA-Kashif-2B-Instruct-v1.0 59.7%
  93. 93 upstage SOLAR-10.7B-v1.0 59.4%
  94. 94 FreedomIntelligence AceGPT-13B-chat 59.2%
  95. 95 FreedomIntelligence AceGPT-v2-70B 58.7%
  96. 96 inceptionai/jais-family-30b-16k 57.6%
  97. 97 Qwen3-14B base model 57.1%
  98. 98 Qwen3-1.7B base model 56.8%
  99. 99 Hagrass LLama3-3.2-instruct-trained 56.8%
  100. 100 AIDC-AI Marco-LLM-AR 54.9%
  101. 101 asas-ai AceGPT-v2-32B 54.9%
  102. 102 FreedomIntelligence AceGPT-v2-32B 54.9%
  103. 103 inceptionai/jais-adapted-70b 54.8%
  104. 104 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 54.8%
  105. 105 cognitivecomputations Dolphin3.0-R1-Mistral-24B 54.7%
  106. 106 01-ZeroOne SUHAIL-14B-preview 54.5%
  107. 107 inceptionai/jais-family-30b-8k 54.3%
  108. 108 Gemma 3 4B base model 54.2%
  109. 109 FreedomIntelligence AceGPT-v1.5-13B-Chat 54%
  110. 110 AIDC-AI Marco-LLM-AR-V3 53.2%
  111. 111 Qwen2-1.5B-Instruct 53.1%
  112. 112 Qwen1.5-7B base model 52.3%
  113. 113 Sakalti Saka-1.5B 51.4%
  114. 114 01-ai/Yi-1.5-9B base model 51.2%
  115. 115 Qwen3-30B-A3B-Instruct Jul 2025 50.9%
  116. 116 FreedomIntelligence AceGPT-7B-chat 49.5%
  117. 117 01-ai/Yi-1.5-9B-32K base model 49%
  118. 118 yellowtown 7B-v0.2 48.5%
  119. 119 Qwen2.5-1.5B 48.4%
  120. 120 FreedomIntelligence AceGPT-13B 48.2%
  121. 121 speakleash Bielik-11B-v2 48%
  122. 122 Gemma 2 9B 47.9%
  123. 123 01-ai/Yi-1.5-34B-32K base model 47.7%
  124. 124 Qwen3-32B 46.8%
  125. 125 DeepSeek-R1-Distill-Qwen-32B 46.5%
  126. 126 Qwen3-0.6B 44.1%
  127. 127 tiiuae/Falcon3-10B-Base base model 44.1%
  128. 128 FreedomIntelligence AceGPT-v1.5-13B 44%
  129. 129 Llama 3.2 3B 42.9%
  130. 130 inceptionai/jais-family-13b 42.9%
  131. 131 Qwen/Qwen1.5-4B base model 42.4%
  132. 132 Phi-4 42.3%
  133. 133 inceptionai/jais-adapted-13b 40.6%
  134. 134 Qwen2-0.5B-Instruct 40.6%
  135. 135 Qwen2.5-0.5B 40.4%
  136. 136 tiiuae/Falcon3-7B-Base base model 37.9%
  137. 137 Naseej noon-7b 36.7%
  138. 138 qwen3-4b-instruct-2507 36.5%
  139. 139 FreedomIntelligence AceGPT-7B 36.4%
  140. 140 Gemma 2 2B IT 34.1%
  141. 141 silma-ai SILMA-9B-Instruct-v1.0 34%
  142. 142 Llama 3.2 1B base model 33.9%
  143. 143 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 33.5%
  144. 144 stabilityai ar-stablelm-2-base 33.1%
  145. 145 inceptionai/jais-adapted-7b 32.9%
  146. 146 TIGER-Lab Qwen2.5-Math-7B-CFT 32.8%
  147. 147 Qwen/Qwen1.5-1.8B base model 32.7%
  148. 148 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 32.7%
  149. 149 DeepSeek-R1-Distill-Qwen-14B 32.6%
  150. 150 riotu-lab ArabianGPT-01B 32.5%
  151. 151 QwQ-32B 31.3%
#EntryScoreSettingDateSource
1 Mushari440 Qwen3-8B-SFT-V2 84.8% Mar 2026 Official leaderboard
2 Mushari440 qwen3-8B-SFT 82.8% Feb 2026 Official leaderboard
3 Applied-Innovation-Center Karnak 79% Feb 2026 Official leaderboard
4 Applied-Innovation-Center AIC-1 79% Sep 2025 Official leaderboard
5 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 78.7% Jul 2025 Official leaderboard
6 AbdulmalekDS qwen72b-ar-lora 78.6% Sep 2025 Official leaderboard
7 Qwen2-72B 78.6% base model Jun 2024 Official leaderboard
8 Lina-Z arabic_llm_test 78.4% Nov 2025 Official leaderboard
9 Qwen2.5-72B 78.2% Sep 2024 Official leaderboard
10 rombodawg Rombos-LLM-V2.5-Qwen-72b 78.2% Feb 2025 Official leaderboard
11 tanliboy lambda-qwen2.5-32b-dpo-test 78.2% Jan 2025 Official leaderboard
12 Sakalti Ultiima-72B 78.2% Feb 2025 Official leaderboard
13 hammh0a Hala-9B 78.2% Feb 2026 Official leaderboard
14 Lina-Z qwen_arabic_ft 78.2% Jan 2026 Official leaderboard
15 MaziyarPanahi calme-2.2-qwen2.5-72b 78.1% Jan 2025 Official leaderboard
16 Qwen2.5-32B 78.1% Sep 2024 Official leaderboard
17 MaziyarPanahi calme-2.1-qwen2.5-72b 78.1% Jan 2025 Official leaderboard
18 huihui-ai Qwen2.5-32B-Instruct-abliterated 78% Jan 2025 Official leaderboard
19 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 78% Dec 2025 Official leaderboard
20 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 78% Sep 2025 Official leaderboard
21 maldv Awqward2.5-32B-Instruct 78% Feb 2025 Official leaderboard
22 maldv Qwentile2.5-32B-Instruct 78% Feb 2025 Official leaderboard
23 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 78% Mar 2025 Official leaderboard
24 TIGER-Lab Qwen2.5-32B-Instruct-CFT 78% Mar 2025 Official leaderboard
25 Gemma 3 27B 77.9% Mar 2025 Official leaderboard
26 Daemontatox Cogito-R1 77.7% Feb 2025 Official leaderboard
27 Aya Expanse 32B 77.6% Oct 2024 Official leaderboard
28 FreedomIntelligence AceGPT-v2-32B-Chat 77.3% Jan 2025 Official leaderboard
29 Qwen2.5-14B 77.2% base model Sep 2024 Official leaderboard
30 FreedomIntelligence AceGPT-v2-70B-Chat 76.6% Jan 2025 Official leaderboard
31 QCRI/Fanar-1-9B-Instruct 76.4% Jun 2025 Official leaderboard
32 freewheelin free-evo-qwen72b-v0.8-re 76.3% Jan 2025 Official leaderboard
33 Sakalti Saka-14B 76.3% Feb 2025 Official leaderboard
34 Sakalti SakaMoe-3x14B-Instruct 75.9% Feb 2025 Official leaderboard
35 CohereForAI/aya-23-35B 75.9% Jan 2025 Official leaderboard
36 Cran-May tempmotacilla-cinerea-0308 75.5% Mar 2025 Official leaderboard
37 Ocelotr Qwen3-8B-GAE 75.5% Jan 2026 Official leaderboard
38 JasperV13 Yehia-7B-Reasoning-preview 75.2% May 2025 Official leaderboard
39 inceptionai/jais-adapted-70b-chat 75.2% Jan 2025 Official leaderboard
40 JasperV13 Yehia-7B-DPO-Reasoning-preview 75.1% May 2025 Official leaderboard
41 AIDC-AI Marco-LLM-AR-V4 75% Mar 2025 Official leaderboard
42 AXCXEPT EZO-Qwen2.5-32B-Instruct 75% Jan 2025 Official leaderboard
43 Mushari440 qwen-3-8B-sft-grpo 74.9% Feb 2026 Official leaderboard
44 Qwen3-8B 74.8% base model Apr 2025 Official leaderboard
45 CohereForAI/c4ai-command-r7b-arabic-02-2025 74.8% Feb 2025 Official leaderboard
46 tanliboy lambda-qwen2.5-14b-dpo-test 74.8% Jan 2025 Official leaderboard
47 airev-ai emirati-14b-v2 74.6% Jan 2025 Official leaderboard
48 recoilme recoilme-gemma-2-9B-v0.4 74.4% Jan 2025 Official leaderboard
49 rombodawg Rombos-LLM-V2.6-Qwen-14b 74.1% Jan 2025 Official leaderboard
50 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 73.8% Jan 2025 Official leaderboard
51 FreedomIntelligence AceGPT-v2-8B-Chat 73.5% Jan 2025 Official leaderboard
52 Qwen2-7B-Instruct 73.2% Jun 2024 Official leaderboard
53 Cran-May T.E-8.1 73.2% Jan 2025 Official leaderboard
54 arcee-ai Arcee-Spark 72.9% Jan 2025 Official leaderboard
55 amine-khelif BARS 72.8% Sep 2025 Official leaderboard
56 JasperV13 Yehia-7B-SFT-Reasoning-preview 72.8% May 2025 Official leaderboard
57 v000000 Qwen2.5-Lumen-14B 72.7% Jan 2025 Official leaderboard
58 recoilme recoilme-gemma-2-9B-v0.2 72.5% Jan 2025 Official leaderboard
59 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 72.5% Jan 2025 Official leaderboard
60 AIDC-AI Marco-LLM-AR-V2 72.4% Mar 2025 Official leaderboard
61 Gemma 2 27B 72.2% Jun 2024 Official leaderboard
62 Qwen2.5-7B 72.2% base model Sep 2024 Official leaderboard
63 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 72% Jan 2025 Official leaderboard
64 SeaLLMs SeaLLM-7B-v2.5 71.4% Jan 2025 Official leaderboard
65 inceptionai/jais-family-30b-8k-chat 71.3% Jan 2025 Official leaderboard
66 inceptionai/jais-family-30b-16k-chat 71.2% Jan 2025 Official leaderboard
67 SeaLLMs SeaLLMs-v3-7B-Chat 71.1% Mar 2025 Official leaderboard
68 Llama 3.1-8B 70.9% Jul 2024 Official leaderboard
69 Navid-AI Yehia-7B-preview 70.8% Mar 2025 Official leaderboard
70 Ocelotr Qwen-8B-SFT 70.7% Dec 2025 Official leaderboard
71 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 70.7% Jan 2025 Official leaderboard
72 Sakalti oxyge1-33B 70.3% Mar 2025 Official leaderboard
73 QCRI/Fanar-1-9B 70.2% Jun 2025 Official leaderboard
74 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 70.2% Jan 2025 Official leaderboard
75 TarjamaN Pronoia-14b-community 70% Feb 2025 Official leaderboard
76 ALLaM-AI/ALLaM-7B-Instruct-preview 69.5% Feb 2025 Official leaderboard
77 Phi-4 Mini 69.3% Feb 2025 Official leaderboard
78 Qwen3-4B 68.5% base model Apr 2025 Official leaderboard
79 inceptionai/jais-adapted-13b-chat 67.3% Jan 2025 Official leaderboard
80 Aya Expanse 8B 66.7% Oct 2024 Official leaderboard
81 inceptionai/jais-family-13b-chat 66.5% Jan 2025 Official leaderboard
82 Qwen1.5-32B 66.4% base model Feb 2024 Official leaderboard
83 Qwen1.5-14B 65.7% base model Feb 2024 Official leaderboard
84 Llama 3.3 70B Instruct 63.7% Dec 2024 Official leaderboard
85 huihui-ai Qwen2.5-7B-Instruct-abliterated 63.6% Jan 2025 Official leaderboard
86 inceptionai/jais-adapted-7b-chat 63.4% Jan 2025 Official leaderboard
87 Slim205 Barka-2b-it 61.8% Jun 2025 Official leaderboard
88 Qwen2.5-3B 61.1% Sep 2024 Official leaderboard
89 Gemma 3 12B 60.7% base model Mar 2025 Official leaderboard
90 Slim205 Barka-9b-it 60.7% Jun 2025 Official leaderboard
91 INSAIT-Institute BgGPT-7B-Instruct-v0.2 60.2% Jan 2025 Official leaderboard
92 silma-ai SILMA-Kashif-2B-Instruct-v1.0 59.7% Feb 2025 Official leaderboard
93 upstage SOLAR-10.7B-v1.0 59.4% Jan 2025 Official leaderboard
94 FreedomIntelligence AceGPT-13B-chat 59.2% Jan 2025 Official leaderboard
95 FreedomIntelligence AceGPT-v2-70B 58.7% Jan 2025 Official leaderboard
96 inceptionai/jais-family-30b-16k 57.6% Jan 2025 Official leaderboard
97 Qwen3-14B 57.1% base model Apr 2025 Official leaderboard
98 Qwen3-1.7B 56.8% base model Apr 2025 Official leaderboard
99 Hagrass LLama3-3.2-instruct-trained 56.8% Aug 2025 Official leaderboard
100 AIDC-AI Marco-LLM-AR 54.9% Feb 2025 Official leaderboard
101 asas-ai AceGPT-v2-32B 54.9% Jan 2025 Official leaderboard
102 FreedomIntelligence AceGPT-v2-32B 54.9% Jan 2025 Official leaderboard
103 inceptionai/jais-adapted-70b 54.8% Jan 2025 Official leaderboard
104 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 54.8% Jan 2025 Official leaderboard
105 cognitivecomputations Dolphin3.0-R1-Mistral-24B 54.7% Feb 2025 Official leaderboard
106 01-ZeroOne SUHAIL-14B-preview 54.5% Jul 2025 Official leaderboard
107 inceptionai/jais-family-30b-8k 54.3% Jan 2025 Official leaderboard
108 Gemma 3 4B 54.2% base model Mar 2025 Official leaderboard
109 FreedomIntelligence AceGPT-v1.5-13B-Chat 54% Jan 2025 Official leaderboard
110 AIDC-AI Marco-LLM-AR-V3 53.2% Mar 2025 Official leaderboard
111 Qwen2-1.5B-Instruct 53.1% Jun 2024 Official leaderboard
112 Qwen1.5-7B 52.3% base model Feb 2024 Official leaderboard
113 Sakalti Saka-1.5B 51.4% Feb 2025 Official leaderboard
114 01-ai/Yi-1.5-9B 51.2% base model Jan 2025 Official leaderboard
115 Qwen3-30B-A3B-Instruct Jul 2025 50.9% Jul 2025 Official leaderboard
116 FreedomIntelligence AceGPT-7B-chat 49.5% Jan 2025 Official leaderboard
117 01-ai/Yi-1.5-9B-32K 49% base model Jan 2025 Official leaderboard
118 yellowtown 7B-v0.2 48.5% Jan 2025 Official leaderboard
119 Qwen2.5-1.5B 48.4% Sep 2024 Official leaderboard
120 FreedomIntelligence AceGPT-13B 48.2% Jan 2025 Official leaderboard
121 speakleash Bielik-11B-v2 48% Jan 2025 Official leaderboard
122 Gemma 2 9B 47.9% Jun 2024 Official leaderboard
123 01-ai/Yi-1.5-34B-32K 47.7% base model Jan 2025 Official leaderboard
124 Qwen3-32B 46.8% Apr 2025 Official leaderboard
125 DeepSeek-R1-Distill-Qwen-32B 46.5% Jan 2025 Official leaderboard
126 Qwen3-0.6B 44.1% Apr 2025 Official leaderboard
127 tiiuae/Falcon3-10B-Base 44.1% base model Feb 2025 Official leaderboard
128 FreedomIntelligence AceGPT-v1.5-13B 44% Jan 2025 Official leaderboard
129 Llama 3.2 3B 42.9% Sep 2024 Official leaderboard
130 inceptionai/jais-family-13b 42.9% Jan 2025 Official leaderboard
131 Qwen/Qwen1.5-4B 42.4% base model Jan 2025 Official leaderboard
132 Phi-4 42.3% Dec 2024 Official leaderboard
133 inceptionai/jais-adapted-13b 40.6% Jan 2025 Official leaderboard
134 Qwen2-0.5B-Instruct 40.6% Jun 2024 Official leaderboard
135 Qwen2.5-0.5B 40.4% Sep 2024 Official leaderboard
136 tiiuae/Falcon3-7B-Base 37.9% base model Feb 2025 Official leaderboard
137 Naseej noon-7b 36.7% Mar 2025 Official leaderboard
138 qwen3-4b-instruct-2507 36.5% Aug 2025 Official leaderboard
139 FreedomIntelligence AceGPT-7B 36.4% Jan 2025 Official leaderboard
140 Gemma 2 2B IT 34.1% Jul 2024 Official leaderboard
141 silma-ai SILMA-9B-Instruct-v1.0 34% Jan 2025 Official leaderboard
142 Llama 3.2 1B 33.9% base model Sep 2024 Official leaderboard
143 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 33.5% Feb 2025 Official leaderboard
144 stabilityai ar-stablelm-2-base 33.1% Jun 2025 Official leaderboard
145 inceptionai/jais-adapted-7b 32.9% Jan 2025 Official leaderboard
146 TIGER-Lab Qwen2.5-Math-7B-CFT 32.8% Mar 2025 Official leaderboard
147 Qwen/Qwen1.5-1.8B 32.7% base model Jan 2025 Official leaderboard
148 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 32.7% Feb 2025 Official leaderboard
149 DeepSeek-R1-Distill-Qwen-14B 32.6% Jan 2025 Official leaderboard
150 riotu-lab ArabianGPT-01B 32.5% Oct 2025 Official leaderboard
151 QwQ-32B 31.3% Mar 2025 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen2.5-72B
base model
78.2%
Qwen2.5-32B
base model
78.1%
Gemma 3 27B
base model
77.9%
Qwen2.5-14B
base model
77.2%
Qwen3-8B
base model
74.8%
Gemma 2 27B
base model
72.2%
Qwen2.5-7B
base model
72.2%
Llama 3.1-8B
base model
70.9%
Qwen3-4B
base model
68.5%
Qwen2.5-3B
base model
61.1%
Qwen3-14B
base model
57.1%
Qwen2.5-1.5B
base model
48.4%
ModelEffortHarnessScoreCost per taskOutput tokens
Qwen2-72B base model 78.6%
Qwen2.5-72B 78.2%
Qwen2.5-32B 78.1%
Gemma 3 27B 77.9%
Aya Expanse 32B 77.6%
Qwen2.5-72B base model 77.2%
Qwen2.5-14B base model 77.2%
Qwen2.5-32B base model 77%
Gemma 3 27B base model 76.1%
Qwen3-8B base model 74.8%
Qwen2-7B-Instruct 73.2%
Qwen2.5-14B 72.3%
Gemma 2 27B 72.2%
Qwen2.5-7B base model 72.2%
Llama 3.1-8B 70.9%
Phi-4 Mini 69.3%
Qwen3-4B base model 68.5%
Aya Expanse 8B 66.7%
Qwen1.5-32B base model 66.4%
Qwen1.5-14B base model 65.7%
Qwen2.5-7B 65.6%
Llama 3.1-8B base model 64.3%
Llama 3.3 70B Instruct 63.7%
Qwen2.5-3B 61.1%
Gemma 3 12B base model 60.7%
Qwen3-14B base model 57.1%
Qwen3-1.7B base model 56.8%
Gemma 3 4B base model 54.2%
Qwen2-1.5B-Instruct 53.1%
Qwen1.5-7B base model 52.3%
Qwen3-30B-A3B-Instruct Jul 2025 50.9%
Qwen2.5-1.5B 48.4%
Gemma 2 9B 47.9%
Qwen2.5-3B base model 47.6%
Qwen3-32B 46.8%
DeepSeek-R1-Distill-Qwen-32B 46.5%
Qwen3-8B 44.6%
Qwen3-0.6B 44.1%
Llama 3.2 3B 42.9%
Phi-4 42.3%
Qwen2-0.5B-Instruct 40.6%
Qwen2.5-0.5B 40.4%
Qwen3-4B 38.7%
qwen3-4b-instruct-2507 36.5%
Gemma 2 2B IT 34.1%
Llama 3.2 1B base model 33.9%
DeepSeek-R1-Distill-Qwen-14B 32.6%
Qwen2.5-0.5B base model 32.4%
Qwen3-14B 32.1%
Gemma 2 27B base model 31.3%
QwQ-32B 31.3%
Qwen2.5-1.5B base model 30.9%

2 models have held number one since its release in Dec 2023

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen
Own leaderboard Not tested since Aug 2025
Qwen2-72B 78.6%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen2-72B Qwen1.5-32B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Qwen2-72B Qwen1.5-32B

Labs report it only in system cards

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
8 Nov 2025 Falcon-Arabic-7B technical report 72.4 ALGhafa, native Arabic subsets (Table 6) Capabilities

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%