Benchmark Almanac

MadinahQA

Applied-Innovation-Center Karnak 88.2% tops its leaderboard Gemma 3 27B 69.4% leads the models we track Only in system cards

Models answer Arabic school exam questions on general Arabic language and grammar.

The dataset card says the questions are part of ArabicMMLU. OALL v2 and TII report it as a separate task.

Made by MBZUAI Released Sep 2024 Knowledge in many languages All tasks public Last in a system card Nov 2025 Last model added Nov 2025 Scores refreshed 2 Nov 2026
Level
School Students, from grade school to high school
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
Dataset Leaderboard huggingface.co
151
Entries on its own leaderboard, 39 models we track
1
System card reports it, from 1 lab

Applied-Innovation-Center Karnak tops the leaderboard with 88.2%

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Gemma 3 27B, at 69.4%.

The newest model with a score here came out in Aug 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

151 entries. Scroll sideways for the rest.

  1. 88.2 Applied-Innovation-Center Karnak
  2. 78.0 Applied-Innovation-Center AIC-1
  3. 77.0 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2
  4. 76.8 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1
  5. 75.9 TarjamaN Pronoia-14b-community
  6. 75.9 AbdulmalekDS qwen72b-ar-lora
  7. 75.7 Lina-Z qwen_arabic_ft
  8. 75.3 rombodawg Rombos-LLM-V2.5-Qwen-72b
  9. 75.2 Sakalti Ultiima-72B
  10. 74.9 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1
  11. 74.4 Lina-Z arabic_llm_test
  12. 74.3 freewheelin free-evo-qwen72b-v0.8-re
  13. 73.8 FreedomIntelligence AceGPT-v2-32B-Chat
  14. 73.6 maldv Qwentile2.5-32B-Instruct
  15. 73.3 QCRI/Fanar-1-9B-Instruct
  16. 71.6 Ocelotr Qwen3-8B-GAE
  17. 71.6 Sakalti SakaMoe-3x14B-Instruct
  18. 71.3 Cran-May tempmotacilla-cinerea-0308
  19. 71.2 hammh0a Hala-9B
  20. 70.0 Sakalti Saka-14B
  21. 69.4 Gemma 3 27B
  22. 68.3 FreedomIntelligence AceGPT-v2-70B
  23. 67.4 recoilme recoilme-gemma-2-9B-v0.4
  24. 66.3 inceptionai/jais-family-30b-16k-chat
  25. 65.6 FreedomIntelligence AceGPT-v2-70B-Chat
  26. 65.5 Slim205 Barka-9b-it
  27. 65.0 Qwen2.5-72B
  28. 64.2 Orion-zhen Qwen2.5-7B-Instruct-Uncensored
  29. 64.2 Mushari440 Qwen3-8B-SFT-V2
  30. 63.9 inceptionai/jais-family-30b-8k-chat
  31. 63.8 CohereForAI/c4ai-command-r7b-arabic-02-2025
  32. 63.5 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2
  33. 63.2 amine-khelif BARS
  34. 63.0 huihui-ai Qwen2.5-7B-Instruct-abliterated
  35. 62.7 Qwen2.5-7B
  36. 62.5 CohereForAI/aya-23-35B
  37. 62.3 Cran-May T.E-8.1
  38. 61.4 Qwen2-72B
  39. 60.6 MaziyarPanahi calme-2.1-qwen2.5-72b
  40. 60.1 QCRI/Fanar-1-9B
  41. 59.9 Qwen2.5-14B
  42. 59.8 tanliboy lambda-qwen2.5-32b-dpo-test
  43. 59.7 asas-ai AceGPT-v2-32B
  44. 59.7 FreedomIntelligence AceGPT-v2-32B
  45. 59.5 Qwen2-7B-Instruct
  46. 58.4 MaziyarPanahi calme-2.2-qwen2.5-72b
  47. 58.0 SeaLLMs SeaLLM-7B-v2.5
  48. 57.5 Mushari440 qwen3-8B-SFT
  49. 57.3 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2
  50. 57.2 Daemontatox Cogito-R1
  51. 57.1 Qwen2.5-32B
  52. 57.0 Ocelotr Qwen-8B-SFT
  53. 57.0 maldv Awqward2.5-32B-Instruct
  54. 56.9 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct
  55. 56.6 TIGER-Lab Qwen2.5-32B-Instruct-CFT
  56. 56.5 huihui-ai Qwen2.5-32B-Instruct-abliterated
  57. 55.9 FreedomIntelligence AceGPT-v2-8B-Chat
  58. 55.3 zetasepic Qwen2.5-32B-Instruct-abliterated-v2
  59. 55.0 JasperV13 Yehia-7B-Reasoning-preview
  60. 54.9 JasperV13 Yehia-7B-DPO-Reasoning-preview
  61. 54.4 Navid-AI Yehia-7B-preview
  62. 54.2 ALLaM-AI/ALLaM-7B-Instruct-preview
  63. 54.0 tanliboy lambda-qwen2.5-14b-dpo-test
  64. 54.0 Qwen3-4B
  65. 53.7 AXCXEPT EZO-Qwen2.5-32B-Instruct
  66. 53.6 inceptionai/jais-adapted-70b-chat
  67. 53.5 Aya Expanse 32B
  68. 53.3 Mushari440 qwen-3-8B-sft-grpo
  69. 53.1 Llama 3.3 70B Instruct
  70. 53.0 inceptionai/jais-family-13b-chat
  71. 53.0 Qwen1.5-32B
  72. 53.0 Gemma 3 12B
  73. 52.8 Qwen3-14B
  74. 52.5 silma-ai SILMA-9B-Instruct-v1.0
  75. 52.2 Qwen3-8B
  76. 51.4 arcee-ai Arcee-Spark
  77. 51.4 Qwen1.5-14B
  78. 51.4 recoilme recoilme-gemma-2-9B-v0.2
  79. 51.3 JasperV13 Yehia-7B-SFT-Reasoning-preview
  80. 50.5 SeaLLMs SeaLLMs-v3-7B-Chat
  81. 48.7 Aya Expanse 8B
  82. 47.1 Phi-4 Mini
  83. 47.0 Qwen2.5-3B
  84. 46.0 AIDC-AI Marco-LLM-AR-V2
  85. 46.0 AIDC-AI Marco-LLM-AR-V4
  86. 45.5 Qwen2-1.5B-Instruct
  87. 45.0 01-ZeroOne SUHAIL-14B-preview
  88. 44.2 inceptionai/jais-adapted-13b-chat
  89. 44.1 Gemma 3 4B
  90. 43.1 Llama 3.1-8B
  91. 43.1 Slim205 Barka-2b-it
  92. 40.2 Qwen1.5-7B
  93. 40.0 Qwen3-1.7B
  94. 40.0 AIDC-AI Marco-LLM-AR
  95. 39.3 Sakalti oxyge1-33B
  96. 39.2 Phi-4
  97. 39.0 Hagrass LLama3-3.2-instruct-trained
  98. 38.8 silma-ai SILMA-Kashif-2B-Instruct-v1.0
  99. 38.7 Llama 3.2 3B
  100. 38.3 INSAIT-Institute BgGPT-7B-Instruct-v0.2
  101. 38.3 FreedomIntelligence AceGPT-v1.5-13B-Chat
  102. 38.2 Qwen2.5-1.5B
  103. 38.1 FreedomIntelligence AceGPT-7B-chat
  104. 38.1 v000000 Qwen2.5-Lumen-14B
  105. 37.9 upstage SOLAR-10.7B-v1.0
  106. 37.9 01-ai/Yi-1.5-9B-32K
  107. 36.6 yellowtown 7B-v0.2
  108. 36.4 v000000 Qwen2.5-14B-Gutenberg-1e-Delta
  109. 36.1 rombodawg Rombos-LLM-V2.6-Qwen-14b
  110. 35.6 01-ai/Yi-1.5-9B
  111. 35.4 FreedomIntelligence AceGPT-13B
  112. 35.4 Qwen3-0.6B
  113. 35.3 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo
  114. 35.3 Naseej noon-7b
  115. 35.0 inceptionai/jais-adapted-70b
  116. 34.8 inceptionai/jais-adapted-7b-chat
  117. 33.7 FreedomIntelligence AceGPT-v1.5-13B
  118. 33.4 tiiuae/Falcon3-10B-Base
  119. 33.3 Gemma 2 27B
  120. 33.1 FreedomIntelligence AceGPT-13B-chat
  121. 33.0 Qwen3-32B
  122. 32.5 speakleash Bielik-11B-v2
  123. 31.8 Qwen/Qwen1.5-4B
  124. 31.8 AIDC-AI Marco-LLM-AR-V3
  125. 31.7 Gemma 2 9B
  126. 31.5 Sakalti Saka-1.5B
  127. 31.1 cognitivecomputations Dolphin3.0-R1-Mistral-24B
  128. 30.9 airev-ai emirati-14b-v2
  129. 30.1 riotu-lab ArabianGPT-01B
  130. 29.9 01-ai/Yi-1.5-34B-32K
  131. 29.8 inceptionai/jais-family-30b-8k
  132. 29.4 DeepSeek-R1-Distill-Qwen-32B
  133. 29.4 QwQ-32B
  134. 29.1 DeepSeek-R1-Distill-Qwen-14B
  135. 29.0 inceptionai/jais-adapted-13b
  136. 28.2 Qwen2-0.5B-Instruct
  137. 27.1 FreedomIntelligence AceGPT-7B
  138. 26.3 inceptionai/jais-family-30b-16k
  139. 26.0 Qwen/Qwen1.5-1.8B
  140. 26.0 TIGER-Lab Qwen2.5-Math-7B-CFT
  141. 25.7 Llama 3.2 1B
  142. 25.1 Qwen3-30B-A3B-Instruct
  143. 25.1 qwen3-4b-instruct-2507
  144. 24.9 tiiuae/Falcon3-7B-Base
  145. 24.8 inceptionai/jais-adapted-7b
  146. 24.8 Qwen2.5-0.5B
  147. 24.8 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1
  148. 24.1 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1
  149. 23.9 inceptionai/jais-family-13b
  150. 21.1 stabilityai ar-stablelm-2-base
  151. 20.1 Gemma 2 2B IT
  1. 1 Applied-Innovation-Center Karnak 88.2%
  2. 2 Applied-Innovation-Center AIC-1 78%
  3. 3 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 77%
  4. 4 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 76.8%
  5. 5 TarjamaN Pronoia-14b-community 75.9%
  6. 6 AbdulmalekDS qwen72b-ar-lora 75.9%
  7. 7 Lina-Z qwen_arabic_ft 75.7%
  8. 8 rombodawg Rombos-LLM-V2.5-Qwen-72b 75.3%
  9. 9 Sakalti Ultiima-72B 75.2%
  10. 10 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 74.9%
  11. 11 Lina-Z arabic_llm_test 74.4%
  12. 12 freewheelin free-evo-qwen72b-v0.8-re 74.3%
  13. 13 FreedomIntelligence AceGPT-v2-32B-Chat 73.8%
  14. 14 maldv Qwentile2.5-32B-Instruct 73.6%
  15. 15 QCRI/Fanar-1-9B-Instruct 73.3%
  16. 16 Ocelotr Qwen3-8B-GAE 71.6%
  17. 17 Sakalti SakaMoe-3x14B-Instruct 71.6%
  18. 18 Cran-May tempmotacilla-cinerea-0308 71.3%
  19. 19 hammh0a Hala-9B 71.2%
  20. 20 Sakalti Saka-14B 70%
  21. 21 Gemma 3 27B 69.4%
  22. 22 FreedomIntelligence AceGPT-v2-70B 68.3%
  23. 23 recoilme recoilme-gemma-2-9B-v0.4 67.4%
  24. 24 inceptionai/jais-family-30b-16k-chat 66.3%
  25. 25 FreedomIntelligence AceGPT-v2-70B-Chat 65.6%
  26. 26 Slim205 Barka-9b-it 65.5%
  27. 27 Qwen2.5-72B base model 65%
  28. 28 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 64.2%
  29. 29 Mushari440 Qwen3-8B-SFT-V2 64.2%
  30. 30 inceptionai/jais-family-30b-8k-chat 63.9%
  31. 31 CohereForAI/c4ai-command-r7b-arabic-02-2025 63.8%
  32. 32 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 63.5%
  33. 33 amine-khelif BARS 63.2%
  34. 34 huihui-ai Qwen2.5-7B-Instruct-abliterated 63%
  35. 35 Qwen2.5-7B 62.7%
  36. 36 CohereForAI/aya-23-35B 62.5%
  37. 37 Cran-May T.E-8.1 62.3%
  38. 38 Qwen2-72B base model 61.4%
  39. 39 MaziyarPanahi calme-2.1-qwen2.5-72b 60.6%
  40. 40 QCRI/Fanar-1-9B 60.1%
  41. 41 Qwen2.5-14B base model 59.9%
  42. 42 tanliboy lambda-qwen2.5-32b-dpo-test 59.8%
  43. 43 asas-ai AceGPT-v2-32B 59.7%
  44. 44 FreedomIntelligence AceGPT-v2-32B 59.7%
  45. 45 Qwen2-7B-Instruct 59.5%
  46. 46 MaziyarPanahi calme-2.2-qwen2.5-72b 58.4%
  47. 47 SeaLLMs SeaLLM-7B-v2.5 58%
  48. 48 Mushari440 qwen3-8B-SFT 57.5%
  49. 49 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 57.3%
  50. 50 Daemontatox Cogito-R1 57.2%
  51. 51 Qwen2.5-32B 57.1%
  52. 52 Ocelotr Qwen-8B-SFT 57%
  53. 53 maldv Awqward2.5-32B-Instruct 57%
  54. 54 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 56.9%
  55. 55 TIGER-Lab Qwen2.5-32B-Instruct-CFT 56.6%
  56. 56 huihui-ai Qwen2.5-32B-Instruct-abliterated 56.5%
  57. 57 FreedomIntelligence AceGPT-v2-8B-Chat 55.9%
  58. 58 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 55.3%
  59. 59 JasperV13 Yehia-7B-Reasoning-preview 55%
  60. 60 JasperV13 Yehia-7B-DPO-Reasoning-preview 54.9%
  61. 61 Navid-AI Yehia-7B-preview 54.4%
  62. 62 ALLaM-AI/ALLaM-7B-Instruct-preview 54.2%
  63. 63 tanliboy lambda-qwen2.5-14b-dpo-test 54%
  64. 64 Qwen3-4B base model 54%
  65. 65 AXCXEPT EZO-Qwen2.5-32B-Instruct 53.7%
  66. 66 inceptionai/jais-adapted-70b-chat 53.6%
  67. 67 Aya Expanse 32B 53.5%
  68. 68 Mushari440 qwen-3-8B-sft-grpo 53.3%
  69. 69 Llama 3.3 70B Instruct 53.1%
  70. 70 inceptionai/jais-family-13b-chat 53%
  71. 71 Qwen1.5-32B base model 53%
  72. 72 Gemma 3 12B base model 53%
  73. 73 Qwen3-14B base model 52.8%
  74. 74 silma-ai SILMA-9B-Instruct-v1.0 52.5%
  75. 75 Qwen3-8B base model 52.2%
  76. 76 arcee-ai Arcee-Spark 51.4%
  77. 77 Qwen1.5-14B base model 51.4%
  78. 78 recoilme recoilme-gemma-2-9B-v0.2 51.4%
  79. 79 JasperV13 Yehia-7B-SFT-Reasoning-preview 51.3%
  80. 80 SeaLLMs SeaLLMs-v3-7B-Chat 50.5%
  81. 81 Aya Expanse 8B 48.7%
  82. 82 Phi-4 Mini 47.1%
  83. 83 Qwen2.5-3B 47%
  84. 84 AIDC-AI Marco-LLM-AR-V2 46%
  85. 85 AIDC-AI Marco-LLM-AR-V4 46%
  86. 86 Qwen2-1.5B-Instruct 45.5%
  87. 87 01-ZeroOne SUHAIL-14B-preview 45%
  88. 88 inceptionai/jais-adapted-13b-chat 44.2%
  89. 89 Gemma 3 4B base model 44.1%
  90. 90 Llama 3.1-8B base model 43.1%
  91. 91 Slim205 Barka-2b-it 43.1%
  92. 92 Qwen1.5-7B base model 40.2%
  93. 93 Qwen3-1.7B base model 40%
  94. 94 AIDC-AI Marco-LLM-AR 40%
  95. 95 Sakalti oxyge1-33B 39.3%
  96. 96 Phi-4 39.2%
  97. 97 Hagrass LLama3-3.2-instruct-trained 39%
  98. 98 silma-ai SILMA-Kashif-2B-Instruct-v1.0 38.8%
  99. 99 Llama 3.2 3B 38.7%
  100. 100 INSAIT-Institute BgGPT-7B-Instruct-v0.2 38.3%
  101. 101 FreedomIntelligence AceGPT-v1.5-13B-Chat 38.3%
  102. 102 Qwen2.5-1.5B 38.2%
  103. 103 FreedomIntelligence AceGPT-7B-chat 38.1%
  104. 104 v000000 Qwen2.5-Lumen-14B 38.1%
  105. 105 upstage SOLAR-10.7B-v1.0 37.9%
  106. 106 01-ai/Yi-1.5-9B-32K base model 37.9%
  107. 107 yellowtown 7B-v0.2 36.6%
  108. 108 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 36.4%
  109. 109 rombodawg Rombos-LLM-V2.6-Qwen-14b 36.1%
  110. 110 01-ai/Yi-1.5-9B base model 35.6%
  111. 111 FreedomIntelligence AceGPT-13B 35.4%
  112. 112 Qwen3-0.6B 35.4%
  113. 113 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 35.3%
  114. 114 Naseej noon-7b 35.3%
  115. 115 inceptionai/jais-adapted-70b 35%
  116. 116 inceptionai/jais-adapted-7b-chat 34.8%
  117. 117 FreedomIntelligence AceGPT-v1.5-13B 33.7%
  118. 118 tiiuae/Falcon3-10B-Base base model 33.4%
  119. 119 Gemma 2 27B 33.3%
  120. 120 FreedomIntelligence AceGPT-13B-chat 33.1%
  121. 121 Qwen3-32B 33%
  122. 122 speakleash Bielik-11B-v2 32.5%
  123. 123 Qwen/Qwen1.5-4B base model 31.8%
  124. 124 AIDC-AI Marco-LLM-AR-V3 31.8%
  125. 125 Gemma 2 9B 31.7%
  126. 126 Sakalti Saka-1.5B 31.5%
  127. 127 cognitivecomputations Dolphin3.0-R1-Mistral-24B 31.1%
  128. 128 airev-ai emirati-14b-v2 30.9%
  129. 129 riotu-lab ArabianGPT-01B 30.1%
  130. 130 01-ai/Yi-1.5-34B-32K base model 29.9%
  131. 131 inceptionai/jais-family-30b-8k 29.8%
  132. 132 DeepSeek-R1-Distill-Qwen-32B 29.4%
  133. 133 QwQ-32B 29.4%
  134. 134 DeepSeek-R1-Distill-Qwen-14B 29.1%
  135. 135 inceptionai/jais-adapted-13b 29%
  136. 136 Qwen2-0.5B-Instruct 28.2%
  137. 137 FreedomIntelligence AceGPT-7B 27.1%
  138. 138 inceptionai/jais-family-30b-16k 26.3%
  139. 139 Qwen/Qwen1.5-1.8B base model 26%
  140. 140 TIGER-Lab Qwen2.5-Math-7B-CFT 26%
  141. 141 Llama 3.2 1B base model 25.7%
  142. 142 Qwen3-30B-A3B-Instruct Jul 2025 25.1%
  143. 143 qwen3-4b-instruct-2507 25.1%
  144. 144 tiiuae/Falcon3-7B-Base base model 24.9%
  145. 145 inceptionai/jais-adapted-7b 24.8%
  146. 146 Qwen2.5-0.5B 24.8%
  147. 147 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 24.8%
  148. 148 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 24.1%
  149. 149 inceptionai/jais-family-13b 23.9%
  150. 150 stabilityai ar-stablelm-2-base 21.1%
  151. 151 Gemma 2 2B IT 20.1%
#EntryScoreSettingDateSource
1 Applied-Innovation-Center Karnak 88.2% Feb 2026 Official leaderboard
2 Applied-Innovation-Center AIC-1 78% Sep 2025 Official leaderboard
3 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.2 77% Sep 2025 Official leaderboard
4 deep-analysis-research D2IL-Arabic-Qwen2.5-72B-Instruct-v0.1 76.8% Jul 2025 Official leaderboard
5 TarjamaN Pronoia-14b-community 75.9% Feb 2025 Official leaderboard
6 AbdulmalekDS qwen72b-ar-lora 75.9% Sep 2025 Official leaderboard
7 Lina-Z qwen_arabic_ft 75.7% Jan 2026 Official leaderboard
8 rombodawg Rombos-LLM-V2.5-Qwen-72b 75.3% Feb 2025 Official leaderboard
9 Sakalti Ultiima-72B 75.2% Feb 2025 Official leaderboard
10 tunny Arabic_Qwen2.5_72B_instruct_finetune_0.1 74.9% Dec 2025 Official leaderboard
11 Lina-Z arabic_llm_test 74.4% Nov 2025 Official leaderboard
12 freewheelin free-evo-qwen72b-v0.8-re 74.3% Jan 2025 Official leaderboard
13 FreedomIntelligence AceGPT-v2-32B-Chat 73.8% Jan 2025 Official leaderboard
14 maldv Qwentile2.5-32B-Instruct 73.6% Feb 2025 Official leaderboard
15 QCRI/Fanar-1-9B-Instruct 73.3% Jun 2025 Official leaderboard
16 Ocelotr Qwen3-8B-GAE 71.6% Jan 2026 Official leaderboard
17 Sakalti SakaMoe-3x14B-Instruct 71.6% Feb 2025 Official leaderboard
18 Cran-May tempmotacilla-cinerea-0308 71.3% Mar 2025 Official leaderboard
19 hammh0a Hala-9B 71.2% Feb 2026 Official leaderboard
20 Sakalti Saka-14B 70% Feb 2025 Official leaderboard
21 Gemma 3 27B 69.4% Mar 2025 Official leaderboard
22 FreedomIntelligence AceGPT-v2-70B 68.3% Jan 2025 Official leaderboard
23 recoilme recoilme-gemma-2-9B-v0.4 67.4% Jan 2025 Official leaderboard
24 inceptionai/jais-family-30b-16k-chat 66.3% Jan 2025 Official leaderboard
25 FreedomIntelligence AceGPT-v2-70B-Chat 65.6% Jan 2025 Official leaderboard
26 Slim205 Barka-9b-it 65.5% Jun 2025 Official leaderboard
27 Qwen2.5-72B 65% base model Sep 2024 Official leaderboard
28 Orion-zhen Qwen2.5-7B-Instruct-Uncensored 64.2% Jan 2025 Official leaderboard
29 Mushari440 Qwen3-8B-SFT-V2 64.2% Mar 2026 Official leaderboard
30 inceptionai/jais-family-30b-8k-chat 63.9% Jan 2025 Official leaderboard
31 CohereForAI/c4ai-command-r7b-arabic-02-2025 63.8% Feb 2025 Official leaderboard
32 huihui-ai Qwen2.5-7B-Instruct-abliterated-v2 63.5% Jan 2025 Official leaderboard
33 amine-khelif BARS 63.2% Sep 2025 Official leaderboard
34 huihui-ai Qwen2.5-7B-Instruct-abliterated 63% Jan 2025 Official leaderboard
35 Qwen2.5-7B 62.7% Sep 2024 Official leaderboard
36 CohereForAI/aya-23-35B 62.5% Jan 2025 Official leaderboard
37 Cran-May T.E-8.1 62.3% Jan 2025 Official leaderboard
38 Qwen2-72B 61.4% base model Jun 2024 Official leaderboard
39 MaziyarPanahi calme-2.1-qwen2.5-72b 60.6% Jan 2025 Official leaderboard
40 QCRI/Fanar-1-9B 60.1% Jun 2025 Official leaderboard
41 Qwen2.5-14B 59.9% base model Sep 2024 Official leaderboard
42 tanliboy lambda-qwen2.5-32b-dpo-test 59.8% Jan 2025 Official leaderboard
43 asas-ai AceGPT-v2-32B 59.7% Jan 2025 Official leaderboard
44 FreedomIntelligence AceGPT-v2-32B 59.7% Jan 2025 Official leaderboard
45 Qwen2-7B-Instruct 59.5% Jun 2024 Official leaderboard
46 MaziyarPanahi calme-2.2-qwen2.5-72b 58.4% Jan 2025 Official leaderboard
47 SeaLLMs SeaLLM-7B-v2.5 58% Jan 2025 Official leaderboard
48 Mushari440 qwen3-8B-SFT 57.5% Feb 2026 Official leaderboard
49 Isaak-Carter Josiefied-Qwen2.5-7B-Instruct-abliterated-v2 57.3% Jan 2025 Official leaderboard
50 Daemontatox Cogito-R1 57.2% Feb 2025 Official leaderboard
51 Qwen2.5-32B 57.1% Sep 2024 Official leaderboard
52 Ocelotr Qwen-8B-SFT 57% Dec 2025 Official leaderboard
53 maldv Awqward2.5-32B-Instruct 57% Feb 2025 Official leaderboard
54 VAGOsolutions SauerkrautLM-Nemo-12b-Instruct 56.9% Jan 2025 Official leaderboard
55 TIGER-Lab Qwen2.5-32B-Instruct-CFT 56.6% Mar 2025 Official leaderboard
56 huihui-ai Qwen2.5-32B-Instruct-abliterated 56.5% Jan 2025 Official leaderboard
57 FreedomIntelligence AceGPT-v2-8B-Chat 55.9% Jan 2025 Official leaderboard
58 zetasepic Qwen2.5-32B-Instruct-abliterated-v2 55.3% Mar 2025 Official leaderboard
59 JasperV13 Yehia-7B-Reasoning-preview 55% May 2025 Official leaderboard
60 JasperV13 Yehia-7B-DPO-Reasoning-preview 54.9% May 2025 Official leaderboard
61 Navid-AI Yehia-7B-preview 54.4% Mar 2025 Official leaderboard
62 ALLaM-AI/ALLaM-7B-Instruct-preview 54.2% Feb 2025 Official leaderboard
63 tanliboy lambda-qwen2.5-14b-dpo-test 54% Jan 2025 Official leaderboard
64 Qwen3-4B 54% base model Apr 2025 Official leaderboard
65 AXCXEPT EZO-Qwen2.5-32B-Instruct 53.7% Jan 2025 Official leaderboard
66 inceptionai/jais-adapted-70b-chat 53.6% Jan 2025 Official leaderboard
67 Aya Expanse 32B 53.5% Oct 2024 Official leaderboard
68 Mushari440 qwen-3-8B-sft-grpo 53.3% Feb 2026 Official leaderboard
69 Llama 3.3 70B Instruct 53.1% Dec 2024 Official leaderboard
70 inceptionai/jais-family-13b-chat 53% Jan 2025 Official leaderboard
71 Qwen1.5-32B 53% base model Feb 2024 Official leaderboard
72 Gemma 3 12B 53% base model Mar 2025 Official leaderboard
73 Qwen3-14B 52.8% base model Apr 2025 Official leaderboard
74 silma-ai SILMA-9B-Instruct-v1.0 52.5% Jan 2025 Official leaderboard
75 Qwen3-8B 52.2% base model Apr 2025 Official leaderboard
76 arcee-ai Arcee-Spark 51.4% Jan 2025 Official leaderboard
77 Qwen1.5-14B 51.4% base model Feb 2024 Official leaderboard
78 recoilme recoilme-gemma-2-9B-v0.2 51.4% Jan 2025 Official leaderboard
79 JasperV13 Yehia-7B-SFT-Reasoning-preview 51.3% May 2025 Official leaderboard
80 SeaLLMs SeaLLMs-v3-7B-Chat 50.5% Mar 2025 Official leaderboard
81 Aya Expanse 8B 48.7% Oct 2024 Official leaderboard
82 Phi-4 Mini 47.1% Feb 2025 Official leaderboard
83 Qwen2.5-3B 47% Sep 2024 Official leaderboard
84 AIDC-AI Marco-LLM-AR-V2 46% Mar 2025 Official leaderboard
85 AIDC-AI Marco-LLM-AR-V4 46% Mar 2025 Official leaderboard
86 Qwen2-1.5B-Instruct 45.5% Jun 2024 Official leaderboard
87 01-ZeroOne SUHAIL-14B-preview 45% Jul 2025 Official leaderboard
88 inceptionai/jais-adapted-13b-chat 44.2% Jan 2025 Official leaderboard
89 Gemma 3 4B 44.1% base model Mar 2025 Official leaderboard
90 Llama 3.1-8B 43.1% base model Jul 2024 Official leaderboard
91 Slim205 Barka-2b-it 43.1% Jun 2025 Official leaderboard
92 Qwen1.5-7B 40.2% base model Feb 2024 Official leaderboard
93 Qwen3-1.7B 40% base model Apr 2025 Official leaderboard
94 AIDC-AI Marco-LLM-AR 40% Feb 2025 Official leaderboard
95 Sakalti oxyge1-33B 39.3% Mar 2025 Official leaderboard
96 Phi-4 39.2% Dec 2024 Official leaderboard
97 Hagrass LLama3-3.2-instruct-trained 39% Aug 2025 Official leaderboard
98 silma-ai SILMA-Kashif-2B-Instruct-v1.0 38.8% Feb 2025 Official leaderboard
99 Llama 3.2 3B 38.7% Sep 2024 Official leaderboard
100 INSAIT-Institute BgGPT-7B-Instruct-v0.2 38.3% Jan 2025 Official leaderboard
101 FreedomIntelligence AceGPT-v1.5-13B-Chat 38.3% Jan 2025 Official leaderboard
102 Qwen2.5-1.5B 38.2% Sep 2024 Official leaderboard
103 FreedomIntelligence AceGPT-7B-chat 38.1% Jan 2025 Official leaderboard
104 v000000 Qwen2.5-Lumen-14B 38.1% Jan 2025 Official leaderboard
105 upstage SOLAR-10.7B-v1.0 37.9% Jan 2025 Official leaderboard
106 01-ai/Yi-1.5-9B-32K 37.9% base model Jan 2025 Official leaderboard
107 yellowtown 7B-v0.2 36.6% Jan 2025 Official leaderboard
108 v000000 Qwen2.5-14B-Gutenberg-1e-Delta 36.4% Jan 2025 Official leaderboard
109 rombodawg Rombos-LLM-V2.6-Qwen-14b 36.1% Jan 2025 Official leaderboard
110 01-ai/Yi-1.5-9B 35.6% base model Jan 2025 Official leaderboard
111 FreedomIntelligence AceGPT-13B 35.4% Jan 2025 Official leaderboard
112 Qwen3-0.6B 35.4% Apr 2025 Official leaderboard
113 Syed-Hasan-8503 Phi-3-mini-4K-instruct-cpo-simpo 35.3% Jan 2025 Official leaderboard
114 Naseej noon-7b 35.3% Mar 2025 Official leaderboard
115 inceptionai/jais-adapted-70b 35% Jan 2025 Official leaderboard
116 inceptionai/jais-adapted-7b-chat 34.8% Jan 2025 Official leaderboard
117 FreedomIntelligence AceGPT-v1.5-13B 33.7% Jan 2025 Official leaderboard
118 tiiuae/Falcon3-10B-Base 33.4% base model Feb 2025 Official leaderboard
119 Gemma 2 27B 33.3% Jun 2024 Official leaderboard
120 FreedomIntelligence AceGPT-13B-chat 33.1% Jan 2025 Official leaderboard
121 Qwen3-32B 33% Apr 2025 Official leaderboard
122 speakleash Bielik-11B-v2 32.5% Jan 2025 Official leaderboard
123 Qwen/Qwen1.5-4B 31.8% base model Jan 2025 Official leaderboard
124 AIDC-AI Marco-LLM-AR-V3 31.8% Mar 2025 Official leaderboard
125 Gemma 2 9B 31.7% Jun 2024 Official leaderboard
126 Sakalti Saka-1.5B 31.5% Feb 2025 Official leaderboard
127 cognitivecomputations Dolphin3.0-R1-Mistral-24B 31.1% Feb 2025 Official leaderboard
128 airev-ai emirati-14b-v2 30.9% Jan 2025 Official leaderboard
129 riotu-lab ArabianGPT-01B 30.1% Oct 2025 Official leaderboard
130 01-ai/Yi-1.5-34B-32K 29.9% base model Jan 2025 Official leaderboard
131 inceptionai/jais-family-30b-8k 29.8% Jan 2025 Official leaderboard
132 DeepSeek-R1-Distill-Qwen-32B 29.4% Jan 2025 Official leaderboard
133 QwQ-32B 29.4% Mar 2025 Official leaderboard
134 DeepSeek-R1-Distill-Qwen-14B 29.1% Jan 2025 Official leaderboard
135 inceptionai/jais-adapted-13b 29% Jan 2025 Official leaderboard
136 Qwen2-0.5B-Instruct 28.2% Jun 2024 Official leaderboard
137 FreedomIntelligence AceGPT-7B 27.1% Jan 2025 Official leaderboard
138 inceptionai/jais-family-30b-16k 26.3% Jan 2025 Official leaderboard
139 Qwen/Qwen1.5-1.8B 26% base model Jan 2025 Official leaderboard
140 TIGER-Lab Qwen2.5-Math-7B-CFT 26% Mar 2025 Official leaderboard
141 Llama 3.2 1B 25.7% base model Sep 2024 Official leaderboard
142 Qwen3-30B-A3B-Instruct Jul 2025 25.1% Jul 2025 Official leaderboard
143 qwen3-4b-instruct-2507 25.1% Aug 2025 Official leaderboard
144 tiiuae/Falcon3-7B-Base 24.9% base model Feb 2025 Official leaderboard
145 inceptionai/jais-adapted-7b 24.8% Jan 2025 Official leaderboard
146 Qwen2.5-0.5B 24.8% Sep 2024 Official leaderboard
147 mobiuslabsgmbh DeepSeek-R1-ReDistill-Llama3-8B-v1.1 24.8% Feb 2025 Official leaderboard
148 mobiuslabsgmbh DeepSeek-R1-ReDistill-Qwen-7B-v1.1 24.1% Feb 2025 Official leaderboard
149 inceptionai/jais-family-13b 23.9% Jan 2025 Official leaderboard
150 stabilityai ar-stablelm-2-base 21.1% Jun 2025 Official leaderboard
151 Gemma 2 2B IT 20.1% Jul 2024 Official leaderboard

Labs report scores for 1 model in their cards. We plot every source in the charts below.

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Gemma 3 27B
base model
69.4%
Qwen2.5-72B
base model
65%
Qwen2.5-7B
base model
62.7%
Qwen2.5-14B
base model
59.9%
Qwen2.5-32B
base model
57.1%
Qwen3-4B
base model
54%
Qwen3-14B
base model
52.8%
Qwen3-8B
base model
52.2%
Qwen2.5-3B
base model
47%
Llama 3.1-8B
base model
43.1%
Qwen2.5-1.5B
base model
38.2%
Gemma 2 27B
base model
33.3%
ModelEffortHarnessScoreCost per taskOutput tokens
Gemma 3 27B 69.4%
Qwen2.5-72B base model 65%
Qwen2.5-7B 62.7%
Qwen2-72B base model 61.4%
Qwen2.5-14B base model 59.9%
Gemma 3 27B base model 59.5%
Qwen2-7B-Instruct 59.5%
Qwen2.5-72B 58.1%
Qwen2.5-32B 57.1%
Qwen2.5-32B base model 55.8%
Qwen3-4B base model 54%
Aya Expanse 32B 53.5%
Llama 3.3 70B Instruct 53.1%
Qwen1.5-32B base model 53%
Gemma 3 12B base model 53%
Qwen3-14B base model 52.8%
Qwen3-8B base model 52.2%
Qwen1.5-14B base model 51.4%
Qwen2.5-7B base model 51.1%
Aya Expanse 8B 48.7%
Phi-4 Mini 47.1%
Qwen2.5-3B 47%
Qwen2-1.5B-Instruct 45.5%
Gemma 3 4B base model 44.1%
Llama 3.1-8B base model 43.1%
Qwen1.5-7B base model 40.2%
Qwen3-1.7B base model 40%
Llama 3.1-8B 39.7%
Phi-4 39.2%
Llama 3.2 3B 38.7%
Qwen2.5-1.5B 38.2%
Qwen2.5-14B 35.6%
Qwen3-0.6B 35.4%
Qwen3-14B 35.1%
Gemma 2 27B 33.3%
Qwen3-32B 33%
Qwen2.5-3B base model 32.1%
Gemma 2 9B 31.7%
Gemma 2 27B base model 29.4%
DeepSeek-R1-Distill-Qwen-32B 29.4%
QwQ-32B 29.4%
DeepSeek-R1-Distill-Qwen-14B 29.1%
Qwen2-0.5B-Instruct 28.2%
Qwen3-4B 27.2%
Llama 3.2 1B base model 25.7%
Qwen3-8B 25.3%
Qwen3-30B-A3B-Instruct Jul 2025 25.1%
qwen3-4b-instruct-2507 25.1%
Qwen2.5-0.5B 24.8%
Qwen2.5-1.5B base model 23.1%
Gemma 2 2B IT 20.1%
Qwen2.5-0.5B base model 19.9%

3 models have held number one since its release in Sep 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Qwen Google DeepMind
Own leaderboard Not tested since Aug 2025
Gemma 3 27B 69.4%

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Falcon-Arabic-7B Gemma 3 27B Qwen2.5-72B Qwen1.5-32B 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) Falcon-Arabic-7B Qwen2.5-72B

Labs report it only in system cards

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark. Some system cards do, below.

One system card reports it

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
8 Nov 2025 Falcon-Arabic-7B technical report 73.6 MadinahQA (Table 6) Capabilities

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%