Benchmark Almanac

AraGen

o1 84.3% leads since December 2024
Not in model cards

Models answer Arabic questions in four tasks: question answering, reasoning, orthographic and grammatical analysis, and safety. An LLM judge scores each answer on the 3C3H measure: correctness, completeness, conciseness, helpfulness, honesty and harmlessness. Only correct answers get credit on the other dimensions.

The test set changes in three-month blind cycles: each round stays private while models are scored, then becomes public. The first round (AraGen-12-24) had 279 questions and the 03-25 round had 340. The v3 results file listed 69 models at its 2026-04 sync. Inception runs the same method for Hindi (HindiGen) in the same space.

Released
December 2024
Kind
Chat and instructions in many languages
Level
Everyday
Scored
Graded by a model
Answers
Open-ended
Tasks
Public, plus a private test set
Strongest model tested
Gemini 3 Pro #55 on the capability index
Last model added
April 2026
Scores refreshed
November 12th, 2026
68
Entries on its own leaderboard, 42 models we track

o1 leads with 84.3% among models released by April 2026

Best score per entry from the benchmark's own leaderboard, read on November 12th, 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in April 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

68 entries. Scroll sideways for the rest.

  1. 84.3 o1
  2. 84.3 GPT-5
  3. 82.2 o3
  4. 81.0 Claude Opus 4
  5. 80.3 Claude Opus 4.5
  6. 78.2 Claude Sonnet 4.5
  7. 78.2 Claude 3.7 Sonnet
  8. 75.6 Claude Sonnet 4
  9. 74.5 GPT-4.1
  10. 72.7 Gemma 4 31B IT
  11. 70.6 o4-mini
  12. 70.5 o1-mini
  13. 68.4 Gemini 2.5 Pro
  14. 67.3 gemma-4-26B-A4B-it
  15. 65.8 Claude Haiku 4.5
  16. 64.2 Gemini 3 Pro
  17. 59.8 o3-mini
  18. 52.4 Jais-2-70B-Chat
  19. 52.1 Llama 3.3 70B Instruct
  20. 52.0 Mistral Large 2
  21. 50.8 calme-2.1-qwen2.5-72b
  22. 50.7 GPT-4o mini
  23. 50.5 Llama 3.2 90B
  24. 50.3 calme-2.2-qwen2.5-72b
  25. 50.0 Llama 3.1-70B
  26. 48.9 Qwen2.5-72B
  27. 46.0 GLM-5
  28. 45.7 Falcon-H1-34B
  29. 45.3 Yehia-7B-Reasoning-preview
  30. 43.2 gpt-oss-120b
  31. 39.8 Mistral Small 3
  32. 36.9 Qwen2.5-14B-Instruct
  33. 35.2 Qwen3-32B
  34. 35.0 AceGPT-v2-8B-Chat
  35. 34.0 Jais-2-8B-Chat
  36. 33.9 Mistral Small 4
  37. 32.3 jais-family-30b-16k-chat
  38. 32.3 Magistral Small 1.2
  39. 30.8 jais-family-30b-8k-chat
  40. 30.6 gpt-oss-20b
  41. 30.0 Phi-4
  42. 29.7 Qwen3-14B
  43. 27.9 c4ai-command-r7b-arabic-02-2025
  44. 26.6 Qwen3-8B
  45. 26.3 Qwen2.5-7B-Instruct
  46. 26.1 Llama 3-70B
  47. 24.9 Llama 3.1-8B
  48. 24.4 Llama 3.2 11B
  49. 24.4 Falcon-H1-7B-Instruct
  50. 24.1 jais-family-13b-chat
  51. 21.3 jais-family-6p7b-chat
  52. 17.6 Ministral 8B
  53. 16.2 SmolLM3-3B
  54. 16.2 Fanar-1-9B-Instruct
  55. 15.9 AceGPT-13B-chat
  56. 15.1 Falcon-H1-3B-Instruct
  57. 15.0 Llama 3.2 3B
  58. 13.2 Qwen2.5-1.5B-Instruct
  59. 12.4 AceGPT-7B-chat
  60. 10.5 Llama 3.2 1B
  61. 9.21 Llama 3-8B
  62. 6.70 SILMA-9B-Instruct-v1.0
  63. 5.21 Qwen2.5-0.5B-Instruct
  64. 5.08 Mistral-7B-Instruct-v0.3
  65. 4.58 Falcon-H1-1.5B-Instruct
  66. 1.57 Qwen3-0.6B
  67. 0.48 Mistral-7B-Instruct-v0.2
  68. 0.0 Qwen3.5-397B-A17B-FP8
  1. 1 o1 UNK 84.3%
  2. 2 GPT-5 UNK 84.3%
  3. 3 o3 UNK 82.2%
  4. 4 Claude Opus 4 UNK 81%
  5. 5 Claude Opus 4.5 UNK 80.3%
  6. 6 Claude Sonnet 4.5 UNK 78.2%
  7. 7 Claude 3.7 Sonnet UNK 78.2%
  8. 8 Claude Sonnet 4 UNK 75.6%
  9. 9 GPT-4.1 UNK 74.5%
  10. 10 Gemma 4 31B IT UNK 72.7%
  11. 11 o4-mini UNK 70.6%
  12. 12 o1-mini UNK 70.5%
  13. 13 Gemini 2.5 Pro May 2025, UNK 68.4%
  14. 14 gemma-4-26B-A4B-it UNK 67.3%
  15. 15 Claude Haiku 4.5 UNK 65.8%
  16. 16 Gemini 3 Pro UNK 64.2%
  17. 17 o3-mini UNK 59.8%
  18. 18 Jais-2-70B-Chat bfloat16 52.4%
  19. 19 Llama 3.3 70B Instruct bfloat16 52.1%
  20. 20 Mistral Large 2 Nov 2024, bfloat16 52%
  21. 21 calme-2.1-qwen2.5-72b bfloat16 50.8%
  22. 22 GPT-4o mini UNK 50.7%
  23. 23 Llama 3.2 90B bfloat16 50.5%
  24. 24 calme-2.2-qwen2.5-72b bfloat16 50.3%
  25. 25 Llama 3.1-70B bfloat16 50%
  26. 26 Qwen2.5-72B bfloat16 48.9%
  27. 27 GLM-5 UNK 46%
  28. 28 Falcon-H1-34B bfloat16 45.7%
  29. 29 Yehia-7B-Reasoning-preview float16 45.3%
  30. 30 gpt-oss-120b bfloat16 43.2%
  31. 31 Mistral Small 3 bfloat16 39.8%
  32. 32 Qwen2.5-14B-Instruct bfloat16 36.9%
  33. 33 Qwen3-32B bfloat16 35.2%
  34. 34 AceGPT-v2-8B-Chat float16 35%
  35. 35 Jais-2-8B-Chat bfloat16 34%
  36. 36 Mistral Small 4 UNK 33.9%
  37. 37 jais-family-30b-16k-chat float32 32.3%
  38. 38 Magistral Small 1.2 UNK 32.3%
  39. 39 jais-family-30b-8k-chat float32 30.8%
  40. 40 gpt-oss-20b bfloat16 30.6%
  41. 41 Phi-4 bfloat16 30%
  42. 42 Qwen3-14B bfloat16 29.7%
  43. 43 c4ai-command-r7b-arabic-02-2025 bfloat16 27.9%
  44. 44 Qwen3-8B bfloat16 26.6%
  45. 45 Qwen2.5-7B-Instruct bfloat16 26.3%
  46. 46 Llama 3-70B bfloat16 26.1%
  47. 47 Llama 3.1-8B bfloat16 24.9%
  48. 48 Llama 3.2 11B bfloat16 24.4%
  49. 49 Falcon-H1-7B-Instruct bfloat16 24.4%
  50. 50 jais-family-13b-chat float32 24.1%
  51. 51 jais-family-6p7b-chat float32 21.3%
  52. 52 Ministral 8B bfloat16 17.6%
  53. 53 SmolLM3-3B bfloat16 16.2%
  54. 54 Fanar-1-9B-Instruct bfloat16 16.2%
  55. 55 AceGPT-13B-chat float16 15.9%
  56. 56 Falcon-H1-3B-Instruct bfloat16 15.1%
  57. 57 Llama 3.2 3B bfloat16 15%
  58. 58 Qwen2.5-1.5B-Instruct bfloat16 13.2%
  59. 59 AceGPT-7B-chat float16 12.4%
  60. 60 Llama 3.2 1B bfloat16 10.5%
  61. 61 Llama 3-8B bfloat16 9.21%
  62. 62 SILMA-9B-Instruct-v1.0 bfloat16 6.70%
  63. 63 Qwen2.5-0.5B-Instruct bfloat16 5.21%
  64. 64 Mistral-7B-Instruct-v0.3 bfloat16 5.08%
  65. 65 Falcon-H1-1.5B-Instruct bfloat16 4.58%
  66. 66 Qwen3-0.6B bfloat16 1.57%
  67. 67 Mistral-7B-Instruct-v0.2 bfloat16 0.48%
  68. 68 Qwen3.5-397B-A17B-FP8 auto 0%

Source: Official AraGen leaderboard

#EntryScoreSettingDateSource
1 o1 84.3% UNK Dec 2024 Official leaderboard
2 GPT-5 84.3% UNK Aug 2025 Official leaderboard
3 o3 82.2% UNK Apr 2025 Official leaderboard
4 Claude Opus 4 81% UNK May 2025 Official leaderboard
5 Claude Opus 4.5 80.3% UNK Nov 2025 Official leaderboard
6 Claude Sonnet 4.5 78.2% UNK Sep 2025 Official leaderboard
7 Claude 3.7 Sonnet 78.2% UNK Feb 2025 Official leaderboard
8 Claude Sonnet 4 75.6% UNK May 2025 Official leaderboard
9 GPT-4.1 74.5% UNK Apr 2025 Official leaderboard
10 Gemma 4 31B IT 72.7% UNK Apr 2026 Official leaderboard
11 o4-mini 70.6% UNK Apr 2025 Official leaderboard
12 o1-mini 70.5% UNK Sep 2024 Official leaderboard
13 Gemini 2.5 Pro May 2025 68.4% UNK May 2025 Official leaderboard
14 gemma-4-26B-A4B-it 67.3% UNK Official leaderboard
15 Claude Haiku 4.5 65.8% UNK Oct 2025 Official leaderboard
16 Gemini 3 Pro 64.2% UNK Nov 2025 Official leaderboard
17 o3-mini 59.8% UNK Jan 2025 Official leaderboard
18 Jais-2-70B-Chat 52.4% bfloat16 Official leaderboard
19 Llama 3.3 70B Instruct 52.1% bfloat16 Dec 2024 Official leaderboard
20 Mistral Large 2 Nov 2024 52% bfloat16 Nov 2024 Official leaderboard
21 calme-2.1-qwen2.5-72b 50.8% bfloat16 Official leaderboard
22 GPT-4o mini 50.7% UNK Jul 2024 Official leaderboard
23 Llama 3.2 90B 50.5% bfloat16 Sep 2024 Official leaderboard
24 calme-2.2-qwen2.5-72b 50.3% bfloat16 Official leaderboard
25 Llama 3.1-70B 50% bfloat16 Jul 2024 Official leaderboard
26 Qwen2.5-72B 48.9% bfloat16 Sep 2024 Official leaderboard
27 GLM-5 46% UNK Feb 2026 Official leaderboard
28 Falcon-H1-34B 45.7% bfloat16 May 2025 Official leaderboard
29 Yehia-7B-Reasoning-preview 45.3% float16 Official leaderboard
30 gpt-oss-120b 43.2% bfloat16 Aug 2025 Official leaderboard
31 Mistral Small 3 39.8% bfloat16 Jan 2025 Official leaderboard
32 Qwen2.5-14B-Instruct 36.9% bfloat16 Official leaderboard
33 Qwen3-32B 35.2% bfloat16 Apr 2025 Official leaderboard
34 AceGPT-v2-8B-Chat 35% float16 Official leaderboard
35 Jais-2-8B-Chat 34% bfloat16 Official leaderboard
36 Mistral Small 4 33.9% UNK Mar 2026 Official leaderboard
37 jais-family-30b-16k-chat 32.3% float32 Official leaderboard
38 Magistral Small 1.2 32.3% UNK Sep 2025 Official leaderboard
39 jais-family-30b-8k-chat 30.8% float32 Official leaderboard
40 gpt-oss-20b 30.6% bfloat16 Aug 2025 Official leaderboard
41 Phi-4 30% bfloat16 Dec 2024 Official leaderboard
42 Qwen3-14B 29.7% bfloat16 Apr 2025 Official leaderboard
43 c4ai-command-r7b-arabic-02-2025 27.9% bfloat16 Official leaderboard
44 Qwen3-8B 26.6% bfloat16 Apr 2025 Official leaderboard
45 Qwen2.5-7B-Instruct 26.3% bfloat16 Official leaderboard
46 Llama 3-70B 26.1% bfloat16 Apr 2024 Official leaderboard
47 Llama 3.1-8B 24.9% bfloat16 Jul 2024 Official leaderboard
48 Llama 3.2 11B 24.4% bfloat16 Sep 2024 Official leaderboard
49 Falcon-H1-7B-Instruct 24.4% bfloat16 Official leaderboard
50 jais-family-13b-chat 24.1% float32 Official leaderboard
51 jais-family-6p7b-chat 21.3% float32 Official leaderboard
52 Ministral 8B 17.6% bfloat16 Oct 2024 Official leaderboard
53 SmolLM3-3B 16.2% bfloat16 Official leaderboard
54 Fanar-1-9B-Instruct 16.2% bfloat16 Official leaderboard
55 AceGPT-13B-chat 15.9% float16 Official leaderboard
56 Falcon-H1-3B-Instruct 15.1% bfloat16 Official leaderboard
57 Llama 3.2 3B 15% bfloat16 Sep 2024 Official leaderboard
58 Qwen2.5-1.5B-Instruct 13.2% bfloat16 Official leaderboard
59 AceGPT-7B-chat 12.4% float16 Official leaderboard
60 Llama 3.2 1B 10.5% bfloat16 Sep 2024 Official leaderboard
61 Llama 3-8B 9.21% bfloat16 Apr 2024 Official leaderboard
62 SILMA-9B-Instruct-v1.0 6.70% bfloat16 Official leaderboard
63 Qwen2.5-0.5B-Instruct 5.21% bfloat16 Official leaderboard
64 Mistral-7B-Instruct-v0.3 5.08% bfloat16 Official leaderboard
65 Falcon-H1-1.5B-Instruct 4.58% bfloat16 Official leaderboard
66 Qwen3-0.6B 1.57% bfloat16 Apr 2025 Official leaderboard
67 Mistral-7B-Instruct-v0.2 0.48% bfloat16 Dec 2023 Official leaderboard
68 Qwen3.5-397B-A17B-FP8 0% auto Official leaderboard

2 models have held number one on the official leaderboard since its release in December 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI
Own leaderboard Not tested since Apr 2026
o1 84.3%

Source: Official AraGen leaderboard

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard Best so far
0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) o1 o1-mini GPT-4o mini Llama 3-70B Mistral-7B-Instruct-v0.2 0% 20% 40% 60% 80% 100% 2024 2025 2026 Near the ceiling (85%) o1 o1-mini

Source: Official AraGen leaderboard

No lab model card reports it yet

We read the headline results of 522 launch posts from 52 labs, and none of them list this benchmark.

More in Multilingual

All 120 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Nemotron 3 Ultra 550B-A55B Base 90.1%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%