Benchmark Almanac

HELM Arabic

llm-x-2026-06-07 92.0 tops its leaderboard Claude Opus 4.7 84.1 leads the models we track Not in model cards

Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU.

Release v1.0.0 came on 2025-12-18 with 29 models. Release v2.2.0 on 2026-06-25 had 41 models. The task list matches OALL v2, but HELM prompts and scores differ. ALRAGE is model-judged.

Made by Stanford CRFM with Arabic.AI Released Dec 2025 Knowledge in many languages All tasks public Last model added Apr 2026 Scores refreshed 2 Nov 2026
Level
School Students, from grade school to high school
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
stanford-crfm/helm 2,961 stars Leaderboard crfm.stanford.edu
41
Entries on its own leaderboard, 28 models we track

llm-x-2026-06-07 tops the leaderboard with 92.0

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Claude Opus 4.7, at 84.1.

The newest model with a score here came out in Apr 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

41 entries. Scroll sideways for the rest.

  1. 92.0 llm-x-2026-06-07
  2. 88.3 incept-arabic-1
  3. 87.2 llm_x
  4. 84.1 Claude Opus 4.7
  5. 82.5 Gemini 2.5 Flash
  6. 82.5 GPT-5.4
  7. 82.3 Gemma 4 31B IT
  8. 81.7 GPT-5.1
  9. 81.5 Claude Sonnet 4.6
  10. 81.4 GPT-4.1
  11. 80.2 Qwen3.5 397B-A17B
  12. 80.2 llm_s
  13. 79.5 Qwen3-235B-A22B-Instruct
  14. 79.5 Gemini 2.5 Flash-Lite
  15. 79.1 Llama 4 Maverick
  16. 79.0 Mistral Large 3
  17. 79.0 GPT-5.4 Mini
  18. 78.8 Claude Haiku 4.5
  19. 78.4 Qwen3-Next-80B-A3B-Instruct
  20. 77.9 DeepSeek-V3.1
  21. 77.7 Cohere Command A
  22. 76.9 Qwen2.5-72B
  23. 76.8 GPT-4.1 mini
  24. 76.4 Llama 3.3 70B Instruct
  25. 75.6 Llama 4 Scout
  26. 74.7 acegpt-v2-70b-chat
  27. 73.1 Qwen3.5-9B
  28. 71.7 Mistral Large 2
  29. 71.6 acegpt-v2-32b-chat
  30. 70.8 allam-7b-instruct-preview
  31. 70.2 GPT-4.1 nano
  32. 68.2 GPT-5.4 Nano
  33. 67.4 jais-adapted-70b-chat
  34. 66.0 Qwen2.5-7B
  35. 64.5 jais-family-30b-16k-chat
  36. 64.3 silma-9b-instruct-v1.0
  37. 63.9 acegpt-v2-8b-chat
  38. 59.9 jais-adapted-13b-chat
  39. 53.1 jais-adapted-7b-chat
  40. 42.4 Falcon3-10B-Instruct
  41. 37.5 Falcon3-7B-Instruct
  1. 1 llm-x-2026-06-07 92.0
  2. 2 incept-arabic-1 88.3
  3. 3 llm_x 87.2
  4. 4 Claude Opus 4.7 84.1
  5. 5 Gemini 2.5 Flash Sep 2025, thinking disabled 82.5
  6. 6 GPT-5.4 82.5
  7. 7 Gemma 4 31B IT thinking disabled 82.3
  8. 8 GPT-5.1 81.7
  9. 9 Claude Sonnet 4.6 81.5
  10. 10 GPT-4.1 81.4
  11. 11 Qwen3.5 397B-A17B thinking disabled 80.2
  12. 12 llm_s 80.2
  13. 13 Qwen3-235B-A22B-Instruct Jul 2025 79.5
  14. 14 Gemini 2.5 Flash-Lite Sep 2025, thinking disabled 79.5
  15. 15 Llama 4 Maverick 79.1
  16. 16 Mistral Large 3 79.0
  17. 17 GPT-5.4 Mini 79.0
  18. 18 Claude Haiku 4.5 78.8
  19. 19 Qwen3-Next-80B-A3B-Instruct 78.4
  20. 20 DeepSeek-V3.1 77.9
  21. 21 Cohere Command A 77.7
  22. 22 Qwen2.5-72B 76.9
  23. 23 GPT-4.1 mini 76.8
  24. 24 Llama 3.3 70B Instruct 76.4
  25. 25 Llama 4 Scout 75.6
  26. 26 acegpt-v2-70b-chat 74.7
  27. 27 Qwen3.5-9B thinking disabled 73.1
  28. 28 Mistral Large 2 Nov 2024 71.7
  29. 29 acegpt-v2-32b-chat 71.6
  30. 30 allam-7b-instruct-preview 70.8
  31. 31 GPT-4.1 nano 70.2
  32. 32 GPT-5.4 Nano 68.2
  33. 33 jais-adapted-70b-chat 67.4
  34. 34 Qwen2.5-7B 66.0
  35. 35 jais-family-30b-16k-chat 64.5
  36. 36 silma-9b-instruct-v1.0 64.3
  37. 37 acegpt-v2-8b-chat 63.9
  38. 38 jais-adapted-13b-chat 59.9
  39. 39 jais-adapted-7b-chat 53.1
  40. 40 Falcon3-10B-Instruct 42.4
  41. 41 Falcon3-7B-Instruct 37.5
#EntryScoreSettingDateSource
1 llm-x-2026-06-07 92.0 Jun 2026 Official leaderboard
2 incept-arabic-1 88.3 Official leaderboard
3 llm_x 87.2 Official leaderboard
4 Claude Opus 4.7 84.1 Apr 2026 Official leaderboard
5 Gemini 2.5 Flash Sep 2025 82.5 thinking disabled Sep 2025 Official leaderboard
6 GPT-5.4 82.5 Mar 2026 Official leaderboard
7 Gemma 4 31B IT 82.3 thinking disabled Apr 2026 Official leaderboard
8 GPT-5.1 81.7 Nov 2025 Official leaderboard
9 Claude Sonnet 4.6 81.5 Feb 2026 Official leaderboard
10 GPT-4.1 81.4 Apr 2025 Official leaderboard
11 Qwen3.5 397B-A17B 80.2 thinking disabled Feb 2026 Official leaderboard
12 llm_s 80.2 Official leaderboard
13 Qwen3-235B-A22B-Instruct Jul 2025 79.5 Jul 2025 Official leaderboard
14 Gemini 2.5 Flash-Lite Sep 2025 79.5 thinking disabled Sep 2025 Official leaderboard
15 Llama 4 Maverick 79.1 Apr 2025 Official leaderboard
16 Mistral Large 3 79.0 Dec 2025 Official leaderboard
17 GPT-5.4 Mini 79.0 Mar 2026 Official leaderboard
18 Claude Haiku 4.5 78.8 Oct 2025 Official leaderboard
19 Qwen3-Next-80B-A3B-Instruct 78.4 Sep 2025 Official leaderboard
20 DeepSeek-V3.1 77.9 Aug 2025 Official leaderboard
21 Cohere Command A 77.7 Mar 2025 Official leaderboard
22 Qwen2.5-72B 76.9 Sep 2024 Official leaderboard
23 GPT-4.1 mini 76.8 Apr 2025 Official leaderboard
24 Llama 3.3 70B Instruct 76.4 Dec 2024 Official leaderboard
25 Llama 4 Scout 75.6 Apr 2025 Official leaderboard
26 acegpt-v2-70b-chat 74.7 Official leaderboard
27 Qwen3.5-9B 73.1 thinking disabled Feb 2026 Official leaderboard
28 Mistral Large 2 Nov 2024 71.7 Nov 2024 Official leaderboard
29 acegpt-v2-32b-chat 71.6 Official leaderboard
30 allam-7b-instruct-preview 70.8 Official leaderboard
31 GPT-4.1 nano 70.2 Apr 2025 Official leaderboard
32 GPT-5.4 Nano 68.2 Mar 2026 Official leaderboard
33 jais-adapted-70b-chat 67.4 Official leaderboard
34 Qwen2.5-7B 66.0 Sep 2024 Official leaderboard
35 jais-family-30b-16k-chat 64.5 Official leaderboard
36 silma-9b-instruct-v1.0 64.3 Official leaderboard
37 acegpt-v2-8b-chat 63.9 Official leaderboard
38 jais-adapted-13b-chat 59.9 Official leaderboard
39 jais-adapted-7b-chat 53.1 Official leaderboard
40 Falcon3-10B-Instruct 42.4 Dec 2024 Official leaderboard
41 Falcon3-7B-Instruct 37.5 Dec 2024 Official leaderboard

2 models have held number one since its release in Dec 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Google DeepMind Anthropic
Own leaderboard Not tested since Apr 2026
Claude Opus 4.7 84.1

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
40.0 60.0 80.0 2025 2026 40.0 60.0 80.0 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%