Benchmark Almanac

HELM Arabic Enterprise

llm-x 82.6 tops its leaderboard Claude Opus 4.7 77.9 leads the models we track Not in model cards

Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute.

Release v1.0.0 came on 2026-05-26 and v1.1.0 on 2026-06-25 with 35 models. Finance questions are machine translated from English textbooks. A model judges the article and legal tasks.

Made by Stanford CRFM with Arabic.AI Released May 2026 Knowledge in many languages Last model added Apr 2026 Scores refreshed 2 Nov 2026
Level
Expert Specialists with years of training, often over hours of work
Scored
Graded by a model A language model scores each response, usually against a rubric.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
stanford-crfm/helm 2,961 stars Leaderboard crfm.stanford.edu
35
Entries on its own leaderboard, 23 models we track

llm-x tops the leaderboard with 82.6

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is Claude Opus 4.7, at 77.9.

The newest model with a score here came out in Apr 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

35 entries. Scroll sideways for the rest.

  1. 82.6 llm-x
  2. 82.5 llm-x-2026-06-07
  3. 77.9 Claude Opus 4.7
  4. 76.7 GPT-5.4
  5. 73.8 Gemma 4 31B IT
  6. 73.7 Claude Sonnet 4.6
  7. 73.1 GPT-5.1
  8. 72.8 DeepSeek-V4-Pro
  9. 72.4 llm-s
  10. 70.2 GPT-5.4 Mini
  11. 69.6 Qwen3-235B-A22B-Instruct
  12. 68.2 Gemini 2.5 Flash
  13. 66.0 Mistral Large 3
  14. 65.9 DeepSeek-V3.1
  15. 65.3 Qwen3.5 397B-A17B
  16. 64.8 Llama 4 Maverick
  17. 63.6 Claude Haiku 4.5
  18. 63.2 GPT-5.4 Nano
  19. 62.6 Gemini 2.5 Flash-Lite
  20. 62.4 Qwen3.5-9B
  21. 62.1 Llama 4 Scout
  22. 62.0 Mistral Large 2
  23. 61.5 Llama 3.3 70B Instruct
  24. 58.0 Cohere Command A
  25. 53.3 allam-7b-instruct-preview
  26. 51.6 jais-adapted-70b-chat
  27. 46.9 acegpt-v2-70b-chat
  28. 46.9 jais-adapted-13b-chat
  29. 45.5 acegpt-v2-32b-chat
  30. 45.3 jais-family-30b-16k-chat
  31. 42.3 jais-adapted-7b-chat
  32. 40.6 acegpt-v2-8b-chat
  33. 39.7 silma-9b-instruct-v1.0
  34. 30.4 Falcon3-7B-Instruct
  35. 28.6 Falcon3-10B-Instruct
  1. 1 llm-x 82.6
  2. 2 llm-x-2026-06-07 82.5
  3. 3 Claude Opus 4.7 77.9
  4. 4 GPT-5.4 76.7
  5. 5 Gemma 4 31B IT thinking disabled 73.8
  6. 6 Claude Sonnet 4.6 73.7
  7. 7 GPT-5.1 73.1
  8. 8 DeepSeek-V4-Pro thinking disabled 72.8
  9. 9 llm-s 72.4
  10. 10 GPT-5.4 Mini 70.2
  11. 11 Qwen3-235B-A22B-Instruct Jul 2025 69.6
  12. 12 Gemini 2.5 Flash Sep 2025, thinking disabled 68.2
  13. 13 Mistral Large 3 66.0
  14. 14 DeepSeek-V3.1 65.9
  15. 15 Qwen3.5 397B-A17B thinking disabled 65.3
  16. 16 Llama 4 Maverick 64.8
  17. 17 Claude Haiku 4.5 63.6
  18. 18 GPT-5.4 Nano 63.2
  19. 19 Gemini 2.5 Flash-Lite Sep 2025, thinking disabled 62.6
  20. 20 Qwen3.5-9B thinking disabled 62.4
  21. 21 Llama 4 Scout 62.1
  22. 22 Mistral Large 2 Nov 2024 62.0
  23. 23 Llama 3.3 70B Instruct 61.5
  24. 24 Cohere Command A 58.0
  25. 25 allam-7b-instruct-preview 53.3
  26. 26 jais-adapted-70b-chat 51.6
  27. 27 acegpt-v2-70b-chat 46.9
  28. 28 jais-adapted-13b-chat 46.9
  29. 29 acegpt-v2-32b-chat 45.5
  30. 30 jais-family-30b-16k-chat 45.3
  31. 31 jais-adapted-7b-chat 42.3
  32. 32 acegpt-v2-8b-chat 40.6
  33. 33 silma-9b-instruct-v1.0 39.7
  34. 34 Falcon3-7B-Instruct 30.4
  35. 35 Falcon3-10B-Instruct 28.6
#EntryScoreSettingDateSource
1 llm-x 82.6 Official leaderboard
2 llm-x-2026-06-07 82.5 Jun 2026 Official leaderboard
3 Claude Opus 4.7 77.9 Apr 2026 Official leaderboard
4 GPT-5.4 76.7 Mar 2026 Official leaderboard
5 Gemma 4 31B IT 73.8 thinking disabled Apr 2026 Official leaderboard
6 Claude Sonnet 4.6 73.7 Feb 2026 Official leaderboard
7 GPT-5.1 73.1 Nov 2025 Official leaderboard
8 DeepSeek-V4-Pro 72.8 thinking disabled Apr 2026 Official leaderboard
9 llm-s 72.4 Official leaderboard
10 GPT-5.4 Mini 70.2 Mar 2026 Official leaderboard
11 Qwen3-235B-A22B-Instruct Jul 2025 69.6 Jul 2025 Official leaderboard
12 Gemini 2.5 Flash Sep 2025 68.2 thinking disabled Sep 2025 Official leaderboard
13 Mistral Large 3 66.0 Dec 2025 Official leaderboard
14 DeepSeek-V3.1 65.9 Aug 2025 Official leaderboard
15 Qwen3.5 397B-A17B 65.3 thinking disabled Feb 2026 Official leaderboard
16 Llama 4 Maverick 64.8 Apr 2025 Official leaderboard
17 Claude Haiku 4.5 63.6 Oct 2025 Official leaderboard
18 GPT-5.4 Nano 63.2 Mar 2026 Official leaderboard
19 Gemini 2.5 Flash-Lite Sep 2025 62.6 thinking disabled Sep 2025 Official leaderboard
20 Qwen3.5-9B 62.4 thinking disabled Feb 2026 Official leaderboard
21 Llama 4 Scout 62.1 Apr 2025 Official leaderboard
22 Mistral Large 2 Nov 2024 62.0 Nov 2024 Official leaderboard
23 Llama 3.3 70B Instruct 61.5 Dec 2024 Official leaderboard
24 Cohere Command A 58.0 Mar 2025 Official leaderboard
25 allam-7b-instruct-preview 53.3 Official leaderboard
26 jais-adapted-70b-chat 51.6 Official leaderboard
27 acegpt-v2-70b-chat 46.9 Official leaderboard
28 jais-adapted-13b-chat 46.9 Official leaderboard
29 acegpt-v2-32b-chat 45.5 Official leaderboard
30 jais-family-30b-16k-chat 45.3 Official leaderboard
31 jais-adapted-7b-chat 42.3 Official leaderboard
32 acegpt-v2-8b-chat 40.6 Official leaderboard
33 silma-9b-instruct-v1.0 39.7 Official leaderboard
34 Falcon3-7B-Instruct 30.4 Dec 2024 Official leaderboard
35 Falcon3-10B-Instruct 28.6 Dec 2024 Official leaderboard

Number one over time

Lab claims count from the day the card came out, independent runs from the model’s release.

Anthropic
Own leaderboard Not tested since May 2026
Claude Opus 4.7 77.9

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
20.0 40.0 60.0 80.0 2025 2026 20.0 40.0 60.0 80.0 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%