Benchmark Almanac

INCLUDE

Claude Opus 5 89.8% leads since Jul 2026 Fading

Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy.

INCLUDE-base has 22,637 exam questions in 44 languages. A lite subset of 11,095 questions also exists, and the dataset cards show scores within about a point of the base set. Reports rarely say which set they used.

In the authors’ words

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

Abstract of INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (arXiv:2411.19799, CC0)
Made by EPFL and Cohere For AI Released Nov 2024 Knowledge in many languages All tasks public Last in a launch post Jun 2026 , dropped after it saturated Last model added Jul 2026
Level
University Undergraduates and entry-level professionals
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Multiple choice
Paper Dataset
25
Models scored in lab cards
13
Launch posts list it, from 3 labs
14
System cards report it, from 4 labs

Claude Opus 5 leads with 89.8% among models released by Jul 2026

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

The newest model with a score here came out in Jul 2026, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

25 models. Scroll sideways for the rest.

  1. 89.8 Claude Opus 5
  2. 87.6 Claude Opus 4.8
  3. 87.0 Claude Opus 4.7
  4. 86.5 Claude Sonnet 5
  5. 86.2 Qwen3.7-Max
  6. 85.6 Qwen3.5 397B-A17B
  7. 85.1 Qwen 3.6 Plus
  8. 83.0 Qwen3.7-Plus
  9. 82.8 Qwen3.5-122B-A10B
  10. 81.0 Qwen3-235B-A22B-Thinking
  11. 80.0 Qwen3-VL-235B-A22B
  12. 78.9 Qwen3-Next-80B-A3B
  13. 78.7 Qwen3-235B-A22B
  14. 74.3 Cohere Command A
  15. 73.7 Qwen3-32B
  16. 71.9 Qwen3-30B-A3B
  17. 71.7 Qwen3-14B
  18. 67.8 Qwen3-8B
  19. 67.3 Granite 4.1 30B
  20. 64.0 Granite 4.0 H Small
  21. 61.8 Qwen3-4B
  22. 58.2 Apertus 70B
  23. 57.2 Gemma 3n E4B
  24. 51.8 Qwen3-1.7B
  25. 42.1 granite-4.0-1b
  1. 1 Claude Opus 5 89.8%
  2. 2 Claude Opus 4.8 87.6%
  3. 3 Claude Opus 4.7 87%
  4. 4 Claude Sonnet 5 86.5%
  5. 5 Qwen3.7-Max 86.2%
  6. 6 Qwen3.5 397B-A17B 85.6%
  7. 7 Qwen 3.6 Plus 85.1%
  8. 8 Qwen3.7-Plus 83%
  9. 9 Qwen3.5-122B-A10B 82.8%
  10. 10 Qwen3-235B-A22B-Thinking Jul 2025 81%
  11. 11 Qwen3-VL-235B-A22B 80%
  12. 12 Qwen3-Next-80B-A3B 78.9%
  13. 13 Qwen3-235B-A22B 78.7%
  14. 14 Cohere Command A 74.3%
  15. 15 Qwen3-32B 73.7%
  16. 16 Qwen3-30B-A3B 71.9%
  17. 17 Qwen3-14B 71.7%
  18. 18 Qwen3-8B 67.8%
  19. 19 Granite 4.1 30B 67.3%
  20. 20 Granite 4.0 H Small 64%
  21. 21 Qwen3-4B 61.8%
  22. 22 Apertus 70B 58.2%
  23. 23 Gemma 3n E4B 57.2%
  24. 24 Qwen3-1.7B 51.8%
  25. 25 granite-4.0-1b 42.1%
#ModelScoreSettingDateSource
1 Claude Opus 5 89.8% Jul 2026 Model card
2 Claude Opus 4.8 87.6% May 2026 Model card
3 Claude Opus 4.7 87% Apr 2026 Model card
4 Claude Sonnet 5 86.5% Jun 2026 Model card
5 Qwen3.7-Max 86.2% May 2026 Model card
6 Qwen3.5 397B-A17B 85.6% Feb 2026 Model card
7 Qwen 3.6 Plus 85.1% Apr 2026 Model card
8 Qwen3.7-Plus 83% Jun 2026 Model card
9 Qwen3.5-122B-A10B 82.8% Feb 2026 Model card
10 Qwen3-235B-A22B-Thinking Jul 2025 81% Jul 2025 Model card
11 Qwen3-VL-235B-A22B 80% Sep 2025 Model card
12 Qwen3-Next-80B-A3B 78.9% Sep 2025 Model card
13 Qwen3-235B-A22B 78.7% Apr 2025 Model card
14 Cohere Command A 74.3% Apr 2025 Model card
15 Qwen3-32B 73.7% May 2025 Model card
16 Qwen3-30B-A3B 71.9% May 2025 Model card
17 Qwen3-14B 71.7% May 2025 Model card
18 Qwen3-8B 67.8% May 2025 Model card
19 Granite 4.1 30B 67.3% Apr 2026 Model card
20 Granite 4.0 H Small 64% Oct 2025 Model card
21 Qwen3-4B 61.8% May 2025 Model card
22 Apertus 70B 58.2% Sep 2025 Model card
23 Gemma 3n E4B 57.2% Jun 2025 Model card
24 Qwen3-1.7B 51.8% May 2025 Model card
25 granite-4.0-1b 42.1% Oct 2025 Model card

7 models have held number one since its release in Nov 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

Cohere Qwen Anthropic
Lab claims Passed 85% in Feb 2026, by Qwen3.5 397B-A17B
Claude Opus 5 89.8%

The best score passed 85% in Feb 2026

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card Best so far
0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 5 Claude Opus 4.7 Qwen3-235B-A22B-Thinking Cohere Command A 0% 20% 40% 60% 80% 100% 2025 2026 Near the ceiling (85%) Claude Opus 5 Qwen3-235B-A22B-Thinking

Labs headline it less often now, last in Jun 2026

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Qwen
Google DeepMind
IBM
DateLaunch postScoreAs the post writes it
1 Jun 2026 Qwen3.7-Plus 83 INCLUDE
20 May 2026 Qwen3.7-Max 86.2 INCLUDE
29 Apr 2026 Granite 4.1 30B 67.3 INCLUDE (5-shot, 14 languages)
2 Apr 2026 Qwen 3.6 Plus 85.1 INCLUDE
24 Feb 2026 Qwen3.5-122B-A10B 82.8 INCLUDE
16 Feb 2026 Qwen3.5 397B-A17B 85.6 INCLUDE
28 Oct 2025 granite-4.0-1b 42.1 INCLUDE (5-shot, 14 languages)
2 Oct 2025 Granite 4.0 H Small 64 INCLUDE (5-shot, 14 languages)
23 Sep 2025 Qwen3-VL-235B-A22B 80 INCLUDE
11 Sep 2025 Qwen3-Next-80B-A3B 78.9 INCLUDE
25 Jul 2025 Qwen3-235B-A22B-Thinking 81 INCLUDE
26 Jun 2025 Gemma 3n E4B 57.2 Include 0-shot (IT)
29 Apr 2025 Qwen3-235B-A22B 78.7 INCLUDE

14 system cards report it, from 4 labs

Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.

DateDocumentScoreAs the document writes itSection
24 Jul 2026 Claude Opus 5 system card 89.8 INCLUDE average accuracy across 44 languages, max effort, five trials, printed on chart (8.16.3, Figure 8.16.3.A) Capabilities
30 Jun 2026 Claude Sonnet 5 system card 86.5 INCLUDE average accuracy across 44 languages, max effort, five trials (8.13.3, Figure 8.13.3.A) Capabilities
28 May 2026 Claude Opus 4.8 system card 87.6 INCLUDE average accuracy across 44 languages, adaptive thinking (8.15.3, Figure 8.15.3.A) Capabilities
16 Apr 2026 Claude Opus 4.7 system card 87 INCLUDE average accuracy across 44 languages, adaptive thinking (8.12.3, Figure 8.12.3.A) Capabilities
26 Nov 2025 Qwen3-VL-235B-A22B technical report 80 INCLUDE (Table 6) Capabilities
17 Sep 2025 Apertus 70B technical report 58.2 INCLUDE, 44 languages (Table 20). INCLUDE V2 beta, 45 more languages, 41.6 Capabilities
14 May 2025 Qwen3-8B technical report 67.8 INCLUDE, thinking mode (Table 17). Non-thinking 62.5 (Table 18) Capabilities
14 May 2025 Qwen3-4B technical report 61.8 INCLUDE, thinking mode (Table 17). Non-thinking 53.8 (Table 18) Capabilities
14 May 2025 Qwen3-32B technical report 73.7 INCLUDE, thinking mode (Table 13). Non-thinking 70.9 (Table 14) Capabilities
14 May 2025 Qwen3-30B-A3B technical report 71.9 INCLUDE, thinking mode (Table 15). Non-thinking 67.8 (Table 16) Capabilities
14 May 2025 Qwen3-235B-A22B technical report 78.7 INCLUDE, thinking mode (Table 11). Non-thinking 75.6 (Table 12) Capabilities
14 May 2025 Qwen3-14B technical report 71.7 INCLUDE, thinking mode (Table 15). Non-thinking 67.8 (Table 16) Capabilities
14 May 2025 Qwen3-1.7B technical report 51.8 INCLUDE, thinking mode (Table 19). Non-thinking 42.6 (Table 20) Capabilities
1 Apr 2025 Cohere Command A technical report 74.3 INCLUDE-44 average over 20 languages (Table 25) Capabilities

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%