Benchmark Almanac

SEA-HELM

Gemini 3.1 Pro 77.4 leads since Feb 2026 Not in model cards

Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety.

HELM releases v1.0.0 (2025-03-17) to v1.2.0 (2025-04-18). AI Singapore also keeps its own SEA-HELM board.

In the authors’ words

With the rapid emergence of novel capabilities in Large Language Models (LLMs), the need for rigorous multilingual and multicultural benchmarks that are integrated has become more pronounced. Though existing LLM benchmarks are capable of evaluating specific capabilities of LLMs in English as well as in various mid- to low-resource languages, including those in the Southeast Asian (SEA) region, a comprehensive and culturally representative evaluation suite for the SEA languages has not been developed thus far. Here, we present SEA-HELM, a holistic linguistic and cultural LLM evaluation suite that emphasises SEA languages, comprising five core pillars: (1) NLP Classics, (2) LLM-specifics, (3) SEA Linguistics, (4) SEA Culture, (5) Safety. SEA-HELM currently supports Filipino, Indonesian, Tamil, Thai, and Vietnamese. We also introduce the SEA-HELM leaderboard, which allows users to understand models' multilingual and multicultural performance in a systematic and user-friendly manner. We make the SEA-HELM evaluation code publicly available.

Abstract of SEA-HELM: Southeast Asian Holistic Evaluation of Language Models (arXiv:2502.14301, CC0)
Made by AI Singapore + Stanford CRFM Released Feb 2025 Reasoning in many languages All tasks public Last model added Sep 2026 Scores refreshed 2 Nov 2026
Level
School Students, from grade school to high school
Scored
Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers
Short answer A number, word or short string matched to a key.
stanford-crfm/helm 2,961 stars Paper Leaderboard crfm.stanford.edu
87
Entries on its own leaderboard, 62 models we track

Gemini 3.1 Pro leads with 77.4 among models released by Sep 2026

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Every run is in the next chart. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders.

The newest model with a score here came out in Sep 2026, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

87 entries. Scroll sideways for the rest.

  1. 77.4 Gemini 3.1 Pro
  2. 77.3 Gemini 3.8 Flash
  3. 77.0 GPT-5.6 Sol
  4. 76.9 GPT-6 Astra
  5. 76.4 Gemini 3.6 Flash
  6. 76.3 GPT-5.5
  7. 76.3 Gemini 3.5 Flash
  8. 76.2 Qwen3.5 397B-A17B
  9. 76.2 Gemma 4 31B IT
  10. 76.0 GPT-5.4
  11. 75.6 Claude Opus 4.8
  12. 75.5 GPT-5.6 Terra
  13. 75.3 Gemini 3.1 Flash-Lite
  14. 74.8 Gemma 4 26B A4B
  15. 74.7 Qwen3.5-122B-A10B
  16. 74.6 Qwen3.6 27B
  17. 74.2 Qwen3.5-27B
  18. 74.1 Claude Sonnet 4.6
  19. 74.0 GLM-5.2
  20. 73.9 Qwen 3.6 35B-A3B
  21. 73.8 Claude Sonnet 5
  22. 73.5 Qwen3.5-35B-A3B
  23. 73.2 Qwen-SEA-LION-v4.5-27B-IT
  24. 73.0 Qwen3.8-Flash-Next
  25. 72.8 GPT-5.4 Mini
  26. 72.6 DeepSeek-V4-Pro
  27. 72.3 MiniMax-M3
  28. 72.1 Muse Glimmer
  29. 72.1 GPT-5.6 Luna
  30. 71.5 Inkling-Small
  31. 71.5 GLM-4.7
  32. 71.3 Inkling
  33. 71.0 Qwen3-VL-235B-A22B-Instruct
  34. 69.8 DeepSeek-V4-Flash
  35. 69.3 Gemma 4 12B
  36. 69.1 Qwen3.5-9B
  37. 68.3 Qwen-SEA-LION-v4-32B-IT
  38. 67.7 Qwen3-32B
  39. 67.0 Qwen3-VL-32B-Instruct
  40. 66.9 GLM-5.1
  41. 66.3 Gemma 4 E4B
  42. 66.1 gpt-oss-120b
  43. 64.9 Claude Haiku 4.5
  44. 64.8 Qwen3-14B
  45. 64.7 Mistral Medium 3.5
  46. 64.5 Gemma 3 27B
  47. 64.3 Gemma-SEA-LION-v4-27B-IT
  48. 64.1 Nemotron 3 Ultra
  49. 63.8 Qwen3.5-4B
  50. 63.7 Llama-SEA-LION-v3-70B-IT
  51. 63.4 Nemotron-SEA-LION-v4.8-120B-A12B
  52. 63.4 Llama 4 Scout
  53. 62.4 Gemma 3 12B
  54. 61.3 Llama 4 Maverick
  55. 60.6 gpt-oss-20b
  56. 60.4 Llama 3.3 70B Instruct
  57. 60.3 Qwen3-8B
  58. 59.2 Gemma-SEA-LION-v4.5-E2B-IT
  59. 58.8 Gemma 4 E2B
  60. 56.7 Command A+
  61. 56.6 Qwen3-VL-8B-Instruct
  62. 56.5 Gemma-SEA-LION-v3-9B-IT
  63. 56.4 Qwen-SEA-LION-v4-8B-VL
  64. 55.2 Apertus-v1.5-70B
  65. 54.9 Llama-SEA-LION-v3.5-8B-R
  66. 54.2 Apertus-v1.5-8B
  67. 53.8 glm-4.7-flash
  68. 52.2 Qwen-SEA-LION-v4-4B-VL
  69. 52.2 Gemma-SEA-LION-v4-4B-VL
  70. 52.1 Mistral Small 4
  71. 51.6 Nemotron-SEA-LION-v4.8-30B-A3B
  72. 50.4 Qwen3-VL-4B-Instruct
  73. 49.3 Nemotron 3 Super
  74. 48.7 Llama-SEA-LION-v3-8B-IT
  75. 47.8 Gemma 3 4B
  76. 46.9 Nemotron 3 Nano 30B-A3B
  77. 46.1 Nemotron 3.5 Lightning 30B-A3B
  78. 44.1 MERaLiON-2-10B
  79. 40.9 Olmo 3.1 32B Instruct
  80. 37.2 Llama 3.1-8B
  81. 36.9 Apertus-SEA-LION-v4-8B-IT
  82. 32.6 Apertus 70B
  83. 27.5 Apertus-8B-Instruct-2509
  84. 25.1 Llama 3.2 3B
  85. 20.0 MERaLiON-2-3B
  86. 19.8 tiny-aya-water
  87. 19.2 tiny-aya-global
  1. 1 Gemini 3.1 Pro 77.4
  2. 2 Gemini 3.8 Flash 77.3
  3. 3 GPT-5.6 Sol 77.0
  4. 4 GPT-6 Astra 76.9
  5. 5 Gemini 3.6 Flash 76.4
  6. 6 GPT-5.5 76.3
  7. 7 Gemini 3.5 Flash 76.3
  8. 8 Qwen3.5 397B-A17B thinking 76.2
  9. 9 Gemma 4 31B IT thinking 76.2
  10. 10 GPT-5.4 76.0
  11. 11 Claude Opus 4.8 75.6
  12. 12 GPT-5.6 Terra 75.5
  13. 13 Gemini 3.1 Flash-Lite 75.3
  14. 14 Gemma 4 26B A4B thinking 74.8
  15. 15 Qwen3.5-122B-A10B thinking 74.7
  16. 16 Qwen3.6 27B thinking 74.6
  17. 17 Qwen3.5-27B thinking 74.2
  18. 18 Claude Sonnet 4.6 74.1
  19. 19 GLM-5.2 74.0
  20. 20 Qwen 3.6 35B-A3B thinking 73.9
  21. 21 Claude Sonnet 5 73.8
  22. 22 Qwen3.5-35B-A3B thinking 73.5
  23. 23 Qwen-SEA-LION-v4.5-27B-IT thinking 73.2
  24. 24 Qwen3.8-Flash-Next 73.0
  25. 25 GPT-5.4 Mini 72.8
  26. 26 DeepSeek-V4-Pro 72.6
  27. 27 MiniMax-M3 72.3
  28. 28 Muse Glimmer 72.1
  29. 29 GPT-5.6 Luna 72.1
  30. 30 Inkling-Small 71.5
  31. 31 GLM-4.7 thinking 71.5
  32. 32 Inkling 71.3
  33. 33 Qwen3-VL-235B-A22B-Instruct 71.0
  34. 34 DeepSeek-V4-Flash thinking 69.8
  35. 35 Gemma 4 12B thinking 69.3
  36. 36 Qwen3.5-9B thinking 69.1
  37. 37 Qwen-SEA-LION-v4-32B-IT 68.3
  38. 38 Qwen3-32B 67.7
  39. 39 Qwen3-VL-32B-Instruct 67.0
  40. 40 GLM-5.1 66.9
  41. 41 Gemma 4 E4B thinking 66.3
  42. 42 gpt-oss-120b 66.1
  43. 43 Claude Haiku 4.5 64.9
  44. 44 Qwen3-14B 64.8
  45. 45 Mistral Medium 3.5 thinking 64.7
  46. 46 Gemma 3 27B 64.5
  47. 47 Gemma-SEA-LION-v4-27B-IT 64.3
  48. 48 Nemotron 3 Ultra thinking 64.1
  49. 49 Qwen3.5-4B thinking 63.8
  50. 50 Llama-SEA-LION-v3-70B-IT 63.7
  51. 51 Nemotron-SEA-LION-v4.8-120B-A12B 63.4
  52. 52 Llama 4 Scout 63.4
  53. 53 Gemma 3 12B 62.4
  54. 54 Llama 4 Maverick 61.3
  55. 55 gpt-oss-20b 60.6
  56. 56 Llama 3.3 70B Instruct 60.4
  57. 57 Qwen3-8B 60.3
  58. 58 Gemma-SEA-LION-v4.5-E2B-IT thinking 59.2
  59. 59 Gemma 4 E2B thinking 58.8
  60. 60 Command A+ 56.7
  61. 61 Qwen3-VL-8B-Instruct 56.6
  62. 62 Gemma-SEA-LION-v3-9B-IT 56.5
  63. 63 Qwen-SEA-LION-v4-8B-VL 56.4
  64. 64 Apertus-v1.5-70B 55.2
  65. 65 Llama-SEA-LION-v3.5-8B-R 54.9
  66. 66 Apertus-v1.5-8B 54.2
  67. 67 glm-4.7-flash thinking 53.8
  68. 68 Qwen-SEA-LION-v4-4B-VL 52.2
  69. 69 Gemma-SEA-LION-v4-4B-VL 52.2
  70. 70 Mistral Small 4 52.1
  71. 71 Nemotron-SEA-LION-v4.8-30B-A3B 51.6
  72. 72 Qwen3-VL-4B-Instruct 50.4
  73. 73 Nemotron 3 Super thinking 49.3
  74. 74 Llama-SEA-LION-v3-8B-IT 48.7
  75. 75 Gemma 3 4B 47.8
  76. 76 Nemotron 3 Nano 30B-A3B thinking 46.9
  77. 77 Nemotron 3.5 Lightning 30B-A3B 46.1
  78. 78 MERaLiON-2-10B 44.1
  79. 79 Olmo 3.1 32B Instruct 40.9
  80. 80 Llama 3.1-8B 37.2
  81. 81 Apertus-SEA-LION-v4-8B-IT 36.9
  82. 82 Apertus 70B 32.6
  83. 83 Apertus-8B-Instruct-2509 27.5
  84. 84 Llama 3.2 3B 25.1
  85. 85 MERaLiON-2-3B 20.0
  86. 86 tiny-aya-water 19.8
  87. 87 tiny-aya-global 19.2
#EntryScoreSettingDateSource
1 Gemini 3.1 Pro 77.4 Feb 2026 Official leaderboard
2 Gemini 3.8 Flash 77.3 Sep 2026 Official leaderboard
3 GPT-5.6 Sol 77.0 Jul 2026 Official leaderboard
4 GPT-6 Astra 76.9 Sep 2026 Official leaderboard
5 Gemini 3.6 Flash 76.4 Jul 2026 Official leaderboard
6 GPT-5.5 76.3 Apr 2026 Official leaderboard
7 Gemini 3.5 Flash 76.3 May 2026 Official leaderboard
8 Qwen3.5 397B-A17B 76.2 thinking Feb 2026 Official leaderboard
9 Gemma 4 31B IT 76.2 thinking Apr 2026 Official leaderboard
10 GPT-5.4 76.0 Mar 2026 Official leaderboard
11 Claude Opus 4.8 75.6 May 2026 Official leaderboard
12 GPT-5.6 Terra 75.5 Jul 2026 Official leaderboard
13 Gemini 3.1 Flash-Lite 75.3 Mar 2026 Official leaderboard
14 Gemma 4 26B A4B 74.8 thinking Apr 2026 Official leaderboard
15 Qwen3.5-122B-A10B 74.7 thinking Feb 2026 Official leaderboard
16 Qwen3.6 27B 74.6 thinking Apr 2026 Official leaderboard
17 Qwen3.5-27B 74.2 thinking Feb 2026 Official leaderboard
18 Claude Sonnet 4.6 74.1 Feb 2026 Official leaderboard
19 GLM-5.2 74.0 Jun 2026 Official leaderboard
20 Qwen 3.6 35B-A3B 73.9 thinking Apr 2026 Official leaderboard
21 Claude Sonnet 5 73.8 Jun 2026 Official leaderboard
22 Qwen3.5-35B-A3B 73.5 thinking Feb 2026 Official leaderboard
23 Qwen-SEA-LION-v4.5-27B-IT 73.2 thinking Official leaderboard
24 Qwen3.8-Flash-Next 73.0 Official leaderboard
25 GPT-5.4 Mini 72.8 Mar 2026 Official leaderboard
26 DeepSeek-V4-Pro 72.6 Apr 2026 Official leaderboard
27 MiniMax-M3 72.3 Jun 2026 Official leaderboard
28 Muse Glimmer 72.1 Aug 2026 Official leaderboard
29 GPT-5.6 Luna 72.1 Jul 2026 Official leaderboard
30 Inkling-Small 71.5 Jul 2026 Official leaderboard
31 GLM-4.7 71.5 thinking Dec 2025 Official leaderboard
32 Inkling 71.3 Jul 2026 Official leaderboard
33 Qwen3-VL-235B-A22B-Instruct 71.0 Sep 2025 Official leaderboard
34 DeepSeek-V4-Flash 69.8 thinking Apr 2026 Official leaderboard
35 Gemma 4 12B 69.3 thinking Jun 2026 Official leaderboard
36 Qwen3.5-9B 69.1 thinking Feb 2026 Official leaderboard
37 Qwen-SEA-LION-v4-32B-IT 68.3 Official leaderboard
38 Qwen3-32B 67.7 Apr 2025 Official leaderboard
39 Qwen3-VL-32B-Instruct 67.0 Official leaderboard
40 GLM-5.1 66.9 Apr 2026 Official leaderboard
41 Gemma 4 E4B 66.3 thinking Mar 2026 Official leaderboard
42 gpt-oss-120b 66.1 Aug 2025 Official leaderboard
43 Claude Haiku 4.5 64.9 Oct 2025 Official leaderboard
44 Qwen3-14B 64.8 Apr 2025 Official leaderboard
45 Mistral Medium 3.5 64.7 thinking Apr 2026 Official leaderboard
46 Gemma 3 27B 64.5 Mar 2025 Official leaderboard
47 Gemma-SEA-LION-v4-27B-IT 64.3 Official leaderboard
48 Nemotron 3 Ultra 64.1 thinking Jun 2026 Official leaderboard
49 Qwen3.5-4B 63.8 thinking Feb 2026 Official leaderboard
50 Llama-SEA-LION-v3-70B-IT 63.7 Official leaderboard
51 Nemotron-SEA-LION-v4.8-120B-A12B 63.4 Official leaderboard
52 Llama 4 Scout 63.4 Apr 2025 Official leaderboard
53 Gemma 3 12B 62.4 Mar 2025 Official leaderboard
54 Llama 4 Maverick 61.3 Apr 2025 Official leaderboard
55 gpt-oss-20b 60.6 Aug 2025 Official leaderboard
56 Llama 3.3 70B Instruct 60.4 Dec 2024 Official leaderboard
57 Qwen3-8B 60.3 Apr 2025 Official leaderboard
58 Gemma-SEA-LION-v4.5-E2B-IT 59.2 thinking Official leaderboard
59 Gemma 4 E2B 58.8 thinking Mar 2026 Official leaderboard
60 Command A+ 56.7 May 2026 Official leaderboard
61 Qwen3-VL-8B-Instruct 56.6 Official leaderboard
62 Gemma-SEA-LION-v3-9B-IT 56.5 Official leaderboard
63 Qwen-SEA-LION-v4-8B-VL 56.4 Official leaderboard
64 Apertus-v1.5-70B 55.2 Official leaderboard
65 Llama-SEA-LION-v3.5-8B-R 54.9 Official leaderboard
66 Apertus-v1.5-8B 54.2 Official leaderboard
67 glm-4.7-flash 53.8 thinking Jan 2026 Official leaderboard
68 Qwen-SEA-LION-v4-4B-VL 52.2 Official leaderboard
69 Gemma-SEA-LION-v4-4B-VL 52.2 Official leaderboard
70 Mistral Small 4 52.1 Mar 2026 Official leaderboard
71 Nemotron-SEA-LION-v4.8-30B-A3B 51.6 Official leaderboard
72 Qwen3-VL-4B-Instruct 50.4 Official leaderboard
73 Nemotron 3 Super 49.3 thinking Mar 2026 Official leaderboard
74 Llama-SEA-LION-v3-8B-IT 48.7 Official leaderboard
75 Gemma 3 4B 47.8 Mar 2025 Official leaderboard
76 Nemotron 3 Nano 30B-A3B 46.9 thinking Dec 2025 Official leaderboard
77 Nemotron 3.5 Lightning 30B-A3B 46.1 Aug 2026 Official leaderboard
78 MERaLiON-2-10B 44.1 Official leaderboard
79 Olmo 3.1 32B Instruct 40.9 Dec 2025 Official leaderboard
80 Llama 3.1-8B 37.2 Jul 2024 Official leaderboard
81 Apertus-SEA-LION-v4-8B-IT 36.9 Official leaderboard
82 Apertus 70B 32.6 Sep 2025 Official leaderboard
83 Apertus-8B-Instruct-2509 27.5 Official leaderboard
84 Llama 3.2 3B 25.1 Sep 2024 Official leaderboard
85 MERaLiON-2-3B 20.0 Official leaderboard
86 tiny-aya-water 19.8 Official leaderboard
87 tiny-aya-global 19.2 Official leaderboard

Scores at each effort level

Models the leaderboard ran at two or more effort levels. We don’t have enough cost data here to say what the extra effort costs.

Lowest effort to highest, light to dark
Qwen3.5 397B-A17B
non-thinking thinking
76.2
Gemma 4 31B IT
non-thinking
76.2
Gemma 4 26B A4B
non-thinking thinking
74.8
Qwen3.5-122B-A10B
non-thinking thinking
74.7
Qwen3.6 27B
non-thinking thinking
74.6
Qwen3.5-27B
non-thinking thinking
74.2
Qwen 3.6 35B-A3B
non-thinking thinking
73.9
Qwen3.5-35B-A3B
non-thinking thinking
73.5
GLM-4.7
non-thinking thinking
71.5
DeepSeek-V4-Flash
non-thinking thinking
69.8
Gemma 4 12B
non-thinking thinking
69.3
Qwen3.5-9B
non-thinking thinking
69.1
ModelEffortHarnessScoreCost per taskOutput tokens
Gemini 3.1 Pro 77.4
Gemini 3.8 Flash 77.3
GPT-5.6 Sol 77.0
GPT-6 Astra 76.9
Gemini 3.6 Flash 76.4
GPT-5.5 76.3
Gemini 3.5 Flash 76.3
Qwen3.5 397B-A17B thinking 76.2
Gemma 4 31B IT thinking 76.2
GPT-5.4 76.0
Claude Opus 4.8 75.6
GPT-5.6 Terra 75.5
Gemini 3.1 Flash-Lite 75.3
Gemma 4 31B IT non-thinking 75.2
Gemma 4 26B A4B thinking 74.8
Qwen3.5-122B-A10B thinking 74.7
Qwen3.6 27B thinking 74.6
Qwen3.5-27B thinking 74.2
Claude Sonnet 4.6 74.1
GLM-5.2 74.0
Qwen3.5 397B-A17B non-thinking 73.9
Qwen 3.6 35B-A3B thinking 73.9
Claude Sonnet 5 73.8
Qwen3.5-35B-A3B thinking 73.5
GPT-5.4 Mini 72.8
DeepSeek-V4-Pro 72.6
MiniMax-M3 72.3
Muse Glimmer 72.1
GPT-5.6 Luna 72.1
Inkling-Small 71.5
GLM-4.7 thinking 71.5
Inkling 71.3
Qwen3-VL-235B-A22B-Instruct 71.0
Qwen3.5-122B-A10B non-thinking 70.9
Gemma 4 26B A4B non-thinking 70.8
Qwen3.5-27B non-thinking 70.8
DeepSeek-V4-Flash thinking 69.8
Gemma 4 12B thinking 69.3
Qwen3.5-9B thinking 69.1
Qwen3-32B 67.7
Qwen3.6 27B non-thinking 67.2
GLM-5.1 66.9
Gemma 4 E4B thinking 66.3
gpt-oss-120b 66.1
Gemma 4 12B non-thinking 65.2
GLM-4.7 non-thinking 64.9
Claude Haiku 4.5 64.9
DeepSeek-V4-Flash non-thinking 64.9
Qwen3-14B 64.8
Mistral Medium 3.5 thinking 64.7
Gemma 3 27B 64.5
Nemotron 3 Ultra thinking 64.1
Qwen3.5-4B thinking 63.8
Llama 4 Scout 63.4
Gemma 4 E4B non-thinking 62.5
Gemma 3 12B 62.4
Nemotron 3 Ultra non-thinking 62.2
Qwen 3.6 35B-A3B non-thinking 61.9
Qwen3.5-35B-A3B non-thinking 61.9
Llama 4 Maverick 61.3
gpt-oss-20b 60.6
Llama 3.3 70B Instruct 60.4
Mistral Medium 3.5 non-thinking 60.4
Qwen3-8B 60.3
Gemma 4 E2B thinking 58.8
Command A+ 56.7
glm-4.7-flash thinking 53.8
Mistral Small 4 52.1
Qwen3.5-9B non-thinking 50.7
Nemotron 3 Super thinking 49.3
Nemotron 3 Super non-thinking 48.0
Gemma 3 4B 47.8
Gemma 4 E2B non-thinking 47.3
Nemotron 3 Nano 30B-A3B thinking 46.9
Nemotron 3.5 Lightning 30B-A3B 46.1
Qwen3.5-4B non-thinking 41.2
Olmo 3.1 32B Instruct 40.9
Llama 3.1-8B 37.2
glm-4.7-flash non-thinking 34.8
Apertus 70B 32.6
Nemotron 3 Nano 30B-A3B non-thinking 26.7
Llama 3.2 3B 25.1

7 models have held number one since its release in Feb 2025

Lab claims count from the day the card came out, independent runs from the model’s release.

Meta Google DeepMind Qwen Z.ai
Own leaderboard Not tested since Sep 2026
Gemini 3.1 Pro 77.4

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
20.0 40.0 60.0 80.0 2025 2026 20.0 40.0 60.0 80.0 2025 2026

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%