Benchmark Almanac

SEAL Multilingual

Gemini 2.0 Pro (December 2024) 1,176 tops its leaderboard o1 1,134 leads the models we track Not in model cards

Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo.

Spanish was a launch board covering Spain, Mexico and Latin America. Japanese, Chinese, Arabic and Korean boards followed. Spanish was deprecated in 2025-03 with 22 models, led by Gemini 2.0 Pro.

Made by Scale AI Released May 2024 Chat and instructions in many languages Tasks kept private Last model added Jan 2025 Scores refreshed 2 Nov 2026
Level
Everyday Most adults, without training
Scored
Graded by people Human experts grade or compare the outputs.
Answers
Open-ended Free-form text, documents, designs or media judged as a whole.
Leaderboard scale.com
22
Entries on its own leaderboard, 19 models we track

Gemini 2.0 Pro (December 2024) tops the leaderboard with 1,176

Best score per entry from the benchmark's own leaderboard, read on 2 Nov 2026. Where a model ran with several settings or harnesses, we keep its best. Names in gray aren't in our model list: fine-tunes, agent systems, products and older models. They don't count toward leaders. The best model we track is o1, at 1,134.

The newest model with a score here came out in Jan 2025, and nobody has tested a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:

22 entries. Scroll sideways for the rest. Bars start at 900.

  1. 1,176 Gemini 2.0 Pro (December 2024)
  2. 1,134 o1
  3. 1,119 Gemini 2.0 Flash
  4. 1,111 o1-preview
  5. 1,108 Gemini 2.0 Flash Thinking
  6. 1,105 Gemini 1.5 Pro (November 2024)
  7. 1,084 GPT-4o
  8. 1,079 o3-mini
  9. 1,069 Gemini 1.5 Pro
  10. 1,067 Gemini 1.5 Pro (August 27, 2024)
  11. 1,067 GPT-4o
  12. 1,034 GPT-4 Turbo
  13. 1,032 Mistral Large 2
  14. 1,020 GPT-4 Turbo
  15. 1,005 Gemini 1.5 Pro
  16. 992 Claude 3.5 Sonnet
  17. 983 Aya Expanse 32B
  18. 980 Gemini 1.5 Flash
  19. 951 Gemma 2 27B
  20. 944 Llama 3.2 90B
  21. 919 Claude 3 Opus
  22. 915 Llama 3.1-405B

Bars start at 900.

  1. 1 Gemini 2.0 Pro (December 2024) 1,176
  2. 2 o1 1,134
  3. 3 Gemini 2.0 Flash Dec 2024 1,119
  4. 4 o1-preview 1,111
  5. 5 Gemini 2.0 Flash Thinking Jan 2025 1,108
  6. 6 Gemini 1.5 Pro (November 2024) 1,105
  7. 7 GPT-4o May 2024 1,084
  8. 8 o3-mini high 1,079
  9. 9 Gemini 1.5 Pro May 2024 1,069
  10. 10 Gemini 1.5 Pro (August 27, 2024) 1,067
  11. 11 GPT-4o Aug 2024 1,067
  12. 12 GPT-4 Turbo Nov 2023 1,034
  13. 13 Mistral Large 2 Jul 2024 1,032
  14. 14 GPT-4 Turbo Apr 2024 1,020
  15. 15 Gemini 1.5 Pro Feb 2024 1,005
  16. 16 Claude 3.5 Sonnet 992
  17. 17 Aya Expanse 32B 983
  18. 18 Gemini 1.5 Flash May 2024 980
  19. 19 Gemma 2 27B 951
  20. 20 Llama 3.2 90B 944
  21. 21 Claude 3 Opus 919
  22. 22 Llama 3.1-405B 915
#EntryScoreSettingDateSource
1 Gemini 2.0 Pro (December 2024) 1,176 Official leaderboard
2 o1 1,134 Dec 2024 Official leaderboard
3 Gemini 2.0 Flash Dec 2024 1,119 Dec 2024 Official leaderboard
4 o1-preview 1,111 Sep 2024 Official leaderboard
5 Gemini 2.0 Flash Thinking Jan 2025 1,108 Jan 2025 Official leaderboard
6 Gemini 1.5 Pro (November 2024) 1,105 Official leaderboard
7 GPT-4o May 2024 1,084 May 2024 Official leaderboard
8 o3-mini 1,079 high Jan 2025 Official leaderboard
9 Gemini 1.5 Pro May 2024 1,069 May 2024 Official leaderboard
10 Gemini 1.5 Pro (August 27, 2024) 1,067 Official leaderboard
11 GPT-4o Aug 2024 1,067 Aug 2024 Official leaderboard
12 GPT-4 Turbo Nov 2023 1,034 Nov 2023 Official leaderboard
13 Mistral Large 2 Jul 2024 1,032 Jul 2024 Official leaderboard
14 GPT-4 Turbo Apr 2024 1,020 Apr 2024 Official leaderboard
15 Gemini 1.5 Pro Feb 2024 1,005 Feb 2024 Official leaderboard
16 Claude 3.5 Sonnet 992 Jun 2024 Official leaderboard
17 Aya Expanse 32B 983 Oct 2024 Official leaderboard
18 Gemini 1.5 Flash May 2024 980 May 2024 Official leaderboard
19 Gemma 2 27B 951 Jun 2024 Official leaderboard
20 Llama 3.2 90B 944 Sep 2024 Official leaderboard
21 Claude 3 Opus 919 Feb 2024 Official leaderboard
22 Llama 3.1-405B 915 Jul 2024 Official leaderboard

5 models have held number one since its release in May 2024

Lab claims count from the day the card came out, independent runs from the model’s release.

OpenAI Google DeepMind
Own leaderboard Not tested since Jan 2025
o1 1,134

Scores over time

All the scores we have, one dot each, so a model tested at several settings appears more than once.

On the benchmark’s own leaderboard
900 950 1,000 1,050 1,100 1,150 2024 2025 900 950 1,000 1,050 1,100 1,150 2024 2025

No lab model card reports it yet

We read the headline results of 479 launch posts from 51 labs, and none of them list this benchmark.

More in Multilingual

All 115 in Multilingual
  • MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
  • WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. K-EXAONE 90.5%
  • MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
  • CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. Solar Open 2 90.7%
  • Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. Inkling 88.7%
  • MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%