Flores
Gemma 3 27B 48.8 leads since Mar 2025 Dropped Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references.
FLORES-200 extended it to 200 languages in 2022, and the Open Language Data Initiative now maintains it.
In the authors’ words
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.
- Level
- University Undergraduates and entry-level professionals
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Open-ended Free-form text, documents, designs or media judged as a whole.
Gemma 3 27B leads with 48.8 among models released by Mar 2025
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
The newest model with a score here came out in Mar 2025, and nobody has reported a newer one. Among the 20 highest-ranked models on the capability index, these have no score yet:
- GPT-6 Astra
- Claude Opus 5.5
- GPT-6.1 Terra
- GPT-6.1 Sol
- and 16 more
- 1 Gemma 3 27B 48.8
- 2 Gemma 3 12B 46.0
- 3 Gemma 3 4B 39.2
- 4
Qwen2.5-72B 39.0 - 5 Qwen2-72B 37.8
- 6
Aya Vision 32B 37.8 - 7 Qwen1.5-72B 35.6
- 8 Gemma 3 1B 29.5
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | Gemma 3 27B | 48.8 | Mar 2025 | Model card | |
| 2 | Gemma 3 12B | 46.0 | Mar 2025 | Model card | |
| 3 | Gemma 3 4B | 39.2 | Mar 2025 | Model card | |
| 4 | Qwen2.5-72B | 39.0 | Sep 2024 | Model card | |
| 5 | Qwen2-72B | 37.8 | Jun 2024 | Model card | |
| 6 | Aya Vision 32B | 37.8 | May 2025 | Model card | |
| 7 | Qwen1.5-72B | 35.6 | Feb 2024 | Model card | |
| 8 | Gemma 3 1B | 29.5 | Mar 2025 | Model card |
4 models have held number one since 2023
Lab claims count from the day the card came out, independent runs from the model’s release.
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs moved on to FLORES-200
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 12 Mar 2025 | Gemma 3 27B | 48.8 | FloRes (PT) |
| 17 Feb 2025 |  Mistral Saba | Arabic-Centric FLORES-101 (chart, metric not stated) | |
| 19 Sep 2024 | Qwen2.5-72B | 39 | Multi-Translation |
| 7 Jun 2024 | Qwen2-72B | 37.8 | Multi-Translation |
| 5 Feb 2024 | Qwen1.5-72B | 35.6 | Multilingual Translation (Flores, BLEU) |
5 system cards report it, from 2 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 13 May 2025 | Aya Vision 32B technical report | 37.8 | FLORES, En to X, spBLEU, 23 languages (Table 5) | Capabilities |
| 25 Mar 2025 | Gemma 3 4B technical report | 39.2 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 27B technical report | 48.8 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 1B technical report | 29.5 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
| 25 Mar 2025 | Gemma 3 12B technical report | 46 | FLoRes, chrF, PT, 1-shot (Table 13) | Capabilities |
Versions
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- WMT24++ In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
K-EXAONE 90.5% - MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.
Inkling 88.7% - MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%