WMT24++
K-EXAONE 90.5% leads since Dec 2025 In use English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET.
Cohere added internal Irish, Maltese and Serbian variants.
In the authors’ words
As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages and dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. The dataset covers four domains: literary, news, social, and speech. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. These results should be confirmed using a human-based evaluation, which we leave for future work.
- Level
- University Undergraduates and entry-level professionals
- Scored
- Graded by a model A language model scores each response, usually against a rubric.
- Answers
- Open-ended Free-form text, documents, designs or media judged as a whole.
K-EXAONE leads with 90.5%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 K-EXAONE 90.5%
- 2
Nemotron 3 Super 86.7% - 3 Nemotron 3 Nano 30B-A3B 86.2%
- 4
Hunyuan-MT-7B 85.8% - 5
Qwen3.7-Max 85.8% - 6 Nemotron 3.5 Super VL 84.8%
- 7 Qwen 3.6 Plus 84.3%
- 8 Nemotron-Cascade 2 30B-A3B 84.1%
- 9
Command A Translate 83.8% - 10 Nemotron 3 Ultra 83.7%
- 11 Command A+ 81%
- 12 Qwen3.5 397B-A17B 78.9%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | K-EXAONE | 90.5% | Dec 2025 | Model card | |
| 2 | Nemotron 3 Super | 86.7% | Mar 2026 | Model card | |
| 3 | Nemotron 3 Nano 30B-A3B | 86.2% | Dec 2025 | Model card | |
| 4 | Hunyuan-MT-7B | 85.8% | Sep 2025 | Model card | |
| 5 | Qwen3.7-Max | 85.8% | May 2026 | Model card | |
| 6 | Nemotron 3.5 Super VL | 84.8% | Oct 2026 | Model card | |
| 7 | Qwen 3.6 Plus | 84.3% | Apr 2026 | Model card | |
| 8 | Nemotron-Cascade 2 30B-A3B | 84.1% | Mar 2026 | Model card | |
| 9 | Command A Translate | 83.8% | Aug 2025 | Model card | |
| 10 | Nemotron 3 Ultra | 83.7% | Jun 2026 | Model card | |
| 11 | Command A+ | 81% | May 2026 | Model card | |
| 12 | Qwen3.5 397B-A17B | 78.9% | Feb 2026 | Model card |
4 models have held number one since its release in Feb 2025
Lab claims count from the day the card came out, independent runs from the model’s release.
The best score passed 85% in Sep 2025
All the scores we have, one dot each, so a model tested at several settings appears more than once.
2 labs put it in a launch post in the last six months
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 15 Oct 2026 | Nemotron 3.5 Super VL | 84.8 | WMT24++ (COMET) |
| 4 Jun 2026 | Nemotron 3 Ultra | 83.7 | WMT24++ (en→xx) |
| 1 Jun 2026 | Qwen3.7-Plus | 84.6 | WMT24++ (XCOMET-XXL, 55 languages) |
| 20 May 2026 | Command A+ | 81 | WMT24++ (xCOMET-XL) |
| 20 May 2026 | Qwen3.7-Max | 85.8 | WMT24++ |
| 2 Apr 2026 | Qwen 3.6 Plus | 84.3 | WMT24++ |
| 19 Mar 2026 | Nemotron-Cascade 2 30B-A3B | 84.1 | WMT24++ (en -> xx) |
| 11 Mar 2026 | Nemotron 3 Super | 86.7 | WMT24++ (en→xx) |
| 24 Feb 2026 | Qwen3.5-122B-A10B | 78.3 | WMT24++ (XCOMET-XXL, 55 languages) |
| 16 Feb 2026 | Qwen3.5 397B-A17B | 78.9 | WMT24++ |
| 31 Dec 2025 | K-EXAONE | 90.5 | WMT24++ |
| 15 Dec 2025 | Nemotron 3 Nano 30B-A3B | 86.2 | WMT24++ (en->xx) |
| 1 Sep 2025 | Hunyuan-MT-7B | 85.8 | WMT24pp (XCOMET-XXL) |
| 28 Aug 2025 | Command A Translate | 83.8 | WMT24++ xCometXL, average over 23 languages, English to L2 |
| 26 Jun 2025 |  Gemma 3n E4B | 50.1 | WMT24++ (ChrF) 0-shot (IT) |
| 12 Mar 2025 | Gemma 3 27B | 55.7 | WMT24++ (ChrF) (PT); IT 0-shot = 53.4 |
18 system cards report it, from 6 labs
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 23 Oct 2026 | Nemotron 3.5 Bolt E4B technical report | 74.4 | WMT24++ English to X xCOMET, average of 36 languages, BF16 non-reasoning. chrF 52.5, X to English xCOMET 83.4 (Table 6) | Capabilities |
| 19 Oct 2026 |  Muse Glimmer technical report | 53.4 | WMT24++, chrF++, high. X to eng 55.7, eng to Y 51.0 (Tables 5 and 9) | Capabilities |
| 15 Oct 2026 | Nemotron 3.5 Super VL technical report | 84.8 | WMT24pp COMET (XCOMET-XXL, en to 9 languages) (Table 10) | Capabilities |
| 12 Sep 2026 | North Small Translate 1.0 technical report | 80.6 | WMT24++ xCOMET-XL (Table 1 and Section 3.2). Agentic 80.7 | Capabilities |
| 5 Aug 2026 | K-EXAONE 2.0 technical report | WMT24++ per-language EN to XX and XX to EN scores for nine languages, no overall (Table 9) | Capabilities | |
| 9 Jun 2026 | Nemotron 3 Ultra technical report | 83.7 | WMT24++ (en to xx) (Table 10) | Capabilities |
| 9 Apr 2026 | EXAONE 4.5 33B technical report | 91.5 | WMT24++ (ko, de, es, ja, vi), metric not stated (Table 3, Reasoning mode) | Capabilities |
| 19 Mar 2026 | Nemotron-Cascade 2 30B-A3B technical report | 84.1 | WMT24++ (en -> xx), XCOMET-XXL, 5 language pairs (Table 1 and A.7) | Capabilities |
| 11 Mar 2026 | Nemotron 3 Super technical report | 86.7 | WMT24++ (en to xx) (Table 5) | Capabilities |
| 5 Jan 2026 | K-EXAONE technical report | 90.5 | WMT24++ (ko, de, es, ja, vi), GPT-5-mini judge score 0 to 100 (Table 3, Reasoning mode). Per direction in Table 6. Non-reasoning 88.0 | Capabilities |
| 23 Dec 2025 | Nemotron 3 Nano 30B-A3B technical report | 86.2 | WMT24++ (en to xx) (Table 3) | Capabilities |
| 26 Sep 2025 | Command A Translate technical report | 83.9 | WMT24++ xComet-XL Avg over 23 languages, English to each (Tables 1 and 2). The launch blog prints 83.8 | Capabilities |
| 5 Sep 2025 | Hunyuan-MT-7B technical report | 85.8 | WMT24pp XCOMET-XXL, 29 English-centric pairs (Table 4). CometKiwi 80.61 | Capabilities |
| 15 Jul 2025 | EXAONE 4.0 32B technical report | 90.7 | WMT24++ (ES), EN and ES pair, GPT-4.1 judge score 0 to 100, Non-reasoning only (Table 4) | Capabilities |
| 25 Mar 2025 | Gemma 3 4B technical report | 46.8 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| 25 Mar 2025 | Gemma 3 27B technical report | 53.4 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| 25 Mar 2025 | Gemma 3 1B technical report | 35.9 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
| 25 Mar 2025 | Gemma 3 12B technical report | 51.6 | WMT24++, chrF, IT, 0-shot (Table 18) | Capabilities |
More in Multilingual
- MMMLU In use MMLU professionally translated into 14 languages. Multiple-choice accuracy. Claude Mythos Preview 92.7%
- MMLU-ProX In use MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. Qwen3.7-Max 87%
- CLIcK In use Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories.
Solar Open 2 90.7% - Global MMLU Lite 3.0 In use Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy.
Inkling 88.7% - MMMB In use Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. LFM2.5-VL-3B 83%
- Alyah In use Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect.
Falcon-Emirati-7B 84.8%