Multilingual
We track 115 benchmarks in multilingual, or 127 if you count each version. They run from everyday level to research frontier level.
Qwen held the top multilingual score 63% of the contested time
That's about 29 of the 46 months since January 2023, averaged over the benchmarks below. The easiest benchmarks are at the top. Versions of one test share a row, with a dark tick at each new version. We stop the clock, and fade the rest of the row, when the best score passes 85%, or at the newest score if 45 days go by without another.
Other labs Hugging Face Alibaba Group OpenAI University of Wisconsin-Madison Shanghai AI Lab Google DeepMind Z.ai Qwen Anthropic Deep Cogito Technology Innovation Institute DeepSeek Cohere Thinking Machines Meta
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Share of contested time at number one
Qwen 63% OpenAI 15% 
Google DeepMind 13% Other labs 3% 
Cohere 2% 
University of Wisconsin-Madison 1% 
DeepSeek 1% Anthropic 1% 
Hugging Face 1% Closed out: 7 benchmarks passed 85%
Qwen 2 
Thinking Machines 1 Anthropic 1 Google DeepMind 1 
Shanghai AI Lab 1 Other labs 1 - Jul 2026 Global MMLU Lite 3.0 Inkling
- May 2026 MMLU-ProX Qwen3.7-Max
- Feb 2026 INCLUDE Qwen3.5 397B-A17B
- Feb 2025 MMMLU Claude 3.7 Sonnet
- Feb 2025 Global MMLU Lite Gemini 2.0 Pro
- Jul 2024 MMBench-CN v1.1 InternVL2-Llama3-76B
- May 2023 MGSM PaLM 2-L
Scores come from each benchmark’s own leaderboard, from Epoch AI’s runs and from labs’ reports on their own models. We date Epoch runs by the model’s release and lab reports by when they came out. Labs pick their own settings, so a lab’s claim can beat an independent run of the same model. Turn off lab claims to see independent runs only.
Other labs Hugging Face Alibaba Group OpenAI University of Wisconsin-Madison Shanghai AI Lab Google DeepMind Qwen Meta Anthropic Cohere
Everyday Most adults, without training
School Students, from grade school to high school
University Undergraduates and entry-level professionals
Expert Specialists with years of training, often over hours of work
Share of contested time at number one, independent runs only
Qwen 58% OpenAI 23% Google DeepMind 7% Other labs 5% Cohere 3% University of Wisconsin-Madison 2% Hugging Face 1% Anthropic 1% Only each benchmark’s own leaderboard and Epoch AI’s runs, dated by the model’s release. We leave out benchmarks that only labs report.
From everyday to research frontier level
Grouped by who could solve the tasks, from things most adults can do to open research problems. The bar under each name is its best score.
Everyday
Most adults, without training
- MMBench-CN v1.1
- Wer wird Millionär? LLM Benchmark
- ECLeKTic
- AlGhafa
- ALRAGE
- IndicGenBench
- SEAL Multilingual
- Global PIQA
- CLUEWSC
- MultiLoKo
- MAXIFE
- BLEnD
- Multilingual Understanding
- Multilingual IFEval
- XQuAD
- IndiVibe
- TydiQA
- XCOPA
- Ko-Arena-Hard v2 (Upstage)
- Alyah
- CMRC
- FLORES-200
- Mandarin-Minority translation (Tencent)
- Multilingual human evaluation
- XNLI
- ArabCulture
- Belebele
- BOUQuET
- Challenging translation human evaluation (Z.ai)
- CulturalBench
- CVQA
- Hunyuan-MT Challenge Testset (Tencent)
- Ko-IFEval (Upstage)
- Language Confusion Benchmark
- Qwen in-house Chinese automatic evaluation
- Structured Translation (Cohere)
- SwitzerlandQA
- ZMultiDialBench
- AL-QASIDA
- ArabCulture-Dialogue
- AraDiCE
- CulturalRiddles
- Dolly Machine Translated v2
- KoBEST
- KoMT-Bench
- mDolly
- MLQA
- Uhura TruthfulQA
- Uhura-Eval
- XStoryCloze
School
Students, from grade school to high school
- MGSM
- SEA-HELM
- ArabicMMLU
- Open Arabic LLM Leaderboard v2
- MadinahQA
- HELM Arabic
- ThaiExam
- CLIcK
- C3
- HAE-RAE Bench
- IndoMMLU
- TurkishMMLU
- MMMB
- CCPM
- CHID
- M3Exam
- 3LM
- CLUE
- EXAMS
- GlobalMGSM
- IrokoBench
- MGSM-Rev2
- Multilingual MMBench
- Open Arabic LLM Leaderboard v1
- QIMMA
- Translated ARC-Challenge, HellaSwag and MMLU (Mistral)
- Uhura ARC-Easy
University
Undergraduates and entry-level professionals
- MMMLU
- Global MMLU Lite
- ArbMMLU-HT
- INCLUDE
- MMLU-ProX
- WMT24++
- Flores
- Global MMLU
- MILU
- NOVA-63
- KMMLU
- MLogiQA
- Okapi MMLU
- Global MMLU Lite 3.0
- KoBALT
- Multilingual Exams
- WMT23
- AMMLU
- JMMLU
- Multilingual MMLU (Meta)
- MSQA
- QwenChineseBench
- WMT25
- WMT26
- FLORES+
- Kaleidoscope
- MENT-SNS
- NTREX-128
- Omnilingual-Gaia2
- Translation Human Scoring (Z.ai)
- ZMultiTransBench
- Global MMLU Lite 2.0
- KMMLU-Hard
- m-ArenaHard
- m-ArenaHard v2.0
- m-ArenaHard v2.1
- MMLU translated with Azure Translate
- MMLU-ProX Lite
- WMT24
Competition
The strongest students, in olympiad-style contests
Expert
Specialists with years of training, often over hours of work
By kind
Knowledge in many languages
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| MMBench-CN v1.1 Models answer the Chinese translation of the MMBench image questions, using CircularEval with shuffled options. Scored by accuracy. | Everyday | Checked automatically | 88.1% | Fading |
| MMMLU MMLU professionally translated into 14 languages. Multiple-choice accuracy. | University | Checked automatically | 92.7% | In use |
| Wer wird Millionär? LLM Benchmark Models play 45 games of the German version of Who Wants to Be a Millionaire, each with 15 questions of rising difficulty. Early questions lean on German wordplay and idioms. | Everyday | Checked automatically | 644444.0 | Not in cards |
| Global MMLU Lite Models answer MMLU questions in 15 languages, with 200 culturally sensitive and 200 culturally agnostic questions per language. All items are human translated or post-edited. Accuracy. | University | Checked automatically | 95.7% | Dropped |
| ECLeKTic Models answer 384 closed-book questions, translated into 12 languages, about facts found in only one language's Wikipedia. Scoring checks transfer across languages. | Everyday | Checked automatically | 56.3% | Dropped |
| ArabicMMLU Models answer 14,575 multiple-choice questions in Modern Standard Arabic across 40 tasks, taken from school exams in North Africa, the Levant and the Gulf. | School | Checked automatically | 71.4% | Dropped |
| Open Arabic LLM Leaderboard v2 Models are scored on native Arabic tasks (AlGhafa native subsets, ArabicMMLU, Exams, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT) and ranked by the average. | School | Checked automatically | 72.4% | Dropped |
| AlGhafa Models answer Arabic multiple-choice tasks on reading comprehension, sentiment, facts and question answering. Scored by normalized log-likelihood accuracy. | Everyday | Checked automatically | 78.6% | System cards only |
| MadinahQA Models answer Arabic school exam questions on general Arabic language and grammar. | School | Checked automatically | 69.4% | System cards only |
| ArbMMLU-HT Models answer a human translation into Arabic of the 57 MMLU subjects. | University | Checked automatically | 72.2% | Not in cards |
| INCLUDE Models answer multiple-choice questions from local academic and professional exams in 44 languages that test regional knowledge. Scored by accuracy. | University | Checked automatically | 89.8% | Fading |
| MMLU-ProX MMLU-Pro translated into up to 29 languages with expert review. Multiple-choice accuracy per language. | University | Checked automatically | 87% | In use |
| HELM Arabic Stanford CRFM runs models on 7 Arabic benchmarks in the HELM harness and ranks them by mean score. The tasks are AlGhafa, ArabicMMLU, Arabic EXAMS, MadinahQA, AraTrust, ALRAGE and human-translated MMLU. | School | Checked automatically | 84.1 | Not in cards |
| ThaiExam Models answer multiple-choice questions from standardized examinations in Thailand, such as ONET, IC, TGAT, TPAT-1 and A-Level. | School | Checked automatically | 75.1% | Not in cards |
| HELM Arabic Enterprise Models do Arabic enterprise tasks in six parts. They write corporate articles, answer finance questions in three formats, and answer legal questions written by Arabic legal experts with and without the statute. | Expert | Graded by a model | 77.9 | Not in cards |
| Global MMLU MMLU translated into 42 languages with cultural-sensitivity annotations. Multiple-choice accuracy. | University | Checked automatically | 94.3% | Dropped |
| MILU Multi-task Indic Language Understanding: culturally grounded multiple-choice questions in 11 Indic languages. | University | Checked automatically | 93.1% | Dropped |
| MultiLoKo Locally relevant knowledge questions in 31 languages, plus translations. Scored by exact match. | Everyday | Checked automatically |  31% | Fading |
| NOVA-63 Models answer questions written by native speakers in 14 languages across 63 academic disciplines. Scoring is accuracy. | University | Checked automatically | 59.1% | Fading |
| KMMLU Models answer 35,030 Korean multiple-choice questions across 45 subjects, collected from original Korean exams. Scoring is accuracy. | University | Checked automatically |  77% | Fading |
| Okapi MMLU Models answer MMLU multiple choice questions that ChatGPT translated into 26 languages. Scoring is accuracy. | University | Checked automatically | 80% | Dropped |
| CLIcK Models answer 1,995 multiple-choice questions on Korean language and culture. Questions come from official Korean exams and textbooks in eleven categories. | School | Checked automatically | 90.7% | In use |
| BLEnD Models answer 52.6k questions about everyday culture in 16 countries and regions across 13 languages. Scoring is accuracy on short-answer and multiple-choice formats. | Everyday | Checked automatically |  66.3% | Dropped |
| Global MMLU Lite 3.0 Models answer 400 human-translated MMLU questions per language in 23 languages. Version 3.0 adds Czech, Hungarian, Oriya, Slovak and Tajik and updates Italian. Accuracy. | University | Checked automatically | 88.7% | In use |
| HAE-RAE Bench Models answer Korean questions on vocabulary, history, general knowledge and reading comprehension. It tests cultural and contextual knowledge that translated benchmarks miss. | School | Checked automatically | 86.2% | Fading |
| Multilingual Exams Models answer multilingual exam and MMLU-style questions. Qwen reports the average of M3Exam, IndoMMLU, ruMMLU, and translated MMLU. | University | Checked automatically | 78.7% | Dropped |
| AMMLU Models answer MMLU multiple-choice questions across 57 subjects, translated into Arabic with GPT-4. Scored by accuracy. | University | Checked automatically | 72.4% | Dropped |
| IndoMMLU Models answer 14,981 multiple-choice questions from Indonesian school and university entrance exams, including nine local languages. Scored by accuracy. | School | Checked automatically | 69.3% | Dropped |
| JMMLU Models answer 7,536 four-way multiple-choice questions in Japanese, from translated MMLU subjects and new Japan-specific subjects. Scored by accuracy. | University | Checked automatically | 80.6% | Dropped |
| TurkishMMLU Models answer over 10,000 multiple choice questions that Turkish curriculum experts wrote for nine high-school subjects. Scoring is accuracy. | School | Checked automatically | 76.1% | Dropped |
| MMMB Models answer multiple-choice questions about images in six languages across 15 categories. The set has about 12,000 questions. | School | Checked automatically | 83% | In use |
| Multilingual MMLU (Meta) Models answer MMLU questions machine-translated into German, French, Italian, Portuguese, Hindi, Spanish and Thai, 5-shot. | University | Checked automatically | 83.2% | System cards only |
| Alyah Models answer 1,173 four-option questions written by native Emirati speakers on greetings, idioms, figurative language, heritage and poetry in Emirati Arabic dialect. | Everyday | Checked automatically |  84.8% | In use |
| MSQA Models answer culture-specific knowledge questions in 11 major languages. | University | Checked automatically |  50.2% | In use |
| QwenChineseBench Models answer questions about real-world Chinese knowledge. Qwen describes it only as an internal Chinese knowledge benchmark. | University | Checked automatically | 84% | Fading |
| CulturalBench Models answer 1,696 human-written and human-verified questions on cultural knowledge across 45 global regions and 17 topics. Scored by accuracy. | Everyday | Checked automatically | 74.2% | System cards only |
| CVQA Vision-language models answer culturally specific visual questions written by native speakers from many countries and languages. | Everyday | Checked automatically | 74.1% | System cards only |
| Kaleidoscope Vision-language models answer 20,911 multiple choice questions from real exams in 18 languages of 8 families. About 55% of the questions need an image. Accuracy. | University | Checked automatically | 41.7% | System cards only |
| M3Exam Models answer 12,317 multiple-choice questions from official school exams in nine languages, about 23 percent needing images. Scored by accuracy. | School | Checked automatically | Dropped | |
| SwitzerlandQA Models answer four-option questions on Swiss geography, history, social life, politics and insurance at national, cantonal and commune level. Items come in five languages. | Everyday | Checked automatically | 67.2% | System cards only |
| AraDiCE Models are tested on Arabic dialect understanding and generation and on cultural awareness of the Gulf, Egypt and the Levant. Datasets were machine translated into dialects and post-edited by people. | Everyday | Checked automatically | System cards only | |
| EXAMS Models answer high school exam questions in 16 languages across many subjects. Arabic reports use the Arabic subset. | School | Checked automatically | Not in cards | |
| Global MMLU Lite 2.0 Models answer 400 human-translated MMLU questions per language in 18 languages, adding Albanian, Burmese and Welsh to the first 15. Accuracy. | University | Checked automatically | Not in cards | |
| IrokoBench Models answer human-translated knowledge, math and inference questions in 16 African languages. The parts are AfriMMLU, AfriMGSM and AfriXNLI. | School | Checked automatically | Not in cards | |
| K-BrowseComp Web browsing agents answer 400 hard search problems grounded in Korean contexts. A 300-problem Verified subset was built and checked by native speakers. | Expert | Checked automatically | Not in cards | |
| KMMLU-Hard Models answer about 5,000 Korean multiple-choice questions from KMMLU that most models got wrong, across many expert subjects. | University | Checked automatically | Not in cards | |
| MMLU translated with Azure Translate Models answer MMLU questions machine translated into 26 languages with Azure Translate. Scoring is 3-shot accuracy per language. | University | Checked automatically | System cards only | |
| MMLU-ProX Lite Models answer 658 MMLU-Pro questions per language, the same items in each of 29 languages. Multiple choice accuracy. | University | Checked automatically | Not in cards | |
| Multilingual MMBench Models answer MMBench dev image questions in English, Chinese, Portuguese, Arabic, Turkish and Russian. Scored as accuracy per language. | School | Checked automatically | System cards only | |
| Multilingual performance on translated expert-domain tasks Models answer an internal evaluation of tasks in expert domains such as science, law and engineering, machine translated into 17 languages. Scored as pass rate per language. | Expert | Graded by a model | System cards only | |
| Open Arabic LLM Leaderboard v1 Models are scored on AlGhafa with native and translated sets, the ACVA and AceGPT sets, and translated MMLU and EXAMS. The board ranks by the average. | School | Checked automatically | Not in cards | |
| QIMMA Models are ranked on a cleaned suite of over 52k Arabic samples. The makers first screened existing Arabic benchmarks for errors with model judges and human review. | School | Checked automatically | Not in cards | |
| Translated ARC-Challenge, HellaSwag and MMLU (Mistral) Models answer French, German, Spanish and Italian versions of ARC-Challenge, HellaSwag and MMLU. | School | Checked automatically | System cards only | |
| Uhura TruthfulQA Models answer TruthfulQA questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | Everyday | Checked automatically | System cards only |
Translation
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| WMT24++ English-to-55-language translation of WMT24 test data. Scored with automatic metrics such as MetricX or xCOMET. | University | Graded by a model |  90.5% | In use |
| Flores Models translate 3,001 English Wikipedia sentences that professionals translated into 101 languages. Scored with BLEU or chrF against references. | University | Checked automatically | 48.8 | Dropped |
| WMT23 Models translate the WMT23 general task test sets between English and languages such as German, Chinese, and Japanese. Model cards score output with BLEURT. | University | Checked automatically | 75.3 | Dropped |
| DiscoX Models translate 200 expert texts between Chinese and English at discourse level, averaging over 1,700 tokens each. Metric-S, a reference-free judge, scores accuracy, fluency and appropriateness. | Expert | Graded by a model | 82% | Fading |
| FLORES-200 Models translate English Wikipedia sentences into about 200 languages. Outputs are scored with BLEU or chrF against human reference translations. | Everyday | Checked automatically | 81.2 | Dropped |
| Mandarin-Minority translation (Tencent) Models translate between Mandarin and Chinese minority languages in a Tencent test set. XCOMET-XXL scores each translation. | Everyday | Checked automatically |  60.8% | Dropped |
| WMT25 Systems translate the WMT25 general task test sets in 16 directions, mostly out of English. New pairs include English to Bhojpuri, Korean and Maasai. Humans evaluate the systems. | University | Graded by people | 48.9 | System cards only |
| WMT26 Systems translate the WMT26 general task test set in 24 directions, mostly out of English, across several domains with added instructions. The shared task ranks systems by human contrastive ESA judgments. | University | Graded by people | 83.6 | In use |
| BOUQuET Models translate handcrafted multi-way paragraphs from several non-English source languages across many registers and domains. Scored with chrF++ against human references. | Everyday | Checked automatically | 36.5 | System cards only |
| Challenging translation human evaluation (Z.ai) People blindly score translations of 100 real cases with slang, nicknames and memes on a 0 to 3 scale. | Everyday | Graded by people | 1.71 | System cards only |
| FLORES+ Models translate English Wikipedia sentences that professionals and community contributors translated into over 200 languages. Outputs are scored with metrics such as chrF against the references. | University | Checked automatically | 47.5 | System cards only |
| Hunyuan-MT Challenge Testset (Tencent) Models translate between Chinese and English across everyday and specialist domains. Expert raters score each translation from 0 to 4 on marked error-prone points. | Everyday | Graded by people | 3.19 | System cards only |
| MENT-SNS Models translate English and Chinese sentences with slang, wordplay and idioms. A GPT-4.1 judge compares each translation with a baseline. | University | Graded by a model | 1013.0 | System cards only |
| NTREX-128 Models translate 1,997 English news sentences into 128 languages, with human reference translations. Scored with metrics such as COMET or chrF. | University | Checked automatically | 68.8 | System cards only |
| Structured Translation (Cohere) Models translate JSON and similar structures, either whole or only named fields. Metrics check parsing, schema match, untranslated or wrongly copied fields and line counts. | Everyday | Checked automatically | 93.7% | System cards only |
| Translation Human Scoring (Z.ai) People score model translations. The GLM-5 report does not describe the test set or the scale. | University | Graded by people | 3.12 | System cards only |
| WMT25 Terminology Translation Task Models translate sentence, paragraph and document level text with a supplied source to target glossary. Scoring checks quality and whether the prescribed target terms appear. | Expert | Checked automatically | 89.7% | System cards only |
| ZMultiTransBench Models translate 1,220 real Chinese source texts into seven languages. A GPT-4.1 judge compares each translation with a fixed baseline. | University | Graded by a model | 1050.0 | System cards only |
| Last Translation Benchmark Strong models translate hard texts chosen to stay unsaturated, and experts assess the outputs instead of automatic metrics that can be reward-hacked. | Research frontier | Graded by people | Not in cards | |
| WMT24 Systems translate the WMT24 general task test sets, mostly English to X plus some X to Y pairs, at paragraph level across several domains. The shared task ranks systems by human ESA judgments. | University | Graded by people | Not in cards |
Reasoning in many languages
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| MGSM 250 GSM8K problems translated into 10 languages. Final-answer accuracy. | School | Checked automatically |  93% | Fading |
| SEA-HELM Models are evaluated in Filipino, Indonesian, Tamil, Thai and Vietnamese across five pillars: NLP classics, LLM-specific tasks, linguistics, culture and safety. | School | Checked automatically | 77.4 | Not in cards |
| MultiNRC Models answer over 1,000 reasoning questions written by native speakers in French, Spanish and Chinese. Questions cover linguistic reasoning, wordplay, cultural knowledge and culturally grounded math. A GPT-4.1 judge compares each answer with a short reference answer. | Expert | Graded by a model | 65.6% | Not in cards |
| Global PIQA Models pick the correct solution to everyday commonsense prompts in over 100 languages, many about local foods and customs. Scoring is accuracy. | Everyday | Checked automatically | 93.4% | Fading |
| CLUEWSC Models decide whether a pronoun in a Chinese sentence refers to a given noun phrase, a Winograd-style task. Scored by accuracy. | Everyday | Checked automatically | 92.8% | Fading |
| MLogiQA Models answer LogiQA logical reasoning questions translated into 10 languages, 80 per language. Scored by accuracy. | University | Checked automatically | 77.1% | Dropped |
| MT-AIME2024 Models solve the 30 AIME 2024 problems translated into 55 languages. A rule-based verifier checks each final answer. | Competition | Checked automatically | 80.8% | Dropped |
| KoBALT Models answer 700 expert-written multiple-choice questions on 24 Korean linguistic phenomena in syntax, semantics, pragmatics, phonology and morphology. | University | Checked automatically | 61.8% | Fading |
| Multilingual Understanding Models answer five multilingual comprehension sets, BELEBELE, XCOPA, XWinograd, XStoryCloze, and PAWS-X. Qwen reports the average accuracy. | Everyday | Checked automatically | 89.6% | Dropped |
| XCOPA Models pick the more plausible cause or effect of a premise from two options, in 11 languages. Scoring is accuracy. | Everyday | Checked automatically | 94.4% | Dropped |
| MT-AIME (Cohere internal) Cohere's internal machine translation of AIME 2025 problems into many languages (via Command A Translate). Answer accuracy. | Competition | Checked automatically | 86% | Fading |
| XNLI Models classify entailment, contradiction or neutral for sentence pairs translated from MultiNLI into 15 languages. | Everyday | Checked automatically | 45.3% | Dropped |
| ArabCulture Models answer commonsense questions in Modern Standard Arabic about daily life in 13 Arab countries. Native speakers wrote and checked the questions across 12 domains. | Everyday | Checked automatically | 65.2% | System cards only |
| Linguini Models solve 160 International Linguistic Olympiad problems with 894 questions in 75 mostly low-resource languages. All evidence needed is in the context. | Competition | Checked automatically | 8.80% | System cards only |
| 3LM Models answer Arabic STEM questions taken from Arabic textbooks and worksheets, plus synthetic STEM questions from the same sources. A third part tests Arabic code generation. | School | Checked automatically | Not in cards | |
| CulturalRiddles Models solve original riddles written by people from 62 cultures in 51 languages. Solving them needs cultural knowledge plus abductive and compositional reasoning. | Everyday | Checked automatically | Not in cards | |
| GlobalMGSM Models solve MGSM grade-school math problems pooled from MGSM-Rev2, AfriMGSM, BenchMAX and other translations, 41 language configs in all. Accuracy on the final number. | School | Checked automatically | Not in cards | |
| KoBEST Models do five Korean tasks designed by professional linguists: BoolQ, COPA, WiC, HellaSwag and SentiNeg. All data is human-annotated. | Everyday | Checked automatically | Not in cards | |
| MGSM-Rev2 Models solve the 250 MGSM grade-school math problems in 10 languages. Google fixed two English errors, rephrased ambiguous items and retranslated every question with Gemini. Accuracy. | School | Checked automatically | Not in cards | |
| Multilingual AIME 2024 (Mistral) Models solve AIME 2024 problems translated into French, Spanish, German, Italian, Russian and Chinese, reasoning in the question language. | Competition | Checked automatically | System cards only | |
| Uhura ARC-Easy Models answer ARC-Easy science questions translated into Amharic, Hausa, Northern Sotho, Swahili and Yoruba. Scoring is accuracy. | School | Checked automatically | System cards only | |
| XStoryCloze Models pick the right ending for short stories from StoryCloze, professionally translated into 10 languages. | Everyday | Checked automatically | Not in cards |
Reading in many languages
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| ALRAGE Models answer Arabic questions from retrieved passages of 40 Arabic books. A Qwen2.5-72B judge scores each answer against a gold answer on a 0 to 10 scale. | Everyday | Graded by a model | 80.7% | System cards only |
| IndicGenBench Models summarize, translate, and answer questions across 29 Indic languages using human-translated versions of CrossSum, FLORES, XQuAD, and XorQA. Each task uses its standard metric. | Everyday | Checked automatically | 77.1 | Dropped |
| C3 Models answer 19,577 multiple-choice questions on Chinese dialogues and texts from Chinese-as-a-second-language exams. Scoring is accuracy. | School | Checked automatically | 97.8% | Dropped |
| XQuAD Models answer 1,190 SQuAD questions over 240 paragraphs, professionally translated so the set is parallel across 11 languages. Scored by F1 and exact match. | Everyday | Checked automatically | 76.8 | Dropped |
| TydiQA Models answer information-seeking questions from Wikipedia articles in 11 typologically diverse languages, across 204K question-answer pairs. Scored by F1. | Everyday | Checked automatically | 31.7% | Dropped |
| CCPM Models pick which of four classical Chinese poetry lines matches a given modern Chinese translation. Scored by accuracy. | School | Checked automatically | 92% | Dropped |
| CMRC Models extract answer spans from Chinese Wikipedia paragraphs for about 20,000 human-written questions. Scoring is exact match and F1. | Everyday | Checked automatically | 76.3% | Dropped |
| Belebele Models answer multiple-choice reading comprehension questions on FLORES passages in 122 language variants. Scored by accuracy. | Everyday | Checked automatically | 87.8% | System cards only |
| CHID Models fill blanks in Chinese passages by choosing the right idiom from candidates. Scored by accuracy. | School | Checked automatically | 72.6% | System cards only |
| CLUE Models are fine-tuned or prompted on nine Chinese language understanding tasks, including sentence classification, sentence-pair inference and machine reading comprehension. The score averages the task results. | School | Checked automatically | Not in cards | |
| MLQA Models extract answers from Wikipedia passages in seven languages, with questions and contexts that can be in different languages. Scored by F1 and exact match. | Everyday | Checked automatically | Not in cards | |
| Uhura-Eval Models answer new reading comprehension questions written in Amharic, Hausa and Yoruba. Scoring is accuracy. | Everyday | Checked automatically | System cards only |
Chat and instructions in many languages
| Benchmark | Level | Scored | Best score | Status |
|---|---|---|---|---|
| SEAL Multilingual Models answer 1,000 private prompts per language, written by native speakers on general and cultural topics. Annotators rate each answer and compare pairs side by side, and boards rank models by Elo. | Everyday | Graded by people | 1,134 | Not in cards |
| MAXIFE Models follow 1,667 verifiable instructions in 23 languages and in cross-lingual settings. Rule-based checks and model judges score compliance. | Everyday | Checked automatically | 89.2% | Fading |
| Multilingual IFEval Models follow IFEval instructions translated into eight languages, 100 examples each, which paid volunteers reviewed. Instruction compliance is checked as in IFEval. | Everyday | Checked automatically | 87% | Dropped |
| IndiVibe Models answer 110 source prompts in chat, STEM, math and coding, translated into 22 Indian languages in native and romanized script. A judge model compares answers pairwise. | Everyday | Graded by a model | 90% | Fading |
| Ko-Arena-Hard v2 (Upstage) Models answer hard user prompts in Korean, judged against a baseline in the style of Arena-Hard v2. Upstage built it in house. | Everyday | Graded by a model | 79.9% | Fading |
| Multilingual human evaluation Human raters score model answers in 10 languages, including Arabic, Korean, and Thai, on a 1 to 5 scale. Results are averaged across languages. | Everyday | Graded by people | 3.93 | Dropped |
| Ko-IFEval (Upstage) Models follow verifiable instructions written in Korean. Upstage adapted IFEval to Korean for the Open Ko-LLM Leaderboard2. | Everyday | Checked automatically | 87.5% | System cards only |
| Language Confusion Benchmark Models answer prompts in 15 languages and must reply in the language the user wants, both in monolingual and crosslingual settings. A language detector scores line-level and word-level pass rates. | Everyday | Checked automatically | 93% | System cards only |
| Omnilingual-Gaia2 Agents run Gaia2 execution, search, ambiguity and adaptability scenarios translated into 10 languages, graded against the scenario oracle with multilingual judge prompts. | University | Checked automatically | 60.4% | System cards only |
| Qwen in-house Chinese automatic evaluation Models answer Qwen in-house Chinese datasets on knowledge, comprehension, coding, math and related skills. Skills and weights change between reports. | Everyday | Graded by a model | 69.6% | System cards only |
| ZMultiDialBench Human annotators score model replies to 141 multilingual dialogue prompts on a 1 to 10 scale with category criteria. | Everyday | Graded by people | 8.57 | System cards only |
| AL-QASIDA Models write in eight dialectal Arabic varieties. Scored on fidelity, understanding, quality and diglossia, including the ADI2 dialect identification score. | Everyday | Checked automatically | System cards only | |
| ArabCulture-Dialogue Models pick the most culturally fitting reply in dialogues from 13 Arab countries, in Modern Standard Arabic and each local dialect. It also has translation and dialect-steering tasks. | Everyday | Checked automatically | Not in cards | |
| Dolly Machine Translated v2 Models answer a curated subset of Dolly prompts translated into 66 languages with newer systems such as Command A Translate. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | Everyday | Graded by a model | Not in cards | |
| KoMT-Bench Models follow multi-turn instructions in Korean on a translated and culturally adapted version of MT-Bench, reviewed by expert linguists. | Everyday | Graded by a model | Not in cards | |
| m-ArenaHard Models answer the 500 Arena-Hard-Auto v0.1 prompts machine-translated into 22 more languages, 23 in all. An LLM judge compares answers pairwise and reports win rates. | University | Graded by a model | System cards only | |
| m-ArenaHard v2.0 Models answer 498 English prompts from Arena-Hard v2.0, translated into 22 more languages for 11,454 prompts in all. An LLM judge compares answers pairwise and reports win rates. | University | Graded by a model | Not in cards | |
| m-ArenaHard v2.1 Models answer the 498 Arena-Hard v2.0 prompts translated into 67 languages, with code and data kept intact. A GPT-4.1 judge rates answers with a rubric in the Tiny Aya report. | University | Graded by a model | Not in cards | |
| mDolly Models answer open-ended Dolly prompts machine-translated into 101 languages, plus human-edited sets in 6 languages and human-written prompts in 7. An LLM judge compares or rates the answers. | Everyday | Graded by a model | Not in cards |