KMMMU
EXAONE 4.5 33B 42.7% Fading Models answer 3,466 college and exam level questions that combine images with Korean text. Questions come from Korean exam sources, some with several images.
A Korean counterpart of MMMU. The dataset repo was created in 2026-02 and the technical report followed in 2026-04.
In the authors’ words
Multimodal evaluation is strengthened when benchmarks reflect the languages, institutions, and disciplinary conventions that shape expert visual reasoning. We introduce KMMMU, a native Korean benchmark built from Korean-native examination materials. KMMMU contains 3,466 questions spanning nine disciplines and nine visual modality categories. For targeted evaluation on challenging questions and Korean-specific knowledge, we additionally provide KMMMU-Adv and KMMMU-Local, respectively. Experiments show that the strongest evaluated open-weight model reaches 42.05% on the KMMMU, while the best evaluated proprietary model achieves 52.42% on the KMMMU-Adv. Performance varies substantially across disciplines and answer formats, with Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that inspected failures often involve convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards, not reasoning depth alone. KMMMU provides a testbed for native Korean expert-level multimodal understanding beyond English-centric or translation-based evaluation.
- Level
- University Undergraduates and entry-level professionals
- Scored
- Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
- Answers
- Multiple choice
EXAONE 4.5 33B leads with 42.7%
Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.
- 1 EXAONE 4.5 33B 42.7%
| # | Model | Score | Setting | Date | Source |
|---|---|---|---|---|---|
| 1 | EXAONE 4.5 33B | 42.7% | Apr 2026 | Model card |
Scores over time
All the scores we have, one dot each, so a model tested at several settings appears more than once.
Labs headline it less often now, last in Apr 2026
Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.
| Date | Launch post | Score | As the post writes it |
|---|---|---|---|
| 9 Apr 2026 | EXAONE 4.5 33B | 42.7 | KMMMU |
One system card reports it
Besides launch posts, we read the system cards, model cards and technical reports labs publish with a release. They report far more, including safety and frontier-risk tests.
| Date | Document | Score | As the document writes it | Section |
|---|---|---|---|---|
| 9 Apr 2026 | EXAONE 4.5 33B technical report | 42.7 | KMMMU (Table 2, Reasoning mode) | Capabilities |
More in Vision
- MMMU In use 11.5K college-level multimodal questions across 30 subjects. Multiple-choice and open accuracy. GPT-5.1 85.4%
- MMMU-Pro In use Harder MMMU: college-level multimodal questions with more options and vision-only inputs to remove text shortcuts. Kimi K3.1 Pro 84.1%
- CharXiv In use Chart understanding on real arXiv figures. Reasoning questions require synthesizing information from complex charts. Claude Fable 5 93.5%
- MATH-Vision (MathVision) In use 3,040 competition math problems with visual context. Answer accuracy.
Qwen 3.8 Max 97.7% - ChartQA In use Models answer human-written and generated questions about chart images that need visual and arithmetic reasoning. Scored by relaxed accuracy with a 5 percent numeric tolerance.
Command A Vision 90.9% - DocVQA In use Models answer 50,000 questions about 12,000 scanned document images by reading their text and layout. Scoring is ANLS, a string similarity to reference answers.
Seed1.5-VL 96.9%