MMLU-Redux 2.0

Not in model cards

Models answer 5,700 MMLU questions from all 57 subjects that experts re-annotated for errors. Flawed questions are marked so scoring can skip or fix them.

Extends the first release from 3,000 questions in 30 subjects to all 57 subjects. Single label fixes in 2025-02 kept scores comparable.

In the authors’ words

Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.

Abstract of Are We Done with MMLU? (arXiv:2406.04127, CC0)

Made by University of Edinburgh Released Aug 2024 Exam questions All tasks public

Level: University Undergraduates and entry-level professionals
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Multiple choice