MultiRC

PaLM 2-L 88.2% Dropped

Models pick all correct answers to questions that need reasoning over several sentences of a paragraph. Scored by F1 over answer options.

Saturated. Also part of SuperGLUE.

In the authors’ words

We present a reading comprehension challenge in which questions can only be answered by taking into account information from multiple sentences. We solicit and verify questions and answers for this challenge through a 4-step crowdsourcing experiment. Our challenge dataset contains 6,500+ questions for 1000+ paragraphs across 7 different domains (elementary school science, news, travel guides, fiction stories, etc) bringing in linguistic diversity to the texts and to the questions wordings. On a subset of our dataset, we found human solvers to achieve an F1-score of 88.1%. We analyze a range of baselines, including a recent state-of-art reading comprehension system, and demonstrate the difficulty of this challenge, despite a high human performance. The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills.

Abstract of Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (ACL N18-1023, CC BY 4.0)

Made by University of Pennsylvania (CogComp) Released Jun 2018 Reading comprehension Public tasks plus a private test set Last in a launch post May 2023 , dropped after it saturated Last model added May 2023

Level: School Students, from grade school to high school
Scored: Checked automatically A program checks the result against an answer key, unit tests or the expected final state.
Answers: Multiple choice

CogComp/multirc 34 stars Paper Dataset cogcomp.seas.upenn.edu

Model scored in lab cards

Launch post lists it, from 1 lab

PaLM 2-L leads with 88.2%

Best score per model as labs report them in their own cards. Labs pick their own settings, so these compare less cleanly than independent runs.

Bars Table

88.2 PaLM 2-L

1 PaLM 2-L 88.2%

#	Model	Score	Setting	Date	Source
1	PaLM 2-L	88.2%		May 2023	Model card

The best score passed 85% in May 2023

All the scores we have, one dot each, so a model tested at several settings appears more than once.

Reported by the lab in its model card

Labs dropped it from launch posts once the best score passed 85%

Launch posts from every lab that has listed it, one tick per post. Tall colored ticks include it, and short gray ones leave it out.

Google DeepMind

Date	Launch post	Score	As the post writes it
10 May 2023	PaLM 2-L	88.2	MultiRC (F1), 1-shot

PaLM 2-L leads with 88.2%

The best score passed 85% in May 2023

Labs dropped it from launch posts once the best score passed 85%

More in Knowledge